Monitor a Python Flask API using Open Telemetry, Prometheus, and Grafana.

Monitor your Flask application effectively with Open Telemetry, Prometheus, and Grafana to gain insights into its performance, while adapting the setup to your specific needs.

Prerequisites

AWS Account with EC2 Instance.
Ensure Python 3, pip, a Python virtual environment, and Docker are installed on the system.

Update the package list.

sudo apt update

Install Python 3, a Python virtual environment, and pip.

sudo apt install -y python3 python3-venv python3-pip

Install Docker to deploy Prometheus and Grafana in containers.

sudo apt install -y docker.io

Start the Docker service and set it to run automatically at startup.

sudo systemctl start docker
sudo systemctl enable docker

Set Up the Flask API

Create a new directory for the project setup and move into that directory.

mkdir flask
cd flask

Since pip doesn't work globally, set up a Python Virtual Environment to isolate your Python environment, making dependency management easier and avoiding conflicts.

python3 -m venv venv

Activate the virtual environment by running the command source venv/bin/activate.

source venv/bin/activate

Inside the virtual environment, install the Flask and Open Telemetry libraries.

pip install flask opentelemetry-api opentelemetry-sdk opentelemetry-instrumentation-flask

Install the necessary package for the OTLP exporter.

pip install opentelemetry-exporter-otlp

Install the Prometheus Flask exporter.

pip install prometheus-flask-exporter

Create the Flask API that includes Prometheus metrics and OpenTelemetry.

nano app.py

Insert the following code into the file.

from flask import Flask
from prometheus_flask_exporter import PrometheusMetrics
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Initialize Flask app
app = Flask(__name__)

# Initialize Prometheus metrics
metrics = PrometheusMetrics(app)

# Initialize OpenTelemetry
trace_exporter = OTLPSpanExporter(endpoint="http://localhost:4317")  # Adjust if using remote OpenTelemetry Collector
provider = TracerProvider()
processor = BatchSpanProcessor(trace_exporter)
provider.add_span_processor(processor)

# Instrument Flask app with OpenTelemetry
FlaskInstrumentor().instrument_app(app)

# Add routes
@app.route('/hello')
def hello():
    return "Hello, World!"

@app.route('/health')
def health():
    return {"status": "healthy"}

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)

Explanation of the code

Importing Necessary Libraries: The code imports libraries like Flask for building web APIs, PrometheusMetrics for exposing HTTP metrics to Prometheus, FlaskInstrumentor for automatic instrumentation with OpenTelemetry, TracerProvider for creating traces and spans, BatchSpanProcessor for efficient span processing, and OTLPSpanExporter for sending spans to an OpenTelemetry Collector.
Flask Application Initialization: A new Flask application instance is created to handle incoming HTTP requests.
Prometheus Metrics Initialization: This sets up Prometheus metrics for the Flask app, allowing it to automatically provide metrics like request counts and latencies at the /metrics endpoint.
OpenTelemetry Setup: The OTLPSpanExporter sends trace data to an OpenTelemetry Collector via gRPC, the TracerProvider manages spans, and the BatchSpanProcessor sends spans in batches to improve performance, with the processor added to the TracerProvider for processing and exporting spans.
Instrumenting the Flask App with OpenTelemetry: The line FlaskInstrumentor().instrument_app(app) automatically adds tracing to the Flask app using OpenTelemetry, so every HTTP request to the application generates trace spans.
The /hello route returns a "Hello, World!" message when accessed via HTTP GET, while the /health route serves as a health check endpoint, returning a JSON response to indicate the application's status.
Running the Flask App: The Flask application starts when the script is run directly, listening on all network interfaces (0.0.0.0) and accessible on port 8080, allowing remote access from any IP address.

Set Up and Configure Prometheus

Download the latest Prometheus Docker image.

sudo docker pull prom/prometheus:latest

Create and navigate to a directory for Prometheus configuration.

mkdir prometheus
cd prometheus

Create the prometheus.yml file in the Prometheus configuration directory.

nano prometheus.yml

Add this configuration.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'flask-metrics'
    scrape_interval: 5s
    static_configs:
      - targets: ['<Public-IP-address>:8080']

Save and exit the file, then run Prometheus with the configuration file mounted.

sudo docker run -d \
  --name=prometheus \
  -p 9090:9090 \
  -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus

Return to the root directory.

cd ..

Set Up Grafana for Visualization

Download the latest Grafana Docker image.

sudo docker pull grafana/grafana:latest

Run the Grafana container.

sudo docker run -d \
--name=grafana \
-p 3000:3000 \
grafana/grafana

Grafana is now accessible at http://<Public-IP>:3000.

Run the Flask API

Start the Flask application.

python app.py

Once the application is running, visit http://<EC2-Instance-IP>:9090 to access the Prometheus web interface.

To verify that Prometheus is collecting metrics from your .NET application, go to the “Targets” page in the Prometheus UI, click on “Status” in the menu bar, then “Targets,” and ensure your “flask-metrics” job is listed and marked as “UP.”

Configure Grafana to visualize metrics

The default login credentials for Grafana are username admin and password admin, and you will be prompted to change the password after your first login, though you can choose to change it or skip it.

After logging in to Grafana, add Prometheus as a data source by clicking on "Connections," then "Data source," searching for Prometheus, selecting it, and clicking on Add new data source.

Set the URL to http://<EC2-Instance-IP>:9090, replacing <EC2-Instance-IP> with your EC2 instance's IP address, leave other settings as default, click “Save & Test” to check Grafana's connection to Prometheus, then click the “Plus” icon, select “Dashboard,” click “Add visualization,” and in the “Query” section, select the Prometheus data source and enter a Prometheus query to get metrics from your Flask API.