Use Prometheus and Grafana to monitor GitLab pipelines

Ankita Lunawat's photo
Ankita Lunawat
·

4 min read

Monitoring GitLab pipelines is important to make sure your development and deployment processes work well. By keeping an eye on pipeline performance and health, you can spot problems like slow builds or failed jobs early, saving time and boosting efficiency.

Prometheus and Grafana are great tools for this. Prometheus gathers and stores metrics data, and Grafana displays it in an easy-to-understand way. In this guide, we will set up Prometheus and Grafana from the beginning to monitor your GitLab pipelines.

Prerequisites

  • AWS Account with Ubuntu EC2 Instance.

  • Python, pip and Docker Installed.

  • Basic knowledge of Gitlab.

sudo apt update

Install Docker and pip if they are not already installed.

sudo apt install -y docker.io python3-pip

Check its version to confirm the installation.

docker --version

pip3 --version

Install Docker-Compose to handle multiple containers.

sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

Change its permissions with this command.

sudo chmod +x /usr/local/bin/docker-compose

Check its version to confirm the installation.

docker-compose --version

Set up the virtual environment.

sudo apt install python3.12-venv

Create a new virtual environment called venv.

python3 -m venv venv

Activate the virtual environment.

source venv/bin/activate

Create Project Structure and Files

Create a folder for your project and go into it.

mkdir Gitlab-monitor
cd Gitlab-monitor

Inside this folder, create a subfolder for Prometheus and go into it.

mkdir Prometheus
cd Prometheus

Create a Prometheus configuration file.

nano prometheus.yml

Add the following code to it.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: monitoring

    static_configs:
      - targets: ['<EC2-Public-IP>:8080']

  - job_name: monitoring_manual

    static_configs:
      - targets: ['<EC2-Public-IP>:8500']

Back to project directory.

cd ..

Create Dockerfile.

nano Dockerfile

Add the following code into it.

# set base image (host OS)
FROM python:3.12

WORKDIR /code

# copy the dependencies file to the working directory
COPY requirement.txt .

# install dependencies
RUN pip install -r requirement.txt

COPY . .


# command to run on container start
CMD [ "python", "gcexporter.py" ]

Create a docker-compose.yml file.

nano docker-compose.yml

Add following code into it.

version: '3.8'
services:
  grafana:
    image: docker.io/grafana/grafana:latest
    ports:
      - 3000:3000
    environment:
       GF_INSTALL_PLUGINS: grafana-polystat-panel,yesoreyeram-boomtable-panel
    links:
      - prometheus

  prometheus:
      image: docker.io/prom/prometheus:v2.28.1
      ports:
       - 9090:9090
      links:
       - gitlab-ci-pipelines-exporter
      volumes:
       - ./Prometheus/prometheus.yml:/etc/prometheus/prometheus.yml

  gitlab-ci-pipelines-exporter:
    image: docker.io/mvisonneau/gitlab-ci-pipelines-exporter:v0.5.2
    ports:
      - 8080:8080
    environment:
      GCPE_CONFIG: /etc/gitlab-ci-pipelines-exporter.yml
    volumes:
      - type: bind
        source: ./gitlab-ci-pipelines-exporter.yml
        target: /etc/gitlab-ci-pipelines-exporter.yml

  customized_exporter:
        build: .
        ports:
            - "8500:8500"

Make a configuration file for the GitLab CI Pipelines Exporter.

nano gitlab-ci-pipelines-exporter.yml

Add the following code to it, and obtain the token from GitLab and the project ID from your project settings.

log:
  level: info

gitlab:
  url: https://gitlab.com
  token: '<gitlab_token>'

# Example public projects to monitor
projects:
  - name: demo/test_pipeline
    # Pull environments related metrics prefixed with 'stable' for this project
    pull:
      environments:
        enabled: true
      refs:
        branches:
          # Monitor pipelines related to project branches
          # (optional, default: true)
          enabled: true

          # Filter for branches to include
          # (optional, default: "^main|master$" -- main/master branches)
          regexp: ".*"
        merge_requests:
          # Monitor pipelines related to project merge requests
          # (optional, default: false)
          enabled: true

Create a file named gcexporter.py to serve as a custom exporter for retrieving the total number of branches in the project.

nano gcexporter.py

Add the following code to it.

import random
import time
import re
import gitlab
import requests
from requests.auth import HTTPBasicAuth
import json
import time
import random
from prometheus_client import start_http_server, Gauge

group_name='demo'
gitlab_server='https://gitlab.com/'
auth_token= '<gitlab _token>'

gl = gitlab.Gitlab(url=gitlab_server, private_token=auth_token)
group = gl.groups.get(group_name)
project_id= <project_id>
project = gl.projects.get(project_id)
gitlab_branch_count = Gauge('gitlab_branch_count', "Number of Branch Count")
def get_metrics():
   gitlab_branch_count.set(len(project.branches.list()))

if __name__ == '__main__':
    start_http_server(8500)
    while True:
        get_metrics()
        time.sleep(43200)

Create a requirement.txt file.

nano requirement.txt

Add the following code into it.

python-gitlab==3.1.1
prometheus-client==0.13.1

Install Python Dependencies

The requirement.txt file lists the dependencies for the custom exporter, so let's run the following command.

pip3 install -r requirement.txt

Use Prometheus and Grafana to Monitor GitLab Pipelines

Build and Start the Services

Build the Docker containers to start the services.

docker-compose build

Start the services by building the Docker containers.

docker-compose up -d

Check the running containers.

docker ps -a

Access the Services

Visit http://<EC2-Public-IP>:9090 to check data collection in the Prometheus UI, verify that both exporters are being scraped on the Status > Targets page, and access the customized metrics at http://<EC2-Public-IP>:8500/metrics and http://<EC2-Public-IP>:8080/metrics.

Configure Grafana and Visualize the metrics

Access Grafana at http://<EC2-Public-IP>:3000, log in with admin as the default Username and Password, navigate to Connections > Data sources, click on Add data source, select Prometheus as the data source, and enter your Prometheus URL.

Click on save and test to see the message Successfully queried the Prometheus API, then go to the “+” icon at the top-right corner, select New dashboard, click on Add visualization, choose Prometheus as the data source, enter the query Metric – gitlab_branch_count to find the branch count in the project, select job = monitoring_manual, and click on Run queries.

AWS#prometheusGitLabGrafana