Building an Automated Monitoring Pipeline with Prometheus, Grafana, and cAdvisor

ABIGAIL MOJUME — Mon, 24 Nov 2025 16:37:20 +0000

How I automated infrastructure monitoring deployment using GitHub Actions, Docker Compose, and container metrics

The Problem

In today's world, monitoring is essential. We need to track CPU, memory, disk, network and metrics to understand system performance. Additionally, monitoring request volumes, response times, and error rates helps identify issues before they impact users.

But here's the challenge: manually setting up monitoring infrastructure every time your code is successfully deployed is time-consuming and error-prone. What if you could automatically deploy your entire monitoring stack whenever new code gets deployed?

That's exactly what I built—an automated monitoring pipeline that deploys Prometheus, Grafana, and cAdvisor as the final step in our deployment process.

The Solution: A Three-Part Pipeline

Our complete CI/CD pipeline consists of:

Code Quality Check (SonarQube) - Ensures code meets quality standards
Application Deployment - Deploys the application to production servers
Monitoring Setup (this article) - Automatically deploys monitoring infrastructure

When code passes quality checks and deploys successfully, the monitoring pipeline triggers automatically, ensuring our newly deployed application is immediately observable.

Directory Structure

Here's what we're building:

dev-prometheus-monitoring
    ├── prometheus.yml          # Monitoring configuration
    ├── docker-compose.yml      # Container orchestration
    ├── monitor.env            # Environment variables
    └── .github/workflows/
        └── deploy-monitoring.yml   # Automation workflow

The components:

Prometheus: Collects and stores metrics
Grafana: Visualizes metrics in dashboards
cAdvisor: Exposes metrics about container
Redis: Stores historical data for cAdvisor
GitHub Actions: Automates deployment
Docker Compose: Orchestrates all containers

Why cAdvisor for Container Monitoring?

For containerized environments, cAdvisor provides critical insights that traditional system monitors can't:

Per-container resource usage: See exactly which container is consuming resources
Container-specific metrics: CPU, memory, network, and filesystem usage for each container
Real-time visibility: Track container performance as workloads scale
Docker integration: Native understanding of Docker containers and their lifecycles

This makes it ideal for microservices architectures where multiple containers run on the same host.

Step 1: Setting Up Your Workspace

I created a dedicated directory for this project on the VM:

mkdir dev-prometheus-monitoring
cd dev-prometheus-monitoring

Step 2: Configuring Prometheus

Create prometheus.yml - this tells Prometheus what to monitor and how often:

global:
  scrape_interval: 15s      # Collect metrics every 15 seconds
  evaluation_interval: 15s   # Evaluate alerting rules every 15 seconds

scrape_configs:
  # Monitor Prometheus itself
  - job_name: 'nvs-prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Monitor Docker containers via cAdvisor
  - job_name: 'nvs-vm'
    static_configs:
      - targets: ['cadvisor:8080']

Understanding the config:

scrape_interval: How frequently Prometheus pulls metrics. 15 seconds balances real-time visibility with resource usage.
job_name: Logical grouping of similar targets
targets: Specific endpoints exposing metrics in the format host:port

The nvs-vm job scrapes cAdvisor, which exposes metrics for all Docker containers running on the host.

Step 3: Creating the Docker Compose File

Docker Compose allows us to define and run multiple containers with a single command. Create docker-compose.yml:

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
    depends_on:
      - cadvisor
    networks:
      - monitoring
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
    env_file:
      - /home/administrator/Desktop/workspace/dev-prometheus-monitoring/monitor.env
    networks:
      - monitoring
    restart: unless-stopped

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    ports:
      - "8080:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    depends_on:
      - redis
    networks:
      - monitoring
    restart: unless-stopped

  redis:
    image: redis:latest
    container_name: redis
    ports:
      - "6379:6379"
    networks:
      - monitoring
    restart: unless-stopped

volumes:
  prometheus-data:
  grafana-data:

networks:
  monitoring:
    driver: bridge

Step 4: Environment Configuration

Create monitor.env to store Grafana configuration:

GF_SECURITY_ADMIN_PASSWORD=your_secure_password
GF_SERVER_ROOT_URL=http://your-server-ip:3000

This keeps sensitive configuration separate from the docker-compose file and makes it easy to change settings without modifying the main configuration.

Step 5: Version Control Setup

Initialize Git and push to GitHub:

git init
git add prometheus.yml docker-compose.yml monitor.env
git commit -m "Initial monitoring setup"
git remote add origin https://github.com/yourusername/dev-prometheus-monitoring.git
git push -u origin main

Create a .gitignore file:

_actions/
*.log
.DS_Store

Why ignore _actions/? This directory contains the self-hosted runner files (explained next), which are large and system-specific.

Step 6: Setting Up Self-Hosted GitHub Actions Runner

Since we're working with on-premise servers, we need a self-hosted runner that can access our internal network.

Steps:

Go to your GitHub repository
Navigate to Settings → Actions → Runners
Click "New self-hosted runner"
Follow the provided commands on your Linux VM:

# Download the runner
mkdir actions-runner && cd actions-runner
curl -o actions-runner-linux-x64-2.311.0.tar.gz -L \
  https://github.com/actions/runner/releases/download/v2.311.0/actions-runner-linux-x64-2.311.0.tar.gz
tar xzf ./actions-runner-linux-x64-2.311.0.tar.gz

# Configure the runner
./config.sh --url https://github.com/yourusername/dev-prometheus-monitoring \
  --token YOUR_TOKEN

# Start the runner
./run.sh

For production, install it as a service:

sudo ./svc.sh install
sudo ./svc.sh start

Why self-hosted? GitHub-hosted runners can't access on-premise servers. Our self-hosted runner bridges the gap between GitHub (cloud) and our infrastructure (on-premise).

Step 7: Adding GitHub Secrets

Store sensitive credentials securely:

Go to repository Settings → Secrets and variables → Actions
Add these secrets:
- GH_USERNAME: Your GitHub username
- GH_TOKEN: Personal access token for GitHub authentication

These will be used to authenticate when cloning the repository during deployment.

Step 8: Creating the Automation Workflow

Create .github/workflows/deploy-monitoring.yml :

name: deploy-monitoring-solution

on:
  repository_dispatch:
    types: [deploy-monitoring]

permissions:
  contents: read
  packages: read

jobs:
  deploy-prometheus:
    name: Deploy Prometheus Stack
    runs-on: [self-hosted, "server-76"]

    steps:
      - name: Checkout orchestrator repo
        uses: actions/checkout@v4

      - name: Clone Prometheus repo
        working-directory: /home/administrator/Desktop/DevOps/dev-prometheus-monitoring
        run: |
          rm -rf dev-prometheus-monitoring
          git clone https://${{ secrets.GH_USERNAME }}:${{ secrets.GH_TOKEN }}@github.com/technotrending/dev-prometheus-monitoring.git -b main

      - name: Build Docker images
        working-directory: /home/administrator/Desktop/workspace/dev-prometheus-monitoring
        run: |
          docker compose up -d

      - name: Cleanup old images
        run: |
          docker image prune -af

Pro tip I learned: Use mkdir -p .github/workflows to create nested directories in one command. The -p flag creates parent directories automatically.

Understanding the workflow:

Trigger

on:
  repository_dispatch:
    types: [deploy-monitoring]

This workflow is triggered by the deployment pipeline. After your application successfully deploys, the deployment workflow sends a repository_dispatch event to trigger this monitoring setup.

Job Steps

Checkout orchestrator repo: Downloads the workflow repository
Clone Prometheus repo:

- Navigates to the DevOps directory
- Removes old copy if it exists
- Clones fresh configuration from GitHub
- Uses secrets for authentication

Build Docker images:

- Changes to the workspace directory
- Runs `docker compose up -d` to start all containers
- The `-d` flag runs them in detached mode (background)

Cleanup old images:

- Removes unused Docker images to save disk space
- Keeps the system clean after updates

Step 9: Integrating with the Deployment Pipeline

In your application deployment workflow, add this step at the end:

- name: Trigger monitoring deployment
  if: success()
  run: |
    curl -X POST \
      -H "Accept: application/vnd.github.v3+json" \
      -H "Authorization: token ${{ secrets.GH_PAT }}" \
      https://api.github.com/repos/yourusername/dev-prometheus-monitoring/dispatches \
      -d '{"event_type":"deploy-monitoring"}'

The flow:

SonarQube checks pass ✓
Application deploys successfully ✓
Deployment workflow sends repository_dispatch event
Monitoring workflow triggers automatically
Prometheus, Grafana, cAdvisor, and Redis deploy

Step 10: Verifying the Setup

Once deployed, access your monitoring stack:

Prometheus UI: `http://your-server-ip:9090`

Navigate to Status → Targets
Both targets should show "UP" status:
- nvs-prometheus (localhost:9090)
- nvs-vm (cadvisor:8080)

Grafana UI: `http://your-server-ip:3000`

Login with credentials from monitor.env
Add Prometheus as a data source:
- Go to Configuration → Data Sources
- Add Prometheus
- URL: http://prometheus:9090
- Click "Save & Test"

cAdvisor UI: `http://your-server-ip:8080`

Browse container metrics directly
View real-time CPU, memory, and network usage per container

Figure 1: Prometheus dashboard showing targets metrics and health status

Import Dashboards

For quick visualization, import pre-built Grafana dashboards:

Go to Create → Import
For Docker monitoring, use dashboard ID: 193
This shows container metrics collected by cAdvisor

Figure 2: Grafana dashboard displaying real-time CPU usage metrics for Docker containers over a 3-hour period

The Complete Flow

Here's what happens end-to-end:

1. Developer commits code
   ↓
2. SonarQube validates code quality
   ↓
3. Application deploys to production
   ↓
4. Deployment workflow sends repository_dispatch
   ↓
5. Monitoring workflow triggers on self-hosted runner
   ↓
6. Runner clones latest monitoring configuration
   ↓
7. Docker Compose starts containers:
   - Redis (for data persistence)
   - cAdvisor (collects container metrics)
   - Prometheus (scrapes and stores metrics)
   - Grafana (visualizes data)
   ↓
8. Prometheus begins scraping cAdvisor every 15s
   ↓
9. Container metrics now visible in Grafana dashboards
   ↓
10. Old Docker images cleaned up automatically

Conclusion

This automated monitoring setup provides:

Immediate Visibility: New deployments are monitored from the moment they start
Resource Optimization: Identify containers consuming excessive resources
Performance Tracking: Historical data shows performance trends over time
Incident Response: Quickly diagnose issues by examining container metrics
Zero Manual Work: Monitoring deploys automatically with each application deployment

The complete pipeline ensures that every time code is deployed, monitoring infrastructure is automatically configured to track it. This eliminates manual setup, reduces errors, and guarantees observability from the moment containers start running.

Resources

Questions? Drop them in the comments below. I'm happy to help troubleshoot or explain any part of this setup in more detail.

If you found this helpful, please consider following me for more DevOps automation guides and infrastructure tutorials.

Forem: ABIGAIL MOJUME