Setting Up Monitoring with Prometheus and Grafana - Part 3 - Grafana

Sat, 18 Apr 2026 00:00:00 +0000

Introduction

In the previous blog posts part 1 and part 2, we have Node Exporter and Prometheus set up. We can already monitor our VMs and get valuable information. However, running PromQL queries repeatedly is not practical for day-to-day monitoring or visualization.

For that, we will deploy Grafana and import a pre-built dashboard so that we can visualize the scraped metrics. By the end of this post, we will have:

A reusable Grafana Ansible role
Provisioning of the data source and dashboards
Docker volume for persisting the Grafana data

Prerequisites

For the demo in this post to work, you need to have:

One or more target VMs
Node Exporter must be preconfigured and set up
Docker to be installed and available on the Prometheus server
community.docker to be installed
An existing Prometheus instance

Architecture Overview

For simplicity, we are going to focus only on Grafana and its immediate components:

A dedicated host for Grafana with an internal IP
Grafana runs as a Docker container and exposes the port 3000
A Docker volume will mount onto the container for persisting grafana-related data

Once all is done, we will have the full architecture setup:

Grafana

Similar to Prometheus, Grafana can be installed directly on the host or deployed as a container. At this point, it is clear that we will deploy Grafana as a container. The Ansible role will be broken down into two phases:

Prepare: Create the directories, render the configuration template and create the Docker volume
Deploy: Pull, run the image and mount the created volume onto the container

Variables

The variables for this role will be split into five categories:

Paths and provisioning
Prometheus integration
Container integration
Storage and volumes
Endpoint

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


---
grafana_provisioning_dir: "/opt/grafana/provisioning"
grafana_datasources_dir: "{{ grafana_provisioning_dir }}/datasources"

grafana_prometheus_datasource: "{{ grafana_datasources_dir }}/prometheus.yml"

# prometheus variables
grafana_prometheus_host: "prometheus"
grafana_prometheus_port: 9090

grafana_dashboards_dir: "{{ grafana_provisioning_dir }}/dashboards"
grafana_dashboard_provider_file: "{{ grafana_dashboards_dir }}/dashboard_provider.yml"
grafana_node_exporter_dashboard_file: "{{ grafana_dashboards_dir }}/node_exporter_full.json"

grafana_image_name: "grafana/grafana"
grafana_image_tag: "13.0"
grafana_image: "{{ grafana_image_name }}:{{ grafana_image_tag }}"

grafana_container_name: "grafana-instance"
grafana_listen_port: 3000
grafana_container_ports:
 - "{{ grafana_listen_port }}:{{ grafana_listen_port }}"

grafana_volume_name: "grafana-data"
grafana_container_volumes:
 - "{{ grafana_volume_name }}:/var/lib/grafana"
 - "{{ grafana_datasources_dir }}:/etc/grafana/provisioning/datasources:ro"
 - "{{ grafana_dashboards_dir }}:/etc/grafana/provisioning/dashboards:ro"

grafana_prometheus_datasource_url: "http://{{ grafana_prometheus_host }}:{{ grafana_prometheus_port }}"
grafana_prometheus_datasource_name: "Prometheus"

grafana_endpoint_url: "http://{{ ansible_host }}:{{ grafana_listen_port }}"

Preparation

Our focus for this phase is:

Creating config directories, which we will later bind-mount into the container
Render the data source and dashboard provisioning templates
Create the Docker volume, which will persist Grafana’s internal state (database, users, plugins). Dashboards and data sources will be provisioned via bind mounts

Dashboard Template

We define the dashboard provider template in templates/dashboards.yml.j2:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


apiVersion: 1

providers:
 - name: "default"
 orgId: 1
 folder: "Infrastructure"
 type: file
 disableDeletion: false
 updateIntervalSeconds: 30
 options:
 path: /etc/grafana/provisioning/dashboards

Prometheus Data Source Template

We define the template in templates/prometheus_datasource.yml.j2 so that we can pre-include our Prometheus data source:

1
2
3
4
5
6
7
8
9


apiVersion: 1

datasources:
 - name: "{{ grafana_prometheus_datasource_name }}"
 type: prometheus
 access: proxy
 url: "{{ grafana_prometheus_datasource_url }}"
 isDefault: true
 editable: false

Later in this series, we will further extend the template to include an Alert data source.

Deployment

The deployment phase consists of two steps:

Pulling the Docker image
Running the container with the volume and bind-mounts included

This can be done using these two tasks:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


---
- name: Pull Grafana Docker image
 community.docker.docker_image:
 name: "{{ grafana_image }}"
 source: pull
 register: grafana_docker_image_pull

- name: Run Grafana container
 community.docker.docker_container:
 name: "{{ grafana_container_name }}"
 image: "{{ grafana_image }}"
 state: started
 restart_policy: always
 recreate: "{{ grafana_docker_image_pull.changed }}"
 ports: "{{ grafana_container_ports }}"
 volumes: "{{ grafana_container_volumes }}"

Main.yml

The role ties the phases together in tasks/main.yml:

1
2
3
4
5
6


---
- name: Run the host-preparation tasks
 ansible.builtin.import_tasks: prepare.yml

- name: Run the Docker setup
 ansible.builtin.import_tasks: deploy.yml

This completes the Grafana role.

Example Playbook

The final step is to actually use our role and run it on our Prometheus host. To do that, we will create playbooks/grafana.yml:

1
2
3
4
5
6
7
8


---
- name: Set up Grafana
 hosts: grafana
 become: true
 roles:
 - grafana
 tags:
 - grafana

Then we run our playbook using the command:

1

ansible-playbook -i inventory playbooks/grafana.yml --tags=grafana

Localhost Deployment

You can perform a safe, local deployment for testing purposes:

1
2
3
4
5
6
7
8


- name: Install Grafana on localhost
 hosts: localhost
 become: false
 gather_facts: true
 roles:
 - grafana
 tags:
 - grafana

Then we can run the playbook using the command:

1

ansible-playbook playbooks/grafana.yml --tags=grafana

You can now access Grafana from the browser at 127.0.0.1:3000 with the default credentials:

1
2


username: admin
password: admin

Verification

Once the playbook finishes execution, we can verify Grafana is up and running by:

Checking running containers:

1

docker ps

Querying the endpoint

1

curl http://<ansible_host>:3000/api/health

If all goes well, you should get:

1
2
3
4
5


{
 "database": "ok",
 "version": "13.0.1",
 "commit": "<hash>"
}

We can also create a dedicated Ansible task for performing the verification for us, making it more suitable for a production environment.

Conclusion

In this post, we built a reusable Ansible role for deploying Grafana as a Docker container while also provisioning the data sources and Node Exporter Full dashboard.

The complete source code for this post can be found on GitHub.

So far, we have a basic monitoring stack setup that can be easily deployed on demand. But we are still missing a few key components to transform it into a more production-level solution. One of these critical components is Alerts, and that will be covered in the next post.

Grafana on HabibiOps