Introduction

In the previous blog posts part 1 and part 2, we have Node Exporter and Prometheus set up. We can already monitor our VMs and get valuable information. However, running PromQL queries repeatedly is not practical for day-to-day monitoring or visualization.

For that, we will deploy Grafana and import a pre-built dashboard so that we can visualize the scraped metrics. By the end of this post, we will have:

• A reusable Grafana Ansible role
• Provisioning of the data source and dashboards
• Docker volume for persisting the Grafana data

Prerequisites

For the demo in this post to work, you need to have:

• One or more target VMs
• Node Exporter must be preconfigured and set up
• Docker to be installed and available on the Prometheus server
• community.docker to be installed
• An existing Prometheus instance

Architecture Overview

For simplicity, we are going to focus only on Grafana and its immediate components:

• A dedicated host for Grafana with an internal IP
• Grafana runs as a Docker container and exposes the port 3000
• A Docker volume will mount onto the container for persisting grafana-related data

Once all is done, we will have the full architecture setup:

Grafana

Similar to Prometheus, Grafana can be installed directly on the host or deployed as a container. At this point, it is clear that we will deploy Grafana as a container. The Ansible role will be broken down into two phases:

• Prepare: Create the directories, render the configuration template and create the Docker volume
• Deploy: Pull, run the image and mount the created volume onto the container

Variables

The variables for this role will be split into five categories:

• Paths and provisioning
• Prometheus integration
• Container integration
• Storage and volumes
• Endpoint

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

---
grafana_provisioning_dir: "/opt/grafana/provisioning"
grafana_datasources_dir: "{{ grafana_provisioning_dir }}/datasources"

grafana_prometheus_datasource: "{{ grafana_datasources_dir }}/prometheus.yml"

# prometheus variables
grafana_prometheus_host: "prometheus"
grafana_prometheus_port: 9090

grafana_dashboards_dir: "{{ grafana_provisioning_dir }}/dashboards"
grafana_dashboard_provider_file: "{{ grafana_dashboards_dir }}/dashboard_provider.yml"
grafana_node_exporter_dashboard_file: "{{ grafana_dashboards_dir }}/node_exporter_full.json"

grafana_image_name: "grafana/grafana"
grafana_image_tag: "13.0"
grafana_image: "{{ grafana_image_name }}:{{ grafana_image_tag }}"

grafana_container_name: "grafana-instance"
grafana_listen_port: 3000
grafana_container_ports:
  - "{{ grafana_listen_port }}:{{ grafana_listen_port }}"

grafana_volume_name: "grafana-data"
grafana_container_volumes:
  - "{{ grafana_volume_name }}:/var/lib/grafana"
  - "{{ grafana_datasources_dir }}:/etc/grafana/provisioning/datasources:ro"
  - "{{ grafana_dashboards_dir }}:/etc/grafana/provisioning/dashboards:ro"

grafana_prometheus_datasource_url: "http://{{ grafana_prometheus_host }}:{{ grafana_prometheus_port }}"
grafana_prometheus_datasource_name: "Prometheus"

grafana_endpoint_url: "http://{{ ansible_host }}:{{ grafana_listen_port }}"

Preparation

Our focus for this phase is:

• Creating config directories, which we will later bind-mount into the container
• Render the data source and dashboard provisioning templates
• Create the Docker volume, which will persist Grafana’s internal state (database, users, plugins). Dashboards and data sources will be provisioned via bind mounts

Dashboard Template

We define the dashboard provider template in templates/dashboards.yml.j2:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

apiVersion: 1

providers:
  - name: "default"
    orgId: 1
    folder: "Infrastructure"
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    options:
      path: /etc/grafana/provisioning/dashboards

Prometheus Data Source Template

We define the template in templates/prometheus_datasource.yml.j2 so that we can pre-include our Prometheus data source:

1
2
3
4
5
6
7
8
9

apiVersion: 1

datasources:
  - name: "{{ grafana_prometheus_datasource_name }}"
    type: prometheus
    access: proxy
    url: "{{ grafana_prometheus_datasource_url }}"
    isDefault: true
    editable: false

Later in this series, we will further extend the template to include an Alert data source.

Deployment

The deployment phase consists of two steps:

• Pulling the Docker image
• Running the container with the volume and bind-mounts included

This can be done using these two tasks:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

---
- name: Pull Grafana Docker image
  community.docker.docker_image:
    name: "{{ grafana_image }}"
    source: pull
  register: grafana_docker_image_pull

- name: Run Grafana container
  community.docker.docker_container:
    name: "{{ grafana_container_name }}"
    image: "{{ grafana_image }}"
    state: started
    restart_policy: always
    recreate: "{{ grafana_docker_image_pull.changed }}"
    ports: "{{ grafana_container_ports }}"
    volumes: "{{ grafana_container_volumes }}"

Main.yml

The role ties the phases together in tasks/main.yml:

1
2
3
4
5
6

---
- name: Run the host-preparation tasks
  ansible.builtin.import_tasks: prepare.yml

- name: Run the Docker setup
  ansible.builtin.import_tasks: deploy.yml

This completes the Grafana role.

Example Playbook

The final step is to actually use our role and run it on our Prometheus host. To do that, we will create playbooks/grafana.yml:

1
2
3
4
5
6
7
8

---
- name: Set up Grafana
  hosts: grafana
  become: true
  roles:
    - grafana
  tags:
    - grafana

Then we run our playbook using the command:

1

ansible-playbook -i inventory playbooks/grafana.yml --tags=grafana

Localhost Deployment

You can perform a safe, local deployment for testing purposes:

1
2
3
4
5
6
7
8

- name: Install Grafana on localhost
  hosts: localhost
  become: false
  gather_facts: true
  roles:
    - grafana
  tags:
    - grafana

Then we can run the playbook using the command:

1

ansible-playbook playbooks/grafana.yml --tags=grafana

You can now access Grafana from the browser at 127.0.0.1:3000 with the default credentials:

1
2

username: admin
password: admin

Verification

Once the playbook finishes execution, we can verify Grafana is up and running by:

• Checking running containers:

1

docker ps

• Querying the endpoint

1

curl http://<ansible_host>:3000/api/health

If all goes well, you should get:

1
2
3
4
5

{
  "database": "ok",
  "version": "13.0.1",
  "commit": "<hash>"
}

We can also create a dedicated Ansible task for performing the verification for us, making it more suitable for a production environment.

Conclusion

In this post, we built a reusable Ansible role for deploying Grafana as a Docker container while also provisioning the data sources and Node Exporter Full dashboard.

The complete source code for this post can be found on GitHub.

So far, we have a basic monitoring stack setup that can be easily deployed on demand. But we are still missing a few key components to transform it into a more production-level solution. One of these critical components is Alerts, and that will be covered in the next post.