Docker on HabibiOps

Setting Up Monitoring with Prometheus and Grafana - Part 3 - Grafana

Sat, 18 Apr 2026 00:00:00 +0000

Introduction

In the previous blog posts part 1 and part 2, we have Node Exporter and Prometheus set up. We can already monitor our VMs and get valuable information. However, running PromQL queries repeatedly is not practical for day-to-day monitoring or visualization.

For that, we will deploy Grafana and import a pre-built dashboard so that we can visualize the scraped metrics. By the end of this post, we will have:

A reusable Grafana Ansible role
Provisioning of the data source and dashboards
Docker volume for persisting the Grafana data

Prerequisites

For the demo in this post to work, you need to have:

One or more target VMs
Node Exporter must be preconfigured and set up
Docker to be installed and available on the Prometheus server
community.docker to be installed
An existing Prometheus instance

Architecture Overview

For simplicity, we are going to focus only on Grafana and its immediate components:

A dedicated host for Grafana with an internal IP
Grafana runs as a Docker container and exposes the port 3000
A Docker volume will mount onto the container for persisting grafana-related data

Once all is done, we will have the full architecture setup:

Grafana

Similar to Prometheus, Grafana can be installed directly on the host or deployed as a container. At this point, it is clear that we will deploy Grafana as a container. The Ansible role will be broken down into two phases:

Prepare: Create the directories, render the configuration template and create the Docker volume
Deploy: Pull, run the image and mount the created volume onto the container

Variables

The variables for this role will be split into five categories:

Paths and provisioning
Prometheus integration
Container integration
Storage and volumes
Endpoint

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33


---
grafana_provisioning_dir: "/opt/grafana/provisioning"
grafana_datasources_dir: "{{ grafana_provisioning_dir }}/datasources"

grafana_prometheus_datasource: "{{ grafana_datasources_dir }}/prometheus.yml"

# prometheus variables
grafana_prometheus_host: "prometheus"
grafana_prometheus_port: 9090

grafana_dashboards_dir: "{{ grafana_provisioning_dir }}/dashboards"
grafana_dashboard_provider_file: "{{ grafana_dashboards_dir }}/dashboard_provider.yml"
grafana_node_exporter_dashboard_file: "{{ grafana_dashboards_dir }}/node_exporter_full.json"

grafana_image_name: "grafana/grafana"
grafana_image_tag: "13.0"
grafana_image: "{{ grafana_image_name }}:{{ grafana_image_tag }}"

grafana_container_name: "grafana-instance"
grafana_listen_port: 3000
grafana_container_ports:
 - "{{ grafana_listen_port }}:{{ grafana_listen_port }}"

grafana_volume_name: "grafana-data"
grafana_container_volumes:
 - "{{ grafana_volume_name }}:/var/lib/grafana"
 - "{{ grafana_datasources_dir }}:/etc/grafana/provisioning/datasources:ro"
 - "{{ grafana_dashboards_dir }}:/etc/grafana/provisioning/dashboards:ro"

grafana_prometheus_datasource_url: "http://{{ grafana_prometheus_host }}:{{ grafana_prometheus_port }}"
grafana_prometheus_datasource_name: "Prometheus"

grafana_endpoint_url: "http://{{ ansible_host }}:{{ grafana_listen_port }}"

Preparation

Our focus for this phase is:

Creating config directories, which we will later bind-mount into the container
Render the data source and dashboard provisioning templates
Create the Docker volume, which will persist Grafana’s internal state (database, users, plugins). Dashboards and data sources will be provisioned via bind mounts

Dashboard Template

We define the dashboard provider template in templates/dashboards.yml.j2:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


apiVersion: 1

providers:
 - name: "default"
 orgId: 1
 folder: "Infrastructure"
 type: file
 disableDeletion: false
 updateIntervalSeconds: 30
 options:
 path: /etc/grafana/provisioning/dashboards

Prometheus Data Source Template

We define the template in templates/prometheus_datasource.yml.j2 so that we can pre-include our Prometheus data source:

1
2
3
4
5
6
7
8
9


apiVersion: 1

datasources:
 - name: "{{ grafana_prometheus_datasource_name }}"
 type: prometheus
 access: proxy
 url: "{{ grafana_prometheus_datasource_url }}"
 isDefault: true
 editable: false

Later in this series, we will further extend the template to include an Alert data source.

Deployment

The deployment phase consists of two steps:

Pulling the Docker image
Running the container with the volume and bind-mounts included

This can be done using these two tasks:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


---
- name: Pull Grafana Docker image
 community.docker.docker_image:
 name: "{{ grafana_image }}"
 source: pull
 register: grafana_docker_image_pull

- name: Run Grafana container
 community.docker.docker_container:
 name: "{{ grafana_container_name }}"
 image: "{{ grafana_image }}"
 state: started
 restart_policy: always
 recreate: "{{ grafana_docker_image_pull.changed }}"
 ports: "{{ grafana_container_ports }}"
 volumes: "{{ grafana_container_volumes }}"

Main.yml

The role ties the phases together in tasks/main.yml:

1
2
3
4
5
6


---
- name: Run the host-preparation tasks
 ansible.builtin.import_tasks: prepare.yml

- name: Run the Docker setup
 ansible.builtin.import_tasks: deploy.yml

This completes the Grafana role.

Example Playbook

The final step is to actually use our role and run it on our Prometheus host. To do that, we will create playbooks/grafana.yml:

1
2
3
4
5
6
7
8


---
- name: Set up Grafana
 hosts: grafana
 become: true
 roles:
 - grafana
 tags:
 - grafana

Then we run our playbook using the command:

1

ansible-playbook -i inventory playbooks/grafana.yml --tags=grafana

Localhost Deployment

You can perform a safe, local deployment for testing purposes:

1
2
3
4
5
6
7
8


- name: Install Grafana on localhost
 hosts: localhost
 become: false
 gather_facts: true
 roles:
 - grafana
 tags:
 - grafana

Then we can run the playbook using the command:

1

ansible-playbook playbooks/grafana.yml --tags=grafana

You can now access Grafana from the browser at 127.0.0.1:3000 with the default credentials:

1
2


username: admin
password: admin

Verification

Once the playbook finishes execution, we can verify Grafana is up and running by:

Checking running containers:

1

docker ps

Querying the endpoint

1

curl http://<ansible_host>:3000/api/health

If all goes well, you should get:

1
2
3
4
5


{
 "database": "ok",
 "version": "13.0.1",
 "commit": "<hash>"
}

We can also create a dedicated Ansible task for performing the verification for us, making it more suitable for a production environment.

Conclusion

In this post, we built a reusable Ansible role for deploying Grafana as a Docker container while also provisioning the data sources and Node Exporter Full dashboard.

The complete source code for this post can be found on GitHub.

So far, we have a basic monitoring stack setup that can be easily deployed on demand. But we are still missing a few key components to transform it into a more production-level solution. One of these critical components is Alerts, and that will be covered in the next post.

Setting Up Monitoring with Prometheus and Grafana - Part 2 - Prometheus

Sun, 05 Apr 2026 00:00:00 +0000

Introduction

In the previous post, we have covered how to deploy Node Exporter using our custom-built Ansible role. Node Exporter is only the first piece of the puzzle in building a monitoring stack solution. The next critical component in the monitoring stack is Prometheus.

By default, Prometheus uses local storage for persisting metrics data. These metrics are stored in a built-in Time Series Database (TSDB). Prometheus also supports integration with various remote storage solutions. For this post, we will focus on local storage and use a Docker volume to persist metrics data even when the container is restarted.

By the end of this post, you will have:

A reusable Prometheus Ansible role
Persistent metrics storage using Docker volumes
Automatic discovery of Node Exporter targets

Prerequisites

For the demo in this post to work, you need to have:

One or more target VMs
Node Exporter must be preconfigured and set up
Docker to be installed and available on the Prometheus server
community.docker to be installed

Architecture Overview

We will first look at the architecture, then move on to implementation.

Ansible host that runs the deployment
Target VMs that have Node Exporter installed and running as a systemd service
A Prometheus host configured to monitor the target VMs

Prometheus follows a pull-based model, periodically scraping metrics from configured targets. The diagram below illustrates the setup:

Prometheus

Prometheus can be installed directly on the host or deployed as a container. By now, it’s recommended to use containerized deployments, and this is what we will be doing. Our Prometheus Ansible role will be broken down into two main phases:

Prepare: Create the directories, render the Prometheus configuration template and create the Docker volume
Deploy: Pull, run the image and mount the created volume onto the container

Core Variables

Just as in the Node Exporter, we will utilize variables to make our role flexible and adaptable without changing the internal implementation. The main variables are:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


prometheus_provisioning_dir: "/opt/prometheus"
prometheus_config_dir: "{{ prometheus_provisioning_dir }}/config"
prometheus_config_file: "{{ prometheus_config_dir }}/prometheus.yml"

prometheus_port: 9090

prometheus_image_name: "prom/prometheus"
prometheus_image_tag: "v3"
prometheus_image: "{{ prometheus_image_name }}:{{ prometheus_image_tag }}"
prometheus_container_name: "prometheus-instance"
prometheus_volume_name: "prometheus-data"

prometheus_container_ports:
 - "{{ prometheus_port }}:{{ prometheus_port }}"

prometheus_container_volumes:
 - "{{ prometheus_volume_name }}:/prometheus"
 - "{{ prometheus_config_dir }}:/etc/prometheus:ro"

We mount the configuration as read-only (/etc/prometheus:ro) to prevent accidental modification from inside the container.

Preparation

Our focus for this phase is:

Creating config directories, which we will later bind-mount into the container
Render the Prometheus configuration template
Create the Docker volume, which will store Prometheus’ TSDB

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


---
- name: Create Prometheus provisioning directories
 ansible.builtin.file:
 path: "{{ item }}"
 state: directory
 mode: "0755"
 loop:
 - "{{ prometheus_provisioning_dir }}"
 - "{{ prometheus_config_dir }}"

- name: Render Prometheus configuration
 ansible.builtin.template:
 src: prometheus.yml.j2
 dest: "{{ prometheus_config_file }}"
 mode: "0644"
 register: prometheus_config

- name: Create Prometheus data volume
 community.docker.docker_volume:
 name: "{{ prometheus_volume_name }}"

Prometheus Config Jinja Template

We will be using the following templates/prometheus.yml.j2 template:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


...
scrape_configs:
 - job_name: "prometheus"
 metrics_path: "/metrics"
 scrape_interval: 15s
 static_configs:
 - targets: ["{{ ansible_host }}:{{ prometheus_port }}"]

 - job_name: "nodes"
 metrics_path: "/metrics"
 scrape_interval: 15s
 static_configs:
 - targets:
 {% for host in groups['targets'] %}
 - "{{ hostvars[host]['ansible_host'] }}:{{ prometheus_node_exporter_port }}"
 {% endfor %}

There are two scrape configs:

Prometheus, allowing it to expose and monitor its own internal metrics
nodes or the target VMs.

The targets are dynamically generated as we loop over the inventory groups['targets'] and we set the metrics endpoint. The metrics endpoint has the format of <ansible_host>:<node_exporter_port>. Each target exposes its metrics via an HTTP endpoint (typically /metrics), which Prometheus periodically scrapes. This is defined in the configuration file via metrics_path: "/metrics".

Below is an example of a rendered prometheus.yml template:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


global:
 scrape_interval: 15s
 evaluation_interval: 15s

scrape_configs:
 - job_name: "prometheus"
 metrics_path: "/metrics"
 scrape_interval: 15s
 static_configs:
 - targets: ["prometheus:9090"]

 - job_name: "nodes"
 metrics_path: "/metrics"
 scrape_interval: 15s
 static_configs:
 - targets:
 - "vm-1:9100"
 - "vm-2:9100"

Rendering of the hosts depends on how ansible_host is defined in your inventory. On localhost, this will be resolved to 127.0.0.1

Deployment

The deployment phase consists of two steps:

Pulling the Docker image
Running the container with the volume and bind-mounts included

This can be done using these two tasks:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


---
- name: Pull Prometheus image
 community.docker.docker_image:
 name: "{{ prometheus_image }}"
 source: pull
 register: prometheus_image_pull

- name: Run Prometheus container
 community.docker.docker_container:
 name: "{{ prometheus_container_name }}"
 image: "{{ prometheus_image }}"
 state: started
 restart_policy: always
 recreate: "{{ prometheus_config.changed or prometheus_image_pull.changed }}"
 ports: "{{ prometheus_container_ports }}"
 volumes: "{{ prometheus_container_volumes }}"

Main.yml

The role ties the phases together in tasks/main.yml:

1
2
3
4
5
6


---
- name: Run Prepare tasks
 ansible.builtin.import_tasks: prepare.yml

- name: Run Deploy tasks
 ansible.builtin.import_tasks: deploy.yml

This completes our Prometheus role.

Example Playbook

The final step is to actually use our role and run it on our Prometheus host. To do that, we will create playbooks/prometheus.yml:

1
2
3
4
5
6
7
8


---
- name: Set up Prometheus
 hosts: prometheus
 become: true
 roles:
 - prometheus
 tags:
 - prometheus

Then we run our playbook using the command:

1

ansible-playbook -i inventory playbooks/prometheus.yml --tags=prometheus

Localhost Deployment

You can perform a safe, local deployment in case you do not have a remote host:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


- name: Install Prometheus on localhost
 hosts: localhost
 become: false
 gather_facts: true
 vars:
 prometheus_provisioning_dir: "./tmp/prometheus"
 ansible_host: "127.0.0.1"
 roles:
 - prometheus
 tags:
 - prometheus

Since we’re running on localhost, we don’t need to explicitly specify an inventory file. So we can run our playbook directly:

1

ansible-playbook playbooks/prometheus.yml --tags=prometheus

You can now access Prometheus from the browser at 127.0.0.1:9090

Verification

Once the playbook finishes execution, we can verify Prometheus is up and running by:

Checking running containers:

1

docker ps

Querying the endpoint

1

curl http://<ansible_host>:9090/-/healthy

If all goes well, you should get Prometheus Server is Healthy. as an output.

We can also create a dedicated Ansible task for performing the verification for us, making it more suitable for a production environment.

We can also access the dashboard from our browser and verify that our targets are being monitored:

Conclusion

In this post, we built a reusable Ansible role for deploying Prometheus as a Docker container. By structuring the role into clear phases and driving it through variables, we ensure maintainability and consistency across environments. The role is also written to be easily adjustable by modifying variables without changing the internal implementation.

The complete source code for this post can be found on GitHub.

By now, we have Node Exporter installed on the target hosts and Prometheus pulling the metrics. The next step is to visualize and explore these metrics using Grafana.

Static Site Deployment on S3 using Containerized LocalStack

Sat, 10 Jan 2026 00:00:00 +0000

Introduction

LocalStack is a local cloud development solution that emulates AWS services in a local environment. This enables developers to build and test various AWS-based solutions on a small scale without requiring an AWS account.

In this post, LocalStack is used to test the deployment of a static website to a S3 bucket. Hugo, a popular static site generator, is used to build the site. The following topics are covered in more detail throughout this post:

Creating a containerized environment that includes:
- LocalStack
- Hugo
S3 bucket creation and configuration for static website hosting
Building and deploying the static site to the S3 bucket
Automating the deployment process

The entire source code and setup is available on GitHub.

This post does not cover the inner workings of Hugo or how to set up a Hugo static site. Instead, it focuses solely on the deployment aspect. Hugo themes can be found here.

Setup of the Containerized Environment

LocalStack provides a pre-built Docker image that’s available on Docker Hub, offering a fully functional local AWS cloud stack in a containerized environment. However, the image doesn’t include any Hugo-related dependencies to build a static website. For that, we will create a new image based on the LocalStack image and include the additional dependencies.

The Dockerfile below implements the initial basic setup:

1
2
3
4
5
6
7
8
9


FROM localstack/localstack

RUN apt-get update && apt-get install -y --no-install-recommends \
 hugo \
 ca-certificates \
 git \
 golang

EXPOSE 4566

Details for installing Hugo can be found from the official documentation

We can then build the image once the Dockerfile is set up by using the command:

1

podman build . -t localstack-gohugo

Finally, we can run the container by using the command:

1
2
3
4
5


podman run \
 -it \
 -p 4566:4566 \
 -v ./hugo-blog:/opt/code/localstack/hugo-blog \
 --name localstack-gohugo localstack-gohugo

./hugo-blog can be an empty directory, in which Hugo can be set up within the container, or it can be a directory that already contains a Hugo website.

Accessing the container can be done by opening a new terminal and executing the following command:

1

podman exec -it localstack-gohugo bash

At this point, we have a running localstack-gohugo container that has the following built-in dependencies set up:

LocalStack
Hugo and its dependencies

However, we do not yet have the actual final build of the website or any deployable, site-related output. In the next sections, we will walk through the deployment process within the container to better understand how the manual process works. Afterward, we will demonstrate a better and enhanced automated way to perform the deployment.

S3 Bucket Setup

Two simple steps are needed to set up the S3 bucket:

Create the bucket
Apply a bucket policy to allow public access

Bucket creation in LocalStack is done using the command below:

1

awslocal s3 mb s3://testwebsite

awslocal is used instead of aws for all AWS-related APIs when using LocalStack.

Once the bucket has been created, a policy has to be set and applied to the bucket so that the website can be publicly accessible. Therefore, the following bucket policy will be set:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Sid": "PublicReadGetObject",
 "Effect": "Allow",
 "Principal": "*",
 "Action": "s3:GetObject",
 "Resource": "arn:aws:s3:::testwebsite/*"
 }
 ]
}

The bucket IAM policy allows anyone ("Principal": "*") to get read-only access ("Action": "s3:GetObject") to the website contents on the S3 bucket ("Resource": "arn:aws:s3:::testwebsite/*").

The official AWS Documentation contains more thorough details regarding the attributes.

Lastly, the policy is applied to the bucket by using the following command:

1

awslocal s3api put-bucket-policy --bucket testwebsite --policy file://bucket_policy.json

Deploying to S3

Several steps are required before we can deploy to our S3 bucket, namely:

Properly set up the hugo.toml configuration file
Build the static site using Hugo: hugo --gc --minify
Deploy the site to the S3-bucket: awslocal s3 sync ./ s3://testwebsite

Configuring hugo.toml

The hugo.toml config file is already properly set when building the image; however, this section explains how to set the configuration file.

The configuration file is human-readable, and we only need to set four parameters to get started. With LocalStack, the S3 website endpoint accepts several formats; the most common is the virtual-hosted style: http://<BUCKET_NAME>.s3-website.localhost.localstack.cloud:4566. Hence, the configuration file looks as follows:

1
2
3
4


baseURL = 'http://testwebsite.s3-website.localhost.localstack.cloud:4566'
languageCode = 'en-us'
title = 'My Hugo Site'
theme = 'mixedpaper'

It’s possible to set a section for deployments; however, it doesn’t seem to be well integrated with LocalStack, so this will be skipped for the time being.

Building the Website

A single command is needed to build the website:

1

hugo

A single command is needed to build the website:

1

hugo

Hugo will process all the Markdown content files and templates, generating a complete, hostable website in the public directory. The source files (Markdown, partials, etc.) will be converted into HTML/CSS/JS, which will be stored in the public directory. For better visualization, the structure of the public folder is presented below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


tree -L 1
.
├── 404.html
├── archives
├── categories
├── favicon.png
├── index.html
├── index.xml
├── links
├── p
├── page
├── post
├── scss
├── search
├── sitemap.xml
├── tags
└── ts

For additional details, check Hugo’s official documentation.

Once the website has been built and all the content is generated, the index page has to be set using the command:

1
2


# inside the generated public directory
awslocal s3 website s3://testwebsite/ --index-document index.html

Deploying to S3 Bucket

Deploying to the S3 bucket simply means uploading the public directory to the bucket. This can be achieved using the s3 sync command as follows:

1

awslocal s3 sync ./ s3://testwebsite

The site is now hosted on the S3 bucket testwebsite. To test, simply open the browser and paste the base URL: http://testwebsite.s3-website.localhost.localstack.cloud:4566.

It might not work on Chrome, Firefox, or other Chromium-based browsers. Edit the /etc/hosts file on your machine and add the line 127.0.0.1 testwebsite.s3-website.localhost.localstack.cloud for it to work. For more info, check the resolved GitHub issue.

Further Automation

To reduce the setup, a helper script was implemented and the Dockerfile was adjusted as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


FROM localstack/localstack

RUN apt-get update \
 && apt-get install -y --no-install-recommends \
 hugo \
 ca-certificates \
 git \
 golang

WORKDIR /opt/code/localstack

COPY . .

RUN chmod +x ./setup.sh \
 && mkdir -p /etc/localstack/init/ready.d \
 && mv ./setup.sh /etc/localstack/init/ready.d/init-aws.sh

EXPOSE 4566

Below is a brief overview of the updated Dockerfile:

Copy the entire working directory into the container so that we can avoid using the -v mount option.
Set the script to executable and move it to /etc/localstack/init/ready.d/init-aws.sh so that it is automatically executed by LocalStack once the container is running.

Check the LocalStack initialization hooks for more info regarding the lifecycle.

With the newly enhanced and improved Dockerfile, the container is ready to use out of the box, without the need to perform any manual tasks.

Setup Script

Below is the implemented script for automating the setup within the container:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


#!/bin/bash

# Define variables
BASE_DIR="/opt/code/localstack"
SITE_DIR="$BASE_DIR/hugo-mixedpaper"
BUCKET="testwebsite"
POLICY_FILE="bucket_policy.json"

# Set up the bucket
awslocal s3 mb s3://$BUCKET
awslocal s3api put-bucket-policy --bucket $BUCKET --policy file://$POLICY_FILE

# Build the site
cd $SITE_DIR && hugo

# Deploy to S3
cd "$SITE_DIR/public" && awslocal s3 sync . s3://$BUCKET

# Set the index page
awslocal s3 website s3://$BUCKET/ --index-document index.html

The script above handles:

Creation of the bucket.
Applying the bucket_policy.json file.
Building the website using hugo and generating the public folder.
Syncing the public folder to the S3 bucket.
Setting the index-document for the bucket.

Once the image is built and the container is running, we can directly access the website at the following endpoint: http://testwebsite.s3-website.localhost.localstack.cloud:4566

Don’t forget to update the /etc/hosts file to add the line 127.0.0.1 testwebsite.s3-website.localhost.localstack.cloud for it to work.

Conclusion

The setup explored in this post is mainly for local development and emulating a production-like environment. In short, we’ve managed to:

Extend the LocalStack container image to integrate Hugo and other dependencies.
Implement a script to automate the setup inside the container by performing the following tasks:
- S3 bucket creation and configuration
- Building and deploying the site to the created S3 bucket

The complete source code used here can be found here on GitHub.