Fio on HabibiOps

I/O Benchmarks with FIO - Part 2 - Automation

Sun, 22 Feb 2026 00:00:00 +0000

Introduction

In the previous post, we had a deep dive over the basics of fio and the different ways it can be fine-tuned to get a proper benchmark. That was also one of the main issues we faced before: there are plenty of combinations and it’s not feasible to test all of them. In this post, we’ll be building on top of our previous knowledge and go over how we can automate this process. Mainly, we will go over:

Auto-generating fio commands.
Parsing all the json files and converting them to a condensed csv summaries.
Plotting the csv file to visualize and analyze the results.

Auto-generating FIO Commands

You might have noticed by now that there are so many different options and combinations to test. What if we want to tinker with the number of jobs and/or iodepth? To see if our disk performance scales linearly and if so, to what extent? How would a different blocksize impact our IO performance? Would the IO run perform consistently for longer periods of time?

We can keep on running one command after the other, but that becomes a pain, error-prone and boring too quickly. For that, we will implement a Python script to generate the commands for us. One approach that I rely on is to have a JSON config file, specifying the different parameters and options that we’d like to test. The config file would look something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53


{
 "benchmark_type": "time",
 "volumes": [
 "/dev/vdb"
 ],
 "ioengines": [
 "libaio"
 ],
 "blocksize": [
 "4k",
 "8k",
 "16k",
 "32k",
 "64k",
 "128k",
 "256k",
 "512k",
 "1024k",
 "2048k",
 "4096k"
 ],
 "numjobs": [
 1,
 2,
 4,
 8,
 16,
 32
 ],
 "iodepths": [
 1,
 2,
 4,
 8,
 16,
 32
 ],
 "direct": [
 1
 ],
 "ramp_time": 5,
 "runtime": [
 30
 ],
 "rw": [
 "read",
 "randread",
 "write",
 "randwrite",
 "rw",
 "randrw"
 ]
}

With this config, a total of 2376 different commands will be generated! So be mindful when setting up the config and be reasonable what to test. Otherwise, the test may take several days to finish.

Parsing the JSON Config File

The method below will load the JSON file and return a JSON object, or a dict in Python:

1
2
3
4
5
6
7
8
9


import json
import itertools

CONFIG_FILE = "config.json"


def load_config(path):
 with open(path, "r") as f:
 return json.load(f)

Generate Commands from JSON Config

Once the JSON config is loaded, we need to pass it to the generate_fio_commands method. The method can be divided into two parts:

Generating the combinations using itertools.product.
Looping over the list and building the fio commands.

Using `itertools.product`

Python has a built-in itertools to generate the Cartesian product of our fio parameters. We’ll be taking advantage of it and using it as follows:

1
2
3
4
5
6
7
8
9


def generate_fio_commands(cfg):
 combinations = itertools.product(
 cfg["volumes"],
 cfg["ioengines"],
 cfg["blocksize"],
 cfg["numjobs"],
 cfg["iodepths"],
 cfg["rw"],
 )

This will generate an array, or a list, containing all the combinations. This simplifies a lot of the work for us, as we would just need to loop over it and start building our command.

You can check the official documentation for further info

Building the FIO Commands

Lastly, we need to loop over the generated list and construct our commands:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


def generate_fio_commands(cfg):
 ...
 for volume, ioengine, blocksize, numjobs, iodepth, rw in combinations:
 name = f"{rw}_{blocksize}_{numjobs}_{iodepth}"
 cmd = (
 "sudo fio "
 f"--name=name "
 f"--filename={volume} "
 f"--ioengine={ioengine} "
 f"--bs={blocksize} "
 f"--numjobs={numjobs} "
 f"--iodepth={iodepth} "
 f"--rw={rw} "
 f"--direct={cfg['direct']} "
 f"--ramp_time={cfg['ramp_time']} "
 f"--runtime={cfg['runtime']} "
 f"--time_based "
 f"--output-format=json "
 f"> {name}.json"
 )
 commands.append(cmd)

 return commands

Implementing the `main` Method

All we need now is to implement the main method and put everything together:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


def main():
 config = load_config(CONFIG_FILE)
 commands = generate_fio_commands(config)

 with open("assets/fio_commands.sh", "w") as f:
 # add the shell header line
 f.write("#!/bin/bash\n")
 for cmd in commands:
 f.write(f"{cmd}\n\n")

 print(f"\nGenerated {len(commands)} fio commands")


if __name__ == "__main__":
 main()

And now we’re done! The script will generate a total of 2376 fio commands and write them to a valid fio_commands.sh file. The only “manual” step that we have to do is to run the generated fio_commands.sh file. Though we can technically extend our Python script to run that bash script, but for now, that will suffice.

Parsing FIO Output

The last piece of the puzzle is to parse the JSON output. Just as before, we’re going to break our script into different sections. What we mainly want to do is:

Load the JSON output (same as before, so will be skipped).
Extract the key metrics.
Print the results/summary.

Extracting the Key Metrics

The main values we care about from the JSON output are mainly:

iops
latency
bandwidth

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


def parse_fio_json(path):
 # load JSON output
 data = load_output(path)
 results = []

 for job in data.get("jobs", []):
 job_name = job.get("jobname", "unknown")

 for io_op in ("read", "write"):
 stats = job.get(io_op, {})
 # ignore and move to the next op
 if not stats or stats.get("io_bytes", 0) == 0:
 continue

 # MB/s
 bandwidth_mbps = stats.get("bw", 0) / 1024
 iops = stats.get("iops", 0)
 latency_ns = stats.get("lat_ns", {}).get("mean", 0)

 results.append({
 "job": job_name,
 "io_op": io_op,
 "bandwidth_mbps": bandwidth_mbps,
 "iops": iops,
 "latency_ns": latency_ns,
 })

 return results

Print the Results

We implement our main method to display the summary of our fio output:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


def main():
 results = parse_fio_json(OUTPUT_FILE)

 for r in results:
 print(
 f"Job: {r['job']}\n"
 f"Mode: {r['io_op']}\n"
 f"BW: {r['bandwidth_mbps']:>.2f} MB/s\n"
 f"IOPS: {r['iops']:>.2f}\n"
 f"Latency: {r['latency_ns'] / 1e6:>.2f} ms"
 )


if __name__ == "__main__":
 main()

Results

To keep things a bit simpler, I’ve generated only 3 commands and have plotted them. The code is still written in a way that’s generic to handle any number of commands, to parse through the results, and plot them as well.

Here’s the config.json that I’ve used:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


{
 "benchmark_type": "time",
 "volumes": [
 "/dev/vdb"
 ],
 "ioengines": [
 "libaio"
 ],
 "blocksize": [
 "4k",
 "64k",
 "1024k"
 ],
 "numjobs": [
 1
 ],
 "iodepths": [
 1
 ],
 "direct": [
 1
 ],
 "ramp_time": 5,
 "runtime": [
 30
 ],
 "rw": [
 "read"
 ]
}

Basically testing small, medium, and large blocksizes and how our I/O behaves accordingly. The plot before summarizes the results for us:

Initially, we can notice that 64k seems to be the sweetspot for the blocksize, having achieved the highest throughput while also maintaining decent IOPs and minimal sacrifice in terms of latency.

Conclusion

By now, you should be able to automate your I/O benchmarking with fio and even adapt the scripts to your needs. The config.json file provides plenty of flexibility to test and try out all sorts of combinations. The complete source code for this post can be found on GitHub.

It’s possible to generate nearly all tests and experiments for all the I/O operations (read, write, rw, etc.) and compare the performance of each configuration to better fine-tune and optimize for your use case. The work can be expanded even further to a full CI/CD pipeline, having each step mentioned in the post (generating commands, executing commands, parsing, etc.) as a separate stage. This concludes our I/O benchmarking, having covered the basics and advanced techniques for automating our tests.

I/O Benchmarking with FIO - Part 1 - Basics

Sun, 08 Feb 2026 00:00:00 +0000

Introduction

Performance is often times an afterthought when it comes to Software Development and DevOps, kind of like how security is. No one seems to consider or care about it until suddenly everything’s too slow and taking forever to work.

More often than not, performance can be boiled down to I/O since storage is frequently the weakest link in the chain. Understanding how I/O works and possible options to fine-tune it can make wonders on your infrastructure. This is exactly what we’ll be discussing in this post today.

We’ll be going over the basics of I/O benchmarking and trying to emulate various workloads while using the open source tool fio, short for Flexible I/O tester.

Setup

Luckily we don’t need a fancy or complicated setup to do our benchmarking. All we need is:

One Linux VM with a distro of your choice. Personally I use Ubuntu 24.04 LTS.
An HDD volume attached to the VM.
fio for I/O benchmarking.

You can run the commands below to set up and mount your volume:

1
2
3
4
5
6
7


sudo mkdir /mnt/dev

# check where your volume's located 
lsblk
# any filesystem would work
sudo mkfs.ext4 /dev/vdb
sudo mount /dev/vdb /mnt/dev

Journaling and existing I/O workloads will affect the benchmark. So make sure to run on a clean system to get accurate measures

Installation

FIO can be installed using the package manager. Check the documentation for your own distro. In my case, it’ll be Ubuntu. So the installation is done using the command:

1

sudo apt install -y fio

FIO Overview

This section will focus mainly on high-level concept explanation. The technical, nitty-gritty details will come after.

fio has an overwhelming number of parameters to simulate all sorts of workloads. We’ll be focusing on a handful of key ones that are enough to cover most scenarios:

bs: block size
size: File size to read/write from
iodepth: Queue size for I/O submissions
numjobs: Number of processes to create and perform the I/O operations
rw: Type of the I/O operation to perform such read, write, rw etc. Check the documentation for further options.
runtime: The duration to perform the I/O test
direct: Boolean attribute, true if it’s set to 1 and 0 otherwise.
ioengine: Two most common engines are io_uring and libaio.
output-format: Defaults to shell-based format. json is recommended for parsing and automated processing.

fio runs in one of two modes. Size-based, until the size is reached. Time-based, until the runtime is exhausted. I’d recommend to set one, not both.

`numjobs` vs `iodepth`

This is a common point of confusion and can be misleading, often leading to very different results. In short:

numjobs controls the process-level parallelism meaning, fio would spawn and create different dedicated processes to perform the I/O task.
iodepth controls the queue size, or, depth, at the job level

The total number of maximum in-flight I/O requests is numjobs * iodepth.

Block size

Setting the correct and proper block size is tricky. It can mess the entire benchmark and give a false impression. Block size can be split into two broad categories:

Small block size: Typically 4k up to 16k in size. Useful for random I/O and database/transactional workloads. AWS uses 4k block size by default for their EBS service
Mid-range block size: Typically 16k up to 64k and is used for mixed workloads. MS SQL Server uses 64k block size by default
Large block size: 128k and up to 4M. Recommended for sequential reads, backups and data warehousing. AWS Redshift uses 1M block size

Each application has its own requirements and the I/O has to be fine-tuned for the workload itself. A good rule of thumb is to use 4k, 64k and 1M for small, middle and large block sizes respectively.

`ioengine`

libaio, which stands for Library Asynchronous I/O, used to be the default I/O engine on Linux. It’s not completely deprecated, since it’s still the recommended I/O engine for testing HDD storage and legacy async I/O behavior. io_uring would be the go-to for SSD and NVMe type storage.

FIO supports plenty of engines, including sync for synchronous I/O, mmap, windowsaio and plenty more. You can find the full list from the documentation.

Always read the documentation when choosing an engine, not all parameters are compatible with every engine.

direct vs non-direct I/O

Direct I/O means that the I/O goes directly to the disk and bypasses any OS/page cache, hence the name. Non-direct I/O, on the other hand, can be considered as buffered-I/O. Meaning that I/O operations are collected together, in a buffer ( OS page cache), and then written (or read) at the same time. Buffered I/O will almost always be faster than non-buffered I/O, though it’s not a fair comparison since they perform different things and serve different purposes. The natural question then becomes, when to use which? Use Direct I/O to test the raw hardware storage performance and non-direct I/O for testing your RAM and caching system.

To sum things up, as a rule of thumb:

Use direct=1 when benchmarking storage hardware
Use direct=0 when benchmarking the cache system

HDD vs SSD Considerations

Different storage types behave very differently, so tuning fio parameters for HDDs and SSDs is important. HDDs, being mechanical, are latency-bound and benefit from low queue depths and single-job workloads ( numjobs=1). Random reads/writes are slow, so small blocks (4k) are used for testing database-like workloads, while sequential tests can use larger blocks (1M) and are useful for user applications/workloads.

SSDs, on the other hand, are throughput-bound, handle high parallelism very well, and can achieve maximum performance with higher iodepth and multiple jobs. block sizes also vary depending on the workload that we’re trying to simulate.

Tuning numjobs, iodepth, and bs parameters according to the storage type is critical to ensure the benchmarks reflect the device’s true performance to avoid erroneous conclusions.

A good rule of thumb:

Storage Type	Recommended `numjobs`	Recommended `iodepth`	Typical `bs`	I/O Engine
HDD	1	1–8	4k (small) / 1M (large)	`libaio`
SSD / NVMe	4–16+	16–64+	4k–128k (random) / 1M+ (sequential)	`io_uring`

Our First FIO Command

We’re finally ready to test our first real FIO command after having built up a lot of background info and knowledge:

1
2
3
4
5
6
7
8
9


sudo fio --name=hello-fio-read \
 --numjobs=1 \
 --iodepth=1 \
 --rw=read \
 --directory=/mnt/dev/ \
 --ioengine=libaio \
 --bs=1M \
 --direct=1 \
 --runtime=30

The command above basically spawns a single process to do a read I/O at /mnt/dev/ directory using the libaio engine with a block size set to 1M. The process will perform O_DIRECT I/O operations (--direct=1) to avoid the cache, and it will do so for a total of 30 seconds. Having specified the --directory, fio will create the file for us and perform the task. Finally, when the process finishes, we get a summary of the output:

1
2
3
4
5
6
7


hello-fio: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=1
Jobs: 1 (f=1): [R(1)][100.0%][r=346MiB/s][r=346 IOPS][eta 00m:00s]
 read: IOPS=166, BW=166MiB/s (174MB/s)(4991MiB/30005msec)
 lat (usec): min=613, max=376663, avg=6008.79, stdev=16419.17
 iops : min= 20, max= 370, avg=164.86, stdev=87.26, samples=59
Run status group 0 (all jobs):
 READ: bw=166MiB/s (174MB/s), 166MiB/s-166MiB/s (174MB/s-174MB/s), io=4991MiB (5233MB), run=30005-30005msec

Some of the output has been removed for maintaining brevity.

Key metrics to focus on are mainly:

lat: Latency
BW: Bandwidth
iops: Number of I/O operations per second

Parameter Comparison

We’ve mentioned earlier the difference between direct and non-direct, big block size and small block size. So let’s see it in practice.

All commands and tests are done on the same machine to get accurate and consistent results

Small Block size vs Large Block size

We’ll be using this base command:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


sudo fio --name=hello-fio \
 --numjobs=1 \
 --iodepth=1 \
 --direct=1 \
 --rw=write \
 --directory=/mnt/dev \
 --ioengine=libaio \
 --runtime=30 \
 --size=1G \
 --time_based

Let’s start with bs=4k, we get:

1
2
3
4
5
6
7


hello-fio: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
 write: IOPS=895, BW=3582KiB/s (3668kB/s)(105MiB/30001msec); 0 zone resets
 lat (usec): min=358, max=29885, avg=1114.65, stdev=844.16
 iops : min= 412, max= 1564, avg=893.86, stdev=304.19, samples=59

Run status group 0 (all jobs):
 WRITE: bw=3582KiB/s (3668kB/s), 3582KiB/s-3582KiB/s (3668kB/s-3668kB/s), io=105MiB (110MB), run=30001-30001msec

And now with bs=1M, we get:

1
2
3
4
5
6


 write: IOPS=69, BW=69.8MiB/s (73.2MB/s)(2094MiB/30011msec); 0 zone resets
 lat (msec): min=6, max=222, avg=14.29, stdev=10.75
 iops : min= 38, max= 84, avg=69.76, stdev= 8.34, samples=59

Run status group 0 (all jobs):
 WRITE: bw=69.8MiB/s (73.2MB/s), 69.8MiB/s-69.8MiB/s (73.2MB/s-73.2MB/s), io=2094MiB (2196MB), run=30011-30011msec

We have the plot below (generated using Python’s matplot lib) to better summarize and visualize the results:

We can notice a pattern. The 4k block size has:

Low throughput (an impressive 3.49 MB/s)
Low latency
High IOPs

While the 1M block size is the complete opposite:

High throughput
High latency
Low IOPs

Direct vs Non-Direct

Now let’s see the difference with the direct parameter. Just as before, we’ll use this base command:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


sudo fio --name=hello-fio \
 --numjobs=1 \
 --iodepth=1 \
 --rw=write \
 --directory=/mnt/dev \
 --ioengine=libaio \
 --bs=4k \
 --runtime=30 \
 --size=1G \
 --time_based

It’s important to run both tests with the SAME block size to get as accurate of a comparison as possible

First, we’ll test direct=1 (same as the command in the earlier section):

1
2
3
4
5
6
7


hello-fio: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
 write: IOPS=895, BW=3582KiB/s (3668kB/s)(105MiB/30001msec); 0 zone resets
 lat (usec): min=358, max=29885, avg=1114.65, stdev=844.16
 iops : min= 412, max= 1564, avg=893.86, stdev=304.19, samples=59

Run status group 0 (all jobs):
 WRITE: bw=3582KiB/s (3668kB/s), 3582KiB/s-3582KiB/s (3668kB/s-3668kB/s), io=105MiB (110MB), run=30001-30001msec

Now running with direct=0 and we get:

1
2
3
4
5
6
7


 write: IOPS=128k, BW=499MiB/s (523MB/s)(14.6GiB/30001msec); 0 zone resets
 lat (usec): min=2, max=2795, avg= 3.15, stdev= 3.20
 bw ( KiB/s): min= 840, max=1247432, per=100.00%, avg=797612.32, stdev=385436.40, samples=37
 iops : min= 210, max=311858, avg=199403.24, stdev=96359.24, samples=37

Run status group 0 (all jobs):
 WRITE: bw=499MiB/s (523MB/s), 499MiB/s-499MiB/s (523MB/s-523MB/s), io=14.6GiB (15.7GB), run=30001-30001msec

Let’s plot the comparison just like before and see:

We have the following comparison:

Direct I/O performs so badly that we can barely see the throughput. We could’ve done some normalization and log scaling, but the point was to show the real drastic difference between the two.

Then again, seeing the results from the non-direct I/O, they are just too good to be true. This is the tricky part when it comes to benchmarking, knowing what results actually make sense, which ones are realistic, and which ones aren’t. Setting direct=0 will throw you off and you’d end up testing a completely different scenario (in this case, the RAM).

Testing Other Operations

Testing for other rw options is straightforward with FIO, we just need to change read to any of the supported options:

write
rw
randread
randwrite

The base command remains the same. So if we want to test our write performance, the command would be adjusted as such:

1
2
3
4
5
6
7
8
9


sudo fio --name=hello-fio-write \
 --numjobs=1 \
 --iodepth=1 \
 --rw=write \
 --directory=/mnt/dev/ \
 --ioengine=libaio \
 --bs=4k \
 --direct=1 \
 --runtime=30

And that’s it, we just had to change one parameter to test a completely different scenario. Which brings us to the next point, I/O testing can be, and is, very tricky because of that. A single parameter change would result in completely different outcomes and scenarios. So it’s important to carefully define what needs to be tested and which parameters to tinker with.

Conclusion

We’ve covered plenty of concepts and topics in this post, providing a solid foundation for running and interpreting fio benchmarks.

The key takeaway is that running a benchmark is only half the work. Understanding what you’re measuring and why is just as important. Small parameter changes can lead to vastly different results, and some trial and error is needed when defining realistic workloads. In the end, benchmark results are only meaningful if the workload matches reality.

In the next post, we’ll take this a step further by automating fio runs and diving deeper into the JSON output, including how to parse and analyze results at scale.

Fio on HabibiOps

I/O Benchmarks with FIO - Part 2 - Automation

Introduction

Auto-generating FIO Commands

Parsing the JSON Config File

Generate Commands from JSON Config

Using itertools.product

Building the FIO Commands

Implementing the main Method

Parsing FIO Output

Extracting the Key Metrics

Print the Results

Results

Conclusion

I/O Benchmarking with FIO - Part 1 - Basics

Introduction

Setup

Installation

FIO Overview

numjobs vs iodepth

Block size

ioengine

direct vs non-direct I/O

HDD vs SSD Considerations

Our First FIO Command

Parameter Comparison

Small Block size vs Large Block size

Direct vs Non-Direct

Testing Other Operations

Conclusion

Using `itertools.product`

Implementing the `main` Method

`numjobs` vs `iodepth`

`ioengine`