I/O Benchmarks with FIO - Part 2

Introduction

In the previous post, we had a deep dive over the basics of fio and the different ways it can be fine-tuned to get a proper benchmark. That was also one of the main issues we faced before: there are plenty of combinations and it’s not feasible to test all of them. In this post, we’ll be building on top of our previous knowledge and go over how we can automate this process. Mainly, we will go over:

Auto-generating fio commands.
Parsing all the json files and converting them to a condensed csv summaries.
Plotting the csv file to visualize and analyze the results.

Auto-generating FIO Commands

You might have noticed by now that there are so many different options and combinations to test. What if we want to tinker with the number of jobs and/or iodepth? To see if our disk performance scales linearly and if so, to what extent? How would a different blocksize impact our IO performance? Would the IO run perform consistently for longer periods of time?

We can keep on running one command after the other, but that becomes a pain, error-prone and boring too quickly. For that, we will implement a Python script to generate the commands for us. One approach that I rely on is to have a JSON config file, specifying the different parameters and options that we’d like to test. The config file would look something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
{
  "benchmark_type": "time",
  "volumes": [
    "/dev/vdb"
  ],
  "ioengines": [
    "libaio"
  ],
  "blocksize": [
    "4k",
    "8k",
    "16k",
    "32k",
    "64k",
    "128k",
    "256k",
    "512k",
    "1024k",
    "2048k",
    "4096k"
  ],
  "numjobs": [
    1,
    2,
    4,
    8,
    16,
    32
  ],
  "iodepths": [
    1,
    2,
    4,
    8,
    16,
    32
  ],
  "direct": [
    1
  ],
  "ramp_time": 5,
  "runtime": [
    30
  ],
  "rw": [
    "read",
    "randread",
    "write",
    "randwrite",
    "rw",
    "randrw"
  ]
}

With this config, a total of 2376 different commands will be generated! So be mindful when setting up the config and be reasonable what to test. Otherwise, the test may take several days to finish.

Parsing the JSON Config File

The method below will load the JSON file and return a JSON object, or a dict in Python:

1
2
3
4
5
6
7
8
9
import json
import itertools

CONFIG_FILE = "config.json"


def load_config(path):
    with open(path, "r") as f:
        return json.load(f)

Generate Commands from JSON Config

Once the JSON config is loaded, we need to pass it to the generate_fio_commands method. The method can be divided into two parts:

Generating the combinations using itertools.product.
Looping over the list and building the fio commands.

Using `itertools.product`

Python has a built-in itertools to generate the Cartesian product of our fio parameters. We’ll be taking advantage of it and using it as follows:

1
2
3
4
5
6
7
8
9
def generate_fio_commands(cfg):
    combinations = itertools.product(
        cfg["volumes"],
        cfg["ioengines"],
        cfg["blocksize"],
        cfg["numjobs"],
        cfg["iodepths"],
        cfg["rw"],
    )

This will generate an array, or a list, containing all the combinations. This simplifies a lot of the work for us, as we would just need to loop over it and start building our command.

You can check the official documentation for further info

Building the FIO Commands

Lastly, we need to loop over the generated list and construct our commands:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
def generate_fio_commands(cfg):
    ...
    for volume, ioengine, blocksize, numjobs, iodepth, rw in combinations:
        name = f"{rw}_{blocksize}_{numjobs}_{iodepth}"
        cmd = (
            "sudo fio "
            f"--name=name "
            f"--filename={volume} "
            f"--ioengine={ioengine} "
            f"--bs={blocksize} "
            f"--numjobs={numjobs} "
            f"--iodepth={iodepth} "
            f"--rw={rw} "
            f"--direct={cfg['direct']} "
            f"--ramp_time={cfg['ramp_time']} "
            f"--runtime={cfg['runtime']} "
            f"--time_based "
            f"--output-format=json "
            f"> {name}.json"
        )
        commands.append(cmd)

    return commands

Implementing the `main` Method

All we need now is to implement the main method and put everything together:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
def main():
    config = load_config(CONFIG_FILE)
    commands = generate_fio_commands(config)

    with open("assets/fio_commands.sh", "w") as f:
        # add the shell header line
        f.write("#!/bin/bash\n")
        for cmd in commands:
            f.write(f"{cmd}\n\n")

    print(f"\nGenerated {len(commands)} fio commands")


if __name__ == "__main__":
    main()

And now we’re done! The script will generate a total of 2376 fio commands and write them to a valid fio_commands.sh file. The only “manual” step that we have to do is to run the generated fio_commands.sh file. Though we can technically extend our Python script to run that bash script, but for now, that will suffice.

Parsing FIO Output

The last piece of the puzzle is to parse the JSON output. Just as before, we’re going to break our script into different sections. What we mainly want to do is:

Load the JSON output (same as before, so will be skipped).
Extract the key metrics.
Print the results/summary.

Extracting the Key Metrics

The main values we care about from the JSON output are mainly:

iops
latency
bandwidth

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
def parse_fio_json(path):
    # load JSON output
    data = load_output(path)
    results = []

    for job in data.get("jobs", []):
        job_name = job.get("jobname", "unknown")

        for io_op in ("read", "write"):
            stats = job.get(io_op, {})
            # ignore and move to the next op
            if not stats or stats.get("io_bytes", 0) == 0:
                continue

            # MB/s
            bandwidth_mbps = stats.get("bw", 0) / 1024
            iops = stats.get("iops", 0)
            latency_ns = stats.get("lat_ns", {}).get("mean", 0)

            results.append({
                "job": job_name,
                "io_op": io_op,
                "bandwidth_mbps": bandwidth_mbps,
                "iops": iops,
                "latency_ns": latency_ns,
            })

    return results

Print the Results

We implement our main method to display the summary of our fio output:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
def main():
    results = parse_fio_json(OUTPUT_FILE)

    for r in results:
        print(
            f"Job: {r['job']}\n"
            f"Mode: {r['io_op']}\n"
            f"BW: {r['bandwidth_mbps']:>.2f} MB/s\n"
            f"IOPS: {r['iops']:>.2f}\n"
            f"Latency: {r['latency_ns'] / 1e6:>.2f} ms"
        )


if __name__ == "__main__":
    main()

Results

To keep things a bit simpler, I’ve generated only 3 commands and have plotted them. The code is still written in a way that’s generic to handle any number of commands, to parse through the results, and plot them as well.

Here’s the config.json that I’ve used:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "benchmark_type": "time",
  "volumes": [
    "/dev/vdb"
  ],
  "ioengines": [
    "libaio"
  ],
  "blocksize": [
    "4k",
    "64k",
    "1024k"
  ],
  "numjobs": [
    1
  ],
  "iodepths": [
    1
  ],
  "direct": [
    1
  ],
  "ramp_time": 5,
  "runtime": [
    30
  ],
  "rw": [
    "read"
  ]
}

Basically testing small, medium, and large blocksizes and how our I/O behaves accordingly. The plot before summarizes the results for us:

Block Size Comparison

Initially, we can notice that 64k seems to be the sweetspot for the blocksize, having achieved the highest throughput while also maintaining decent IOPs and minimal sacrifice in terms of latency.

Conclusion

By now, you should be able to automate your I/O benchmarking with fio and even adapt the scripts to your needs. The config.json file provides plenty of flexibility to test and try out all sorts of combinations. The complete source code for this post can be found on GitHub.

It’s possible to generate nearly all tests and experiments for all the I/O operations (read, write, rw, etc.) and compare the performance of each configuration to better fine-tune and optimize for your use case. The work can be expanded even further to a full CI/CD pipeline, having each step mentioned in the post (generating commands, executing commands, parsing, etc.) as a separate stage. This concludes our I/O benchmarking, having covered the basics and advanced techniques for automating our tests.