Infinite I/O

Workload generation: You're doing it wrong

Posted by Matt Brender on Apr 10, 2014
Find me on:

spinaltap-11We’ve all heard a conversation like this one before:

“We need to test this new system.”

“Okay, I’ll run IOmeter and send you the report.”

Here are a handful of takeaways about IOmeter and what it offers you as a tool of the trade:

Problem: IOmeter is designed to stress storage throughput with no view of latency

Why it’s a problem: IOmeter is run on a Windows VM. This VM is on the hypervisor, which leverages the network to communicate to storage. To stress storage—like it often is stressed in production environments—no other layer of this puzzle can be the bottleneck. Tracing the bottleneck has only gotten more complex with virtualization.

IOmeter also pushes IOPS with no regard for latency. Total throughput without the accompanying quality of service is only half the story.

Problem: Running IOmeter usually means running one instance of IOmeter

Why it’s a problem: A single VM trying to drive max IOPS in an environment is not at all related to a real distributed workload in a VMware environment. Users often have 2 to 16 different hosts with anywhere from 10 to 100 VMs per host. Creating more workers on a single session won’t solve the problem either—that’s just more local threads. You would need to spin up dozens to hundreds of VMs running IOmeter to see a realistic design.

Problem: By default, IOmeter runs with a queue depth of one

Why it’s a problem: Operating systems that are resource constrained result in a long queue depth. A strong server-side caching solution reduces the queue depth, enabling the VM to increase it’s effective throughput while reducing latency. This real-world benefit of server-side caching is never possible to see running IOmeter with a queue depth of one.

The last problem we’ll cover today:

The goal of running IOmeter against a product like Infinio, or any other server-side caching solution, is to test whether it benefits your production environment. IOmeter was designed for a time when data was purely address-based. Today’s caching is content-based in a way IOmeter does not represent.

IOmeter data is pure chaos—random seed files are spread across whatever data set is chosen. I am not aware of any production workloads that run like a basic IOmeter test. Enterprise environments have a natural amount of duplicate content.

Said another way, testing a cache with IOmeter tests frequency of access of the same address. To verify Infinio is useful to your environment, you need to test frequency of access of the same content. That concludes, before we even begin, that these tests measure whether or not a product is good at offloading or accelerating IOmeter, not your production workload.

That’s not helpful.

It’s not your fault

The challenge of creating realistic workload generators is nontrivial.

There are great ways to run synthetic workload generators to produce real-world workload. The team at Infinio—specifically Peter Smith—has built a Linux system that pushes I/O in realistic patterns. This is the load generation that we recommend our customers use, and we’ll elaborate on it in a future post.

Until then, remember to stay suspect of IOmeter tests.



Hat tip to my colleague, Sheryl Koenigsberg, for the catchy title. You should follow her on Twitter. You can find me there too.

Matt is a Sales Engineer at Infinio

Topics: Talking Tech