Benchmarking Java Methods using JMH with Gradle

JMH (Java Microbenchmarking Harness) is a library for benchmarking Java code. Running a benchmark using a library such as JMH is better than simply measuring execution time of methods using System.currentTimeMillis() or System.nanoTime().

The easiest way to run a benchmark using JMH is to use Maven/Gradle plugins. This article shows how to write and run a simple benchmark using the Gradle plugin.

I suppose you already have a Gradle project in which you want to use JMH, but if you wish to try out the walkthrough in this article, you can checkout the code from here as a starting point – which was used in the other article ‘Measuring Execution Time of Java Methods‘. It is a simple project with a class called ‘RandomNumbers’ that creates a list of 100 random numbers and stores it in memory when an object is instantiated.

Add The Plugin

The latest version of this plugin is 0.5.3 at the time of writing this post. You should probably use whichever latest version is available now.

plugins {
    id 'me.champeau.jmh' version '0.6.5'

Write Your Benchmarks

The plugin expects all our benchmarks are put in a src/jmh directory. I like this because it is similar to how we would keep our unit tests – in a separate src/tests folder. So create a jmh directory inside the src, create folder structure to represent your package inside this. That is, if your package name is com.example.mybenchmark, then create com/example/mybenchmark directory inside the jmh directory.

Create a java file for putting your benchmarks in. This is a benchmark –

public class RandomNumbersBenchmark {
    public void initializeRandomNumbers(Blackhole bh) {
        bh.consume(new RandomNumbers());

new RandomNumbers() is the piece of code that we are benchmarking. We are benchmarking the performance of creating new RandomNumbers objects. The Blackhole.consume() method ensures that JVM optimisations don’t come in the way of our benchmark. Without it, the JVM can see that we are not actually using the new RandomNumbers object, and optimise our code accordingly. JVMs do many such things and benchmarking frameworks like JMH give us ways to minimise JVM optimisations from hurting our benchmark numbers.

Run the Benchmarks

Running the benchmarks is quite simple. Just execute ./gradlew jmh from the project directory (where the build.gradle file is). You should be able to see the benchmark running and displaying it’s metrics as it runs. And the output would end with a summary like so –

Benchmark                                       Mode  Cnt     Score    Error  Units
RandomNumbersBenchmark.initializeRandomNumbers  avgt   25  2056.808 ± 84.686  ns/op

The full project for trying out can be downloaded from GitHub –

Screenshot of a JMH Benchmark Running


There are a ton of annotations in JMH which you can refer to in the documentation. But I’ll briefly write about the annotations used in the above sample.

@State is used to assign a “Scope” for the benchmark. Possible scopes are Benchmark, Group and Thread.

The @Benchmark annotation identifies this method as a benchmark. Similar to the @Test annotation that marks Unit tests.

@OutputTimeUnit annotation specifies what time unit your benchmark output should be in.

@BenchmarkMode annotation specifies in which modes the benchmark would run. Note that this annotation can supply multiple modes. Possible modes are AverageTime, SampleTime, SingleShotTime, Throughput and All.

For reference, all annotations and their meanings can be found in the JavaDoc for JMH.

Why JMH?

Microbenchmarking is a big step above simple time capturing. JMH does things like warmup, iterating and managing threads for us. It also helps us avoid common traps while benchmarking – for example, side stepping from JVM optimisations so that our benchmarks are accurate.

About Me

Yup. I just deleted the blog that used to be here. Not that it was a good one, but I had not blogged in it for quite some time and now when I looked at it, it felt like reading someone else’s blog. I’m in a different career now. I was into making reports and dashboards then and now for the last 3 years I’ve returned back to good old programming. I’m a Java developer, creating API services that involve reactive programming, data streams, graph databases and document databases. I like this better. I’d like programming things that involve pictures, animations and sounds even more. But what I do is good.

Then why did I come here and delete the blog that was just sitting there, not harming anyone?

I’ve started feeling like I’m no longer great at my job. I’ve forgotten many old lessons. I’m stuck between being a manager who has to organise tickets, being a leader who has to coach developers, and being a developer who has to be very very good at coding. Juggling between the three is taking a toll, but I don’t think it’s unfair. That’s the trade and I need to get good at it. Also, I’ve lost the habit of doing hobby projects – fun things, coding-related, that have nothing to do with my job. I think those are the ones I learned the most from.

I’ll just be writing whatever I find interesting at that time.