How to use JMH with gradle?

Question

I want to use JMH, an OpenJDK microbenchmark tool, with gradle. However, Im getting the NPE on compilation. On the other hand, JMH works when using from maven.

I am not posting any build.gradle as it is basic - apply java plugin and add dependency on JHM tool (org.openjdk.jmh:jmh-core:0.2).

I have tried whats written here without success.

What else I have to do? I think something with setting the agent, but I still didnt figure it out.

Exception:

:compileJava
java.lang.NullPointerException
at org.openjdk.jmh.processor.internal.GenerateMicroBenchmarkProcessor.validMethodSignature(GenerateMicroBenchmarkProcessor.java:502)

score 33 · Accepted Answer · edited Aug 13 '22 at 18:05

Just finished my "masterpiece". No uber-jars, no plugins, code base separated from main & test, benchmarks compilation hooked to main, but does not run automatically in the mainstream lifecycle. Simple, explicit, and hackable, vanilla gradle.

I run it directly from IntelliJ, to run on a box you probably will need the uber-jar back :-)

Before doing it I have spent a fair amount of time trying to get that plugin work, but it's way too clunky for my taste.

Step-by-step breakdown below.

Define a new sourceSet called jmh with classpath hooked to that of the main sourceSet

sourceSets {
    jmh {
        java.srcDirs = ['src/jmh/java']
        scala.srcDirs = ['src/jmh/scala']
        resources.srcDirs = ['src/jmh/resources']
        compileClasspath += sourceSets.main.runtimeClasspath
    }
}

Define dependencies for it (at minimum JMH and its annotation processor).

dependencies {
    ...
    jmhImplementation 'org.openjdk.jmh:jmh-core:1.35'
    jmhImplementation 'org.openjdk.jmh:jmh-generator-annprocess:1.35'
}

Define a task jmh of type JavaExec

task jmh(type: JavaExec, dependsOn: jmhClasses) {
    main = 'org.openjdk.jmh.Main'
    classpath = sourceSets.jmh.compileClasspath + sourceSets.jmh.runtimeClasspath
}

Hook jmhClasses task to run after classes to make sure benchmarks are compiled with the rest of the code

classes.finalizedBy(jmhClasses)

So simple. Perfect! One thing that I ran across: you need to ensure that you've configured IntelliJ to [use separate module per source set](https://www.jetbrains.com/help/idea/gradle.html), otherwise it will not find your JMH source sets. — BrassyPanache, Jan 31 '19 at 22:20
I would only add the requirement of `jmhAnnotationProcessor 'org.openjdk.jmh:jmh-generator-annprocess:1.35'` in the dependencies. — Olivier Grégoire, Sep 29 '22 at 17:02

score 26 · Answer 2 · edited Jul 13 '15 at 15:31

26

Currently you can just use dedicated plugin jmh-gradle-plugin

It requires minimal configuration and allows you to run JMH benchmarks as well as build benchmarks artifact

edited Jul 13 '15 at 15:31

Antag99

401
4
7

answered Jul 16 '14 at 23:26

Jakub Kubrynski

13,724
6
60
85

2

no possibility to run individual tests, no possibility to use separate jvm_args per test, etc. If you just want to "play" with JMH - sure, if you plan to be serious about it - this is a big NOGO. – Eugene Feb 10 '20 at 21:10
2

It looks like the plugin now can be configured to run individual tests, use separate JVM args etc. See the [configuration options](https://github.com/melix/jmh-gradle-plugin#configuration-options). – jcsahnwaldt Reinstate Monica Aug 24 '21 at 19:25
In addition to the configuration options (described above), it's also possible to tweak a lot of these things through JMH's annotations. You can also run `./gradlew jmhJar` in your project and then use the JMH cli as you would with a JMH jar file built with maven. e.g. `java -jar build/libs/my-project-0.0.1-SNAPSHOT-jmh.jar MySpecificBenchmarkClass $insertYourFavoriteOptionsHere`. Seems to me the gradle plugin has all the bells & whistles a maven project would. – jonnybot Jun 17 '22 at 14:41

igr · Answer 3 · 2014-03-19T08:45:56.397

My bad, I was trying to benchmark a method that has an argument - of course JMH will not know what to pass :) Once when I created a void method with no arguments, everything worked.

My build.gradle:

defaultTasks 'build'

apply plugin: 'java'
apply plugin: 'shadow'

buildscript {
    repositories {
        mavenCentral()
        maven {
            name 'Shadow'
            url 'http://dl.bintray.com/content/johnrengelman/gradle-plugins'
        }
    }
    dependencies {
        classpath 'org.gradle.plugins:shadow:0.7.4'
    }
}

jar {
    manifest {
        attributes 'Main-Class': 'org.openjdk.jmh.Main'
    }
}

repositories {
    mavenCentral()
}


build.doLast {
    tasks.shadow.execute()
}

shadow {
    outputFile = new File('build/libs/microbenchmarks.jar')
}

ext {
    lib = [
        ... other dependencies...
        jmh:            'org.openjdk.jmh:jmh-core:0.2'
    ]
}

dependencies {
    compile lib... other dependencies...
    compile lib.jmh
}

sourceCompatibility = 1.7

Build tests and jar:

gw clean build

and then run them with:

java -jar build/libs/microbenchmarks.jar ".*" -wi 2 -i 10 -f 2 -t 16

UPDATE

From recent versions of JMH, you would also need to add dependency to:

org.openjdk.jmh:jmh-generator-annprocess:0.5.4

and you can use shadow 0.8.

Yeah, but also JMH should have printed more sane error message. Can you try to feed the erroneous benchmark you had before to 1.0-SNAPSHOT version of jmh (requires building from source)? I have rebuilt the significant part of the validation code there. — Aleksey Shipilev, Dec 07 '13 at 17:56
Yeah, NPE was confusing to me, I thought I did something wrong. Sure, I will try 1.0-SNAPSHOT but have first to prepare everything with online repo, for other ppl. After that I can play with local. — igr, Dec 07 '13 at 18:08

barfuin · Answer 4 · 2021-08-02T13:38:29.590

6

I made a very small example project to clone and modify as you like. It's a full working example:
https://gitlab.com/barfuin/jmh-gradle-example

It requires no shadow Jars and no plugins, while still running the benchmark in a dedicated JVM. The project also includes some extra Gradle tasks for printing the classpath, the JMH options, etc., stuff that may help to understand what's going on.

edited Aug 02 '21 at 13:38

answered Jan 05 '21 at 13:51

barfuin

16,865
10
85
132

It's a lot of boilerplate that can be avoided by using the JMH Gradle plugin. – Abhijit Sarkar Dec 06 '22 at 21:55
It's quite lean, no plugin, not much code either. Take a look! – barfuin Dec 07 '22 at 17:07
I’ve already looked at it, and have come to the conclusion posted above. IMO, there’s no need to reinvent the wheel because “no plugin” isn’t necessarily an advantage here. – Abhijit Sarkar Dec 07 '22 at 18:29
Well, [9 lines of code](https://gitlab.com/barfuin/jmh-gradle-example/-/blob/5a68fb2d847d9bfd9bf85abf76f5d01f71a522e4/build.gradle#L52-69) achieve the result, without a plugin. – barfuin Dec 07 '22 at 20:19

score 0 · Answer 5 · answered Aug 16 '19 at 15:21

0

If you're an IntelliJ user, perhaps the easiest way to make it work, without all that workarounds, is to use the IDE plugin:

https://github.com/artyushov/idea-jmh-plugin

Add the dependencies
Create your benchmark
be happy :)

answered Aug 16 '19 at 15:21

diogo

3,769
1
24
30

How to use JMH with gradle?

5 Answers5

UPDATE

Linked