Parallelizing gradle tasks like GNU Makefile

Question

I followed the tutorial in https://guides.gradle.org/using-the-worker-api/ and got the example working. Summary: It's about parallelization of tasks in a gradle run, supported since version 4.0, and the gradle team obviously thinks they have a nice solution.

This is, in contrast, what a Makefile does (with the same paths, for comparison):

all: md5 sha256

clean:
    $(RM) -r build

build/md5/%.md5: src/%
    mkdir -p build/md5/
    md5sum $< >$@
    sleep 3

build/sha256/%.sha256: src/%
    mkdir -p build/sha256/
    sha256sum $< >$@
    sleep 3

md5: $(patsubst src/%,build/md5/%.md5,$(wildcard src/*))
sha256: $(patsubst src/%,build/sha256/%.sha256,$(wildcard src/*))

As you can see: i added also sha256 tasks (targets in Makefile "language"). The dependency tree for the all target looks like this:

all + md5    + build/md5/feynman.txt.md5
    |        | build/md5/oppenheimer.txt.md5
    |        \ build/md5/einstein.txt.md5
    \ sha256 + build/sha256/feynman.txt.sha256
             | build/sha256/oppenheimer.txt.sha256
             \ build/sha256/einstein.txt.sha256

All targets in level 3 (the filename targets) are executed in parallel, when called with make -j 6 or higher, the resulting time is 3 seconds (with the sleep)

In contrast, the extended build.gradle:

task md5(type: CreateMD5) {
    destinationDir = file("${buildDir}/md5") 
    source file("src/") 
}

task sha256(type: CreateMD5 /*enough for the example*/) {
    destinationDir = file("${buildDir}/sha256") 
    source file("src/") 
}

task all(dependsOn: [md5, sha256])

doesn't scale that well. It has two parallelizable tasks which in parallel process 3 files each because the tasks are designed to do that, but gradle is not smart enough to parallelize the two tasks md5 and sha256 even when they are completely independent. So the resulting time is 6 seconds (2*3)

Now: for a more complex scenario, I need to chain even more different actions and parallelize them. for example 10 slightly different chains of dependent pack->upload->remote-build tasks which should all run in parallel until everything is up and then continue with non-parallel testing.

In GNU make, this is all no problem, but our developer team is Java only and they don't like the idea to use a 15* years old pure binary tool for something that looks like it's a job for gradle.

Is there really no chance to set up that type of parallelization in gradle?

not yet, but now I tried. the result is 6s no matter if `-Dorg.gradle.parallel=true`, `-Dorg.gradle.parallel=false` or none — Daniel Alder, Mar 21 '18 at 14:14

score 0 · Answer 1 · answered Dec 24 '18 at 23:23

This is default Worker API feature starting from Gradle 4.0:

Once all of the work for a task action has been submitted, it is safe to exit the task action. The work will be executed asynchronously and in parallel (up to the setting of max-workers). Of course, any tasks that are dependent on this task (and any subsequent task actions of this task) will not begin executing until all of the asynchronous work completes. However, other independent tasks that have no relationship to this task can begin executing immediately.

TLDR: Independent tasks that use Worker API to submit work asynchronously (make sure to not call workerExecutor.wait() in task action) will be executed in parallel automatically.

Parallelizing gradle tasks like GNU Makefile

1 Answers1