11

I'm working on a fairly big application, with ~260k LOC and roughly 1800 unit tests on the suite. A little background:

  1. Multi-modular Maven project
  2. All tests run on a separate module and JaCoCo is used to check coverage.
  3. Tests are put together in some suites and then the suites are included on Surefire's configuration.
  4. Using SpringJUnit4ClassRunner to run the tests
  5. Surefire is currently configured to use the same VM to build and test (forkCountequals one and reuseForks equals true). Doing this I am sure I'm using the advantages of Spring's Static Context Cache (reuse the same ApplicationContext on every test)

When we run tests on our development VMs, using Oracle JDK 7u79/80 we see very fast tests. For example, a fairly big test class with 50+ test methods takes something around 1:30m (including context initialisation time) to run.

Our basic Dev VM is something like:

  • Dual-Core i5 processor (with HyperThreading, so 4 virtual threads)
  • 8GB RAM
  • Mid Range, SATA HDD (5k or 7k2 RPM)
  • Windows 7 x64 running Oracle JDK on Client mode (-client)

We use Jenkins as our CI server and Maven (3.2) takes care of the build process. We have a Master + 2 Slaves architecture. All the VMs are exactly the same:

  • 8 Xeon E5 Cores (Real cores)
  • 8 GB RAM
  • SSD LUN serving the VMs (1.2GB/s mean throughput)
  • Debian Linux 8 x64
  • Oracle JDK on Server mode

Remember those tests that usually take 1:30 min to run on our Eclipse's? Here on the server they take more than 15 minutes to run! Here's what I already tried to do (without success so far.):

  • Set MAVEN_OPTS with fixed heap size (2GB heap), large PermGen space, tuned GC settings, /dev/urandom as random seed
  • Installed a 32 bits JDK to use the same Client mode we use on the Dev Machines
  • Tuned on Surefire's configuration to increase memory, tune GC, etc. (As I'm not using a forked VM to run the tests anymore I took it out because it wouldn't change anything.)

Bottom line, is there any concrete reason on why Surefire's execution is s much slower than running JUnit on Eclipse? I'm scratching my head over this for a few days now and it's really starting to annoy me! I hate it when the solution seems to be so close but yet it is so far.

I couldn't test with Maven on my Dev machine because I couldn't allocate all the memory necessary to do it, but running chunks of classes (not the whole 1.8k tests suite) still reveals to be quite slower than running on Eclipse.

I understand Maven has all those phases and so on, but the difference should't be this big. Don't you agree?

Any input will be highly appreciated. I can provide any more information you find necessary!

PS: Surefire v2.17, Maven 3.2.2, JUnit 4.12, Spring Test 3.2.13

Thanks a lot!

UPDATE 1

I tried deactivating JaCoCo on the CI server to see if it affected the build times. It doesn't. The execution times remain the same.

Gustavo Ramos
  • 1,324
  • 1
  • 12
  • 23
  • Find an environment when you can run tests in both maven and Eclipse to eliminate problem area to either build tool or an environment. Make sure you use (or do not use) JaCoCo in both cases as it can slow test execution a lot. If none of that helps, try comparing maven build performance with and without Jenkins. – Oliver Gondža Jun 10 '15 at 08:40
  • Hi @ogondza. Thanks for you input. Do you think JaCoCo could so drastically increase the execution time? I will give a try disabling the agent on Surefire and will post the results here. Thanks a lot! – Gustavo Ramos Jun 10 '15 at 08:49
  • I've never encountered that much of a difference. Usually a few seconds between surefire and an IDE. If something is that much slower... thats indeed weird. Is there a virus scanner on the CI server that needs to scan every new jar file hanging around? (I've seen this on local machines taking 40 minutes instead of 10 minutes on the CI server) – wemu Jun 10 '15 at 08:49
  • Hi @wemu nah. As this is a VM on a DMZ we didn't install any antivirus or security daemon which could be causing this. Thanks for your input! – Gustavo Ramos Jun 10 '15 at 08:54
  • if you run the build on the command line (not inside jenkings) is it still slow? (to verify the ci server does not have an issue with something) – wemu Jun 11 '15 at 20:43
  • wemu: I have the same problem and still slow without jenkins (using failsafe plugin which use surefire behind). The accepted answer is not a option for me since I tried in a non multi-module project and I have the same problem. – baraber Dec 07 '16 at 19:19

3 Answers3

2

I know this is an old thread, but maybe its still interesting: If your workload uses much stdout or stderr, surefire will create three times as big arrays as your output is, in order to save your unescaped output into them. This causes surefire to take much more time than executing the same test in eclipse or gradle.

I created a fork https://github.com/DaGeRe/maven-surefire and will see whether it is possible to merge this into surefire. My personal tests went from ~1100 ms to ~370ms with the patched plugin.

David Georg Reichelt
  • 963
  • 1
  • 15
  • 36
1

You can try to reproduce the remote surefire runs by running mvn test locally. Are you sure that you are running with the same settings, regarding forking?

I know the fork settings of surefire are quite complicated and changed a bit over the time (see here for a detailed desciption: https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html).

If the tests get forked each time and don't reuse forks, the SpringTestRunner might spend a long time on initializing the application again and again.

Are you sure that it's the surefire tests that really slow things down?

Chrstian Beutenmueller
  • 807
  • 6
  • 5
  • Hey @cbeutenmueller, thanks for the long explanation. :) I'm sure the fork settings are all right. As `forkMode` was deprecated, I'm now using a combination of `forkCount=0` and `reuseForks=true`. Actually that second parameter doesn't changes anything because i only have one VM. I'm sure the configuration is all right because I can see the context being loaded only once :) I've installed a Stopwatch rule on my tests to measure the time, so I'm sure they're guilty :( Thanks! – Gustavo Ramos Jun 10 '15 at 10:33
  • the jacoco maven plugin states not to use forkCount=0. see http://www.eclemma.org/jacoco/trunk/doc/maven.html – wemu Jun 10 '15 at 13:36
1

I have made some progress on this. As I said above, this is a multi-modular (20+) Maven application. We keep all our tests separated on one module, but Surefire's configuration were being done on the parent POM, rather on the test module's POM.

After trying everything we decided to bring the test execution configuration to this module alone, skipping all the others. This brought the suite execution from ~45' to 22'. We're scratching our heads over this, and once we understand what is going on I'll post here again.

TL;DR: This image explains everything. (off-topic, safe for work)

Thanks for all your input!

Gustavo Ramos
  • 1,324
  • 1
  • 12
  • 23
  • 1
    NB: **NOT safe for work**... your image is fine, but some of the ads down the right hand side of the page certainly aren't! – David Lavender Sep 21 '16 at 11:59