1

I'm using TestNG (in Netbeans 8) to unit test a Java Keccak implementation, and currently the test suite contains 51,840 tests.

When I instruct TestNG to run every test in the package, initially CPU usage is tickling the 100% ceiling, and the test counter climbs like a rocket. But after a minute or so CPU usage clambers down and seems happy to linger between 30% and 60%, and the test counter climbs at a frustratingly more leisurely pace. (CPU utilisation is as reported by KDE System Monitor.)

The tests are all of the same nature (binary operations for hash calculation), and TestNG has been instructed to run in parallel mode with the following being found in the NGSuite.xml file:

<suite name="Cryptography" verbose="1" parallel="classes" thread-count="8">

Have others seen this behaviour, and is there an established explanation for it?

Bobulous
  • 12,967
  • 4
  • 37
  • 68

1 Answers1

1

TestNG works in 2 steps:

  1. Configuration, where it looks for tests and sort them.
  2. Run tests, according to the configuration.

Step 1 may consume a lot because TestNG has to scan the classpath in order to find tests and it builds its graph of tests, which can take times when you have a lot of tests.

Fyi, some perf issues already exist: https://github.com/cbeust/testng/issues?q=is%3Aopen+is%3Aissue+label%3Aperf%2Fenhancement

juherr
  • 5,640
  • 1
  • 21
  • 63
  • I suspect it's not TestNG making up the difference when the CPU is running at 100% on all four cores, because the counter climbs impressively rapidly while CPU usage is showing at 100%, and the climb rate visibly becomes sluggish as the CPU rate drifts down towards 35%. Based on the outstanding issues you link, it sounds like TestNG might indeed lose focus when the number of tests is very high. Not a disaster, but it would be nice to see the full CPU resource used the whole way through the test run. – Bobulous May 28 '17 at 00:15
  • Feel free to share your observations on the appropriate issues if you have some. Maybe it could help to understand/fix the problem :) – juherr May 29 '17 at 07:52