12

My current project has a policy of 100% code coverage from its unit tests. Our continuous integration service will not allow developers to push code without 100% coverage.

As the project has grown, so has the time to run the full test suite. While developers typically run a subset of tests relevant to the code they are changing, they will usually do one final full run before submitting to CI, and the CI server itself also runs the full test suite.

Unit tests by their nature are highly parallelizable, as they are self-contained and stateless from test to test. They return only two pieces of information: pass/fail and the lines of code covered. A map/reduce solution seems like it would work very well.

Are there any Python testing frameworks that will run tests across a cluster of machines with code coverage and combine the results when finished?

Joe Shaw
  • 22,066
  • 16
  • 70
  • 92
  • Did you find anything that does what you wanted? – Ian Goodfellow Aug 30 '12 at 14:45
  • @stair314 Sadly no. We've bought some more time by optimizing our unit tests themselves and cutting out a large amount of unnecessary work. We've also added machines to our CI cluster so we can build (and test) in parallel, but we still run all the tests every time. Hopefully at some point we'll be able to focus some development effort at this problem. – Joe Shaw Sep 07 '12 at 14:29
  • https://pypi.org/project/pytest-xdist/ – Mauricio Scheffer Aug 01 '22 at 13:38

3 Answers3

4

I don't know of any testing frameworks that will run tests distributed off a group of machines, but nose has support for parallelizing tests on the same machine using multiprocessing.

At minimum that might be a good place to start to create a distributed testing framework

cnelson
  • 1,355
  • 11
  • 14
4

I think there is no framework that matches exactly to your needs.

I know py.test has xdist plugin which adds distributed test executors. You can use it to write your CI infrastructure on top of it.

Mairbek Khadikov
  • 7,939
  • 3
  • 35
  • 51
1

Not exactly what you are looking at, but the closest I could recall is from the Hadoop groups is using JUnit for testing with Hadoop. Here is the mail. As mentioned in the mail search for gridunit papers.

Unit testing with Hadoop in a distributed way is very interesting. Any frameworks around this would be very useful, but developing a framework shouldn't be very difficult. If interested let me know.

Praveen Sripati
  • 32,799
  • 16
  • 80
  • 117