phpunit - testing is painfully slow

Question

I am diving deeper and deeper in the world of unit testing.

One issue I encountered, and this is where I would like feedback, is when one runs multiple test suites, maybe it is just me but I need to use the parameter --process-isolation for my tests to pass. I can run any of my suites individually without a problem, but running the 6-7 suites I have so far with 180 assertions spread between them fails if I run without --process-isolation. The problem is that using this parameter makes the test run last for 35 mins, versus the usual 2.5 minutes. That's a loooong wait.

The problem is related to using mocked DI containers for specific tests and containers are not properly re-initialised when tests suites are running chained. Static properties set on DI-Container to test for expected failures make the tests in following suite fail. The container has a parameter that can hold contained object in a static var, to return the same instance at every call. A singleton in disguise. And this runs fine on application level, it's just a nuisance for testing.

I could avoid that container parameter and code the application to not use static properties, but avoiding a useful language construct for the sake of a methodology seems like overkill.

Maybe I am doing something wrong (I sure hope so!) but I have the impression if one wants to run tests with the SUT in a clean state for every test, there is no getting around using --process-isolation. This makes testing very time consuming and takes the joy out of it a little bit. I have bypassed the issue somewhat by running suites and tests individually when I am coding, and running the suite in background before major commits.

Is what I am experiencing normal, and is there a way to counter this? How do you testers out there ensure testing time is reasonable? How are statics handled so as to not influence testing?

Any insight appreciated/comment appreciated.

cweiske · Accepted Answer · 2016-04-14T09:47:32.423

You have several problems.

The first is process isolation. Normally, it should not be necessary and you only want to use it to find out which specific test is the one that fatally breaks your tests. As you noted yourself, it's awfully slow, which is something you cannot fix. You might want to disable backing up global vars which saves some milliseconds per test, though.

The second problem, which leads to your first problem, is that your code is not testable because static vars are kept during tests - my most-hated singleton problem. You can solve that problem by providing a "cleanup" or "reset" method in your dependency containers. Those will get called from the setUp() method in your main test case class and reset everything to a clean state.

Speed

Regarding the runtime of tests - I recently wrote a blog entry about finding out which tests were too slow. Generally, tests are too slow if you can't run them after saving the file or each commit on your own box. 10 seconds is barely acceptable for me. The more tests you have, the slower will running them be.

If you really have 35 minutes, then split up your tests into sensible groups so that you can run the necessary ones on your own machine - only the tests that test the code you changed. Pyrus, the next-gen PEAR installer, has the nifty feature to automatically detect and run the tests that need to be run, depending on what files you changed. PHPUnit does not have that, but you can emulate that by hand and phpunit --group .. :)

Always take care of mocking web services and databases, or at least running the database with only necessary data for each single test. Waiting 3 seconds for a web services response in a test that's verifying if you can save the user to database is something you never want.

Very insightful, thanks for the feedback, nice to know there are ways to correct my issues. I aprreciate. — Stephane Gosselin, May 29 '11 at 13:06
I can't really add any more except to stress that solving the problem that blocks running tests in a single process--while seemingly overkill--will make you a much happier tester. Our legacy code makes use of a lot of static variables (painful for testing). Instead of fixing them all at once I wrote simple "setup actions" that perform setup/teardown for every test. It's a shotgun approach that buys us time to test now and come back later with a scalpel to handle each case elegantly. :) — David Harkness, May 29 '11 at 20:19
Came here to post a major thumbs up to your [blog post](http://cweiske.de/tagebuch/visualizing-phpunit-runs.htm). Using the phing task you mentionned there clued me in on the test that was choking. I have taken note of adding a reset method in the container but would of preferred not having test related code with production code. Although this method could easily be [stripped out with sed](http://www.cyberciti.biz/faq/sed-howto-remove-lines-paragraphs/) with svn pre-commit hook. Testing time has gone back down to reasonable 96 secs, will optimize DB tests some more by using sqlite this week. — Stephane Gosselin, May 29 '11 at 20:31
Blog entry link is dead but you read it from here : https://web.archive.org/web/20111124071100/http://phpadvent.org/2009/testing-with-phpt-and-xunit-by-greg-beaver — Eray, Apr 13 '16 at 14:03

score 4 · Answer 2 · answered Nov 01 '17 at 18:29

One of the things I usually do when I test with MySQL instead of SQLite's :memory: is I add Hash::setRounds(5); inside tests/CreatesApplication.php Trait like this. I experienced this will make the tests especially with MySQL so much faster:

public function createApplication()
{
    $app = require __DIR__ . '/../bootstrap/app.php';

    $app->make(Kernel::class)->bootstrap();

    // TODO: DON'T FORGET TO IMPORT HASH OBJECT ON TOP
    Hash::setRounds(5);

    return $app;
}

It looks like `Hash` is a Laravel class - so only relevant if you're testing Laravel code that uses password hashing. — bdsl, May 13 '21 at 19:47

score 2 · Answer 3 · edited Apr 19 '22 at 12:37

2

A few tricks;

filtering your test cases for example if you wanna test one file, just need to to

    phpunit --filter 'Default_My_Test'

Removing code coverage in your phpunit.xml file. If you wanna get a code coverage do:

    phpunit --coverage-html ./report reportTest

edited Apr 19 '22 at 12:37

ikhvjs

5,316
2
13
36

answered Jul 06 '11 at 07:22

Orod Semsarzadeh

149
1
6

phpunit - testing is painfully slow

3 Answers3

Speed