Running a unit test in isolation in PHP

Question

I've created a basic unit test suite in PHP. It shows whether or not a test passed and the time it took to execute the test. It works great, but I've run into a problem and I'm not sure how to address it. Each initial unit test of each unit test class always appears to take more time to execute than the others in that class, usually by a significant factor.

As an example:

TestClass1

Test1.1 (0.042395)
Test1.2 (0.000392)
Test1.3 (0.000325)
Test1.4 (0.000645)

TestClass2

Test2.1 (0.042395)
Test2.2 (0.000392)
Test2.3 (0.000325)
Test2.4 (0.000645)

I'm assuming that the initial test time is also factoring in the loading of resources, but I can't seem to make the first test consistent with the other times. Each test has setUp and tearDown methods that are run. I've also tried repeating the exact same test under a different name and it runs in the expected shorter time, so the first test always has some sort of overhead.

PHP doesn't have any sort of unloadClass or unloadFile function, so I'm not sure if I can recreate a consistent test environment for each test if the first test has the overhead of loading resources into the memory. I've tried preloading files and an initial class instantiation, but neither seems to make an impact.

Are there other possibilities that I might be missing that could be affecting the initial test time? What is the proper way to address this?

Note: I'm aware of existing PHP unit testing frameworks, but I'm trying to do something a little different, so please don't say, "Just use PHPUnit", as it's not helpful in helping me solve this issue.

why do you care about the running time of unit tests? you don't use it for performance tests, do you? — Karoly Horvath, Sep 15 '11 at 23:24
Are you using APC or similar? Perhaps the initial long first run is due to compilation, which is then cached for subsequent runs. — Alex Howansky, Sep 15 '11 at 23:25

score 2 · Answer 1 · answered Sep 15 '11 at 23:41

2

Any benchmarks you're doing in PHP code are going to be skewed. I would suggest looking into using Xdebug's profiling features to get more accurate benchmarks: http://xdebug.org/docs/profiler

answered Sep 15 '11 at 23:41

Matthew Turland

763
3
11

+1 - I'm using Xdebug for other parts of the test suite, so it would be a good option. However, it doesn't look like the profiler has a PHP interface, which makes it less desirable an option as I'd prefer not to have to use an additional program like KCacheGrind or other additional dependencies. – VirtuosiMedia Sep 15 '11 at 23:55
Could you expand a little on why the benchmarks will be skewed if they're done in PHP code? – VirtuosiMedia Sep 16 '11 at 00:00

Running a unit test in isolation in PHP

1 Answers1