10

I just started using Codeception after years of writing unit tests in plain PHPUnit. One thing that is bugging me, that I can't find a way to control the order in which the tests are invoked.

In pure old PHPUnit I was building the test structure manually like this:

$suite = new PHPUnit_Framework_TestSuite();
$suite->addTest('MyFirstTest');
$suite->addTest('MySecondTest');

and the test would be invoked in the order which they were added to the suite. Codeception on the other hand seems to be iterating through directories and running every test it can find.

I would like to be able to control the order of the tests on two levels:

  1. The order in which different kind of tests are invoked (i.e. I would like to run unit tests before acceptance tests)
  2. I would like to control the order of tests invoked in specific test type (in similar manner the PHPUnit builds suites)

Ad. 2: Let's say I have two tests in acceptance directory:

AbcCept.php
WebGuy.php
XyzCept.php

I want to be able to run the XyzCept.php before AbcCept.php. Is this even possible?

And to anticipate picky comments: yes, I know that tests should be able to run in any order, and not depend on each other, but that's not what I'm asking.

Maciej Sz
  • 11,151
  • 7
  • 40
  • 56

4 Answers4

8

Files get sorted by name (I assume we are talking about files from the same directory). In other words if you need to run the test XyzCept.php before AbcCept.php you re-name the XyzCept.php to, let's say, AazCept.php.

akond
  • 15,865
  • 4
  • 35
  • 55
  • 1
    Ok, thanks. That was kinda obvious. I actually sort them numerically right now (by adding prefixes `001`, `002` etc.). But what about complex directory structure, where unit test classes are in namespaces (PSR-0)? – Maciej Sz Dec 22 '13 at 09:17
  • 1
    All files are sorted by their full name, i.e. path + file name. – akond Dec 22 '13 at 22:01
2

For the ones who are searching for a proper solution and bump into this issue.
I believe, what you truly want to achieve, is that a specific test ran before another test. The order itself is probably not that important. In order to do that, you can add the @dependency annotation to a test.
Full documentation can be found here: https://codeception.com/docs/07-AdvancedUsage#Dependencies. Example:

class ModeratorCest {

    public function login(AcceptanceTester $I)
    {
        // logs moderator in
    }

    /**
     * @depends login
     */
    public function banUser(AcceptanceTester $I)
    {
        // bans user
    }
}
Bob Claerhout
  • 781
  • 5
  • 24
  • This did not work for me in Codeception acceptance tests. – Bob Ray Feb 23 '21 at 22:04
  • That's strange, maybe try to run the above example? – Bob Claerhout Feb 25 '21 at 10:07
  • That might tell us that login ran first, but we'd have no way of knowing if the @ depends had any effect. My situation may have been complicated by the fact the I was using @ example annotation at the same time. – Bob Ray Feb 26 '21 at 17:29
  • I've just re-read the docs on using the @ depends annotation. I don't believe it controls the order of execution of the methods. What it does is prevent dependent tests from running if the test they depend on fails. So in the example above, if the login(0) test fails, the banUser() test will never execute. – Bob Ray Feb 26 '21 at 17:37
  • I agree that it will prevent the banUser() from running when the login() test fails but in order to know whether this test succeeded, it should have been executed, right? Otherwise there would be no way of telling whether the test failed or not. The documentation on the suffle (https://codeception.com/docs/07-AdvancedUsage#Shuffle) config property tells you that the tests will be shuffled except for the ones having a dependency. – Bob Claerhout Feb 28 '21 at 12:25
  • My thinking is that the tests normally are run in the order they are written (as in the example above, and depends just keeps the dependent test from running after a failure. So depends may only affect running order is shuffle is on. It would be easy to test with debug on. If I have time, I'll check. – Bob Ray Mar 01 '21 at 14:06
  • In my acceptance tests, I have a test which depends on a test which is in the same file but further down the file. That test in itself is dependent on another test from another file and all tests are executed in the correct order and when 1 of them fails, the tests execution is stopped and the consecutive tests are skipped so it is working as you would expect it. – Bob Claerhout Mar 02 '21 at 15:42
1

You can create a batch script that runs the tests in any order you like, e.g.,

cd \path\to\test\dir
codecept run unit
codecept run acceptance

or

cd \path\to\test\dir
codecept run unit
codecept run acceptance XycCept.php
codecept run acceptance AbcCept.php

See the "Run" command on this page.

Bob Ray
  • 1,105
  • 10
  • 20
  • Note that some tests will call batch files of their own. In that case the process may not return to your batch script, so some tests down the list may not run. If that happens, use "call" instead of "run." – Bob Ray Feb 23 '21 at 22:03
0

To select a specific suite, or specify order to run the different test-suites you can use the following syntax (no spaces):

codecept run unit,functional,acceptance
SmilingSisyphus
  • 11
  • 3