83

I want to run each selected py.test item an arbitrary number of times, sequentially.
I don't see any standard py.test mechanism for doing this.

I attempted to do this in the pytest_collection_modifyitems() hook. I modified the list of items passed in, to specify each item more than once. The first execution of a test item works as expected, but that seems to cause some problems for my code.

Further, I would prefer to have a unique test item object for each run, as I use id (item) as a key in various reporting code. Unfortunately, I can't find any py.test code to duplicate a test item, copy.copy() doesn't work, and copy.deepcopy() gets an exception.

Can anybody suggest a strategy for executing a test multiple times?

nkr
  • 3,026
  • 7
  • 31
  • 39
Martin Del Vecchio
  • 3,558
  • 2
  • 27
  • 36

7 Answers7

122

One possible strategy is parameterizing the test in question, but not explicitly using the parameter.

For example:

@pytest.mark.parametrize('execution_number', range(5))
def run_multiple_times(execution_number):
    assert True

The above test should run five times.

Check out the parametrization documentation: https://pytest.org/latest/parametrize.html

Frank T
  • 8,268
  • 8
  • 50
  • 67
  • 2
    This will execute the test the number of times specified in the test file. It doesn't achieve my goal of executing a test an arbitrary number of times, as specified by a command-line option. – Martin Del Vecchio Feb 22 '14 at 02:37
  • 1
    My apologies, I completely misunderstood your question. I think I've figured out how to do what you want, which I'll add as another answer (since this one is completely wrong :). – Frank T Feb 23 '14 at 02:14
  • 37
    I'm glad this answer is here. I think it adds to the context for future users looking for possible solutions. – moorecm Dec 01 '15 at 14:36
  • 2
    Nice solution if you want to "hardcode" number of times test should be executed – Anthony Feb 15 '18 at 12:56
  • what is particularly cool about this is the ability to use n to say nth-time {X} happens. This is just creating a single parameter test though, so you can use any single-dimensional data-source such as an array to make the tests more readable. I just took a series of canned names and this automatically interpolates them so that you know which precise one is failing.Very nice feature for test readability – MrMesees Mar 27 '19 at 10:50
  • 1
    This is particularly useful for debugging the occasional "flakey" test that fails 1 out 25 times due to side effects of fixture data etc. – shacker Sep 02 '20 at 23:30
76

The pytest module pytest-repeat exists for this purpose, and I recommend using modules where possible, rather than re-implementing their functionality yourself.

To use it simply add pytest-repeat to your requirements.txt or pip install pytest-repeat, then execute your tests with --count n.

jsj
  • 9,019
  • 17
  • 58
  • 103
  • 12
    `warnings.warn("Repeating unittest class tests not supported")` https://github.com/pytest-dev/pytest-repeat/blob/9587709fdcc0b4af1b82dd82ac4374f93fac2810/pytest_repeat.py#L44-L45 – user7610 Aug 07 '18 at 08:03
41

In order to run each test a number of times, we will programmatically parameterize each test as the tests are being generated.

First, let's add the parser option (include the following in one of your conftest.py's):

def pytest_addoption(parser):
    parser.addoption('--repeat', action='store',
        help='Number of times to repeat each test')

Now we add a "pytest_generate_tests" hook. Here is where the magic happens.

def pytest_generate_tests(metafunc):
    if metafunc.config.option.repeat is not None:
        count = int(metafunc.config.option.repeat)

        # We're going to duplicate these tests by parametrizing them,
        # which requires that each test has a fixture to accept the parameter.
        # We can add a new fixture like so:
        metafunc.fixturenames.append('tmp_ct')

        # Now we parametrize. This is what happens when we do e.g.,
        # @pytest.mark.parametrize('tmp_ct', range(count))
        # def test_foo(): pass
        metafunc.parametrize('tmp_ct', range(count))

Running without the repeat flag:

(env) $ py.test test.py -vv
============================= test session starts ==============================
platform darwin -- Python 2.7.5 -- py-1.4.20 -- pytest-2.5.2 -- env/bin/python
collected 2 items 

test.py:4: test_1 PASSED
test.py:8: test_2 PASSED

=========================== 2 passed in 0.01 seconds ===========================

Running with the repeat flag:

(env) $ py.test test.py -vv --repeat 3
============================= test session starts ==============================
platform darwin -- Python 2.7.5 -- py-1.4.20 -- pytest-2.5.2 -- env/bin/python
collected 6 items 

test.py:4: test_1[0] PASSED
test.py:4: test_1[1] PASSED
test.py:4: test_1[2] PASSED
test.py:8: test_2[0] PASSED
test.py:8: test_2[1] PASSED
test.py:8: test_2[2] PASSED

=========================== 6 passed in 0.01 seconds ===========================

Further reading:

Allan Lewis
  • 308
  • 3
  • 13
Frank T
  • 8,268
  • 8
  • 50
  • 67
  • While this method may work, I used your suggestion to find a simpler way, which doesn't require any parametrized arguments, etc. I will post it and accept it as the answer. Thanks for your suggestion; it led me directly to my solution. – Martin Del Vecchio Feb 26 '14 at 20:36
  • 1
    Working on a new project, so I came here to figure out how I had done this previously. I now understand that this solution does not require me to declare a fixture parameter to each test. So I am selecting this answer as correct, and I am using your strategy on my new project. Thanks! – Martin Del Vecchio Jul 20 '17 at 18:48
  • How does one repeat the entire test file in order? Here test 1 is run 3 times, what about running 1,2,3 (in order) 3 times? – mungayree Apr 30 '20 at 23:09
  • Can it be made to repeat tests in the order though? I mean, if you want to call two tests, you usually do `pytest -k "test_foo or test_bar"` and it results in `foo` and `bar` being in order. However, when I use your code and do a `pytest --repeat=99 -k "test_foo or test_bar"`, instead of calling `foo` and `bar` repeatedly it calls 99 times `foo` and then goes on to call `bar`. – Hi-Angel Aug 17 '23 at 09:58
8

Based on Frank T's suggestion, I found a very simple solution in the pytest_generate_tests() callout:

parser.addoption ('--count', default=1, type='int', metavar='count', help='Run each test the specified number of times')

def pytest_generate_tests (metafunc):
    for i in range (metafunc.config.option.count):
        metafunc.addcall()

Now executing py.test --count 5 causes each test to be executed five times in the test session.

And it requires no changes to any of our existing tests.

vvvvv
  • 25,404
  • 19
  • 49
  • 81
Martin Del Vecchio
  • 3,558
  • 2
  • 27
  • 36
  • 2
    metafunc.addcall() is deprecated, which is why I preferred .parametrize() in my implementation. See also: https://pytest.org/latest/parametrize.html#_pytest.python.Metafunc.addcall – Frank T Mar 04 '14 at 22:03
  • Your solution required me to add a parameter/funcarg to each of my 1,000 existing tests, which I didn't want to do. There is likely a .parametrize() equivalent to my simple solution, but I couldn't figure it out. – Martin Del Vecchio Mar 05 '14 at 14:51
  • 6
    Working on a new project, so I came here to figure out how I had done this previously. I now understand that your solution does not require me to declare a fixture parameter to each test. So I am selecting your original answer as correct, and I am using your strategy on my new project. Thanks! – Martin Del Vecchio Jul 20 '17 at 18:48
6

While pytest-repeat (the most popular answer) doesn't work for unittest class tests, pytest-flakefinder does:

pip install pytest-flakefinder
pytest --flake-finder --flake-runs=5 tests...

Before finding test-flakefinder, I wrote a little hack of a script that does a similar thing. You can find it here. The top of the script includes instructions to how it can be run.

stason
  • 5,409
  • 4
  • 34
  • 48
3

Based on what I've seen here, and given that I already do some filtering of tests in pytest_collection_modifyitems, my method of choice is the following. In conftest.py

def pytest_addoption(parser):
    parser.addoption('--count', default=1, type=int, metavar='count', help='Run each test the specified number of times')


def pytest_collection_modifyitems(session, config, items):
    count = config.option.count
    items[:] = items * count  # add each test multiple times
user7610
  • 25,267
  • 15
  • 124
  • 150
0

I've been looking for a simple solution for a long time. I just had to run one test n-number of times. He shouldn't have fallen once. I solved this problem as simply and stupidly as possible, but it worked for me.

def run_multiple_times(number):
    i = 0
    while i < number:
        #test
        i += 1
run_multiple_times(5)
Artyom Vasilyev
  • 1