31

I have some Selenium Webdriver GUI tests that run on every check-in in our Trac/Bitten build environment. For various silly reasons, these are fragile and re-running the failed test always works (unless it doesn't, and then I have an actually failing test).

How do I re-run these failed tests? The GUI tests take 10-15 minutes to run, so re-running all of them is a pain.

Here is the build step:

<step id="guitest" description="Run gui tests">
    <sh:exec executable="gui_tests.sh" />
</step>

and gui_tests.sh

# Setup environment stuff deleted
nosetests python/*Tests.py -v

We are working to make the GUI tests more robust, but my understanding is that this is life with GUI tests.

stvsmth
  • 3,578
  • 2
  • 28
  • 30
  • If you want to do it all in one step, Flaky plugin may help: http://stackoverflow.com/questions/7234086/python-unittest-retry-on-failure-with-nose/30306934#30306934 – Zymotik May 20 '15 at 08:47

1 Answers1

61

EDIT June 2021: Surprised this is still getting votes. If at all possible, consider using ptyest with the --last-failed option.

It turns out this is trivial with nose. First run the command and include the --with-id flag.

# Environment stuff deleted
nosetests python/*Tests.py -v --with-id
nosetests python/*Tests.py -v --failed

If, say, four tests fail, running with the --failed flag will run only those four tests. If you then correct three, running with --failed will run only the remaining failed test.

Running --with-id again resets the tracking of failed tests and runs all the found tests.

I searched quite a bit for this before I wandered into writing my own plugin. When I was researching how to register my plug-in I found some code that happened to be the testid plugin provided by nose, which provides the functionality above. Sheesh. I am documenting this here, hoping to save someone else some time.

Of course, now that I know about it, it's so obviously found with:

nosetests --help

I scanned the docs initially, but didn't think it would be so easy, so I did so half-heartedly.

stvsmth
  • 3,578
  • 2
  • 28
  • 30
  • 2
    You *can* actually run this without `--with-id` initially. If running with `--failed` and the last test didn't capture the information it'll run all tests. – Ian Clark Jun 03 '16 at 10:06
  • What if I just want to run say the first failed test? If I used the -x option it works, but forgets all the "failed" ids, so that I have to run *all* the tests again in order to find the next failing one. Is this poor interaction of -x and --failed intentional? Can I get around it somehow? I don't want to run all of my failing tests every time because there are a lot of them, and digging through the output is confusing. It just wastes a lot of time. I need to do something like save a list of failing tests and use it with -x option... – Ben Farmer Jun 01 '21 at 00:30
  • Like there is the option --id-file=FILE, but this just seems to change the filename where the ids are stored, I don't see a way to use the same file repeatedly without overwriting the ids in it. – Ben Farmer Jun 01 '21 at 00:43