5

The problem is:

Manipulating external state in a test file reduces the possible parallelism of your test suite.

What I'm testing is precisely manipulating external state (and then resetting it).

I'd still like to run in parallel most of the time. I'd then just like to skip the tests that would manipulate external state, which might make others fail.

So I would work around this problem by detecting the parallelism condition and then skip the test that would manipulate external state, thereby causing others to fail. Is that doable? How?

Note that just parsing the environment variable HARNESS_OPTIONS from your script isn't sufficient to detect the condition: prove -j3, for instance, won't set it.

Update

While there are lots of ways to let your test script know it's being run in a parallel harness (see Brian's answer below), there isn't one standard way of doing so, which I thought there might be (but didn't phrase this properly into my question).

I was thinking of something like, hmm, well, I read it in your (great!) Effective Perl Programming book, brian, something like maintainer tests, not sure of the term now, which you'd typically only run if you're the maintainer of the module, not during installation on a user's computer. Seem to remember there was a convention of using some environement variable for that.

brian d foy
  • 129,424
  • 31
  • 207
  • 592
Lumi
  • 14,775
  • 8
  • 59
  • 92

4 Answers4

4

The prove command creates the test harness object itself, so it passes arguments directly to the constructor of the test harness class (usually TAP::Harness). You don't get to see those arguments.

There are several ways you can go here:

  • Write a wrapper for prove to also set the environment variable if you use -j3.
  • Make your own subclass of Test::Harness to do the same if it gets the jobs argument and specify the subclass in your config for prove.
  • Don't use prove (my favorite).
  • Use a semaphore from your test programs to note that another test program has control
  • Move the external interactions parts to a common library that can tell its callers if someone already has control
  • Many other things
brian d foy
  • 129,424
  • 31
  • 207
  • 592
  • can you investigate `$0`, is that sufficient? – Joel Berger Feb 20 '12 at 16:46
  • @JoelBerger, `$0` just has the name of the test script, whether running under `prove` or not. Not sure what you mean. @brian, thanks, looks like there's no standard way or variable to detect the parallel condition. Wasn't sure about that. Then, building a custom solution is justified. [`TAP::Harness`](https://metacpan.org/module/TAP::Harness) has a `rules` configuration option, which is marked as experimental but looks promising. – Lumi Feb 20 '12 at 17:14
  • @Lumi, of course your're right, I think that was the nyquil talking. – Joel Berger Feb 20 '12 at 20:41
0

I'm working on alternate solution to this problem over at this question: How to run some but not all tests in a Perl test suite in parallel?.

My path is to patch prove to add an option like --serial-tests FILE which would reference a file listing tests which must always be run in serial. You could "set and forgot it" and then use the "-j" option as you wish. If you happen to specify one of these tests, the rest of the tests would be run in parallel, but these tests would still be run in serial.

This seems preferable to me to either avoiding running the tests, or splitting them into their own test run.

Tonight I have a working proof-of-concept which I'll share with the maintainers of prove. I'll post further progress notes through the other Q/A page.

Community
  • 1
  • 1
Mark Stosberg
  • 12,961
  • 6
  • 44
  • 49
0

i think it might be possible to do this using caller - (caller(4))[0] will im assuming return the arguments of what it was called from - which might be j3 - it’ll be worth comparing with the other values of caller though.

I think it worth me mentioning it on this thread since this thread is also often the first hit for detecting when a script is called from a harness or test, that (caller)[1] is the best way ive seen to know if something was started as part of a test

pacifist
  • 712
  • 4
  • 13
0

I had a somewhat similar need for detecting execution within prove. I wanted to suppress output from Test::More::diag() if I was running the entire test suite using prove. But when executing a test individually by hand, I do want to see these diagnostic messages. I do this by checking to see if STDERR is attached to a terminal.

diag("prove will not see this message") if (-t STDERR);

I think this makes sense beyond just use within the prove environment, the idea being, if STDERR is being displayed to a human, then assume interactive use and be more chatty.

KillerRabbit
  • 173
  • 1
  • 8