0

I have an instance where I have several hundred iterations I need to run of the same @Test using a @RunWith(Parameterized.class). Each takes about 10 minutes, so this overall test takes most of the day.

The problem is, the system we are running it against only has keeps logs for about 30 minutes, so if an iteration fails in the middle of the test we do not really have anything other than the stack trace that java gives for the failure, and which gives very little information about the state of the system that we are testing.

I tried adding a rule to the test like this:

...
@Rule
public TestWatcher logPuller = new TestWatcher() {
     @Override
     protected void failed(final Throwable e, final Description description)
     {
         downloadLogs(description);
     }
};
...
@Parameters()
public static Collection<Object[]> generateData() {
    return Arrays.asList(new Object[][] {.....}
...
@Test
public void runTest(){...}

But this only pulls the logs at the end of all of the iterations, rather than at the end of each failed iteration, and with the failure being long gone from the logs that are pulled, so the log shows no signs of what was happening at the time.

Is there a way of accomplishing this and keeping my parameterized input intact?

Allen
  • 127
  • 1
  • 1
  • 9
  • I have to say that seems like both an excessive amount of time for the test to run, and an excessive number of interactions. I have had issues getting parameterised tests to run with rules, perhaps the test itself could just download the logs. Not idea, but it will be executed each time the test case runs – Gavin Jun 29 '16 at 22:38
  • I don't have a lot of experience with JUnit or Parameterization, so please forgive me if this comment is clueless. Is your issue the fact that the references to the other system's state which are made at that time of failure (at the iteration) become worthless by the time all iterations have completed, or that the code that reports the error occurs only after all the iterations are done and the state is no longer meaningful? Why can't the iteration be given its own try/catch, and state be read in the catch clause and saved to a custom log file? – Phil Freihofner Jun 29 '16 at 23:48
  • @Gavin This is a stress test for the system, so we want it to run for a long time, this is why there are so many permutations. – Allen Jun 30 '16 at 12:32
  • @PhilFreihofner this is basically what the TestWatcher is, I have overwritten the failed method that gets called when the test gets into a failed state it performs the action, but not when the iterations of the test fail. I am trying to have it work on the iterations rather than the whole test. – Allen Jun 30 '16 at 12:37
  • I can't tell from the code you show when the tester is instantiated nor how granular the check for the error state is. Are these aspects built into the syntax of JUnit? Maybe that is the point of your question. If so, can you build a simpler example for troubleshooting: write a method call (random.nextInt(100) where return of 0 marks a "fail") within a for loop of 100,000, and get the test to trigger on the iteration rather than at the end of the loop. If this is hard to do, at least being a simpler case the question will be easier for others to understand and help out. – Phil Freihofner Jun 30 '16 at 17:17

0 Answers0