1

Is there something different about how Espresso runs tests after the first one when using parameterised tests?

I have two parameters and they both run just fine if I do them on their own (if I comment one of them out and just have one parameter), or if I reverse the order the first one always passes.

But for some reason when I run them together the second one always fails. Actually, one time it didn't fail. Without changing anything it worked one time. But just once. So it's possible maybe there's a bug but I'm inclined to think it's something I'm doing (or not doing).

The actual problem that occurs is that a field gets set to null halfway through the test (according to the debugger). But I can't seem to figure out how or why, especially since it doesn't happen on the first test. So I can't find anything obvious in the code that's doing it, but I also can't see what's different with Espresso either. Any ideas?

I've followed the debugger where it proved to me that the value is set, and then for some reason when it pauses the current Activity to launch a dialog the value gets set to null. But only on the second test. In this exact same situation on the first test it doesn't do that. Why?

Edit: Ok I've narrowed it down further but still not sure why it's doing this. It's a bit strange but I've discovered through the debugger, that onDetach in the first test is being called after onResume for the second test. Why might that be?

Anyway that's definitely why I'm having a problem as the code that's making the value null originates from onDetach, and the value is originally set from a method in onResume. So that explains it. But why is this happening?

Issue logged here: https://code.google.com/p/android/issues/detail?id=235247

Michael Vescovo
  • 3,741
  • 4
  • 32
  • 45
  • Could you share your code? Are cleaning up after the test so that the second test starts with the same state as the first one? – Piotr Zawadzki Feb 24 '17 at 12:25
  • I was really after more of a general answer. Like what could possibly be different on the second test. Just some ideas of what to look for. As far as cleaning up, yes I've done that for everything that seems like it could possibly be effected. But the problem occurs during (not at the start of) the test when it for some reason sets the value to null. During the first part of the test it uses this value successfully, and then out of no where Espresso sets the value to null and the second part fails. In the first run of the test it doesn't set the value to null, only on the second run. – Michael Vescovo Feb 24 '17 at 12:50
  • I just discovered something else rather interesting. If I run all the tests in the class then this test that was only failing the second time fails both times. So I think there is definitely something I'm not doing with cleanup. I'm out of ideas as to what else to clean up though. I tried previously to investigate what happens with this activityTestRule but I didn't find anything except this part about setting the flag for multiple_tasks. However that didn't do anything (I used add so it will keep the new_task flag). I also tried setting them both explicitly. – Michael Vescovo Feb 24 '17 at 13:10
  • Without actually seeing the code it's hard to tell where the issue lies I'm afraid :/ – Piotr Zawadzki Feb 24 '17 at 13:13
  • Yeah I realise it's a hard question without the code. But still, if someone has experienced this before they may have some advice. – Michael Vescovo Feb 24 '17 at 13:18
  • Hi @PiotrZawadzki if you still want to look at this I've included some links in the comments to Code-Apprentice. I'm not sure if it's a coincidence or not that Code-Apprentice is on here now (since they are from Google) but I had another one of these problems and eventually decided to log a bug report. https://code.google.com/p/android/issues/detail?id=235247&can=4&colspec=ID%20Status%20Priority%20Owner%20Summary%20Stars%20Reporter%20Opened – Michael Vescovo Feb 28 '17 at 20:29

1 Answers1

0

Ok I found a solution. Since onDestroy from the previous test is being called after onResume in the current test, I moved the code in onDestroy that clears my listeners to the start of onResume. Now when the other code in onResume runs it's not nullified by the bad timing of onDestroy. The listeners are accessed via the repository which stays active the whole time.

Hopefully this all makes sense to anyone reading in case they face a similar problem. So far I haven't noticed any side effects of this workaround. All the tests pass and the actual app works as well when running manually.

Edit: This workaround is now causing problems so I have to unaccept my answer. The reason is because when I call another Activity for result, when it comes back the listeners are killed in onResume. In this case onDestroy isn't killed yet so would not cause this problem.

Edit 2 and current solution: this question was a bit hard to test for readers so I created a test app and a new simpler question. I've since found a working solution that I've explained in this post: Espresso not waiting till Activity is destroyed, before creating a new one for the next test

I believe based on reading some Meta Q&A's that SO prefers that questions aren't deleted so I'll just leave this question here.

Community
  • 1
  • 1
Michael Vescovo
  • 3,741
  • 4
  • 32
  • 45
  • 1
    This does not seem like a very good solution. These two callback methods serve two very different purposes. Moving coffee from one two three other is highly suspect. Unfortunately, I do not have a better solution atm. – Code-Apprentice Feb 28 '17 at 17:44
  • Can you provide some evidence for your claim about the order of the execution of your callbacks? – Code-Apprentice Feb 28 '17 at 17:46
  • I agree that it's not a good solution, however at the time it seemed like maybe that's what I was supposed to do, and it did fix the problem (and there were no other answers). Little did I know, possibly the same problem would return for a different test http://stackoverflow.com/questions/42494674/espresso-test-passes-individually-fails-when-run-in-the-suite but I'm not certain at this point that they are related. They might have the same cause, I'm not sure. – Michael Vescovo Feb 28 '17 at 20:08
  • The order of execution I found simply by running the debugger. I put a break point in the onResume and OnDestroy/OnDetach methods and to my surprise I discovered that onDestroy from the previous test (the only explanation) was being called after onResume from the current test. You can see it set the value to null before it continues with the rest of the test, at which point it of course fails. You can test it yourself from this commit https://github.com/mvescovo/item-reaper/commit/e3f2eb9824e7d91a1d7405ce4f0fd6415a2f2fda – Michael Vescovo Feb 28 '17 at 20:13
  • Run the addItem_ShowsItemInList() test from the "EditItemScreenTest" class after reversing the changes in the aforementioned commit. – Michael Vescovo Feb 28 '17 at 20:25