Testing behavior not consistent when watching actor for termination

Question

When I write tests that involve subscribing to events on the Eventstream or watching actors and listning for "Terminated", the tests work fine running them 1 by 1 but when I run the whole testsuite those tests fail. Tests also works if each of those tests are in a separate test class with Xunit.

How come?

A repo with those kind of tests: https://github.com/Lejdholt/AkkaTestError

I tested this with NUnit, and if I run all tests together `GivenProcessExist_WhenProcessterminates_ShouldLogRemovingProcess` reliably failes everytime. It always works if I run one test alone. But all tests together run fine, if I reorder them, executing `GivenProcessExist_WhenProcessterminates_ShouldLogRemovingProcess` as first one. — Haukinger, Dec 02 '16 at 10:04

score 1 · Accepted Answer · edited Aug 28 '17 at 11:40

Took a look at your repository. I can reproduce the problems you are describing.

It feels like a bug in the TestKit, some timing issue somewhere. But its hard to pin down. Also, not all unit test frameworks are created equally. The testkit uses its own TaskDispatcher to enable the testing of what are normally inherently asynchronous processed operations. This sometimes causes some conflicts with the testframework being used. Is also coincidentally why akka.net also moved to XUnit for their own CI process.

I have managed to fix your problem, by not using the TestProbe. Although im not sure if the problem lies with the TestProbe per say, or the fact that your where using an global reference (your 'process' variable). I suspect that the testframework, while running tests in parrallel, might be causing some wierd things to happen with your testprobe reference.

Example of how i changed one of your tests:

     [Test]
    public void GivenAnyTime_WhenProcessTerminates_ShouldLogStartRemovingProcess()
    {
        IProcessFactory factory = Substitute.For<IProcessFactory>();
        var testactor = Sys.ActorOf<FakeActor>("test2");
        processId = Guid.NewGuid();
        factory.Create(Arg.Any<IActorRefFactory>(), Arg.Any<SupervisorStrategy>()).Returns(testactor);
        manager = Sys.ActorOf(Props.Create(() => new Manager(factory)));

        manager.Tell(new StartProcessCommand(processId));

        EventFilter.Info("Removing process.")
            .ExpectOne(() => Sys.Stop(testactor));
    }

It should be fairly self explanatory on how you should change your other test. The FakeActor is nothing more then an empty ReceiveActor implementation.

Testing behavior not consistent when watching actor for termination

1 Answers1