6

I'd like some ideas about how I should test some objects that can block, waiting for another participant. The specific unit to be tested is the channel between the participants, The the participants themselves are mock fixtures for the purposes of the tests.

It would be nice to validate that the participants do deadlock when they are expected to, but this is not terribly important to me, since what happens after the deadlock can reasonably be described as undefined.

More critical would be to verify that the defined interactions from the participants do not deadlock.

In either case, I'm not really sure what the optimal testing strategy should be. My current notion is to have the test runner fire off a thread for each participant, sleep for a while, then discover if the child threads have returned. In the case they have not returned in time, assume that they have deadlocked, and safely terminate the threads, and the test fails (or succeeds if the deadlock was expected).

This feels a bit probabalistic, since there could be all sorts of reasons (however unlikely) that a thread might take longer than expected to complete. Are there any other, good ways of approaching this problem?

EDIT: I'm sure a soundness in testing would be nice, but I don't think I need to have it. I'm thinking in terms of three levels of testing certainty.

  • "The actual behavior has proven to match the expected behavior" deadlock cannot occur
  • "The actual behavior matched the expected behavior" deadlock did not occur in N tests
  • "The actual behavior agrees with the expected behavior" N tests completed within expected deadline

The first of course is a valuable test to pass, but ShiDoiSi's answer speaks to the impracticality of that. The second one is significantly weaker than the first, but still hard; How can you establish that a network of processes has actually deadlocked? I'm not sure that's any easier to prove than the first (maybe a lot harder)

The last one is more like what I have in mind.

SingleNegationElimination
  • 151,563
  • 33
  • 264
  • 304
  • 1
    The book [Growing Object-Oriented Software Guided by Tests](http://www.growing-object-oriented-software.com/) has some tricks about how to run tests on multithreaded code. Sorry that I can't share something more profound, but I never tested something to react gracefuly to a deadlock. I've only created tests to ensure that there were no deadlocks. – Augusto Jun 05 '11 at 00:20
  • Testing for no deadlocks is the same as testing for deadlocks; The test must still complete, even if there happen to be deadlocks, (unless you are willing to `kill` a test runner that has failed its test and deadlocked.) – SingleNegationElimination Jun 13 '11 at 02:38
  • This question somehow reminded me of the Infinite Cookie Machine: http://www.davedoyle.com/prof/pastProjects/Nerd/humor/csci_sesame.html – Ates Goral Jun 17 '11 at 17:52

4 Answers4

5

The only way to reliably test for deadlocks is to instrument the locking subsystem to detect and report them. The last time I had to do this, we built a debug version of it that recorded which threads held which locks and checked for potential deadlocks on every lock-obtain call. It can be a heavyweight operation in a system with a lot of locking going on, but we found it to be so valuable that we reorganized the subsystem so we could turn it on and off with a switch at runtime, even in production builds.

Ross Patterson
  • 9,527
  • 33
  • 48
2

The academic community will probably tell you (in fact it IS telling you right now ;) that you should do a faithful abstraction into some so-called model checking-framework (CSP, pi-calculus). That would then simulate abstract executions (exhaustive search through all possible scheduler interleavings). Of course the trick is to make sure that the abstraction IS actually faithful. You are no longer checking the actual source of your program, but the source in some other language.

Otherwise, some heavy-handed approach like using Java Path Finder/Explorer (which does something very similar) for the particular language comes to mind. Similar research prototypes exist for C, and Intel and other companies are also in this business with specialised tools.

You are looking at one of the hot topics in Computer Science research, and for non-trivial/real systems, neither exhaustive testing nor formal verification are easily applicable to real code.

A valuable approach could be to instrument your code so that it will actually detect a deadlock, and potentially try to recover. For detecting deadlocks, the FreeBSD kernel uses a set of C-macros that track lock usage and report potential violations through the witness(4) mechanism. But again, errors that only occur rarely, will only be rarely spotted.

(Disclaimer: I'm not involved in any of the commercially tools linked above---I just added them to give you a feeling for the difficulty of the problem you are facing.)

Volker Stolz
  • 7,274
  • 1
  • 32
  • 50
0

For testing if there is no deadlock, you could use the equivalent of NUnit's TimeoutAttribute, which aborts and fails a test if execution time exceeds an upper limit. You could come *up with a good timeout value e.g if the test doesn't complete within 30s - something is wrong.

I'm not sure (or I haven't come across a situation) about asserting that a deadlock has occurred. Deadlocks are usually undesirable. I'm stumped on how to write a unit test that fails unless the test blocks - unit tests are usually supposed to be fast and non-blocking.

Gishu
  • 134,492
  • 47
  • 225
  • 308
  • 1
    Testing for an actual deadlock is less critical to me than testing for its absence. It doesn't look like the test framework I'm using (python's unittest) has this primitive, but it'd be simple enough to add it. – SingleNegationElimination Jun 13 '11 at 04:14
0

Since you've already done enough abstraction to mock out the participants, why not take it further and abstract out your thread synchronization (mutex, semaphore, whatnot)?

When you think about what constitutes a deadlock, you could use a specialized, deadlock-aware thread synchronizer in your tests. By "deadlock-aware", I don't mean that it should detect deadlocks the brute-force way by using timeouts etc., but have awareness of the situations that lead to deadlocks by way of flags, counters etc. It could detect deadlocks, while optionally providing the expected thread synchronization functionality. What I'm basically saying is, use instrumented thread synchronization for your tests...

This is all too abstract and easier said than done. And I don't claim to have successfully done it. I might simply be being silly here. But perhaps if you could provide just one (incomplete) test, the problem can be attacked in more concrete terms.

Ates Goral
  • 137,716
  • 26
  • 137
  • 190