17

I'm spending a bit of time with Erlang, and I'm wanting to apply TDD to code I'm writing.

While EUnit in the standard lib provides a nice traditional unit testing framework for testing regular style code, there doesn't seem to be anything to help specifically with testing concurrent code, which is used a LOT in Erlang.

Note that we're talking Erlang here, which uses message passing (as opposed to shared state) for communication between concurrent processes, so techniques for unit testing concurrent code in shared state languages may not be applicable.

Anyone found a good way to test concurrent code in Erlang?

madlep
  • 47,370
  • 7
  • 42
  • 53

6 Answers6

9

I just found some cool newly (as of 2011) developed software for testing concurrent Erlang applications called Concuerror. There is a few papers on it and a repository on github. Apparently it works by using its own scheduler and systematically testing different interleaving between the processes.

Also worth mentioning is Dialyzer (DIscrepancy AnaLYZer for ERlang) (Papers, Tutorial, Manual) , which is a tool for static analysis of the code for finding errors. This has support for detecting some concurrency-errors too (see paper).

I have not tested any of these myself, though dialyzer seems to be relatively mature software. Both programs have a GUI for working with the tests.

PS. For the non-concurrent parts, EUnit and QuickCheck (there is a free version also) should work fine.

Hjulle
  • 2,471
  • 1
  • 22
  • 34
5

The question is a bit vague ("Erlang is concurrent, test it with Erlang!") but I'll try to elaborate a bit.

Testing Erlang code can range from the straightforwardly simple (the right input produces the right output) to setting up complex test harnesses that verify your component behaves the way it should. What is best for your given situation depends totally on the requirements you have and the amount of black box / white box testing you want to do.

Part of the beauty of Erlang is the ability to make concurrency transparent. Consider the following example (a function that parallelizes the summation of a list of lists):

deep_sum(ListOfLists) ->
     Parent = self(),
     [spawn(fun() -> Parent ! lists:sum(List) end) || List <- ListOfLists],
     lists:sum([receive Sum -> Sum end || _ <- ListOfLists]).

You would typically test this with a very simple EUnit test case:

deep_sum_test() ->
     ?assertEqual(0,  deep_sum([0,  0,  0,  0])),
     ?assertEqual(40, deep_sum([10, 10, 10, 10]).

Now, let's say we have a bit more explicit API to this functionality: a process pool as argument:

deep_sum(Pool, ListOfLists) ->
     distribute_lists(Pool, ListOfLists),
     lists:sum([receive Sum -> Sum end || _ <- ListOfLists]).

distribute_lists(Pool, ListOfLists) -> distribute_lists(Pool, Pool, ListOfLists).

distribute_lists([P|Pool], All, [L|ListOfLists]) ->
     P ! {self(), L},
     distribute_lists(Pool, All, ListOfLists);
distribute_lists([], All, ListOfLists) ->
     distribute_lists(All, All, ListOfLists);
distribute_lists(_Pool, _All, []) ->
     ok.

When testing this, we have to deal with faking this process pool:

deep_sum_test() ->
     Pool = [spawn_link(fun() -> fake_pool(1) end) || _ <- lists:seq(1, 3)],
     ?assertEqual(4, deep_sum(Pool, [lists:seq(1, 3) || _ <- list:seq(1, 4)]),
     ?assertEqual(7, deep_sum(Pool, [lists:seq(1, 3) || _ <- list:seq(1, 7)]),
     [P ! stop || P <- Pool].

fake_pool(CannedResponse) ->
     receive
         {From, _L} -> From ! CannedResponse;
         stop -> ok
     end,
     fake_pool(CannedResponse).

As you can see, testing concurrency programs in Erlang can take different shapes. These are extremely simple examples, but with Erlang's built in concurrency primitives it is very easy to create the kind of test harness you want, abstracting at the right levels.

I usually find TDD to be orthogonal to wether you're testing concurrent code or not, so said testing techniques can be used for normal unit testing as well.

Adam Lindberg
  • 16,447
  • 6
  • 65
  • 85
2

The one tool I know of that you can use to test Erlang programs under concurrent scenarios is QuickCheck: http://www.quviq.com/

You can specify your expected behaviour using properties, and the tool can exercise your Erlang program over a range of inputs and time to ensure the properties are satisfied.

Unfortunately it's a commercial tool

Also have a look at the ProTest project: http://www.protest-project.eu

Update: The ProTest project have released a survey, "Results for: Erlang testing tools survey for the ProTest project"

bjnortier
  • 2,008
  • 18
  • 18
1

Because errors in concurrent code manifest themselves depending on the order various parts get executed, I think it's best not to rely on testing to uncover your bugs in the concurrent part of your code. Such bugs are very easy to slip through and extremely difficult to locate. See for instance how the double-locking technique, which was widely cited and used as an efficient method for implementing lazy initialization in a multithreaded environment, was later found to be broken. It's far better to use appropriate abstractions and techniques to make the concurrent parts of your code correct "by design".

Diomidis Spinellis
  • 18,734
  • 5
  • 61
  • 83
  • This advice is true for a language like Java with shared state concurrency. However Erlang uses message passing, and you have control over when data is sent to other processes, and in what order it is received. Basically more control. It feels like there should be code/test patterns for this. – madlep Dec 20 '08 at 12:32
  • True, the lack of shared state saves you from data corruption or inconsistency problems. But you still got to worry about synchronization. Am I correct? – Diomidis Spinellis Dec 20 '08 at 14:32
0

Adding to what Diomidis said, the only real way to gain some confidence with your concurrent code is to run extended tests under constantly varying conditions; and even then, it's only proof that it didn't fail under those conditions.

Lawrence Dol
  • 63,018
  • 25
  • 139
  • 189
-1

You could isolate out the part you would like to test with mocks by using this: http://github.com/charpi/erl_mock/tree/master

I have no experience with erl_mock though.