0

I have made a genetic algorithm in Java, it's in the form of a library that can be added to several applications. During development I've made some (jUnit) tests that could be functional tests, but they don't have asserts because the algorithm is nondeterministic. So, they are not suitable for automatic testing, and when run a time later you must spend time to see what to look at the solution.

Solutions are vehicle routes, can be print out to XLS format, and actual tests are made at the time that a necessity arises, so you know what to look for in that very moment. They have a distance value and a time value, that have a reasonable values for every example, and the route itself, that must not have certain zig-zags though sometimes there just must be zig-zags.

I could just assert that that values are in that reasonable ranges, but these are not clear and I wouldn't be sure I don't left in bad solutions. It would be great to know what people is doing if anything or what do you think about this matter.

EDIT: To clarify the goal:

  • Unit testing is pretty solved. I could isolate random number generation code from logic code in diferent functions, and I test logic functions passing them non random numbers from testing code.
  • I want to have some functional tests ala Cucumber or Selenium tests, with no fake data. Real input that generates real output. (I'm not saying I wanna use Cucumber or Selenium specifically, only saying that what I asking for is not unit testing and that the behavior of the algorithm must be real and no faked in any way)

The library is used in diferent projects, and the input data of every project follows the same model, but it's biased in diferent ways. One project data can have lots of repetitions of the same point, while another project don't. One can have vehicles that reach the limit capacity very fast, while at others this limit hardly can be reached.

The goal is to have sample data from every project which is inserted in, in order to have fast feedback of the behavior of the algorithm while modifying it (like modifying or adding mutation or crossover operators, or adding new parametrization). Thus its behavior must not be faked in any way.

Given all this, the problem I ask about is to find a way to make effective asserts of the whole behavior. I wanted to ask how other have dealt with this problem, or what do you guys can say about it.

Pedro R.
  • 142
  • 1
  • 11

2 Answers2

1

Unittests are there to test the behavior of units within your code. Those are perfectly testable even if the overall algorithm is non-deterministic. What you're trying to achieve (i believe) is having a way of automating integration tests. For this to work you either need to setup a scenario where the outcome is deterministic (certainly possible, for example, can my implementation resolve the max value of the sum of 2 bytes), or you need to fake your algorithm into being deterministic (by fixing both the input, output and randomly generated values).

Based on your edit:

My implementations of GA never contained real-life tests, only unit and integration tests. The first would proof the theoretical correctness of the implementation while the second would proof the cohesion of the implementation. Both factors tell me something about the quality of the code but not the performance (where performance is the generated output from the optimum output).

If i get you right, you want to measure the performance of your implementation to see the effects of modifications. For this to work you first of all need sample input data and optimal output data. I figure you have the first one. The second one might be more tricky though. Secondly, you need to run multiple times for each test set. Since GA's are unpredictable (based on your configured mutation rate). This all might come down to a point where just having a simple sample test set would be more efficient.

Anyways, as for answering your question: After doing multiple runs, your assertion should verify that the mean error (compared to the optimal output) is within an acceptable range.

Polity
  • 14,734
  • 2
  • 40
  • 40
  • I manage to dodge the nondeterministic thing in unit testing. The way it's coded I could isolate the generation of random numbers. Every time random numbers are used, I could generate them and put the code with the logic that works with them in another function. This way I can test the logic function passing it the numbers I want to pass from test cases. But what I want now is test the real behavior of the algorithm with real data, in order to see while at maintenance, if at some point the behavior with some data becomes "weird". – Pedro R. Dec 26 '12 at 17:00
  • Thanks. Yeah, doing multiple runs and assert on the mean error and maybe other statistic values can be a good approach. Thank you very much. – Pedro R. Dec 27 '12 at 21:05
1

In general use, your pseudo-random number generator is seeded with something difficult for an attacker to exactly predict - usually based on the computer's clock or uptime. If, however, you seed your pseudo-random number generator with a constant - it will always return the same results. So you could unit test with a seeded pRNG. That will provide a decent - not perfect, but decent - guard against regressions.

Joel Rein
  • 3,608
  • 1
  • 26
  • 32
  • Library already has unit tests. What I want to test is the real behavior with real data to keep it under control while modifying it. I'll update the test to clarify the goal. – Pedro R. Dec 26 '12 at 17:02