Mutation testing in cpp without modifying/recompiling the code

Question

Currently I have some tests that are blamed to not catch bugs very well. I want to do mutation testing in order to detect them (and prevent from adding new useless ones), but without the time-inefficient loop: change the code -> recompile -> run tests -> change the code -> recompile -> run tests... etc.

Initially I wanted to mutate somehow binary elf files directly (no recompilation), but as later posts suggested, it makes no sense.

The key point on testing WHAT is to be tested. Some code would help! — fiscblog, Jun 20 '13 at 09:00

Adam · Answer 1 · 2016-11-14T10:31:41.360

4

Ok, I was able to solve it partially, by splitting the mutation testing into 4 main stages:

instrument the code in all mutations with python/clang-tooling (selected C++ expressions are wrapped in a special macro, that delegates calls to a mutation class, which generates ID for each mutation, controls activation of mutation operators, etc.)
recompile the code (only once)
run tests in parallel, with all mutations inactive, and obtain IDs of all mutations (if there are failing tests, put them on an ignore list),
run tests in parallel while switching mutations in the run-time (by ID obtained in the previous step), and gather statistics (mutant kill ratio, etc.)

The implementation is done in python and C++, it is around ~1700 lines of code (with tests) + minor adaptations in the production code (CMake, and the gtest main.cpp file). It supports only a couple of simple mutations, but it still makes fun :)

edited Nov 14 '16 at 10:31

answered Jun 17 '15 at 09:39

Adam

316
2
11

What you have done, is it something that can be reused or are you using some existing tools? I am browsing around for mutation testing solutions for C++ and not finding much. – Zitrax Nov 06 '15 at 10:11
hmm, the code itself is protected by the license, but I will try to find some spare time to create a new, clean public project with a proper license. – Adam Dec 28 '15 at 10:26
1

Did you publish your mutation test tool somewhere? – Niels Lohmann Sep 14 '17 at 07:20

score 2 · Answer 2 · answered Jun 20 '13 at 09:08

Assuming that tests run quickly, and the number of runs is big enough (~1M, ~1k ??), I should get a rough estimate of the hit rate of potential bugs??

No. Your "one-bit error somewhere in the elf binary" could corrupt anything (from elf format to data segments to call stacks and so on). You will not get any rough estimates on the number of bugs that way, but a rough estimate of the chances of a corrupted executable to execute (which says nothing about your application at all).

Currently I have lots of tests that are blamed to not catch bugs at all.

This is something you will have to address directly, and there are no shortcuts for it: you will have to establish new goals for your tests, refactor code to support them and implement them.

Mutation testing in cpp without modifying/recompiling the code

2 Answers2