7

Can anyone provide suggestions of tools that can be used to make mutations within a Java program at a source-code (not byte-code) level? I need to seed my source code with faults. I would prefer an application with a GUI, if one exists.

Thomas Owens
  • 114,398
  • 98
  • 311
  • 431
Konstantin Milyutin
  • 11,946
  • 11
  • 59
  • 85
  • This has little application in the real world – Bohemian Jun 27 '11 at 01:11
  • Simply replacing some random characters by others could do, but in most cases this will be letal (i.e. the resulting program will not be compilable, if you did't hit a string or comment or replaced really luckily). – Paŭlo Ebermann Jun 27 '11 at 01:11
  • 7
    "This has little application in the real world" is a poor excuse. When tensor, topology or calculus mathematics were developed, they had very "little real world use". This is an interesting question. Code mutation has very real world applications. – Blessed Geek Jun 27 '11 at 01:32
  • 1
    It might be useful if you could provide suggestions (i.e. examples) of mutations you would like and the attributes of the result - should it still compile, should the errors be regularly placed, etc. – Charles Goodwin Jun 27 '11 at 01:40
  • 3
    @Bohemian: It is applied in the real world for determining test coverage. – Don Roby Jun 27 '11 at 01:55
  • How is this question too localized? Mutant (or mutation) testing is a valid testing technique that can be used to test software applications. However, it's not commonly used since it's time-intensive (and therefore, for most projects, cost-intensive as well). – Thomas Owens Jun 27 '11 at 02:05
  • https://pitest.org is the number 1 Mutation Testing tool for Java today (2023), but this is done at _byte code_ level rather than _source code_ level – 8bitjunkie Mar 11 '23 at 14:12

7 Answers7

4

I've found https://pitest.org, which I quite like. With MuJava I have exceptions caused by @Override annotations. Seems that it doesnt' support Java 5/6.

8bitjunkie
  • 12,793
  • 9
  • 57
  • 70
Konstantin Milyutin
  • 11,946
  • 11
  • 59
  • 85
  • 3
    Good overview of PIT here: http://dev.theladders.com/2013/02/mutation-testing-with-pit-a-step-beyond-normal-code-coverage/ – Kyri Sarantakos Mar 05 '13 at 15:25
  • Broken link above. Baeldung have a recent article (2022) https://www.baeldung.com/java-mutation-testing-with-pitest – 8bitjunkie Mar 11 '23 at 14:25
4

If your requirement really is for source code mutation then the options I'm aware of are

Jester - Provides a limited number of mutation operators that are quite unstable. Jeff Offutt described it as a very expensive way to apply branch testing. I believe it is possible to define your own operators however, so this probably isn't entirely fair.

Mu Java - Doesn't support Java 5

Judy - Don't know much about this other than that it exists

Lava - Sounds pretty basic and I don't think it supports java 5

From the answer further down I see your now using PIT. I think that's a truly great choice (I wrote it so may have a certain bias), but it's a byte code mutation system.

henry
  • 5,923
  • 29
  • 46
3

When I took a software engineering class on testing at GMU, we used Mu Java, for one of the chapters. HTH

Foon
  • 6,148
  • 11
  • 40
  • 42
3

This is the technique used by the open source test coverage tool Jester. A look at it's source might be helpful if you are unable to find a suitable tool and want to build one yourself.

Don Roby
  • 40,677
  • 6
  • 91
  • 113
2

"This has little application in the real world". Disagree strongly, I contract at a very big media company in the UK and PIT (http://pitest.org/) is extremely useful for mutation testing. Most of the CI builds are set fail without 85% coverage PIT mutation testing. Re: "replacing some random characters..." Mutation does not have to mean that. The PIT mutations are restricted to compilable changes.

Noctiluque
  • 846
  • 9
  • 7
1

The Major mutation framework (Major's website) provides a compiler-integrated mutator and a mutation analyzer for JUnit tests.

Major's mutator is integrated in the Java 7 compiler. It provides several mutation operators and supports two options for mutating source code:

  1. Generate and embed all mutants during compilation
  2. Generate mutants and export mutated source files
Rene Just
  • 81
  • 2
1

Here it is a benchmark :

MμClipse only supports JUnit 3 and is no longer maintained. Jester as for it, is laborious and requires a complicated configuration; plus is not maintained anymore. The best tool I could find is Javalanche I had wrote a entire article about this !

Nicolas De Nayer
  • 259
  • 2
  • 6
  • Since 2013, it appears MuClipse is now abandonware: https://sourceforge.net/projects/muclipse (MuClipse was an Eclipse plugin wrapper for MuJava, which itself became abandonware in 2016) – 8bitjunkie Mar 11 '23 at 14:30