STAMP Unit Test Amplification Virtual Lab Guide

Prerequisites

You need to have:

  • JDK 8 or later
  • git
  • Apache Maven

What is it about ?

Descartes performs "extreme mutation" : it removes code in a software project, then passes the JUnit test suite.

Are you sure your tests fail if your code is removed ? Sounds like common sense... but in most real-life projects, many tests don't fail when code vanishes ! Descartes detects that, and tells you where it happens, so you can fix.

DSpot generates JUnit tests to complete an existing suite (it looks for missing assertions). In complement with Descartes, it can help detect some mutations that existing tests did not cover.

In this tutorial, you will run Descartes on a sample project to detect flaws in tests, then use DSpot to fix some of them.

Check your unit tests effectiveness with Descartes

As described in Descartes home page, you can evaluate the capability of your test suite to detect bugs using extreme mutation testing.

First of all, clone the example project DHell from STAMP official repository:

git clone https://github.com/STAMP-project/dhell

Then go to the DHell project folder:

cd dhell

You will find several pom.xml files each preconfigured to try the tools with specific configurations.

To try Descartes quickly, you can run the shell script run_pitest_descartes.sh.

The console output will show you the effect of applying Descartes to a Java project:

 
stamper@stamper-lab1:~/STAMP-project/dhell$ ./run_pitest_descartes.sh
Oct 18, 2018 2:18:34 PM eu.stamp_project.examples.dhell.HelloApp <init>
INFO: MyPrintCount = 1 - MyTracesName = myHelloApp.traces
Oct 18, 2018 2:18:34 PM eu.stamp_project.examples.dhell.HelloApp <init>
...

14:18:36
[INFO] Scanning for projects...
Downloading from central: https://repo.maven.apache.org/maven2/org/pitest/pitest-maven/1.4.0/pitest-maven-1.4.0.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/pitest/pitest-maven/1.4.0/pitest-maven-1.4.0.pom (5.5 kB at 16 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/pitest/pitest-parent/1.4.0/pitest-parent-1.4.0.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/pitest/pitest-parent/1.4.0/pitest-parent-1.4.0.pom (9.0 kB at 54 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/pitest/pitest-maven/1.4.0/pitest-maven-1.4.0.jar
Downloaded from central: https://repo.maven.apache.org/maven2/org/pitest/pitest-maven/1.4.0/pitest-maven-1.4.0.jar (78 kB at 646 kB/s)
[INFO]
[INFO] ~-~-~-~-~-~-~-~-~-~-~-~-~-< eu.stamp-project.examples:hello_app >~-~-
[INFO] Building hello_app 1.2.1
[INFO] ~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-[ jar ]~-~-~-~-~-~-~-~-~-~-~-~-~-
[INFO]
[INFO] ~-~-- pitest-maven:1.4.0:mutationCoverage (default-cli) @ hello_app ~-~--
Downloading from central: https://repo.maven.apache.org/maven2/eu/stamp-project/descartes/1.2/descartes-1.2.pom

...

[INFO] Found plugin : Mutant export plugin

[INFO] Found shared classpath plugin : Engine for extreme mutation operators
[INFO] Found shared classpath plugin : Default mutation engine
[INFO] Found shared classpath plugin : JUnit plugin
[INFO] Found shared classpath plugin : TestNG plugin
[INFO] Adding eu.stamp-project:descartes to SUT classpath
[INFO] Adding org.pitest:pitest to SUT classpath
[INFO] Mutating from /home/daniele/Progetti/STAMP/dhell/target/classes
[INFO] Defaulting target classes to match packages in build directory
14.18.41 PIT >> INFO : Verbose logging is disabled. If you encounter an problem please enable it before reporting an issue.
14.18.41 PIT >> INFO : Sending 6 test classes to minion
14.18.41 PIT >> INFO : Sent tests to minion
14.18.41 PIT >> INFO : MINION : 14.18.41 PIT >> INFO : Checking environment

14.18.41 PIT >> INFO : MINION : 14.18.41 PIT >> INFO : Found  11 tests

14.18.41 PIT >> INFO : MINION : 14.18.41 PIT >> INFO : Dependency analysis reduced number of potential tests by 0

14.18.41 PIT >> INFO : MINION : 14.18.41 PIT >> INFO : 11 tests received

14.18.41 PIT >> INFO : MINION : Oct 18, 2018 2:18:41 PM eu.stamp_project.examples.dhell.HelloApp <init>

14.18.41 PIT >> INFO : MINION : Oct 18, 2018 2:18:41 PM eu.stamp_project.examples.dhell.HelloApp <init>
INFO: MyPrintCount = 1 - MyTracesName = foo1.traces

\14.18.41 PIT >> INFO : Calculated coverage in 0 seconds.
14.18.42 PIT >> INFO : Created  4 mutation test units
/stderr  : ott 18, 2018 2:18:42 PM eu.stamp_project.examples.dhell.HelloApp <init>
INFORMAZIONI: MyPrintCount = 8 - MyTracesName = foo3.traces
stderr  : ott 18, 2018 2:18:42 PM eu.stamp_project.examples.dhell.HelloApp <init>
...

|14.18.42 PIT >> INFO : Completed in 1 seconds
~================================================================================
- Timings
~================================================================================
> scan classpath : < 1 second
> coverage and dependency analysis : < 1 second
> build mutation tests : < 1 second
> run mutation analysis : < 1 second
~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~
> Total  : 1 seconds
~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~
~================================================================================
- Statistics
~================================================================================
>> Generated 18 mutations Killed 11 (61%)
>> Ran 26 tests (1.44 tests per mutation)
~================================================================================
- Mutators
~================================================================================
> ""
>> Generated 1 Killed 1 (100%)
> KILLED 1 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~
> 0
>> Generated 1 Killed 0 (0%)
> KILLED 0 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~

...
~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~
> false
>> Generated 2 Killed 2 (100%)
> KILLED 2 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~
> true
>> Generated 2 Killed 0 (0%)
> KILLED 0 SURVIVED 2 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~
[INFO] ~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-
[INFO] BUILD SUCCESS
[INFO] ~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-
[INFO] Total time: 5.126 s
[INFO] Finished at: 2018-10-18T14:18:42+02:00
[INFO] ~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-
14:18:42
  

Descartes generates a detailed HTML output, available in target/pit-reports: there, you will find a sub-directory (its name is the date/time of the moment Descartes was run, like target/pit-reports/201907181115/ ). Open the "index.html" file that you find there, in your web browser:

dhell-descartes.png

This is Descartes statistics: browse the HTML files, to see what Descartes discovered.

For example, in HelloApp.java, line 124, method "returnSomething()" can be emptied (its content replaced by "return 1"), and JUnit tests detect nothing: you are prompted to fix the corresponding tests !

If you want to understand better how to use Descartes, you can open the run_pitest_descartes.sh script in the Leafpad available in the VM and study the content:

descarted_script.png

Amplify your unit tests with DSpot

As described in DSpot home page, you can enhance the capability of your test suite to detect bugs applying DSpot to it. DSpot will generate missing assertions to your test cases, taking them in input as well as your Java project.

If you still haven't done, clone the example project DHell from STAMP official repository:

git clone https://github.com/STAMP-project/dhell

Then go to the DHell project folder:

cd dhell

You will find several pom.xml files each preconfigured to try the tools with specific configurations.

To try DSpot quickly, you can run the shell script *run_dspot.sh.

The console output will show you the effect of applying Descartes to a Java project:

stamper@stamper-lab1:~~/STAMP-project/dhell$ ./run_dspot.sh

Now, you can just copy the test generated by DSpot to enhance the JUnit test suite:

stamper@stamper-lab1:~~/STAMP-project/dhell$ cp dspot-out/eu/stamp_project/examples/dhell/AmplHelloAppTest.java src/test/java/eu/stamp_project/examples/dhell/

Then run the JUnit test suite:

stamper@stamper-lab1:~~/STAMP-project/dhell$ mvn test

Run Descartes again to see what's been fixed

To compare results, save the content of target/pit-reports somewhere (as the next "mvn clean" would delete it - and Descartes scripts do that).

Then, as before, run the shell script run_pitest_descartes.sh again.

Have a look at target/pit-reports: the statistics are better than the 1st time, because DSpot fixed something !

 

Site maintained by OW2