STAMP Unit Test Amplification Virtual Lab Guide

Prerequisites

You need to have:

  • JDK 8 or later
  • git
  • Apache Maven

Check your unit tests effectiveness with Descartes

As described in Descartes home page, you can evaluate the capability of your test suite to detect bugs using extreme mutation testing.

First of all, clone the example project DHell from STAMP official repository:

git clone https://github.com/STAMP-project/dhell

Then go to the DHell project folder:

cd dhell

You will find several pom.xml files each preconfigured to try the tools with specific configurations.

To try Descartes quickly, you can run the shell script run_pitest_descartes.sh.

The console output will show you the effect of applying Descartes to a Java project:

 
stamper@stamper-lab1:~/STAMP-project/dhell$ ./run_pitest_descartes.sh
Oct 18, 2018 2:18:34 PM eu.stamp_project.examples.dhell.HelloApp <init>
INFO: MyPrintCount = 1 - MyTracesName = myHelloApp.traces
Oct 18, 2018 2:18:34 PM eu.stamp_project.examples.dhell.HelloApp <init>
...

14:18:36
[INFO] Scanning for projects...
Downloading from central: https://repo.maven.apache.org/maven2/org/pitest/pitest-maven/1.4.0/pitest-maven-1.4.0.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/pitest/pitest-maven/1.4.0/pitest-maven-1.4.0.pom (5.5 kB at 16 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/pitest/pitest-parent/1.4.0/pitest-parent-1.4.0.pom
Downloaded from central: https://repo.maven.apache.org/maven2/org/pitest/pitest-parent/1.4.0/pitest-parent-1.4.0.pom (9.0 kB at 54 kB/s)
Downloading from central: https://repo.maven.apache.org/maven2/org/pitest/pitest-maven/1.4.0/pitest-maven-1.4.0.jar
Downloaded from central: https://repo.maven.apache.org/maven2/org/pitest/pitest-maven/1.4.0/pitest-maven-1.4.0.jar (78 kB at 646 kB/s)
[INFO]
[INFO] ~-~-~-~-~-~-~-~-~-~-~-~-~-< eu.stamp-project.examples:hello_app >~-~-
[INFO] Building hello_app 1.2.1
[INFO] ~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-[ jar ]~-~-~-~-~-~-~-~-~-~-~-~-~-
[INFO]
[INFO] ~-~-- pitest-maven:1.4.0:mutationCoverage (default-cli) @ hello_app ~-~--
Downloading from central: https://repo.maven.apache.org/maven2/eu/stamp-project/descartes/1.2/descartes-1.2.pom

...

[INFO] Found plugin : Mutant export plugin

[INFO] Found shared classpath plugin : Engine for extreme mutation operators
[INFO] Found shared classpath plugin : Default mutation engine
[INFO] Found shared classpath plugin : JUnit plugin
[INFO] Found shared classpath plugin : TestNG plugin
[INFO] Adding eu.stamp-project:descartes to SUT classpath
[INFO] Adding org.pitest:pitest to SUT classpath
[INFO] Mutating from /home/daniele/Progetti/STAMP/dhell/target/classes
[INFO] Defaulting target classes to match packages in build directory
14.18.41 PIT >> INFO : Verbose logging is disabled. If you encounter an problem please enable it before reporting an issue.
14.18.41 PIT >> INFO : Sending 6 test classes to minion
14.18.41 PIT >> INFO : Sent tests to minion
14.18.41 PIT >> INFO : MINION : 14.18.41 PIT >> INFO : Checking environment

14.18.41 PIT >> INFO : MINION : 14.18.41 PIT >> INFO : Found  11 tests

14.18.41 PIT >> INFO : MINION : 14.18.41 PIT >> INFO : Dependency analysis reduced number of potential tests by 0

14.18.41 PIT >> INFO : MINION : 14.18.41 PIT >> INFO : 11 tests received

14.18.41 PIT >> INFO : MINION : Oct 18, 2018 2:18:41 PM eu.stamp_project.examples.dhell.HelloApp <init>

14.18.41 PIT >> INFO : MINION : Oct 18, 2018 2:18:41 PM eu.stamp_project.examples.dhell.HelloApp <init>
INFO: MyPrintCount = 1 - MyTracesName = foo1.traces

\14.18.41 PIT >> INFO : Calculated coverage in 0 seconds.
14.18.42 PIT >> INFO : Created  4 mutation test units
/stderr  : ott 18, 2018 2:18:42 PM eu.stamp_project.examples.dhell.HelloApp <init>
INFORMAZIONI: MyPrintCount = 8 - MyTracesName = foo3.traces
stderr  : ott 18, 2018 2:18:42 PM eu.stamp_project.examples.dhell.HelloApp <init>
...

|14.18.42 PIT >> INFO : Completed in 1 seconds
~================================================================================
- Timings
~================================================================================
> scan classpath : < 1 second
> coverage and dependency analysis : < 1 second
> build mutation tests : < 1 second
> run mutation analysis : < 1 second
~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~
> Total  : 1 seconds
~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~
~================================================================================
- Statistics
~================================================================================
>> Generated 18 mutations Killed 11 (61%)
>> Ran 26 tests (1.44 tests per mutation)
~================================================================================
- Mutators
~================================================================================
> ""
>> Generated 1 Killed 1 (100%)
> KILLED 1 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~
> 0
>> Generated 1 Killed 0 (0%)
> KILLED 0 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~

...
~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~
> false
>> Generated 2 Killed 2 (100%)
> KILLED 2 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~
> true
>> Generated 2 Killed 0 (0%)
> KILLED 0 SURVIVED 2 TIMED_OUT 0 NON_VIABLE 0
> MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
> NO_COVERAGE 0
~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~
[INFO] ~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-
[INFO] BUILD SUCCESS
[INFO] ~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-
[INFO] Total time: 5.126 s
[INFO] Finished at: 2018-10-18T14:18:42+02:00
[INFO] ~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-
14:18:42
  

If you want to understand better how to use Descartes, you can open the run_pitest_descartes.sh script in the Leafpad available in the VM and study the content:

descarted_script.png

Amplify your unit tests with DSpot

As described in DSpot home page, you can enhance the capability of your test suite to detect bugs applying DSpot to it. DSpot will generate missing assertions to your test cases, taking them in input as well as your Java project.

If you still haven't done, clone the example project DHell from STAMP official repository:

git clone https://github.com/STAMP-project/dhell

Then go to the DHell project folder:

cd dhell

You will find several pom.xml files each preconfigured to try the tools with specific configurations.

To try DSpot quickly, you can run the shell script *run_dspot.sh.

The console output will show you the effect of applying Descartes to a Java project:

stamper@stamper-lab1:~~/STAMP-project/dhell$ ./run_dspot.sh