Blog

Apr 24 2019

Cyrille Artho, KTH Associate Professor

Mutation testing can serve as indicators on test quality and relevance

STAMP_Cyrille_Artho_KTH.jpg
How do you see mutation testing tools being adopted in business projects, as an effective method with significant benefits on distributed software updates?

I think mutation testing has its uses in specialized settings, where the software is small (or well-modularized). However, many companies struggle with the fact that they can't keep up with testing, so we also see the opposite (test selection), where tests are not analyzed as much in depth, but instead, the goal is to be as light-weight as possible. From this point of view, mutation techniques should in general also be efficient and light-weight.

Do you expect mutation testing to improve software quality and maintainability?

Mutation testing can serve as indicators on test quality and relevance, so even if used only periodically, it could have its uses. For example, tests that are ineffective no longer have to be maintained.


Do you see a potential complementarity between mutation tests and model-based test tools? For instance, could software-defined networks or IoT applications benefit from the combinaison of Modbat and Descartes? 

We already use mutation testing in this context, to see how effective a tool or model is at detecting defects. Usually, there are not enough real defects to go around, or it is time-consuming to find enough failing unit tests in a bug tracker that still work today. So it is easier to generate artificial failures. Of course, they may not have all the characteristics of the real thing. I see mutation testing as a good way to test not only the quality of the inputs, but especially the quality of the test oracle as well.

Business software are now based on a DevSecOps approach, using open source infrastructure stacks. Do you see any downside in this trend? 

Many frameworks are relying on shared open source software. The open collaborations have greatly improved the quality of distributed software. The only downside I can see: if you have security problems now, thousands of companies are affected. During the last ten years, the open source mouvement has grown, with a natural selection of the best components. Many OSS innovations are supported by the industry. However, I wish industrials share their concerns and let the community know when they fix software issues. Most of the companies are still too secretive. They often use open source software, but don’t contribute back. More mutual relationship would be beneficial to improve the quality of the codes. Also the contributing organisations would become more attractive as employers. 

A word about yourself and your organization

Since August 2016, I am Associate Professor at the KTH Royal Institute of Technology in Stockholm. My main interests are software verification and software engineering. During my Master's thesis and my Ph.D. thesis at ETH Zurich, I investigated different approaches for finding faults in multi-threaded programs. I worked for two years at Tokyo's National Institute of Informatics as a Postdoctoral Researcher. Then, from April 2007 to July 2016, I worked as Senior Researcher at the National Institute of Advanced Industrial Science and Technology (AIST) in Tokyo and Osaka. In recent work, I extended my work on concurrent software to networked programs and I developed my own model-based test tool Modbat, which is specialized for generating tests for networked software.

Apr 17 2019

STAMP talk at Devoxx France, in Paris

STAMP will be presentated at DeVoxx France, through two conferences. See details below (in french). 

Date: 17-19 April 2019
Place: Paris
Registration: https://www.devoxx.fr/


Presentation 1: L'Intégration Continue comme partenaire pour des suggestions d'amélioration des tests
Speaker: Caroline Landry (INRIA)
Conference track: Agilité, méthodologie et tests
Abstract: Le CI (Intégration Continue) exécute en continu les tests unitaires et d'intégration pour donner aux développeurs un retour d'information sur la qualité du code. Cependant, il fournit peu de rétroaction sur la qualité des cas de test eux-mêmes. Cet exposé présente deux techniques automatiques qui fonctionnent dans le CI afin de suggérer les faiblesses et les améliorations possibles des cas de test.
La première technique est basée sur du test par mutation. Elle identifie les méthodes pseudo-testées : méthodes qui sont couvertes par la suite de tests mais qui peuvent être complètement supprimées sans qu'aucun cas de test n'échoue. Ceci indique que les cas de test qui couvrent ces méthodes doivent être améliorés pour observer et spécifier le comportement de manière plus approfondie. La seconde technique est basée sur l'amplification de test : elle suggère des améliorations possibles des cas de test pour mieux tester les méthodes pseudo-testées. Les deux techniques sont implémentées pour Java et sont disponibles sous forme d'outils open source : Descartes et DSpot.

Presentation 2: Tests fonctionnels avec Docker et TestContainers
Speaker: Vincent Massol (XWiki)
Conference track: Tools in action
Abstract: En tant que développeur, qu'il est bon d'être capable de débugguer sur sa machine un problème survenant en production, dans une configuration spécifique ! C'est ce que permet le framework TestContainers. Il permet de piloter Docker directement depuis ses tests JUnit et donc d'avoir un mécanisme extrêmement efficace pour déployer ses tests fonctionnels dans un environnement donné.

Cette session présentera TestContainers, appliqué à un cas réel avec une démonstration de comment l'utiliser pour effectuer des tests impliquant une base de données, un moteur de Servlet et plus. Au programme: Intégration JUnit5, création d'images Docker custom, enregistrement automatique de vidéos des tests, intégration avec un job Jenkins pipeline pour itérer sur les différentes configurations à tester.

 

Apr 12 2019

Mutation Testing Workshop at Ericsson HQ in Stockholm

Date: Friday 12 April, 2019
Place: Ericsson Offices, building 18, Torshamngatan 23 Kista Stockholm
Room: LM Junior Conference room
Hours: 09:00AM - 04:00PM
Speaker: Oscar Luis Vera Pérez (Inria Rennes) More…

Apr 11 2019

Test Automation Research for Industry Workshop in Stockholm

TESTOMAT, STAMP, XIVT & Software Center are organising in Stockholm a one-day workshop with leading researchers in test automation. They're sharing their work to help businesses reach the next level of test automation. This unique event consists of presentations as well as showcasing tools with topics covering latest research in test automation.
It is an opportunity for industry representatives and experts to collaborate with researchers and to discuss how the latest test automation research can transform the current costlier and less effective way of testing.

The workshop is taking place 11 April 2019, from 8:30AM to 4:00PM at
Ericsson Group HQ
Building 21 Torshamngatan
Kista, Stockholm
Sweden

Talk: Software Testing Amplification in DevOps
Speaker: Benoit Baudry, KTH and STAMP Project

Apr 09 2019

Plenary meeting in Stockholm, April 9-10

Meeting Place: KTH central campus
map
Lindstedtsvägen 5
Room:D31 More…

Apr 04 2019

Lars Thomas Boye, Tellu IoT Senior Software Developer

Test amplification has a great potential to cover multiple contexts and configurations

Lars_Thomas_TellU.jpg

How would you present STAMP?

For me, STAMP is about helping organizations who develop software to test that software, thereby increasing software quality. It uses amplification to address the need to do more testing while also reducing the effort needed to increase test coverage and quality. For a company selling software, or services based on software developed in-house, software quality is vital, and testing is the way to insure quality. Efficiency and agility are very important in today’s world, and DevOps is used to be able to quickly deploy changes made to the code. Efficient DevOps rely on automation, and good testing is vital to make sure new changes can be deployed without breaking existing features. We need good test coverage and we need to know that the quality of the coverage is good, that bugs in covered code really are discovered by tests. STAMP is about amplification, which I believe can help in several ways. Developers can write more tests, but this is costly and time-consuming, so if new tests can be generated from existing tests, stack traces or other existing data then this is a large gain. It is also not feasible to manually cover all variations, especially when considering complex systems which can run in different contexts and configurations, and here amplification has great potential.

 

What is your role in STAMP?

I work for a small Norwegian company called Tellu. Tellu’s software platform for our TelluCloud service is a STAMP project use case. This means that my main responsibility is in Work Package 5, where I work on instrumenting testing for TelluCloud, test the STAMP tools and providing feedback to the STAMP developers and researchers. I also have roles in two of the work packages developing the STAMP tools, WP2 and WP3. Here I work more directly with the tool developers on adopting the tools for the TelluCloud use case.
My role is mainly to provide a real-world use case to test and validate the tools. Hopefully this will help the tool development reach mature tools with a wide area of application. 

What key innovation do you bring or help to develop? 

The TelluCloud use case is quite demanding and need both unit test and configuration test amplification. We use industry standard technologies such as Java, Maven and Jenkins, and are moving towards DevOps with continuous integration and delivery. At the start of the STAMP project, a new version of TelluCloud was in development, based on a micro-service architecture. This is the version we have been testing in the project. Splitting up the system into micro-services made the development and maintenance of each part more manageable, but it also introduced new challenges. The complexity of the system as a whole increased, especially with respect to configuration and deployment, as there are many more parts to deploy, including queues and other infrastructure components. In addition to unit testing, testing of each micro-service and of the system as a whole became paramount. The micro-service version of TelluCloud is now in production. Testing done in STAMP has helped make this happen.

A word about yourself and your organization

I am a senior software developer at Tellu IoT, a small software company in Norway. Tellu provides TelluCloud, a cloud platform for collecting and processing data with a focus on IoT. We provide TelluCloud to service providers and other partners in different domains, for integration in their solutions. Our current commercial focus is e-health, communal care and personal safety and security domains.

I have a master’s degree in software engineering from NTNU (the Norwegian University of Science and Technology). I have worked at Tellu since 2008. Here, I work on various parts of our technology stack, both the TelluCloud server-side platform and mobile apps. Tellu is quite research oriented, having originated as a spinoff from Ericsson’s research department. I have participated in a number of research projects, often EU projects.

My involvement in STAMP concerns the work I do on software testing at Tellu. I have implemented a testing framework for running component and system tests on TelluCloud, which is based on pushing messages to the system under test and analyzing message outputs. Tellu’s overall objective in STAMP is to improve and automate testing for TelluCloud, to ensure its correctness and robustness in a cost-effective manner so that Tellu can continue to develop and deploy new versions without introducing bugs which disrupt the services. One STAMP tool, Descartes, has already helped to improve our test quality. It detects pseudo-tested methods – methods in our code which are executed in tests but where breaking the method is not detected by the test. Learning about mutation testing and pseudo-tested methods has been very useful for us and means we can now quantity test quality and correct mistakes. We are also very interested in configuration testing, as we need to be able to bring up the TelluCloud system in various configurations for running component and system tests. 

 

Apr 02 2019

Search-Based Test Case Implantation for Testing Untested Configurations

Publication: Information and Software Technology
Authors: Dipesh Pradhan, Shuai Wang, Tao Yue, Shaukat Ali, Marius Liaaen
URL: https://www.sciencedirect.com/science/article/pii/S0950584919300540?dgcid=rss_sd_all

Title: Search-Based Test Case Implantation for Testing Untested Configurations

Context
Modern large-scale software systems are highly configurable, and thus require a large number of test cases to be implemented and revised for testing a variety of system configurations. This makes testing highly configurable systems very expensive and time-consuming.

Objective
Driven by our industrial collaboration with a video conferencing company, we aim to automatically analyze and implant existing test cases (i.e., an original test suite) to test the untested configurations.

Method
We propose a search-based test case implantation approach (named as SBI) consisting of two key components: 1) Test case analyzer that statically analyzes each test case in the original test suite to obtain the progr am dependence graph for test case statements and 2) Test case implanter that uses multi-objective search to select suitable test cases for implantation using three operators, i.e., selection, crossover, and mutation (at the test suite level) and implants the selected test cases using a mutation operator at the test case level including three operations (i.e., addition, modification, and deletion).

Results
We empirically evaluated SBI with an industrial case study and an open source case study by comparing the implanted test suites produced by three variants of SBI with the original test suite using evaluation metrics such as statement coverage (SC), branch coverage (BC), and mutation score (MS). Results show that for both the case studies, the test suites implanted by the three variants of SBI performed significantly better than the original test suites. The best variant of SBI achieved on average 19.3% higher coverage of configuration variable values for both the case studies. Moreover, for the open source case study, the best variant of SBI managed to improve SC, BC, and MS with 5.0%, 7.9%, and 3.2%, respectively.

Conclusion
SBI can be applied to automatically implant a test suite with the aim of testing untested configurations and thus achieving higher configuration coverage.

Mar 22 2019

Configuration Tests: the JHipster web development stack use case

SpringerESE

Title: Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack
Authors: Axel Halin, Alexandre Nuttinck, Mathieu Acher, Xavier Devroey, Gilles Perrouin, Benoit Baudry
Publication: Springer Empirical Software Engineering, April 2019, Vol24

A group of software researchers, partially involved in the EU-funded STAMP project, has published interesting results based on configuration tests on JHipster, a popular open source application generator to create Spring Boot and Angular/React projects. 

Abstract: Many approaches for testing configurable software systems start from the same assumption: it is impossible to test all configurations.
This motivated the definition of variability-aware abstractions and sampling techniques to cope with large configuration spaces. Yet, there is no theoretical barrier that prevents the exhaustive testing of all configurations by simply enumerating them if the effort required to do so remains acceptable. Not only this: we believe there is a lot to be learned by systematically and exhaustively testing a configurable system. In this case study, we report on the first ever endeavour to test all possible configurations of the industry-strength, open source configurable software system JHipster, a popular code generator for web applications. We built a testing scaffold for the 26,000+ configurations of JHipster using a cluster of 80 machines during 4 nights for a total of 4,376 hours (182 days) CPU time. We find that 35.70% configurations fail and we identify the feature interactions that cause the errors. We show that sampling strategies (like dissimilarity and 2-wise):
(1) are more effective to find faults than the 12 default configurations used in the JHipster continuous integration;
(2) can be too costly and exceed the available testing budget. We cross this quantitative analysis with the qualitative assessment of JHipster’s lead developers.

Read the full article on Springer website

Mar 20 2019

BreizhCamp 2019 STAMP & CI testing automation 

Date: 20/3/2019
Place: ISTIC, Rennes University, France - Map

Talk title: The Continuous Integration as a partner for test improvement suggestions
Speaker: Caroline Landry (INRIA)

More…

Mar 12 2019

STAMP Webinar with Industry DevOps

LaDefense_cb16_cgi_tower.jpg
Date: 12 March 2019
Presenter: Benjamin Danglot, Inria researcher
Language: French

With 25 external participants from CGI digital service company, this webinar provided the opportunity to run STAMP toolset demos and to answer to the questions asked by industrial developers, testers and DevOps professionals. 

CGI is a Canadian global information technology (IT) consulting, systems integration, outsourcing, and solutions company headquartered in Montreal, Quebec, Canada. In 1976, Serge Godin and André Imbeau founded CGI, which originally stood for “Conseillers en gestion et informatique” (translating to “Consultants in management and information technology”).
In 2018, the group revenue reached $11.5 billion, up 6.1% or 4.6% in constant currency, with adjusted EBIT of $1.7 billion, or 14.8% of revenue. More…