Blog

Jun 14 2019

DSpot amplification visualization

In this post we share a visualization of the unit test amplification process. In the example, DSpot is used to amplify two unit tests from JSoup. More…

Jun 12 2019

STAMP Breakout Session & Talks at OW2con'19

Date: 12-13 June 2019
Place: Orange Gardens, Paris Châtillon
Registration: https://www.ow2con.org

During the OW2 open source community annual conference, the STAMP project is represented in a couple of talks:

  • STAMP test automation for Continuous Integration, a talk by Caroline Landry (Inria)
  • Functional testing with Docker and TestContainers, a talk by Vincent Massol (XWiki) 

The STAMP project also offers a dedicated Breakout Session: Improve your tests with STAMP
Date: 13 June 2019 from 9:15 to 10:45

The breakout session will demonstrate STAMP testing tools to DevOps Teams.
Bring your source code, test suites and Java stack traces, and try STAMP tools to automatically improve your tests:

  • Descartes: test your JUnit tests and find their weaknesses
  • DSpot: automatically generate new JUnit tests or new assertions to amplify your test suite
  • CAMP: automatically generate and run in new configurations from your existing Docker files
  • Botsing: automatically generate new JUnit test case from a Java stack trace to reproduce your crash
  • STAMP automated issue reporting with fix suggestions (demo on Gitlab)

Speakers: 

  • Franck Chauvel, Research Scientist, SINTEF
  • Benjamin Danglot, Researcher, Inria
  • Pouria Derakhshanfar, PhD Student, TU Delft
  • Pierre-Yves Gibello, Engineer, OW2
  • Caroline Landry, Inria, Project Manager
  • Oscar Luis Vera-Perez, Researcher, Inria

More about the program and free registration: https://www.ow2con.org

STAMP Testing Tools Among Mature OSS Featured at OW2con'19

More than 30 presentations and breakout sessions showing open source software industrial maturity
 are offered at OW2con’19, the free OW2 community conference, June 12-13, 2019.

Paris, June 12, 2019 - OW2, the international community dedicated to develop and to promote an open source code base for information systems announces the program of its 2019 annual conference. With this year theme  “Open Source: Toward industry Maturity”, OW2con’19 offers two days of project presentations and demonstrations, round tables, workshops and prestigious guest speakers. 

The keynote have been assigned to open source strategists from global companies including Deutsche Telekom, Engineering, Ericsson, IBM, Nokia, Orange and Siemens and from organisations such as DINSIC and Free Software Foundation Europe. 

The program is punctuated by the presentations of latest improvements in OW2 projects and allows for diverse topics such as Software Testing, Remote Collaborations, Artificial Intelligence, IoT, multi-cloud, etc. The open source governance makes a marked entry into the program this year. 

The conference also shows the state of European open souce research through AEGIS, CROSSMINER, DECODER, FASTEN, MELODIC, REACHOUT, and STAMP EC-funded projects.

More highlights include the breakout sessions with practical themes such as STAMP DevOps test automation, collaborative edition with Libre Office and XWiki Content Management Platform, Knowage Business Intelligence, FOSSology code analysis, C* operator for Kubernetes. open source licenses understanding and OSS project assesment. 

« By focusing on the industrial maturity of open source software, the OW2con'19 program is intended, beyond the open source community, for all business users who, from now on, can’t ignore open source in their IT strategy », explains Cedric Thomas, CEO d’OW2

Finally, not to be missed, the OW2con’19 Best Project Awards ceremony will be followed by a friendly networking break with speakers and multiple members of the community.

OW2con’19 is being held June 12-13 at Orange Gardens, the Orange innovation center in Paris-Châtillon. Registration is free, thanks to our sponsors, and should be done online where the program is being updated: https://www.ow2con.org/2019

Jun 03 2019

Production Traffic For Testing

ist

Publication: Information and Software Technology
Authors: Jeff Anderson, Maral Azizi, Saeed Salem, and Hyunsook Do
URL: https://www.sciencedirect.com/science/article/abs/pii/S0950584919301223?dgcid=rss_sd_all

Title: On the Use of Usage Patterns from Telemetry Data for Test Case Prioritization

In an original work about Production Traffic for Testing, Jeff Anderson, Maral Azizi, Saeed Salem, and Hyunsook Do present a new opportunity in the area of regression testing techniques. Here is the abstract:

Context: Modern applications contain pervasive telemetry to ensure reliability and enable monitoring and diagnosis. This presents a new opportunity in the area of regression testing techniques, as we now have the ability to consider usage profiles of the software when making decisions on test execution. Objective: The results of our prior work on test prioritization using telemetry data showed improvement rate on test suite reduction, and test execution time. The objective of this paper is to further investigate this approach and apply prioritization based on multiple prioritization algorithms in an enterprise level cloud application as well as open source projects. We aim to provide an effective prioritization scheme that practitioners can implement with minimum effort. The other objective is to compare the results and the benefits of this technique factors with code coverage-based prioritization approaches, which is the most commonly used test prioritization technique. 

Method: We introduce a method for identifying usage patterns based on telemetry, which we refer to as “telemetry fingerprinting.” Through the use of various
algorithms to compute fingerprints, we conduct empirical studies on multiple software products to show that telemetry fingerprinting can be used to more effectively prioritize regression tests. 

Results: Our experimental results show that the proposed techniques were able to reduce over 30 percent in regression test suite run times compared to the coverage-based prioritization technique in detecting discoverable faults. Further, the results indicate that fingerprints are effective in identifying usage patterns, and that the fingerprints can be applied to improve regression testing techniques.

Conclusion: In this research, we introduce the concept of fingerprinting software usage patterns through telemetry. We provide various algorithms to compute fingerprints and conduct empirical studies that show that fingerprints are effective in identifying distinct usage patterns. By applying these techniques, we believe that regression testing techniques can be improved beyond the current state-of-the-art, yielding additional cost and quality benefits.

May 29 2019

STAMP Software Testing Practices at ICSE 2019

Conference: ICSE '19, the 41st ACM/IEEE International Conference on Software Engineering
Place: Montréal, QC, Canada
Date: 29-30 May 2019
Speakers: Benoit Baudry (KTH), Martin Monperrus (KTH), Benjamin Danglot (Inria), Oscar Vera Perez (Inria)

  

May 24 2019

Jeff Offutt, Professor of Software Engineering at George Mason University (Virginia)

Mutation testing can improve software reliability, including security

Jeff_Offutt.jpg
Where are the business challenges solved by mutation testing?

Mutation can help develop excellent tests. Mutation creates subtle variations of the program-under-test (called mutants), and challenges the tester to design or find test inputs that distinguish the mutants from the original. The resulting test set will find faults that lurk in complicated expressions and hard-to-reach sections of the code. Mutation tests ensure statement and branch coverage, but go far beyond them in their ability to reveal faults in the code.

Who are the main users adopting mutation testing tools in their software production chain? 

Mutation is expensive, impossible to use without substantial tool support, and we still don't know the most efficient and effective way to apply it. It's mostly used by safety-critical software companies (aerospace for example) and financial institutions.

Do you expect mutation testing to improve software quality, maintainability, security, or any other aspect? 

Mutation testing can improve software reliability, including security. From my understanding of DevOps, it cannot exist without test automation. So, by definition, DevOps teams will collaborate on testing automation tools.


Where do you see the main enablers and motivators for automated test automation?

ENABLERS

  • Education: Many developers know very little about test automation and many testers do not know how to automate.
  • Tools: Good tool support make the transition to test automation simpler and cheaper.

MOTIVATORS

  • Competition means quality matters more. Software companies, and companies that depend on software, succeed only if their software works well.
  • Knowledge of the value of better software, unit testing, and test automation.

Would you appreciate more collaborations with STAMP project partners ? On which aspects ? 

I'm a researcher and educator, not a developer. But I am hoping for the day when fully automated testing integrates directly with compiling inside IDEs.

A word about yourself and your organization

I'm leading projects in test automation, mutation testing, and computing education.

Jeff Offutt received the Outstanding Faculty Award from the State Council of Higher Education for Virginia in 2019 and the George Mason University Teaching Excellence Award, Teaching With Technology, in 2013. He leads the MS in Software Engineering program at GMU (George Mason University), teaches Software Engineering courses at all levels and has developed new courses on several Software Engineering subjects, including web engineering, software testing, construction, design, usability, experimentation, and analysis. His current research interests include software testing, test automation, usable security, software engineering education, analysis and testing of web applications.

May 17 2019

STAMP Workshop for EC-DGIT

Title: STAMP software testing for DevOps pipeline
Speakers: Prof. Benoit Baudry (KTH), Caroline Landry and Oscar Luis Vera Pérez (INRIA)
Place: Brussels
Date: 17/5/2019 2:30PM-5:30PM

After Introduction talks from European Commission DGIT Director, Head of Unit DIGIT B4 and Head of sector Software development, STAMP project coordinator Benoit Baudry presents STAMP EC-funded project. Then Oscar Luis Vera Pérez details Extreme Mutation examples using Descartes, an outcome of the STAMP Project, based on PITest to offer faster mutation analysis results. The EC-DG INFORMATICS Software Engineering team seizes this opportunity to ask technical questions about DevOps mutation testing to improve their own Java test environment.

  • Registered participants: 100 (70+ attendees) 

    

May 15 2019

STAMP Banners 

Please use the feedback form URL as image destination link: https://www.stamp-project.eu/view/main/betatestingsurvey/
Alternatively, you can use the github tool URL you want to reinforce in your message. 

STP_Feedback_Banner.jpg
STP_Feedback_DSpot_Banner.jpg
STP_Feedback_Descartes_Banner.jpg
STP_Feedback_CAMP_Banner.jpg
STP_Feedback_Botsing_Banner.jpg

May 06 2019

DSpot meetup in Stockholm

Title: DSpot: Improved software testing using AI
Speaker: Professor Benoit Baudry (KTH)
Place: Birger Jarlsgatan 7, Stockholm
Date: 6/5/2019 17:30-19:00

DSpot analyzes existing test cases and automatically generates variants of these test cases. The variants cover more execution paths and observe more states than the original tests. These tests are meant to be used in the context of regression testing. DSpot can be integrated in the continuous integration to spot issues related to one commit as early as possible.

Apr 24 2019

Cyrille Artho, KTH Associate Professor

Mutation testing can serve as indicators on test quality and relevance

STAMP_Cyrille_Artho_KTH.jpg
How do you see mutation testing tools being adopted in business projects, as an effective method with significant benefits on distributed software updates?

I think mutation testing has its uses in specialized settings, where the software is small (or well-modularized). However, many companies struggle with the fact that they can't keep up with testing, so we also see the opposite (test selection), where tests are not analyzed as much in depth, but instead, the goal is to be as light-weight as possible. From this point of view, mutation techniques should in general also be efficient and light-weight.

Do you expect mutation testing to improve software quality and maintainability?

Mutation testing can serve as indicators on test quality and relevance, so even if used only periodically, it could have its uses. For example, tests that are ineffective no longer have to be maintained.


Do you see a potential complementarity between mutation tests and model-based test tools? For instance, could software-defined networks or IoT applications benefit from the combinaison of Modbat and Descartes? 

We already use mutation testing in this context, to see how effective a tool or model is at detecting defects. Usually, there are not enough real defects to go around, or it is time-consuming to find enough failing unit tests in a bug tracker that still work today. So it is easier to generate artificial failures. Of course, they may not have all the characteristics of the real thing. I see mutation testing as a good way to test not only the quality of the inputs, but especially the quality of the test oracle as well.

Business software are now based on a DevSecOps approach, using open source infrastructure stacks. Do you see any downside in this trend? 

Many frameworks are relying on shared open source software. The open collaborations have greatly improved the quality of distributed software. The only downside I can see: if you have security problems now, thousands of companies are affected. During the last ten years, the open source mouvement has grown, with a natural selection of the best components. Many OSS innovations are supported by the industry. However, I wish industrials share their concerns and let the community know when they fix software issues. Most of the companies are still too secretive. They often use open source software, but don’t contribute back. More mutual relationship would be beneficial to improve the quality of the codes. Also the contributing organisations would become more attractive as employers. 

A word about yourself and your organization

Since August 2016, I am Associate Professor at the KTH Royal Institute of Technology in Stockholm. My main interests are software verification and software engineering. During my Master's thesis and my Ph.D. thesis at ETH Zurich, I investigated different approaches for finding faults in multi-threaded programs. I worked for two years at Tokyo's National Institute of Informatics as a Postdoctoral Researcher. Then, from April 2007 to July 2016, I worked as Senior Researcher at the National Institute of Advanced Industrial Science and Technology (AIST) in Tokyo and Osaka. In recent work, I extended my work on concurrent software to networked programs and I developed my own model-based test tool Modbat, which is specialized for generating tests for networked software.