4.4 Article

Analysing the fitness landscape of search-based software testing problems

Journal

AUTOMATED SOFTWARE ENGINEERING
Volume 24, Issue 3, Pages 603-621

Publisher

SPRINGER
DOI: 10.1007/s10515-016-0197-7

Keywords

Test data generation; Genetic algorithms; Fitness landscape characterisation

Funding

  1. Australian Research Council
  2. Australian Government
  3. Australian Research Council's Discovery Projects funding scheme [DE140100017]
  4. Australian Research Council [DE140100017] Funding Source: Australian Research Council

Ask authors/readers for more resources

Search-based software testing automatically derives test inputs for a software system with the goal of improving various criteria, such as branch coverage. In many cases, evolutionary algorithms are implemented to find near-optimal test suites for software systems. The result of the search is usually received without any indication of how successful the search has been. Fitness landscape characterisation can help understand the search process and its probability of success. In this study, we recorded the information content, negative slope coefficient and the number of improvements during the progress of a genetic algorithm within the EvoSuite framework. Correlating the metrics with the branch and method coverages and the fitness function values reveals that the problem formulation used in EvoSuite could be improved by revising the objective function. It also demonstrates that given the current formulation, the use of crossover has no benefits for the search as the most problematic landscape features are not the number of local optima but the presence of many plateaus.

Authors

I am an author on this paper
Click your name to claim this paper and add it to your profile.

Reviews

Primary Rating

4.4
Not enough ratings

Secondary Ratings

Novelty
-
Significance
-
Scientific rigor
-
Rate this paper

Recommended

Article Computer Science, Information Systems

A comprehensive empirical evaluation of generating test suites for mobile applications with diversity

Thomas Vogel, Chinh Tran, Lars Grunske

Summary: By conducting a fitness landscape analysis, researchers were able to avoid trial-and-error experiments and find a more suitable configuration for SAPIENZ. The new approach, SAPIENZ(DIV), achieves better or at least similar test results in terms of faults and coverage compared to SAPIENZ, but typically produces longer test sequences and requires more execution time.

INFORMATION AND SOFTWARE TECHNOLOGY (2021)

Article Computer Science, Software Engineering

History-based Model Repair Recommendations

Manuel Ohrndorf, Christopher Pietsch, Udo Kelter, Lars Grunske, Timo Kehrer

Summary: In Model-driven Engineering, models are primary development artifacts that are heavily edited and may become temporarily inconsistent. There are multiple methods to resolve inconsistencies, with the suitable one depending on various factors. By analyzing inconsistencies in the version history and generating repair recommendations, past incomplete edits can be automatically fixed.

ACM TRANSACTIONS ON SOFTWARE ENGINEERING AND METHODOLOGY (2021)

Article Computer Science, Software Engineering

Quantitative Verification of Stochastic Regular Expressions

Sinem Getir Yaman, Esteban Pavese, Lars Grunske

Summary: This article introduces a probabilistic verification algorithm for stochastic regular expressions over a probabilistic extension of ACTL*, including a model checking algorithm and semantics on the probabilistic action logic for SREs. The focus is on defining SREs through local probabilistic functions, allowing for local verification of properties and reuse of results for global verification. The article demonstrates how to model a system with SREs and verify it using probabilistic action logic, along with a preliminary performance evaluation based on the reachability algorithm's execution time.

FUNDAMENTA INFORMATICAE (2021)

Article Computer Science, Software Engineering

An Experimental Assessment of Using Theoretical Defect Predictors to Guide Search-Based Software Testing

Anjana Perera, Aldeida Aleti, Burak Turhan, Marcel Bohme

Summary: This paper proposes a new SBST technique called PreMOSA, which combines coverage information with defect prediction information to determine where to increase test coverage in the CUT. Experimental results show that PreMOSA is more effective and efficient than DynaMOSA in detecting bugs, detecting up to 8.3% more bugs on average.

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING (2023)

Article Computer Science, Software Engineering

Inputs From Hell: Learning Input Distributions for Grammar-Based Test Generation

Ezekiel Soremekun, Esteban Pavese, Nikolas Havrikov, Lars Grunske, Andreas Zeller

Summary: Grammars can generate structured test inputs that are syntactically correct. By assigning probabilities to productions, the distribution of input elements can be controlled. Through learning from common inputs, uncommon inputs, and failure-inducing inputs, inputs similar or dissimilar to the sample can be generated for different testing purposes. The evaluation shows the effectiveness of these methods on different input formats.

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING (2022)

Article Computer Science, Information Systems

VUDENC: Vulnerability Detection with Deep Learning on a Natural Codebase for Python

Laura Wartschinski, Yannic Noller, Thomas Vogel, Timo Kehrer, Lars Grunske

Summary: This article introduces VUDENC, a deep learning-based vulnerability detection tool that automatically learns vulnerable code features with high accuracy. By applying the word2vec model and LSTM network, VUDENC can classify vulnerable code fragments finely and provide confidence levels for its predictions.

INFORMATION AND SOFTWARE TECHNOLOGY (2022)

Review Computer Science, Information Systems

A systematic literature review on counterexample explanation

Arut Prakash Kaleeswaran, Arne Nordmann, Thomas Vogel, Lars Grunske

Summary: This article provides an overview of the current state of counterexample explanation techniques, including the contexts, methods, and evaluation. Most studies provide graphical or trace explanations and localize errors in model checkers. However, there is a lack of research on probabilistic and real-time systems, domain-specific models, and user studies for evaluation.

INFORMATION AND SOFTWARE TECHNOLOGY (2022)

Article Computer Science, Software Engineering

Feature-based software design pattern detection

Najam Nazar, Aldeida Aleti, Yaokun Zheng

Summary: Software design patterns are standard solutions to common problems in software design and architecture, and automatic detection of design patterns can improve efficiency. The new design pattern detection method DPDF, using machine learning classifiers and code features, achieves precision over 80% and recall over 79%, outperforming existing methods.

JOURNAL OF SYSTEMS AND SOFTWARE (2022)

Article Computer Science, Information Systems

A property specification pattern catalog for real-time system verification with UPPAAL

Thomas Vogel, Marc Carwehl, Genaina Nunes Rodrigues, Lars Grunske

Summary: This work presents a comprehensive specification pattern catalog for UPPAAL, which supports qualitative and real-time requirements and covers all corresponding patterns of existing catalogs. The catalog is integrated with UPPAAL, allowing for the specification of requirements using patterns and providing an automated generator for translating these requirements into observer automata and TCTL formulas.

INFORMATION AND SOFTWARE TECHNOLOGY (2023)

Article Computer Science, Software Engineering

RaQuN: a generic and scalable n-way model matching algorithm

Alexander Schultheiss, Paul Maximilian Bittner, Alexander Boll, Lars Grunske, Thomas Thuem, Timo Kehrer

Summary: Model matching algorithms are crucial for identifying common elements in input models. Existing n-way matching algorithms often suffer from scalability issues, but our proposed algorithm utilizes multi-dimensional search trees to efficiently find suitable match candidates, leading to significant improvements in both performance and matching quality.

SOFTWARE AND SYSTEMS MODELING (2023)

Article Computer Science, Software Engineering

Instance Space Analysis of Search-Based Software Testing

Neelofar Neelofar, Kate Smith-Miles, Mario Andres Munoz, Aldeida Aleti

Summary: Search-based software testing (SBST) is a mature area with techniques developed to tackle the challenging task of software testing. SBST techniques have been successfully applied in the industry to generate test cases for large and complex software systems. However, their effectiveness depends on the problem being addressed. This paper revisits the evaluation of SBST techniques using Instance Space Analysis (ISA) to visualize and assess their strengths and weaknesses across a broad range of problem instances from common benchmark datasets. The paper also examines the diversity and quality of benchmark datasets used in experimental evaluations.

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING (2023)

Proceedings Paper Computer Science, Interdisciplinary Applications

Automatically finding Metamorphic Relations in Computational Material Science Parsers

Sebastian Mueller, Valentin Gogoll, Anh Duc Vu, Timo Kehrer, Lars Grunske

Summary: Software testing is crucial but often faces difficulties in determining the expected outcome. This study explores the feasibility of using metamorphic testing to automatically identify relations between input and output pairs in the exciting-NOMAD parser, a widely used software package in computational material science. The analysis of the discovered metamorphic relations focuses on their quantity and quality, and the developed tool and used data are made available through a replication package.

2022 IEEE 18TH INTERNATIONAL CONFERENCE ON E-SCIENCE (ESCIENCE 2022) (2022)

Proceedings Paper Computer Science, Software Engineering

A Comparative Evaluation on the Quality of Manual and Automatic Test Case Generation Techniques for Scientific Software - A Case Study of a Python Project for Material Science Workflows

Daniel Truebenbach, Sebastian Mueller, Lars Grunske

Summary: Writing software tests is crucial for ensuring software quality, and automated test case generation is particularly helpful for researchers in the scientific field. In this case study, the efficacy of automatic test case generation approaches for the ASE Python project in material sciences was investigated. The results showed that while automated methods improved the original test suite, none were able to achieve the coverage reached by the manually created test suite.

15TH SEARCH-BASED SOFTWARE TESTING WORKSHOP (SBST 2022) (2022)

Proceedings Paper Computer Science, Software Engineering

BeDivFuzz: Integrating Behavioral Diversity into Generator-based Fuzzing

Hoang Lam Nguyen, Lars Grunske

Summary: This paper discusses the performance evaluation metric of fuzzers and emphasizes the importance of both coverage and behavioral diversity. Introducing BeDivFuzz, a feedback-driven fuzzing technique that achieves better behavioral diversity by employing mutation strategies based on validity and behavioral diversity.

2022 ACM/IEEE 44TH INTERNATIONAL CONFERENCE ON SOFTWARE ENGINEERING (ICSE 2022) (2022)

Proceedings Paper Computer Science, Artificial Intelligence

Sampling What Matters: Relevance-guided Sampling of Event Logs

Martin Kabierski, Hoang Lam Nguyen, Lars Grunske, Matthias Weidlich

Summary: This paper introduces a relevance-guided sampling approach for event logs, which learns the characteristics of event data to determine its relevance for conformance checking, resulting in improved quality of samples and conformance checking results compared to baseline strategies.

2021 3RD INTERNATIONAL CONFERENCE ON PROCESS MINING (ICPM 2021) (2021)

No Data Available