Mutation Testing – ISSTA'16 https://issta2016.cispa.saarland Fri, 29 Jul 2016 12:43:05 +0000 en-US hourly 1 https://wordpress.org/?v=4.5.3 Mutation-Aware Fault Prediction https://issta2016.cispa.saarland/mutation-aware-fault-prediction/ Thu, 12 May 2016 08:15:03 +0000 https://issta2016.cispa.saarland/?p=446 Read More]]> David Bowes, Tracy Hall, Mark Harman, Yue Jia, Federica Sarro, and Fan Wu

We introduce mutation-aware fault prediction, which leverages additional guidance from metrics constructed in terms of mutants and the test cases that cover and detect them. Mutation-aware fault prediction can significantly (p ≤ 0.05) improve fault prediction performance. We report the results of 12 sets of experiments, applying 4 different predictive modelling techniques to 3 large real-world systems (both open and closed source). Mutation metrics lie in the top 5% most frequently relied upon defect predictors in 10 of the 12 sets of experiments, and provide the majority of the top ten defect predictors in 9 of the 12 sets of experiments.

]]>
Predictive Mutation Testing https://issta2016.cispa.saarland/predictive-mutation-testing/ Thu, 12 May 2016 08:15:08 +0000 https://issta2016.cispa.saarland/?p=448 Read More]]> Jie Zhang, Ziyi Wang, Lingming Zhang, Dan Hao, Lei Zang, Shiyang Cheng, and Lu Zhang

Mutation testing is a powerful methodology for evaluating test suite quality. In mutation testing, a large number of mutants are generated and executed against the test suite to check the ratio of killed mutants. Therefore, mutation testing is widely believed to be a computationally expensive technique. To alleviate the efficiency concern of mutation testing, in this paper, we propose predictive mutation testing (PMT), the first approach to predicting mutation testing results without mutant execution. In particular, the proposed approach constructs a classification model based on a series of features related to mutants and tests, and uses the classification model to predict whether a mutant is killed or survived without executing it. PMT has been evaluated on 163 real-world projects under two application scenarios (i.e., cross-version and cross-project). The experimental results demonstrate that PMT improves the efficiency of mutation testing by up to 151.4X while only incurring a small accuracy loss on mutant execution result prediction, indicating a good tradeoff between efficiency and effectiveness of mutation testing.

]]>
Threats to the Validity of Mutation-Based Test Assessment https://issta2016.cispa.saarland/threats-to-construct-validity-of-mutation-based-test-assessment/ Thu, 12 May 2016 08:15:12 +0000 https://issta2016.cispa.saarland/?p=450 Read More]]> Mike Papadakis, Christopher Henard, Mark Harman, Yue Jia, and Yves Le Traon

Much research on software testing and test techniques relies on experimental studies based on mutation testing. In this paper we reveal that such studies are vulnerable to a potential threat to construct validity, leading to possible Type I errors; incorrectly rejecting the Null Hypothesis. Our findings indicate that Type I errors occur, for arbitrary experiments that fail to take countermeasures, approximately 62% of the time. Clearly, a Type I error would potentially compromise any scientific conclusion. We show that the problem derives from such studies’ combined use of both subsuming and subsumed mutants. We collected articles published in the last two years at three leading software engineering conferences. Of those that use mutation-based test assessment, we found that 68% are vulnerable to this threat to validity.

]]>