A Metric for Measuring Test Input Generation Effectiveness of Test Generation Methods for Boolean Expressions

Abstract

The literature includes several methods to generate test inputs for Boolean expressions. The effectiveness of those methods needs to be analyzed by extensive comparisons. To this end, mutation analysis is often benefited by applying a distinctively selected set of mutants on each test generation method. Mutation analysis provides substantive information about the effectiveness of a test suite by indicating the percentage of killed mutants, which is a common metric. However, as we claim and show in this paper, this metric alone is not sufficient to demonstrate the effectiveness of the methods. For a test generation method, the amount of generated test inputs is also an important attribute to evaluate effectiveness. To the best of our knowledge, there is no metric that measures the effectiveness within a scale taking into account several attributes. In this study, we propose a new metric to measure the effectiveness of test input generation methods, which takes into account both the number of killed mutants and the number of test inputs. We demonstrate our new metric on three well-known test input generation methods for Boolean expressions.