WC prioritization (paper title)

Objective

The present study evaluates the total and additional WC techniques aimed at prioritizing test cases for multiple aggregate changes, in order to assess their prioritization results and efficiency as compared with other change-based approach and a baseline (random prioritization), from the point of view of software test engineers.

Research Questions and Metrics

RQ1. Are the test suites prioritized with the WC techniques more efficient in identifying faults in mutants of the System Under Test (SUT) with multiple modifications?

For this measurement, we apply well-known metrics Average Percentage Faults Detected (APFD) [Elbaum et al.] and F-measure [Zhou].

RQ2. What is the answer of RQ1, taking into account mutants with a single modification?

Also using APFD and F-Measure.

RQ3. Do the test suites prioritized with the WC techniques maintain spreading of failed test cases as presented by the Changed Blocks technique?

For this answer, we apply metric F-Spreading [Alves et al.] and proposed two metrics for assessing previously-undetected defect anticipation and grouping, Group-measure and Group-spreading.

RQ4. What is the answer of RQ3, considering the spreading of test cases for a single modification?

Also using F-Spreading, Group-spreading and Group-Measure.

RQ5. What is the difference in execution time for the prioritization algorithms?

Execution time as measured in exact clone environments.

Setup

  • Changed Blocks implementation
  • Total WC implementation
  • Additional WC implementation

Systems and mutants

  • JBehave [code,test suite]
  • JMock [code, test suite]
  • XML-Unit [code, test suite]
  • XML-Matchers [code, test suite]
  • Mutants with a single change
  • Mutants with multiple changes in the same class
  • Mutants with multiple changes in the same package
  • Mutants with multiple changes in several packages

Data

  • Charts
  • Results for RQ1 [xls,csv]
  • Results for RQ2 [xls,csv]
  • Results for RQ3 [xls,csv]
  • Results for RQ4 [xls,csv]
  • Results for RQ5 [xls,csv]
  • R script for statistical analysis

Main conclusions

  • Performed better ...
  • Perform equally ...