To compare the automatic categorization with our manual categorization we used the metric matches:
where x is an experimental unit; Total of Agreements (x) is the total of coincidences between automatic and manual categorization of nonconformances; and Total of Categorized Nonconformances(x) is the total of categorized nonconformances for both approaches – manual (baseline) and automatic.
Table 1 presents the results for matches for all experimental units that we used.
Table 1. matches results for each experimental unit and the mean of this metric.
The mean of matches metric – used to compare the results from manual and automatic categorization – was 0.73. Despite some limitations, our approach is a step ahead of contract violations correction aided by automation.
Tiago Massoni > JmlOk2 > Enhancing Conformance Checking for Contract-Based Programs > Experiment Results >