zero false positives and a recall of 98.8% with three false negatives. There
is no clear second-best performing tool, as GumTree 3.0 greedy has the second
highest precision with 98.5% and GumTree 2.1.0 has the second highest recall
with 59.2%. Notably, IJM does not perform as good as it does for method
declarations, since it has the lowest recall among all tools (44.5%), but it
has the third-best precision (98.2%). IJM also applies partial matching for
field declarations, but it processes all field declaration subtrees together
in a single iteration, without processing separately those that have identical
signatures (i.e., same type and name). After some investigation, we found that
IJM’s lower recall is due to some non-identical field declarations being moved
to newly introduced inner classes within the same file. The first phase of IJM
that executes GumTree on the entire files to find the largest identical
subtrees, fails to match the moved field declarations, since they are non-
identical (e.g., they have differences in modifiers). In the phase where IJM
performs partial matching for field declarations, it includes only the fields
declared within the same class, and thus it is unable to match attributes
moved to other classes, even if these classes are within the same file (i.e.,
inner classes). The other GumTree-based tools search for similar sub-trees in
the entire file, and thus are able to match these moved field declarations
regardless of their class container. Although, RefactoringMiner follows a
similar approach to IJM by first matching field declarations within the same
classes (Figure 11, Phase 2), it has an additional phase at the end (Figure
11, Phase 4) that detects moved field declarations between different classes.
Moreover, by utilizing field references, RefactoringMiner can match fields
with co-occurring changes in their name, type, modifiers and initializer
(i.e., when most or every part of the field declaration subtree has changed),
while the other tools fail as the subtree similarity of such field
declarations is very small, which explains their lower recall values.

RQ3 findings: Our analysis shows that all tools have a relatively high
accuracy in matching type/enum declarations, but RefactoringMiner is the only
tool with 100% precision and recall. However, when it comes to matching method
and field declarations RefactoringMiner still has an almost perfect accuracy,
while the recall of the other tools drops significantly. In particular, all
AST diff tools, except for RefactoringMiner, have considerably low recall in
matching field declarations (44-59%). Therefore, we can conclude that
utilizing programming language clues, such as the signatures and references of
program elements, can improve significantly the mapping accuracy for method
and field declarations. Based on their overall program element mapping
accuracy, the tools are ranked as follows: (1) RefactoringMiner 3.0
(F-score=99.9%) (2) IJM (F-score=92.4%) (3) GumTree 2.1.0 (F-score=89.4%) (4)
GumTree 3.0 simple (F-score=89.3%) (5) GumTree 3.0 greedy (F-score=88.8%) (6)
MTDiff (F-score=87.6%)

###  5.4. RQ4: Refactoring-related mappings

####  5.4.1. Experimental procedure

For this research question, we investigate how different refactoring types
affect the mapping accuracy of AST diff tools. To isolate the refactoring
effect, we carefully select the AST sub-trees that could be potentially
affected by each particular refactoring transformation, and compute the
accuracy of the AST diff tools on the subset of mappings included within the
selected AST sub-trees. The AST sub-trees considered for each refactoring type
are listed below:

  * •

Extract Method: The source method declaration from which the method is
extracted on the left and right side of the diff, and the extracted method
itself on the right side of the diff.

  * •

Inline Method: The target method declaration to which the method is inlined on
the left and right side of the diff, and the inlined method itself on the left
side of the diff.

  * •

Extract Variable: The pairs of statements that reference the extracted
variable, and the extracted variable declaration itself on the right side of
the diff.

  * •

Inline Variable: The pairs of statements that reference the inlined variable,
and the inlined variable declaration itself on the left side of the diff.

  * •

Rename Variable/Parameter + Change Variable/Parameter Type + Parameterize
Variable: The pairs of statements that reference the variable/parameter, and
the variable/parameter declaration itself on the left and right side of the
diff. Parameterize Variable is a refactoring that commonly occurs along with
Extract Method and captures local variables that become parameters in the
extracted method. In many cases, the name of the parameterized variable
changes in the extracted method. There are also a few cases where Parameterize
Variable occurs within the same method by eliminating a local variable and
introducing a parameter in its place.

  * •

Replace Variable with Field: The pairs of statements that reference the
variable/field, the variable declaration itself on the left side of the diff,
and the field declaration on the right side of the diff.

  * •

Add Parameter: The method declaration to which the parameter is added on the
left and right side of the diff, and the parameter declaration itself on the
right side of the diff. The reason we include the complete method declaration
is because the addition of a parameter is usually accompanied with newly added
functionality affecting existing statements or adding new statements, which
should be properly matched by the diff tool.

  * •

Rename Method: The renamed method declaration on the left and right side of
the diff. The reason we include the complete method declaration is because
renaming might involve some functionality change affecting existing statements
or adding/deleting statements, which should be properly matched by the diff
tool.

We included in our analysis only the refactoring types with ten or more
instances, as having a small number of instances poses a threat to the
generalizability of our findings. Moreover, we excluded all refactoring types
that move code between different files, such as Move Method, Pull Up Method,
Move Field, because we investigate the accuracy of the tools on inter-file
mappings separately in the next Section 5.5. Therefore, all examined
refactoring types have a local effect within the same file. Finally, in cases
where different refactoring instances of the same type affect exactly the same
AST subtrees, we keep only one instance in our results to avoid recounting the
same effect. For example, when duplicated code is extracted from the same
method, all Extract Method instances affect the same source method and the
same extracted method, and thus we keep only one instance in our results.

####  5.4.2. Results and discussion

Table 8. Precision and recall for refactoring-related mappings in both datasets considering only statement mappings Refactoring |  | RM 3.0 | GT 3.0 greedy | GT 3.0 simple | GT 2.1.0 | IJM | MTDiff
---|---|---|---|---|---|---|---
Type | # | Precision | Recall | Precision | Recall | Precision | Recall | Precision | Recall | Precision | Recall | Precision | Recall
Extract Method | 178 | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{89.9}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{73.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{90.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{73.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{89.2}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{71.8}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{92.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{67.3}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{87.2}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{70.4}$ |
Rename Variable | 77 | $100$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{98.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{53.8}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{41.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{53.1}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{40.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{51.1}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{40.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{59.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{31.1}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{51.1}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{41.0}$ |
Extract Variable | 75 | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{98.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{98.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{79.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{68.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{81.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{71.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{79.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{68.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{75.2}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{62.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{80.8}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{76.6}$ |
Change Var Type | 47 | $100$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{97.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{95.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{85.9}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{95.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{84.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{91.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{82.1}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{85.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{46.2}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{82.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{79.5}$ |
Add Parameter | 45 | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.9}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{68.9}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{73.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{69.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{71.3}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{54.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{57.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{73.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{73.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{94.2}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{94.6}$ |
Param Variable | 30 | $100$ |  | $100$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{84.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{81.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{87.9}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{86.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{84.8}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{72.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{66.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{21.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{70.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{62.4}$ |
Inline Method | 28 | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{98.8}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{90.1}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{89.8}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{92.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{90.8}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{85.9}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{84.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{92.9}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{86.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{90.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{87.3}$ |
Inline Variable | 22 | $100$ |  | $100$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{50.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{29.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{66.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{47.1}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{54.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{35.3}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{53.8}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{41.2}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{61.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{47.1}$ |
Rename Parameter | 19 | $100$ |  | $100$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{87.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{87.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{89.1}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{85.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{87.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{83.3}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{86.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{77.1}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{89.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{87.5}$ |
Change Param Type | 18 | $100$ |  | $100$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{89.1}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{82.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{89.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{87.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{87.1}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{78.3}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{93.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{84.1}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{95.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{89.9}$ |
Rename Method | 15 | $100$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.1}$ |  | $100$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{95.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{98.2}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{97.3}$ |  | $100$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{98.2}$ |  | $100$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{95.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{94.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{95.5}$ |
Replace Var w Field | 10 | $100$ |  | $100$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{89.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{85.0}$ |  | $100$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{75.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{81.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{85.0}$ |  | $100$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{70.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{82.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{70.0}$ |
Overall | 564 | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{84.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{73.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{85.2}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{73.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{80.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{69.3}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{87.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{67.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{87.2}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{74.7}$ |

Based on the results shown in Table 8, RefactoringMiner has consistently the
highest precision and recall for all examined refactoring types. The second-
best performing tool differs for different sets of refactoring types. For
instance, MTDiff has the second-best accuracy (based on F-score) for
parameter-related refactorings (Add Parameter, Rename Parameter, Change
Parameter Type), as well as Extract Variable. GumTree 3.0 simple has the
second-best accuracy for Extract Method, Inline Method, Parameterize Variable,
and Inline Variable. GumTree 3.0 greedy has the second-best accuracy for
variable-related refactorings (Rename Variable, Change Variable Type, Replace
Variable with Field), but it is only slightly better than the third-best
accuracy by GumTree 3.0 simple. Notably, there is no refactoring type for
which IJM has the second-best accuracy, and for the reasons explained in
Sections 5.1 and 5.3 it has the lowest recall value for Extract Method,
Extract Variable, Rename Variable, Change Variable Type, Parameterize
Variable, and Rename Parameter refactorings. GumTree 2.1.0 has the second-best
accuracy for only one refactoring type, namely Rename Method, but all tools
have a quite similar performance on this refactoring type, and thus there is
no clear advantage for GumTree 2.1.0.

Since some of the refactoring types listed in Table 8 involve sub-expression
mappings (see Section 3.3.1), we decided to further investigate the
performance of the tools at sub-expression level. However, due to the
compatibility issues among the tools discussed in Section 4.2, the tools that
can provide comparable sub-expression mappings are those using the AST
generator gumtree.gen.jdt-3.0.0, namely RefactoringMiner and the greedy/simple
versions of GumTree 3.0. Table 9 shows the precision and recall of the three
tools at a more fine-grained level that additionally includes sub-expression
mappings.

Table 9. Precision and recall for refactoring-related mappings in both datasets considering statement and sub-expression mappings Refactoring |  | RM 3.0 | GT 3.0 greedy | GT 3.0 simple
---|---|---|---|---
Type | # | Precision | Recall | Precision | Recall | Precision | Recall
Extract Method | 178 | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.3}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.3}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{74.9}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{63.8}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{79.2}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{63.4}$ |
Rename Variable | 77 | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.1}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{98.3}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{38.9}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{21.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{49.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{20.7}$ |
Extract Variable | 75 | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{96.8}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{97.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{43.9}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{51.9}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{55.2}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{50.5}$ |
Change Variable Type | 47 | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{98.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{96.9}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{55.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{50.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{78.9}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{46.4}$ |
Parameterize Variable | 30 | $100$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{58.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{56.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{74.2}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{60.8}$ |
Inline Method | 28 | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.1}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{98.8}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{72.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{82.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{82.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{83.8}$ |
Inline Variable | 22 | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{98.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{98.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{37.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{40.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{53.8}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{42.0}$ |

By comparing Tables 9 and 8, we can observe that RefactoringMiner maintains
high accuracy levels with some small decrease in its precision and recall.
However, GumTree 3.0 greedy has a considerable drop in its precision, and has
consistently a larger drop in precision than GumTree 3.0 simple. This result
shows that the greedy version of GumTree 3.0 often forces incorrect mappings
at the sub-expression level that harm considerably its precision. On the other
hand, the same greedy nature helps to slightly increase the recall over its
non-greedy counterpart for some refactoring types, but this small improvement
in recall comes with a much larger cost in its precision. Overall, we can
conclude that GumTree simple has a clearly better performance than GumTree
greedy when we take into consideration sub-expression mappings.

RQ4 findings: RefactoringMiner has consistently the highest precision and
recall for all examined refactoring types. Regarding the second-best
performance, each tool has an advantage over a different set of refactoring
types. MTDiff has an advantage in parameter-related refactorings and extract
variable. GumTree 3.0 simple has an advantage in extract/inline method.
GumTree 3.0 greedy has an advantage in local-variable-related refactorings.
Based on their overall refactoring mapping accuracy, the tools are ranked as
follows: (1) RefactoringMiner 3.0 (F-score=99.5%) (2) MTDiff (F-score=80.5%)
(3) GumTree 3.0 simple (F-score=78.9%) (4) GumTree 3.0 greedy (F-score=78.7%)
(5) IJM (F-score=76.3%) (6) GumTree 2.1.0 (F-score=74.6%)

###  5.5. RQ5: Inter-file mappings

####  5.5.1. Experimental procedure

For this research question, we focus our analysis on inter-file mappings,
i.e., AST nodes that have a corresponding node in a file other than the one
they are located in. More specifically, we filter all AST node mappings marked
as inter-file mappings in our benchmark $S_{B}$, and compute the accuracy of
the AST diff tools on this particular subset of mappings. Typically, inter-
file mappings are caused by refactorings moving methods and fields between
existing files (i.e., Move Method, Move Field), or to new files (i.e., Extract
Class, Extract Superclass).

####  5.5.2. Results and discussion

Table 10. Precision and recall for inter-file mappings in both datasets RM 3.0 | GT 3.0 greedy | GT 3.0 simple | GT 2.1.0 | IJM | MTDiff
---|---|---|---|---|---
Precision | Recall | Precision | Recall | Precision | Recall | Precision | Recall | Precision | Recall | Precision | Recall
$100$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{0.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{0.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{0.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{0.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{0.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{0.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{0.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{0.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{0.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{0.0}$ |
0 | FPs | 6 | FNs | 14 | FPs | 1432 | FNs | 11 | FPs | 1432 | FNs | 8 | FPs | 1432 | FNs | 4 | FPs | 1432 | FNs | 21 | FPs | 1432 | FNs

As shown in Table 10, all tools except for RefactoringMiner have a precision
and recall of 0% with zero true positives, as they do not support inter-file
mappings. Moreover, these tools have a relatively small number of false
positives, which correspond to moved statements forcefully matched with newly
added statements within the same file, as shown in the example of Figure 9.
Therefore, the lack of support for inter-file mappings mainly affects the
recall of the tools, but it can potentially affect their precision if there is
newly added code in the file. RefactoringMiner has a perfect precision, as all
moves it detected were correct, but it missed one method move, thus leading to
6 false negative statement mappings.

RQ5 findings: All tools except for RefactoringMiner have a precision and
recall of 0% with zero true positives, as they do not support inter-file
mappings. Moreover, the tools that are more greedy in nature (GumTree 3.0
greedy and MTDiff) have a larger number of false positive mappings, as they
tend to forcefully match the moved statements with newly added statements
within the same file. Based on their inter-file mapping accuracy, the tools
are ranked as follows (the tools having an F-score of 0% are ranked based on
their number of false positives in ascending order): (1) RefactoringMiner 3.0
(F-score=99.8%) (2) IJM (4 false positives) (3) GumTree 2.1.0 (8 false
positives) (4) GumTree 3.0 simple (11 false positives) (5) GumTree 3.0 greedy
(14 false positives) (6) MTDiff (21 false positives)

###  5.6. RQ6: Overall accuracy

####  5.6.1. Experimental procedure

For this research question, we compute the overall accuracy of the tools by
including all AST node mappings in our benchmark $S_{B}$. Moreover, given the
compatibility issues among the tools discussed in Section 4.2, we compute the
accuracy at two levels of granularity. At the coarse-grained level, we include
mappings between statements and program elements, as all tools can provide
this information. At the fine-grained level, we include sub-expression
mappings in addition to the mappings considered at the coarse-grained level.
The tools that can provide comparable sub-expression mappings are those using
the AST generator gumtree.gen.jdt-3.0.0, namely RefactoringMiner and the
greedy/simple versions of GumTree 3.0.

####  5.6.2. Results and discussion

Table 11. Overall mapping accuracy considering only statement mappings | RM 3.0 | GT 3.0 greedy | GT 3.0 simple | GT 2.1.0 | IJM | MTDiff
---|---|---|---|---|---|---
Dataset | Precision | Recall | Precision | Recall | Precision | Recall | Precision | Recall | Precision | Recall | Precision | Recall
Defects4J | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.8}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.2}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{98.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.1}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{98.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{98.9}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{98.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.0}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{98.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{98.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{98.2}$ |
Refactoring | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.8}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{91.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{74.1}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{91.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{73.9}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{90.2}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{70.9}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{92.8}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{71.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{93.3}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{75.7}$ |
Overall | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.8}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{97.3}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{91.2}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{97.2}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{91.3}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{96.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{90.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{97.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{90.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{97.1}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{91.6}$ |

As we can observe from Table 11, at the coarse-grained level all tools exhibit
a very high accuracy in the Defects4J dataset with RefactoringMiner having the
highest precision and recall. However, in the Refactoring dataset the
differences in accuracy are considerably larger with RefactoringMiner having a
+6.5% precision and +24% recall than the second-best performing tool, which is
MTDiff. This result proves our argument that bug fixing commits are a low
barrier for assessing the accuracy of AST diff tools, and thus more
challenging datasets should be used, such as the Refactoring benchmark we
introduce in this paper. Our analysis in the previous research questions has
shown that the lack of support for multi-mappings and inter-file mappings
contributes a considerable number of false negatives for all tools other than
RefactoringMiner. Moreover, the lack of language-awareness contributes false
negatives in method and field declaration mappings, and the lack of
refactoring-awareness contributes false negatives in statement mappings,
especially for variable-related refactorings. Although MTDiff seems to have
the second-best performance based on F-score, it is hard to make a safe
conclusion about which tool is actually the second-best performing one, as all
tools (with the exception of RefactoringMiner) have a quite similar overall
F-score at statement level ranging between 93.5-94.3%. However, there is some
indication that MTDiff has a slightly better performance in the Refactoring
dataset with +1.6% precision and recall than the third-best performance.

Table 12. Overall mapping accuracy considering statement and sub-expression mappings | RM 3.0 | GT 3.0 greedy | GT 3.0 simple
---|---|---|---
Dataset | Precision | Recall | Precision | Recall | Precision | Recall
Defects4J | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.3}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{97.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{93.1}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{98.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{97.8}$ |
Refactoring | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.2}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{84.1}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{70.2}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{86.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{72.4}$ |
Overall | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.3}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{93.8}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{86.1}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{95.2}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{90.0}$ |

At the fine-grained level that additionally includes sub-expression mappings,
we can observe by comparing Table 12 with Table 11 that the precision and
recall of all tools drops. However, GumTree 3.0 greedy has the largest
accuracy drop among the three tools, which means that it tends to force more
incorrect sub-expression mappings, but also miss more sub-expression mappings
due to its greedy nature. RefactoringMiner has still a remarkable overall
precision and recall of 99.7% and 99.3%, respectively, showing that it
performs very well at the sub-expression level too. More specifically, in the
Refactoring dataset it has a +13% precision and +27% recall than the second-
best performing tool, which is GumTree 3.0 simple. GumTree 3.0 simple has
consistently better precision and recall than GumTree 3.0 greedy in both
datasets, when considering sub-expression mappings, while the accuracy
differences between these tools are very small, when considering only
statement mappings.

RQ6 findings: All tools except for RefactoringMiner, have a quite similar
F-score at statement level ranging between 93.5-94.3%. RefactoringMiner brings
significant accuracy improvements, especially in the Refactoring dataset and
when considering sub-expression mappings, with +13% in precision and +27% in
recall than the second-best performing tool. GumTree 3.0 simple has
consistently better precision and recall than GumTree 3.0 greedy in both
datasets, when considering sub-expression mappings. Based on their overall
mapping accuracy, the tools are ranked as follows: (1) RefactoringMiner 3.0
(F-score=99.7% at statement level and 99.5% at sub-expression level) (2)
MTDiff (F-score=94.3% at statement level) (3) GumTree 3.0 simple
(F-score=94.2% at statement level and 92.6% at sub-expression level) (4)
GumTree 3.0 greedy (F-score=94.2% at statement level and 89.8% at sub-
expression level) (5) IJM (F-score=93.9% at statement level) (6) GumTree 2.1.0
(F-score=93.5% at statement level)

###  5.7. RQ7: Perfect diff rate

####  5.7.1. Experimental procedure

For this research question, we investigate the ability of the tools to
generate perfect diffs. A diff generated by a tool $T$ is considered perfect,
if the set of AST node mappings in $S_{T}$ is identical to the benchmark
mappings $S_{B}$ (i.e., $S_{T}=S_{B}$). Therefore, perfect diff rate is the
percentage of commits for which a tool generated the perfect diff. Since even
small diff imperfections can cause confusion to the developer trying to
understand the changes intended by the commit author, perfect diff rate
essentially measures the ability of the tools to generate noise-free or
confusion-free diffs, as the perfect diff reflects the commit author
intentions in the most accurate way.

####  5.7.2. Results and discussion

Table 13. Perfect diff rate considering only statement mappings Dataset | RM 3.0 | GT 3.0 greedy | GT 3.0 simple | GT 2.1.0 | IJM | MTDiff
---|---|---|---|---|---|---
Defects4J | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{89.3}$ |  | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{75.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{72.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{75.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{75.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{68.3}$ |
Refactoring | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{81.9}$ |  | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{14.9}$ |  | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{13.8}$ |  | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{11.2}$ |  | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{13.8}$ |  | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{5.9}$ |
Overall | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{87.9}$ |  | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{63.9}$ |  | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{61.2}$ |  | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{63.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{63.9}$ |  | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{56.4}$ |
Table 14. Perfect diff rate considering statement and sub-expression mappings Dataset | RM 3.0 | GT 3.0 greedy | GT 3.0 simple
---|---|---|---
Defects4J | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{85.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{18.1}$ |  | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{63.5}$ |
Refactoring | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{69.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{4.8}$ |  | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{8.5}$ |
Overall | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{82.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{15.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{53.0}$ |

Table 13 shows the perfect diff rate of the tools considering only statement
mappings (i.e., coarse-grained level). As we can observe from Table 13, all
tools are able to generate the perfect diff for a relatively large percentage
of cases from the Defects4J dataset, with RefactoringMiner having the highest
rate with 89.3% and the remaining tools ranging between 68-75%. Regarding the
Refactoring dataset, with the exception of RefactoringMiner that achieves a
rate of 81.9%, the rates of the other tools decrease dramatically and range
between 6-15%. This result is another proof that bug fixing commits are a low
barrier for assessing the accuracy of AST diff tools, and thus more
challenging datasets should be used, such as the Refactoring benchmark we
introduce in this paper. Table 14 shows the perfect diff rate of the tools
considering statement and sub-expression mappings (i.e., fine-grained level).
By comparing Table 14 with Table 13, we can observe that all tools have a
lower perfect diff rate. This is reasonable as the tools need to correctly
match all fine-grained sub-expression mappings to achieve the perfect diff.
However, GumTree 3.0 greedy is the tool that has by far the largest decrease
in its perfect diff rate. This result is consistent with our findings in RQ4
and RQ6, showing that the greedy version of GumTree 3.0 very often forces
incorrect mappings at the sub-expression level. On the other hand,
RefactoringMiner achieves a perfect diff rate of 70% in the Refactoring
dataset, and 85.5% in the Defects4J dataset, which is a remarkable
achievement, given the challenge of matching correctly all fine-grained sub-
expression mappings.

RQ7 findings: RefactoringMiner has the highest perfect diff rate in both
levels of granularity (i.e., with and without including sub-expression
mappings). All tools, except for RefactoringMiner, exhibit a very low perfect
diff rate for the Refactoring dataset ranging between 6-15%. Based on their
overall perfect diff rate, the tools are ranked as follows: (1)
RefactoringMiner 3.0 (87.9%) (2) IJM (63.9%) (3) GumTree 3.0 greedy (63.9%)
(4) GumTree 2.1.0 (63.4%) (5) GumTree 3.0 simple (61.2%) (6) MTDiff (56.4%)

###  5.8. RQ8: Execution time

####  5.8.1. Experimental procedure

For this research question, we measure the execution time of each tool on each
dataset in our benchmark. We executed each tool separately on each commit of
the Defects4J and Refactoring datasets, and recorded the time taken to
generate the AST diff using the System.nanoTime Java method to get the start
time and end time of the diff generation process for each commit. All tools
were executed on the same machine with the following specifications: Apple M2
@ 3.49GHz, 16GB LPDDR5-6400 RAM, 512GB SSD, MacOS Sonoma 14.2, Java 17.0.6
x64.

####  5.8.2. Results and discussion

Figures 22 and 23 show the execution time distribution of each tool on the
Defects4J and Refactoring datasets, respectively. Clearly, GumTree 3.0 simple
is the fastest tool, followed by IJM and GumTree 2.1, which have a very
similar performance, as IJM is an extension of GumTree 2.1. RefactoringMiner
comes in fourth place with an execution time that is 2.5 times slower on
median and 3.3 times slower on average than GumTree (simple). Still, it has
execution times within the same order of magnitude as the three faster tools,
and it can process each bug fixing commit in the Defects4J dataset in less
than 800 ms. MTDiff comes in the fifth place, and GumTree greedy (i.e., the
default algorithm in GumTree 3.0 configuration) has the slowest execution time
among all tools, and is an order of magnitude slower than the GumTree simple
(13.5 times slower on median and 21.3 times slower on average).

Figure 22. Execution time per Defects4J commit in milliseconds. Figure 23.
Execution time per Refactoring commit in milliseconds.

As explained before, the Refactoring benchmark includes inter-file mappings,
and RefactoringMiner is the only tool supporting such mappings with the cost
of additional processing. Therefore, its execution time differences with the
other tools are expected to be larger than those observed in the Defects4J
dataset. Still, we can observe from Figure 23 that the tools are ranked in the
same order in terms of their execution time as in the Defects4J dataset, but
RefactoringMiner is now 4.8 times slower on median and 6.8 times slower on
average than GumTree 3.0 simple. We inspected the 5 commits in which
RefactoringMiner has an execution time above 1 second, and they either
involved inter-file mappings, or very large files (i.e., 15K LOC).

Table 15. Results of the Wilcoxon signed-rank test on the execution time distributions of the tools in both datasets X Y | IJM | GumTree 2.1.0 | RefactoringMiner | MTDiff | GumTree 3.0 greedy
---|---|---|---|---|---
GumTree 3.0 simple | $<$ | $<$ | $<$ | $<$ | $<$
IJM |  | $<$ | $<$ | $<$ | $<$
GumTree 2.1.0 |  |  | $<$ | $<$ | $<$
RefactoringMiner |  |  |  | $<$ | $<$
MTDiff |  |  |  |  | $<$

To create a ranking of the tools based on their execution time, for each
pairwise combination of tools, we applied the Wilcoxon signed-rank test on
their execution time distributions with the following null hypothesis: “the
execution time of tool X is smaller than the execution time of tool Y”. We
selected this test, because the execution time distributions are matched or
dependent samples, meaning that we have execution times for the same commits
in all distributions. Table 15 shows all 15 pair combinations of tools that
the statistical test was applied for. The rows of the table represent tool X,
while the columns represent tool Y in the statistical test. The symbol $<$
indicates that the null hypothesis was accepted with significance at 95
percent confidence level, and thus we can conclude that tool X is faster than
Y.

RQ8 findings: RefactoringMiner is slower than GumTree 3.0 simple, GumTree 2.1
and IJM, but still its execution time is within the same order of magnitude.
GumTree 3.0 simple is an order of magnitude faster than GumTree 3.0 greedy,
and given it has also higher precision and recall, it should be preferred over
GumTree greedy. Based on their execution times, the tools are ranked as
follows: (1) GumTree 3.0 simple (2) IJM (3) GumTree 2.1.0 (4) RefactoringMiner
3.0 (5) MTDiff (6) GumTree 3.0 greedy

###  5.9. RQ9: Accuracy comparison between RefactoringMiner 3.0 and 2.0

####  5.9.1. Experimental procedure

As discussed in Section 3.2, we made some changes in the statement mapping
process of RefactoringMiner 2.0 (Tsantalis et al., 2022), which is the core of
the tool. With this research question, we want to investigate what is the
effect of these changes on the accuracy of RefactoringMiner 3.0 over the prior
version 2.0. Since RefactoringMiner 2.0 does not have AST diff generation
capabilities, we focused our comparison on features that both versions 2.0 and
3.0 support, namely the matching of program element declarations (i.e., type,
method, field declarations) and statements. Therefore, we utilized our
benchmark at the appropriate level of granularity to compute the accuracy of
versions 2.0 and 3.0 on program element declaration and statement mappings,
respectively.

####  5.9.2. Results and discussion

Table 16. Precision and recall for statement mappings and program element declaration mappings |  | RM 3.0 | RM 2.0
---|---|---|---
| Dataset | Precision | Recall | Precision | Recall
Statement | Defects4J | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.8}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.2}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{86.3}$ |
mappings | Refactoring | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.8}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.5}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{96.4}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{89.8}$ |
Overall | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.8}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.6}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{98.3}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{87.3}$ |
Declaration | Defects4J | $100$ |  | $100$ |  | $100$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{98.1}$ |
mappings | Refactoring | $100$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.1}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{93.8}$ |
Overall | $100$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.9}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{99.7}$ |  | ${{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}{}{}{}{}{}}{{}{}{}{}{}}}{96.5}$ |

As we can observe from Table 16, RefactoringMiner 3.0 has a slightly higher
precision and considerably higher recall in comparison to version 2.0 with
respect to statement mappings. The differences are more evident in the
Refactoring dataset, where RefactoringMiner 3.0 has a +3.4% precision and
+9.7% recall than version 2.0, while in the Defects4J dataset version 3.0 has
a +0.6% precision and +13.3% recall than version 2.0. We performed a thorough
investigation to understand why version 2.0 performs worse than version 3.0 in
statement mappings, and we found four main reasons. First, there are some
program elements that contain statements, but are not processed by version
2.0. Such program elements are initializer blocks and the initializers of
field declarations and enum constants, which are used to initialize fields and
are executed before class constructors. RefactoringMiner supported the
processing of initializer blocks in version 2.3, the diff of field declaration
initializers in version 2.2, and the diff of enum constant initializers in
version 2.3.2. In particular, some commits from project JSoup in the Defects4J
dataset have enum constants intialized with anonymous classes containing a
large number of statements, which explains the lower recall observed in the
Defects4J dataset compared to the Refactoring dataset. Second, there are some
refactoring types that contribute statement mappings, but are not supported by
version 2.0. For example, Replace Anonymous with Class is a refactoring that
contains statement mappings between the methods of the anonymous class and the
methods of the introduced class, and was supported in version 3.0. Third, we
found cases in version 2.0 diffs, where repeated statements within the body of
a method were not accurately matched. The improved sorting criteria discussed
in Section 3.2.1 in cases of ties (i.e., when there are multiple possible
matches for a given statement), reduced both false positives and negatives,
thus contributing to improved precision and recall. The fourth reason is
related to the replacement algorithm used by RefactoringMiner to match non-
identical statements. As explained in Section 3.1 (Phase 2) and in (Tsantalis
et al., 2022), RefactoringMiner applies syntactically valid AST node
replacements within a pair of non-identical input statements, and matches
these statements only if they become textually identical after the application
of replacements. This design choice avoids the use of arbitrary code
similarity thresholds to match non-identical statements. Throughout the
evolution of the tool from version 2.0 to 3.0, the kinds of supported
replacements increased, thus enabling the matching of statement pairs that
could not be matched before, as the replacement algorithm could not make them
textually identical. For example, version 2.3 supports the replacement of
variables with boolean literals and the replacement of variables with class
instance creations, while these kinds of replacements were not supported in
previous versions of the tool. In conclusion, the first, second and fourth
reasons mainly affect the recall of the tool, while the third reason affects
both precision and recall.

Regarding program element declaration mapping accuracy, we can observe from
Table 16 that the precision of versions 2.0 and 3.0 is rather close, as
version 3.0 has a +0.9% precision in the Refactoring dataset, but both
versions have the same precision in the Defects4J dataset. However, the
differences in recall are larger, especially in the Refactoring dataset, where
version 3.0 has a +6% recall than version 2.0. Again, we performed a thorough
investigation to explain the observed differences in recall, and we found two
main reasons. The first reason is that version 2.0 is not processing the
initializers of field declarations and enum constants to find differences, as
already explained in the previous paragraph. However, in some cases field
declarations and enum constants are initialized with an anonymous class, and
thus version 2.0 is missing to match the method declarations within the
anonymous classes in the field/enum-constant initializers. The second reason
is related to an inherent limitation of version 2.0, which is not able to
match method declarations with empty bodies, as it requires to find at least
one statement mapping in order to match a pair of methods. Developers commonly
instantiate interfaces with anonymous classes (e.g., when implementing
listeners for various events) and leave some methods with empty
implementation. Version 2.0 is unable to match such methods with an empty body
within anonymous class declarations. This issue was fixed in version 3.0 of
RefactoringMiner by matching the empty methods within anonymous classes based
on their signature similarity. Both of these reasons affect the recall of the
tool in matching method declarations within anonymous classes.

RQ9 findings: Our analysis has shown that RefactoringMiner 2.0 was not
processing parts of the AST, as its matching process is fully guided based on
language-dependent features and clues. In other words, RefactoringMiner is not
trying to randomly match code fragments between the left and right ASTs based
on their similarity, but it is utilizing the program structure, the program
element declaration types, and language clues (e.g., method calls, method
signatures, field references) to reason about and guide the matching process
on specific parts of the ASTs. In version 3.0, we enhanced the aforementioned
reasoning/guiding algorithm to process parts of the AST that were previously
not handled (e.g., initializer blocks, field declaration initializers), thus
improving the recall of the tool by 10-13% in statement mappings and by 2-6%
in program element declaration mappings. Moreover, the changes in the
statement mapping process of RefactoringMiner discussed in Section 3.2
improved both precision and recall of the tool, as RefactoringMiner 3.0 makes
less mistakes when breaking ties for statements with multiple possible
matches.

###  5.10. Discussion

In this section, we provide insights based on our experiment findings.

How to eliminate semantically incompatible mappings: The fact that AST diff
tools can match semantically incompatible AST nodes is something already known
in the literature. As a matter of fact, IJM (Frick et al., 2018) attempted to
overcome the semantic violations related to name identifiers by modifying the
structure of AST (i.e, by merging SimpleName leaf AST nodes with their parent
nodes). However, as discussed in RQ1, this change in the AST structure made
the depth of the AST shorter, and affected negatively the accuracy of IJM on
matching statements of height equal to 1 (i.e., method invocation statements
with a merged name node), as GumTree employs a default threshold for the
hyperparameter $minHeight$ (i.e., length of the longest path from one leaf to
the root of the subtree), which is equal to 2.

Based on our findings, changing the AST structure is not the correct approach
to eliminate semantically incompatible AST node mappings. A better approach is
to introduce the semantic role of the AST nodes in the program (e.g., method
call name, variable name, type name) as a node property, and match two nodes
of the same AST type, only if they have the same or a compatible semantic role
property. Alternatively, semantic compatibility can be forced for AST nodes
playing multiple semantic roles (e.g., SimpleName) by checking whether their
direct parents have the same AST type.

Greed is bad: GumTree greedy and MTDiff represent algorithms trying to
maximize the number of generated AST node mappings. GumTree greedy applies a
non-conservative approach that matches AST subtrees of height equal to 1 in
the bottom-up phase of the algorithm. MTDiff uses 5 optimizations to generate
additional move edit operations. Based on our findings, GumTree greedy has
consistently lower accuracy than its non-greedy counterpart (i.e., GumTree
simple) in all examined research questions. More specifically, when compared
to the simple version, GumTree greedy has lower accuracy in multi-mappings
(RQ1), more semantically incompatible mappings (RQ2), lower accuracy in
program element mappings (RQ3), lower accuracy in refactoring-related mappings
(RQ4), more false positives in inter-file mappings (RQ5), lower overall
accuracy (RQ6), and much lower perfect diff rate when considering sub-
expression mappings (RQ7). The same findings hold when comparing MTDiff with
its less-greedy counterpart, which is GumTree 2.1.0. More specifically, when
compared to GumTree 2.1.0, MTDiff has lower accuracy in multi-mappings (RQ1),
more semantically incompatible mappings (RQ2), lower accuracy in program
element mappings (RQ3), more false positives in inter-file mappings (RQ5), and
lower perfect diff rate (RQ7). In addition, their greedy nature has
considerable runtime cost, as GumTree greedy and MTDiff are an order of
magnitude slower than all other AST diff tools in both Defects4J and
Refactoring datasets (RQ8).

GumTree 3.0 is a very established tool, widely used by researchers and
practitioners in downstream tasks. In GumTree 3.0 released version, the
default algorithm is the greedy one. We strongly recommend to the GumTree
development team to change the default algorithm to the simple version, as the
majority of researchers and practitioners are using this tool with its default
configuration.

Partial matching is a good idea: Partial matching is essentially a divide-and-
conquer approach introduced by IJM to divide the AST into different sections
of declarations based on their kind (e.g., import declarations, method
declarations, field declarations), and then apply the tree matching process
separately on each section of declarations. Based on the discussion of RQ3
results, this approach improved significantly the program element mapping
accuracy for method declarations compared to GumTree 2.1.0 (i.e., the tool
extended by IJM). However, we also found that partial matching was not so
effective for field declarations, as it fails to match slightly modified field
declarations moved to inner classes within the same file. Thus, IJM requires
an additional post-processing step to match field declarations between
different classes declared within the same file.

Although partial matching requires some basic language-awareness as input to
the algorithm (i.e., the main program declarations in each programming
language), as well as some criteria to match declarations of the same kind
(i.e., signature-based matching), it is a very effective approach to narrow-
down the matching process to different sections of the AST instead of the
whole AST.

Language-awareness is helpful to guide the matching process: GumTree and all
tools based on it, apply a language-independent AST node matching algorithm,
which tries to “blindly” find the largest identical (or similar) AST sub-trees
in a pair of files without relying on language clues to guide the matching
process. The premise is that being language-independent, it is possible to
devise a solution that can support any programming language (i.e., silver
bullet). However, based on our findings, being 100% language-independent may
negatively affect the accuracy of the generated AST node mappings, due to the
“blind” matching process. Programming language clues, such as method calls,
field accesses, and refactoring transformations can guide the matching process
to more accurate mappings. For example, when in the place of a deleted piece
of code, there is a call to a newly added method in a file, it is very likely
that the deleted piece of code has been moved to this newly added method
(i.e., Extract Method refactoring). Therefore, instead of “blindly” trying to
match the deleted piece of code within the entire file, the matching process
can focus on the code within the body of this newly added method. This
approach can avoid false mappings with newly added code in other parts of the
AST that is coincidentally similar to the deleted code, and guide the matching
process towards the correct mappings.

Obviously, computing bindings between references (i.e., method calls and field
accesses) and their corresponding declarations (i.e., method and field
declarations), as well as detecting refactoring operations, are rather
challenging tasks for some programming languages, especially at commit (or
pull request) level, where we have a small portion of the project’s code
available for analysis and it is not possible to build the code and get these
bindings from the compiler. If we are able to overcome these challenges, then
the design and development of hybrid AST diff tools that combine language-
independent and language-specific features seems a very promising direction.

We need more challenging benchmarks: Based on our analysis of RQ6 and RQ7
results, all examined tools achieve a higher accuracy in the Defects4J dataset
compared to the Refactoring dataset. Moreover, the differences in their
accuracy are rather small in the Defects4J dataset compared to the Refactoring
dataset. These results show that bug-fixing commits, which typically include
small (a few lines of code), local (typically located within a single method),
and rather simple changes, are not suitable for assessing the accuracy of AST
diff tools, as they offer AST matching problem instances of rather low
complexity and challenge. However, the majority of AST diff tools have been
evaluated in bug fixing datasets, which makes the evaluation findings
unreliable and hard to interpret. Moreover, these evaluations were not
objective, as they were based on the number of generated mappings and the size
of the resulting edit script, without validating whether the mappings and edit
actions are correct and reflect the actual changes intended by the commit
author.

We call the AST differencing community to follow our lead and contribute new
and more challenging benchmarks, as these can help advance the capabilities of
our tools, and reveal limitations that have not been discovered yet. To this
end, we have developed a benchmarking infrastructure and tooling (Alikhanifard
and Tsantalis, 2023) that can automate parts of this labour-intensive process.

###  5.11. Limitations

To provide an overview of the limitations of our work, we examined the AST
diffs for 15 commits in which our tool has some of its lowest accuracy values,
despite having higher accuracy than the other tools. We found four commonly
occurring issues:

  1. (1)

Our tool may generate inaccurate mappings in test code, because unit tests
share a lot of identical or similar statements (e.g., assertions, test object
creations), and thus there is more ambiguity in the matching process, as a
given statement may have multiple candidate matching statements. Moreover,
developers apply some test-specific refactorings in test code (Soares et al.,
2023), which are not supported by RefactoringMiner yet. As this is a common
issue that all AST diff tools exhibit to some extent, the AST diff community
should develop special techniques for matching test code, and create
benchmarks with commits including test files to evaluate the accuracy of the
tools on test changes.

  2. (2)

The algorithm described in Section 3.2.2 for generating multi-mappings when
duplicated code is moved out of or moved into conditionals does not cover
scenarios that follow a different nesting pattern than the expected one. For
example, in a commit from project apache/hbase (Zhang, 2023) the statements in
lines L164-168 are duplicated 3 times in R152-159, R163-169 and R174-181, but
the conditional statements do not form an if-else-if chain as expected, but
are sequential if statements. Thus, our algorithm can be improved to cover
more scenarios.

  3. (3)

Despite the improvements described in Section 3.2.1.1 for handling ties in
leaf statement mappings, we still observe some false mappings for jump
statements (e.g., break, continue, return), especially when the control flow
of the program is heavily restructured, as in the case of a commit from
project Nutz (Chen, 2015).

  4. (4)

A limitation that is inherited by GumTree simple (used in Algorithm 2) is its
inability to match string literals with small variations (i.e., only identical
string literals can be matched). However, greedy algorithms, such as GumTree
(greedy) and MTDiff can overcome this limitation with the risk of having more
false mappings. For pairs of statements where we have high confidence that
they correspond to a correct mapping, we could utilize algorithms that can
match AST nodes with different values, especially for string literals with
small variations.

Arguably, someone could claim that the main limitation of this work is
language specificity, as it builds upon RefactoringMiner (a tool supporting
Java programs) to obtain the statement mappings. Our counter-argument and main
point of this work is that AST diff should not be treated in a language-
agnostic manner, as language-awareness provides very useful information for
improving the AST mapping accuracy, thus leading to a diff output that
reflects more accurately the changes intended by the commit author. Through
illustrative examples and experiment results, we have shown that by utilizing
language clues (i.e., method calls and signatures, field references), the
semantic role of AST nodes in the program, refactoring information, and by
supporting multi-mappings and inter-file mappings, we can achieve considerable
improvements in accuracy.

###  5.12. Threats to Validity

External validity: The benchmark used for our experiments includes nearly
1,000 cases (800 bug fixes from 17 different projects and 188 refactoring
commits from 105 different projects). This dataset ensures a certain level of
diversity with respect to the domain of the projects and the characteristics
of the changes (i.e., refactorings vs. bug fixes).

Internal validity: Although we did our best effort to reduce bias in the
construction of our benchmark by incorporating the input of six different AST
diff tools, we cannot claim the oracle is completely unbiased, as all
validators are co-authors of this work. The vast majority of the AST mappings
in the benchmark were straightforward to validate using well-defined diff
assessment criteria (Section 4.5). Only around 3% of the examined cases
included debatable mappings, which were resolved after extensive discussions
between the two authors of the paper, and by asking the opinion of our lab
members, when the co-authors were not able to reach a consensus.

Verifiability: We make the source code of RefactoringMiner (Tsantalis, 2023)
and our benchmark (Alikhanifard and Tsantalis, 2023) publicly available to
enable the replication of our experiments and facilitate future research on
AST diff techniques.

##  6\. Related Work

The most significant and influential work on AST differencing is GumTree by
Falleri et al. (Falleri et al., 2014). GumTree is a language-independent
abstract syntax tree matcher, and generates the AST node mappings between two
trees in two phases. In the first (top-down) phase, it detects the largest
identical subtrees. In order to find the largest identical subtrees, GumTree
must compare all sub-tree combinations within two files. Due to the recursive
nature of trees, the runtime complexity is prone to grow very fast. To achieve
a better runtime, GumTree exploits a hashing technique. There are two hash
values associated with every tree. The first one is calculated by considering
the label and the value of the tree and the second one is calculated based on
the label only (structure hash). The structure hash is useful to find
structurally isomorphic trees excluding the actual node values. Hash values
for non-leaf nodes are dependent on the values of their children. In the first
phase, both labels and values are considered, as GumTree aims to find the
largest identical subtrees. GumTree starts with the highest trees available in
both files and performs a comparison by checking their hash values. This
comparison goes step by step deeper into the inner nodes, until a minimum
height equal to 2 is reached. The tuning of the minimum height parameter is
essential to avoid matching small trees that are leaves, but have identical
values. The second (bottom-up) phase aims to match the trees that are not
matched previously, but a fair amount of their children are matched. More
specifically, two trees are matched if the ratio of their common descendants
is greater or equal to 0.5.

Iterative Java Matcher (IJM) (Frick et al., 2018) is built upon GumTree and
aims to improve the quality of the generated move and update actions,
specifically for Java projects. The first enhancement (partial matching)
restricts the scope of matching to selected parts of the source code to
decrease the chance of mismatches. Instead of applying the GumTree algorithm
on the entire Java files, IJM provides partial input to GumTree by pruning
specific parts of the Java file, such as the import statements, type
declarations, enumeration declarations, method declarations, and field
declarations. The second enhancement (merging name nodes) addresses another
limitation of GumTree, which is prone to match nodes whose parents correspond
to different AST types. Since this limitation mostly occurs for the SimpleName
AST type, IJM combines SimpleName nodes with their parent nodes to prevent
inaccurate mappings. The last enhancement (name-aware matching) improves the
bottom-up phase of GumTree by considering the string similarity of the names
of the nodes in addition to their node types.

Move-optimized Tree Differencing (MTDiff) (Dotzler and Philippsen, 2016) is
based on ChangeDistiller (Fluri et al., 2007) and uses 5 optimizations, namely
identical subtree, longest common subsequence, unmapped leaves, inner node,
and leaf move, to shorten the resulting edit scripts. When applied to Gumtree,
RTED, JSync, and ChangeDistiller, they led to shorter scripts for 18-98% of
the changes in the histories of 9 open-source software repositories.

CLDiff (Huang et al., 2018) uses GumTree (without any changes in its
implementation) to obtain AST mappings and edit actions, and generates concise
code differences by grouping fine-grained code differences at or above the
statement level. Finally, it links the related concise code differences
according to five predefined links, namely Def-Use linking changes in
declarations with changes in their references, Abstract-Method linking changes
in abstract methods with changes in their sub-class implementations, Override-
Method linking changes in overridden methods with changes in their overriding
methods, Implement-Method linking changes in interface methods with changes in
their implementing methods, and Systematic-Change linking similar changes
caused by systematic edits (e.g., refactoring).

##  7\. Conclusions

In this work, we presented a novel AST diff tool based on RefactoringMiner
(Tsantalis et al., 2022) that resolves all limitations of the current tools:
(1) it supports multi-mappings, (2) it avoids matching semantically
incompatible AST nodes, (3) it uses language clues to guide the matching
process, (4) it is refactoring-aware, (5) it supports diff at the commit
level. To evaluate our tool and compare it with the state-of-the-art tools, we
created the first benchmark of AST node mappings, including 800 bug-fixing
commits from Defects4J (Just et al., 2014) and 188 refactoring commits from
the Refactoring Oracle (Tsantalis et al., 2022), following a very meticulous
and well-defined process. Our evaluation showed that our tool achieved a
considerably higher precision and recall, especially for refactoring commits,
it does not generate semantically incompatible AST node mappings, and has an
execution time that is comparable with that of the faster tools. More
specifically, our tool has the best performance in 7 out of 8 diff quality
dimensions covered by our evaluation framework, including multi-mapping
accuracy (RQ1), semantically compatible mappings (RQ2), program element
mapping accuracy (RQ3), refactoring-related mapping accuracy (RQ4), inter-file
mapping accuracy (RQ5), overall mapping accuracy (RQ6), and perfect diff rate
(RQ7). The only category our tool does not have the best performance is
execution time (RQ8), where it ranks fourth out of 6 tools.

Implications for researchers: The accurate AST node mappings generated by
RefactoringMiner can improve the performance of several downstream tasks that
rely on AST diff information. For instance, automated program repair (Koyuncu
et al., 2020; Yang et al., 2022; Meng et al., 2013; Jiang et al., 2023) and
library/API migration (Lamothe et al., 2022; Fazzini et al., 2020) techniques
mine the change history of open-source projects to extract bug-fixing patches
and API usage migration scenarios, respectively, based on AST edit scripts.
The AST diff accuracy affects the quality of the extracted change patterns.
Our benchmark of AST node mappings can be used to evaluate the accuracy of
such novel AST diff tools.

Implications for practitioners: The exceptional accuracy of RefactoringMiner’s
AST diff can enable fine-grained and refactoring-aware code evolution analysis
(Jodavi and Tsantalis, 2022b), such as tracking a specific statement (e.g.,
code block) in the commit change history of a project, and finding the commits
it changed. Codoban et al. (Codoban et al., 2015) surveyed 217 developers and
found that 85% of them consider software history important to their
development activities and 61% need to refer to history at least several times
a day. An interesting direction is to develop an intelligent AST diff
visualization UI based on the output of RefactoringMiner, which can overlay
semantic and refactoring information on top of the changes (Escobar et al.,
2022; Huang et al., 2018). Software developers can use such an intelligent
change visualization tool to conduct their code reviews, saving considerable
effort and time in understanding the changes, and thus improving their overall
productivity.

##  8\. Future Research

We believe that a diff reflecting with high accuracy the changes intended by
the commit author is an important foundation upon which intelligent code
review assistants could be built, as having accurate mappings between two
versions of code establishes what portion of the code remained the same (even
if it was refactored) and what portion of the code is truly deleted or added.
However, there are still missing components to enable intelligent assistants.
The first component is navigation assistance.

  * •

In what order and granularity should we navigate the code reviewer over the
changes to help her understand better the “story” and motivation behind the
changes?

  * •

Should we start from behavior-preserving changes (i.e., refactorings) to give
a higher-level view of the design changes intended by the commit author, and
then proceed with non-behavior-preserving changes to give more insights about
the new features or bug fixes intended by the commit author?

  * •

How should we deal with overlapping behavior-preserving and non-behavior-
preserving changes?

  * •

Should we group relevant changes and present them as one change to the code
reviewer?

The second component is explanation assistance. Ideally, the code reviewer
would like to know why a particular change happened. Some of these
explanations can come directly from diff-related information. For example,
multi-mappings can be associated with the intention of eliminating duplicated
code. Inversion, splitting and merging of conditional statements can be
associated with the intention of re-structuring the control-flow of the
program. However, for changes that cannot be explained by refactorings, and
especially for changes implementing a new functionality or fixing a bug, the
explanation is not straightforward. For such changes we could utilize commit
message information, linked issues in the commit message, or discussions from
a Pull Request involving the commit. If the aforementioned information is not
helpful, we could utilize Large Language Models (LLMs) to provide an
explanation. This would require to investigate what is the best way to provide
the required input and context to the LLM, what is the best approach to prompt
the LLM, and how should we deal with known LLM limitations, such as
hallucination and limited ability to control and reproduce the model’s
responses.

Another research direction is to extend RefactoringMiner, so that it is able
to support more programming languages. RefactoringMiner’s core statement
mapping algorithm (Figure 12) is mostly language-independent, as it uses a
string replacement algorithm to match statements that are textually different,
and a set of heuristics to match method calls and object creations operating
upon abstractions. For example, it already abstracts method invocations,
object creations, and method references into objects of the same class, namely
AbstractCall, and thus similar constructs from other programming languages
could be represented by the same abstraction.

###### Acknowledgements.

The authors would like to thank the members of the Refactoring research group
at Concordia University, Victor Veloso, Tayeeb Hasan, Pedram Nouri, Palash
Borhan Uddin, Diptopol Dam, and Mosabbir Khan Shiblu, for providing their
opinion on debatable AST mappings whenever needed. This research was partially
supported by NSERC grant RGPIN-2018-05095.

## References

  * (1)
  * Aalizadeh (2021) Mohammad Sadegh Aalizadeh. 2021\.  _Automatic Motivation Detection for Extract Method Refactoring Operations_.  Master’s thesis. Concordia University.  https://spectrum.library.concordia.ca/id/eprint/988716/
  * Alikhanifard and Tsantalis (2023) Pouria Alikhanifard and Nikolaos Tsantalis. 2023.  _DiffBenchmark_.  https://github.com/pouryafard75/DiffBenchmark
  * Alrubaye et al. (2019) Hussein Alrubaye, Mohamed Wiem Mkaouer, and Ali Ouni. 2019\.  MigrationMiner: An Automated Detection Tool of Third-Party Java Library Migration at the Method Level. In _2019 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. 414–417.  https://doi.org/10.1109/ICSME.2019.00072
  * Briand et al. (1996) Lionel C. Briand, Christiane M. Differding, and H. Dieter Rombach. 1996.  Practical guidelines for measurement-based process improvement.  _Software Process: Improvement and Practice_ 2, 4 (1996), 253–280.  https://doi.org/10.1002/(SICI)1099-1670(199612)2:4<253::AID-SPIP53>3.0.CO;2-G
  * Chen (2015) Wendal Chen. 2015\.  _Nutzam Nutz_.  https://github.com/nutzam/nutz/commit/6599c748ef35d38085703cf3bd41b9b5b6af5f32#diff-d47d95ef90d7f70d88b06a02d8505b58a81c7c1174c2494e56c9b09d090bc6edL308
  * Codoban et al. (2015) Mihai Codoban, Sruti Srinivasa Ragavan, Danny Dig, and Brian Bailey. 2015.  Software history under the lens: A study on why and how developers examine it. In _2015 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. 1–10.  https://doi.org/10.1109/ICSM.2015.7332446
  * Decker et al. (2020) Michael John Decker, Michael L. Collard, L. Gwenn Volkert, and Jonathan I. Maletic. 2020\.  SrcDiff: A Syntactic Differencing Approach to Improve the Understandability of Deltas.  _J. Softw. Evol. Process_ 32, 4 (apr 2020), 31 pages.  https://doi.org/10.1002/smr.2226
  * Dotzler and Philippsen (2016) Georg Dotzler and Michael Philippsen. 2016.  Move-Optimized Source Code Tree Differencing. In _Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering_ (Singapore, Singapore) _(ASE ’16)_. Association for Computing Machinery, New York, NY, USA, 660–671.  https://doi.org/10.1145/2970276.2970315
  * Ebert et al. (2021) Felipe Ebert, Fernando Castor, Nicole Novielli, and Alexander Serebrenik. 2021.  An Exploratory Study on Confusion in Code Reviews.  _Empirical Softw. Engg._ 26, 1 (jan 2021), 48 pages.  https://doi.org/10.1007/s10664-020-09909-5
  * Ellis et al. (2023) Max Ellis, Sarah Nadi, and Danny Dig. 2023.  Operation-Based Refactoring-Aware Merging: An Empirical Evaluation.  _IEEE Transactions on Software Engineering_ 49, 4 (2023), 2698–2721.  https://doi.org/10.1109/TSE.2022.3228851
  * Eschner et al. (2023) Johannes Eschner, Peter Mindek, and Manuela Waldner. 2023\.  Illustrative Motion Smoothing for Attention Guidance in Dynamic Visualizations.  _Computer Graphics Forum_ 42, 3 (2023), 361–372.  https://doi.org/10.1111/cgf.14836 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14836
  * Escobar et al. (2022) Ronald Escobar, Juan Pablo Sandoval Alcocer, Hagen Tarner, Fabian Beck, and Alexandre Bergel. 2022\.  Spike – A code editor plugin highlighting fine-grained changes. In _2022 Working Conference on Software Visualization (VISSOFT)_. IEEE Press, 167–171.  https://doi.org/10.1109/VISSOFT55257.2022.00026
  * Falleri et al. (2014) Jean-Rémy Falleri, Floréal Morandat, Xavier Blanc, Matias Martinez, and Martin Monperrus. 2014\.  Fine-Grained and Accurate Source Code Differencing. In _Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering_ (Vasteras, Sweden) _(ASE ’14)_. Association for Computing Machinery, New York, NY, USA, 313–324.  https://doi.org/10.1145/2642937.2642982
  * Fan et al. (2021) Yuanrui Fan, Xin Xia, David Lo, Ahmed E. Hassan, Yuan Wang, and Shanping Li. 2021\.  A Differential Testing Approach for Evaluating Abstract Syntax Tree Mapping Algorithms. In _Proceedings of the 43rd International Conference on Software Engineering_ (Madrid, Spain) _(ICSE ’21)_. IEEE Press, 1174–1185.  https://doi.org/10.1109/ICSE43902.2021.00108
  * Fazzini et al. (2020) Mattia Fazzini, Qi Xin, and Alessandro Orso. 2020.  APIMigrator: An API-Usage Migration Tool for Android Apps. In _Proceedings of the IEEE/ACM 7th International Conference on Mobile Software Engineering and Systems_ (Seoul, Republic of Korea) _(MOBILESoft ’20)_. Association for Computing Machinery, New York, NY, USA, 77–80.  https://doi.org/10.1145/3387905.3388608
  * Fluri et al. (2007) Beat Fluri, Michael Wursch, Martin Pinzger, and Harald Gall. 2007.  Change Distilling:Tree Differencing for Fine-Grained Source Code Change Extraction.  _IEEE Transactions on Software Engineering_ 33, 11 (2007), 725–743.  https://doi.org/10.1109/TSE.2007.70731
  * Fowler (1999) Martin Fowler. 1999\.  _Refactoring: Improving the Design of Existing Code_.  Addison-Wesley, Boston, MA, USA.
  * Fowler (2015) Martin Fowler. 2015\.  _Refactoring with Loops and Collection Pipelines_.  https://martinfowler.com/articles/refactoring-pipelines.html
  * Fowler (2018) Martin Fowler. 2018\.  _Refactoring: Improving the Design of Existing Code_ (2 ed.).  Addison-Wesley, Boston, MA, USA.
  * Frick (2023) Veit Frick. 2023\.  _IJM GitHub repository_.  https://github.com/VeitFrick/IJM
  * Frick et al. (2018) Veit Frick, Thomas Grassauer, Fabian Beck, and Martin Pinzger. 2018.  Generating Accurate and Compact Edit Scripts Using Tree Differencing. In _2018 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE Press, 264–274.  https://doi.org/10.1109/ICSME.2018.00036
  * Fujimoto et al. (2020) Akira Fujimoto, Yoshiki Higo, Junnosuke Matsumoto, and Shinji Kusumoto. 2020.  Staged Tree Matching for Detecting Code Move across Files. In _Proceedings of the 28th International Conference on Program Comprehension_ (Seoul, Republic of Korea) _(ICPC ’20)_. Association for Computing Machinery, New York, NY, USA, 396–400.  https://doi.org/10.1145/3387904.3389289
  * Gerrit Code Review (2023) Gerrit Code Review. 2023\.  _Side-by-Side Diff Screen_.  https://gerrit-review.googlesource.com/Documentation/user-review-ui.html#diff-view
  * GitHub Docs (2023) GitHub Docs. 2023\.  _Pull requests_.  https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-comparing-branches-in-pull-requests#diff-view-options
  * He et al. (2021) Hao He, Runzhi He, Haiqiao Gu, and Minghui Zhou. 2021\.  A Large-Scale Empirical Study on Java Library Migrations: Prevalence, Trends, and Rationales. In _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Athens, Greece) _(ESEC/FSE 2021)_. Association for Computing Machinery, New York, NY, USA, 478–490.  https://doi.org/10.1145/3468264.3468571
  * Higo et al. (2017) Yoshiki Higo, Akio Ohtani, and Shinji Kusumoto. 2017\.  Generating Simpler AST Edit Scripts by Considering Copy-and-Paste. In _Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering_ (Urbana-Champaign, IL, USA) _(ASE ’17)_. IEEE Press, 532–542.
  * Huang et al. (2018) Kaifeng Huang, Bihuan Chen, Xin Peng, Daihong Zhou, Ying Wang, Yang Liu, and Wenyun Zhao. 2018.  ClDiff: Generating Concise Linked Code Differences. In _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering_ (Montpellier, France) _(ASE ’18)_. Association for Computing Machinery, New York, NY, USA, 679–690.  https://doi.org/10.1145/3238147.3238219
  * Jiang et al. (2023) Yanjie Jiang, Hui Liu, Xiaoqing Luo, Zhihao Zhu, Xiaye Chi, Nan Niu, Yuxia Zhang, Yamin Hu, Pan Bian, and Lu Zhang. 2023\.  BugBuilder: An Automated Approach to Building Bug Repository.  _IEEE Transactions on Software Engineering_ 49, 4 (2023), 1443–1463.  https://doi.org/10.1109/TSE.2022.3177713
  * Jodavi and Tsantalis (2022a) Mehran Jodavi and Nikolaos Tsantalis. 2022a.  Accurate Method and Variable Tracking in Commit History. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Singapore, Singapore) _(ESEC/FSE 2022)_. Association for Computing Machinery, New York, NY, USA, 183–195.  https://doi.org/10.1145/3540250.3549079
  * Jodavi and Tsantalis (2022b) Mehran Jodavi and Nikolaos Tsantalis. 2022b.  Accurate Method and Variable Tracking in Commit History. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Singapore, Singapore) _(ESEC/FSE 2022)_. Association for Computing Machinery, New York, NY, USA, 183–195.  https://doi.org/10.1145/3540250.3549079
  * Just et al. (2014) René Just, Darioush Jalali, and Michael D. Ernst. 2014\.  Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs. In _Proceedings of the 2014 International Symposium on Software Testing and Analysis_ (San Jose, CA, USA) _(ISSTA 2014)_. Association for Computing Machinery, New York, NY, USA, 437–440.  https://doi.org/10.1145/2610384.2628055
  * Ketkar et al. (2019) Ameya Ketkar, Ali Mesbah, Davood Mazinanian, Danny Dig, and Edward Aftandilian. 2019.  Type Migration in Ultra-Large-Scale Codebases. In _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_. IEEE Computer Society, Los Alamitos, CA, USA, 1142–1153.  https://doi.org/10.1109/ICSE.2019.00117
  * Ketkar et al. (2022) Ameya Ketkar, Oleg Smirnov, Nikolaos Tsantalis, Danny Dig, and Timofey Bryksin. 2022.  Inferring and Applying Type Changes. In _Proceedings of the 44th International Conference on Software Engineering_ (Pittsburgh, Pennsylvania) _(ICSE ’22)_. Association for Computing Machinery, New York, NY, USA, 1206–1218.  https://doi.org/10.1145/3510003.3510115
  * Ketkar et al. (2020) Ameya Ketkar, Nikolaos Tsantalis, and Danny Dig. 2020\.  Understanding Type Changes in Java. In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Virtual Event, USA) _(ESEC/FSE 2020)_. Association for Computing Machinery, New York, NY, USA, 629–641.  https://doi.org/10.1145/3368089.3409725
  * Koyuncu et al. (2020) Anil Koyuncu, Kui Liu, Tegawendé F. Bissyandé, Dongsun Kim, Jacques Klein, Martin Monperrus, and Yves Le Traon. 2020.  FixMiner: Mining Relevant Fix Patterns for Automated Program Repair.  _Empirical Softw. Engg._ 25, 3 (may 2020), 1980–2024.  https://doi.org/10.1007/s10664-019-09780-z
  * Kurbatova et al. (2021) Zarina Kurbatova, Vladimir Kovalenko, Ioana Savu, Bob Brockbernd, Dan Andreescu, Matei Anton, Roman Venediktov, Elena Tikhomirova, and Timofey Bryksin. 2021.  RefactorInsight: Enhancing IDE Representation of Changes in Git with Refactorings Information. In _2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. 1276–1280.  https://doi.org/10.1109/ASE51524.2021.9678646
  * Lamothe et al. (2022) Maxime Lamothe, Weiyi Shang, and Tse-Hsun Peter Chen. 2022\.  A3: Assisting Android API Migrations Using Code Examples.  _IEEE Transactions on Software Engineering_ 48, 2 (2022), 417–431.  https://doi.org/10.1109/TSE.2020.2988396
  * Le et al. (2016) Xuan Bach D. Le, David Lo, and Claire Le Goues. 2016\.  History Driven Program Repair. In _2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER)_. 213–224.  https://doi.org/10.1109/SANER.2016.76
  * Liu et al. (2021) Zhongxin Liu, Xin Xia, Meng Yan, and Shanping Li. 2021\.  Automating Just-in-Time Comment Updating. In _Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering_ (Virtual Event, Australia) _(ASE ’20)_. Association for Computing Machinery, New York, NY, USA, 585–597.  https://doi.org/10.1145/3324884.3416581
  * MacLeod et al. (2018) Laura MacLeod, Michaela Greiler, Margaret-Anne Storey, Christian Bird, and Jacek Czerwonka. 2018\.  Code Reviewing in the Trenches: Challenges and Best Practices.  _IEEE Software_ 35, 4 (2018), 34–42.  https://doi.org/10.1109/MS.2017.265100500
  * Martinez et al. (2023) Matias Martinez, Jean-Rémy Falleri, and Martin Monperrus. 2023.  Hyperparameter Optimization for AST Differencing.  _IEEE Transactions on Software Engineering_ 49, 10 (2023), 4814–4828.  https://doi.org/10.1109/TSE.2023.3315935
  * McLead (2022) Mason McLead. 2022\.  _Global Code Time Report_.  https://www.software.com/reports/code-time-report
  * Meng et al. (2013) Na Meng, Miryung Kim, and Kathryn S. McKinley. 2013\.  LASE: Locating and Applying Systematic Edits by Learning from Examples. In _Proceedings of the 2013 International Conference on Software Engineering_ (San Francisco, CA, USA) _(ICSE ’13)_. IEEE Press, 502–511.
  * Murphy-Hill et al. (2012) Emerson Murphy-Hill, Chris Parnin, and Andrew P. Black. 2012\.  How We Refactor, and How We Know It.  _IEEE Transactions on Software Engineering_ 38, 1 (2012), 5–18.  https://doi.org/10.1109/TSE.2011.41
  * Myers (2023) Eugene W. Myers. 2023\.  AnO(ND) Difference Algorithm and Its Variations.  _Algorithmica_ 1, 1–4 (mar 2023), 251–266.  https://doi.org/10.1007/BF01840446
  * Negara et al. (2013) Stas Negara, Nicholas Chen, Mohsen Vakilian, Ralph E. Johnson, and Danny Dig. 2013.  A Comparative Study of Manual and Automated Refactorings. In _Proceedings of the 27th European Conference on Object-Oriented Programming_ (Montpellier, France) _(ECOOP’13)_. Springer-Verlag, Berlin, Heidelberg, 552–576.  https://doi.org/10.1007/978-3-642-39038-8_23
  * Negara et al. (2012) Stas Negara, Mohsen Vakilian, Nicholas Chen, Ralph E. Johnson, and Danny Dig. 2012\.  Is It Dangerous to Use Version Control Histories to Study Source Code Evolution?. In _26th European Conference on Object-Oriented Programming_ (Beijing, China) _(ECOOP’12)_. Springer-Verlag, Berlin, Heidelberg, 79–103.  https://doi.org/10.1007/978-3-642-31057-7_5
  * Nugroho et al. (2020) Yusuf Sulistyo Nugroho, Hideaki Hata, and Kenichi Matsumoto. 2020\.  How Different Are Different Diff Algorithms in Git? Use –Histogram for Code Changes.  _Empirical Softw. Engg._ 25, 1 (jan 2020), 790–823.  https://doi.org/10.1007/s10664-019-09772-z
  * Ossendrijver et al. (2022) Rick Ossendrijver, Stephan Schroevers, and Clemens Grelck. 2022\.  Towards Automated Library Migrations with Error Prone and Refaster. In _Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing_ (Virtual Event) _(SAC ’22)_. Association for Computing Machinery, New York, NY, USA, 1598–1606.  https://doi.org/10.1145/3477314.3507153
  * Ossendrijver et al. (2023) Rick Ossendrijver, Stephan Schroevers, and Clemens Grelck. 2023\.  Automating Library Migrations with Error Prone and Refaster.  _SIGAPP Appl. Comput. Rev._ 23, 1 (apr 2023), 5–19.  https://doi.org/10.1145/3594264.3594265
  * Pantiuchina et al. (2020) Jevgenija Pantiuchina, Fiorella Zampetti, Simone Scalabrino, Valentina Piantadosi, Rocco Oliveto, Gabriele Bavota, and Massimiliano Di Penta. 2020.  Why Developers Refactor Source Code: A Mining-Based Study.  _ACM Trans. Softw. Eng. Methodol._ 29, 4, Article 29 (sep 2020), 30 pages.  https://doi.org/10.1145/3408302
  * RefactoringMiner AST Diff Gallery (2023a) RefactoringMiner AST Diff Gallery. 2023a.  _12 duplicated code fragments extracted from the same method_.  https://github.com/tsantalis/RefactoringMiner/wiki/AST-Diff-Gallery#12-duplicated-code-fragments-extracted-from-the-same-method-in-commit-click-to-view-diff
  * RefactoringMiner AST Diff Gallery (2023b) RefactoringMiner AST Diff Gallery. 2023b.  _36 duplicated code fragments extracted from the same method_.  https://github.com/tsantalis/RefactoringMiner/wiki/AST-Diff-Gallery#36-duplicated-code-fragments-extracted-from-the-same-method-in-commit-click-to-view-diff
  * Rosa et al. (2021) Giovanni Rosa, Luca Pascarella, Simone Scalabrino, Rosalia Tufano, Gabriele Bavota, Michele Lanza, and Rocco Oliveto. 2021.  Evaluating SZZ Implementations Through a Developer-Informed Oracle. In _Proceedings of the 43rd International Conference on Software Engineering_ (Madrid, Spain) _(ICSE ’21)_. IEEE Press, 436–447.  https://doi.org/10.1109/ICSE43902.2021.00049
  * Shan et al. (2022) Qianhua Shan, David Sukhdeo, Qianying Huang, Seth Rogers, Lawrence Chen, Elise Paradis, Peter C. Rigby, and Nachiappan Nagappan. 2022.  Using Nudges to Accelerate Code Reviews at Scale. In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Singapore, Singapore) _(ESEC/FSE 2022)_. Association for Computing Machinery, New York, NY, USA, 472–482.  https://doi.org/10.1145/3540250.3549104
  * Shen et al. (2021) Bo Shen, Wei Zhang, Christian Kästner, Haiyan Zhao, Zhao Wei, Guangtai Liang, and Zhi Jin. 2021.  SmartCommit: A Graph-Based Interactive Assistant for Activity-Oriented Commits. In _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Athens, Greece) _(ESEC/FSE 2021)_. Association for Computing Machinery, New York, NY, USA, 379–390.  https://doi.org/10.1145/3468264.3468551
  * Silva et al. (2021) Danilo Silva, João Paulo da Silva, Gustavo Santos, Ricardo Terra, and Marco Tulio Valente. 2021\.  RefDiff 2.0: A Multi-Language Refactoring Detection Tool.  _IEEE Transactions on Software Engineering_ 47, 12 (2021), 2786–2802.  https://doi.org/10.1109/TSE.2020.2968072
  * Silva et al. (2016) Danilo Silva, Nikolaos Tsantalis, and Marco Tulio Valente. 2016\.  Why We Refactor? Confessions of GitHub Contributors. In _Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering_ (Seattle, WA, USA) _(FSE 2016)_. Association for Computing Machinery, New York, NY, USA, 858–870.  https://doi.org/10.1145/2950290.2950305
  * Soares et al. (2023) Elvys Soares, Márcio Ribeiro, Rohit Gheyi, Guilherme Amaral, and André Santos. 2023.  Refactoring Test Smells With JUnit 5: Why Should Developers Keep Up-to-Date?  _IEEE Transactions on Software Engineering_ 49, 3 (2023), 1152–1170.  https://doi.org/10.1109/TSE.2022.3172654
  * South et al. (2022) Laura South, David Saffo, Olga Vitek, Cody Dunne, and Michelle A. Borkin. 2022.  Effective Use of Likert Scales in Visualization Evaluations: A Systematic Review.  _Computer Graphics Forum_ 41, 3 (2022), 43–55.  https://doi.org/10.1111/cgf.14521 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.14521
  * Tan and Bockisch (2019) Liang Tan and Christoph Bockisch. 2019.  A Survey of Refactoring Detection Tools. In _Proceedings of the Workshops of the Software Engineering Conference 2019, Stuttgart, Germany, February 19, 2019_ _(CEUR Workshop Proceedings, Vol. 2308)_ , Stephan Krusche, Kurt Schneider, Marco Kuhrmann, Robert Heinrich, Reiner Jung, Marco Konersmann, Eric Schmieders, Steffen Helke, Ina Schaefer, Andreas Vogelsang, Björn Annighöfer, Andreas Schweiger, Marina Reich, and André van Hoorn (Eds.). CEUR-WS.org, 100–105.  https://ceur-ws.org/Vol-2308/emls2019paper02.pdf
  * Tsantalis (2023) Nikolaos Tsantalis. 2023\.  _RefactoringMiner_.  https://github.com/tsantalis/RefactoringMiner
  * Tsantalis et al. (2022) Nikolaos Tsantalis, Ameya Ketkar, and Danny Dig. 2022.  RefactoringMiner 2.0.  _IEEE Transactions on Software Engineering_ 48, 3 (2022), 930–950.  https://doi.org/10.1109/TSE.2020.3007722
  * Tsantalis et al. (2018) Nikolaos Tsantalis, Matin Mansouri, Laleh Eshkevari, Davood Mazinanian, and Danny Dig. 2018\.  Accurate and Efficient Refactoring Detection in Commit History. In _2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE)_ _(ICSE ’18)_. IEEE Press, 483–494.  https://doi.org/10.1145/3180155.3180206
  * Yang et al. (2022) Yilin Yang, Tianxing He, Yang Feng, Shaoying Liu, and Baowen Xu. 2022.  Mining Python Fix Patterns via Analyzing Fine-Grained Source Code Changes.  _Empirical Softw. Engg._ 27, 2 (mar 2022), 37 pages.  https://doi.org/10.1007/s10664-021-10087-1
  * Zhang (2023) Duo Zhang. 2023\.  _Apache HBase_.  https://github.com/apache/hbase/commit/29cf51d8391099f5628a52d2c3306932758d4108