RevRec [6] | P&R |  | P&R |  | P&R |  |  |
WRC [152] |  | ACC |  |  |  |  |  |
PR-CF [128] |  | P&R |  | P&R |  | P&R | P&R |
Fejzer et al. [136] | P&R |  | P&R |  |  |  |  |
TRFPre [147] |  |  |  | ACC | ACC |  |  | ACC
WhoDo [140] |  |  |  |  | P&R |  |  |

Other studies—presented in Table 13—compared a reviewer recommender with
alternative baselines, e.g. a simple heuristic or a standard learning
technique. For instance, Balachandran [108] developed RevHistRECO, which is a
heuristic inspired by the observed manual process of the reviewer assignment,
adopted as a baseline for his proposed Review Bot. Yu et al. [106], in turn,
extended and implemented as baseline recommenders based on existing
techniques, such as information retrieval (IR). Some of the baseline
approaches in this table, e.g. xFinder [157] and CoreDevRec [158], are not in
our SLR, because their purpose is not to recommend code reviewers and are not
in the context of MCR.

Table 13: Results of comparisons between a reviewer recomender and a baseline technique. Each row indicates that an approach in the first column outperformed baselines in the third column, with respect to the metrics listed in the second column. The metrics are accuracy (ACC), F-measure (F1), mean reciprocal rank (MRR), and precision and recall (P&R). Approach | Measurement |  Outperformed Baselines
---|---|---
Review Bot [108] | ACC |  RevHistRECO
IR+CN [106] | P&R and F1 |  SVM-based, IR-based, FL-based, IR-based+CN-based, and FL-based+CN-based
cHRev [120] | P&R, F1, and MRR |  xFinder and RevCom
WRC Algorithm [152] | ACC |  Line 10 Rule, Number of Changes, Expertise Recommender, Code Ownership, Expertise Cloud, and Degree-of-Authorship
TRFPre [147] | ACC |  CoreDevRec and ACRec

##### Code Checking

While reviewer recommenders are usually compared to similar approaches, eight
out of ten offline studies [109, 114, 117, 119, 138, 141, 142, 148] that
evaluate work to support code checking use a manually built ground truth.
Hanam et al. [145], however, used additional tools to build a ground truth. In
both cases, it is used to identify false positives and false negatives given
as output of the approaches to support code checking. The identified false
positives are then analyzed by the researchers, who examine their cause so
that the proposed approach can be improved. For instance, Barnett et al. [109]
and Luna Freire et al. [138] highlighted which type of code change was not
considered by their approach to distinguish trivial and non-trivial changes.
The other offline evaluation [143] of a code checking approach then mixed the
manual analysis of output and the comparison with an existing tool. As key
findings, Huang et al. [143] found that the proposed approach has higher
accuracy and better conciseness as well as required less time in the study.

Finding 18: Evaluations of approaches to support MCR mainly focus on the
validation of the output using data analysis. This strategy—named _offline
evaluation_ —is usually used when assessing reviewer recommenders and code
checking support. Most offline evaluations use data from OSS projects.
Precision and recall are the most frequent metrics in this type of evaluation.
Considering reviewer recommenders, RevFinder has been the most used as a
baseline.

Finding 19: Most conclusions of the evaluation studies are related to the
feasibility and effectiveness of an approach to support MCR based on data
analysis. The results of the evaluation of a newly-developed proposal
outperforms baselines, achieving better accuracy, precision, and/or recall.

###  6.3 Experiments

A smaller amount of MCR approaches (in comparison to offline evaluations) have
been evaluated by means of experiments. They are performed in controlled
environments and involve subjects, being them students, professionals, or
both. We next discuss the following aspects of the design of these studies:
(i) independent and dependent variables; and (ii) participants. As in the
previous section, we also summarize their findings.

####  6.3.1 Study Design

##### Variables of the experiments

The experimental studies to evaluate an MCR approach rely on various
variables. Five of these studies adopted a within-subjects design, in which
all participants performed review tasks using both the proposed approach and
another selected for comparison. In the experiment of Zhang et al. [111], the
analysis of systematic changes in the code was performed using the proposed
CRITICS and also, as baseline, the diff and search features of Eclipse.
Similarly, Hanam et al. [145] asked the study participants to perform a review
task supported by the proposed change impact analysis tool SemCIA and also by
other tools, SynCIA and UnixDiff. In two studies, the participants reviewed a
code change with and without the support of the proposed approach output, i.e.
with or without partitions [114] and with or without a hint of salient class
[141]. Lastly, participants of the study of Huang et al. [143] conducted the
review with the proposed tool ClDiff and with GumTree, the state-of-the-art
tool to generate fine-grained code differences. In all these cases, the
researchers measured the correctness of the output of the review task and the
time spent. More specifically, in two of these studies [141, 143] the
experiment analyzed the degree of understanding the changes.

In contrast with these studies, two evaluations followed a between-subjects
design. Khandelwal et al. [149] evaluated the effect of gamification on code
review, organizing the participants into five groups, each of them using
either a gamified or a non-gamified review tool. The researchers then measured
the subject’s interest by the number of review comments, the usefulness of
review comments, the number of identified bugs, the number of identified code
smells, and the time spent. In Menarini et al. [131]’s experiment, the
intervention group used the proposed Getty approach, while the control group
used GitHub resources. From these interactions, the code review process
resulting from the used supporting approach has been assessed.

Finally, Runeson and Wohlin [154] performed an experiment to compare three
alternative capture-recapture methods, which are used to estimate the number
of bugs in a code after going through review. Participants had to review a
target code and point out bugs. Based on this, the researchers analyzed the
identified bugs and inspected the errors in the estimation of the evaluated
methods.

##### Sample size

The experiments performed to evaluate MCR approaches considered varying
numbers of subjects to participate in the studies. In Table 11, where we show
the number of projects used in offline evaluations, we also detail the
descriptive statistics of the number of subjects in experiments. The study
that involved the highest number of subjects (183) was conducted by Khandelwal
et al. [149], while Runeson and Wohlin [154] experiment involved the smallest
sample, with 8 subjects. Moreover, considering the background of subjects,
three of the experiments [149, 111, 114] were conducted solely with students;
the other three experiments [131, 154, 145] involved professionals in addition
to students. In the experimented conducted by Huang et al. [141], the
participants are characterized as people engaged in computer-related work with
programming experience, but it is not clear whether they are professionals,
students, or both.

####  6.3.2 Findings

Experiments performed in the context of MCR focus on evaluating particular
aspects of the proposed approaches. Thus, the key findings target the value
promoted by each approach. Two experiments [111, 114] provide evidence of a
positive impact in the review process due to the proposed approach, such as by
improving the correctness and time spent in the review activity. Similarly,
three studies [141, 145, 143] give evidence of a specific positive impact of
the proposed approach on the understanding of code changes. In contrast,
Khandelwal et al. [149] found no evidence that gamified tools promote a
positive impact on the subjects’ interest. Additionally, grounded on
observations of the experiment execution, Menarini et al. [131] indicated that
semantically-assisted code review is feasible and effective.

In addition to the main collected data, three experiments also include a
follow-up study with the participants to collect their perceptions about the
proposed approach. By means of a survey [149, 111] or an interview [131], the
subjective opinion of the participants suggests that the approaches might
indeed help the code review process.

###  6.4 RQ3 - How have MCR approaches been evaluated and what were the
reached conclusions?

Most of the approaches proposed to support MCR have been evaluated using
offline studies, with the goal of validating the output of the technique or
tool, commonly using historical data from code review as input. From 37
evaluation papers, 29 (78.38%) present an offline study. In 12 of 29 cases,
these validations consider the effectiveness by measuring precision and recall
of the output compared to the stored information. In 10 of 29 offline studies,
the validations consider the accuracy. The reached conclusions are usually
related to a newly-developed approach that outperforms baselines by achieving
better effectiveness in an offline evaluation. There are also few reports of
evaluations with user studies (13 of 37 evaluations), and they are mainly
focused on observing the effects and the opinion of the proposed approach.

##  7 Discussion

The literature on MCR reviewed in our SLR allowed us to identify and analyze
the researched aspects of the practice, classify the existing approaches to
support it, and understand how these supporting approaches have been
evaluated. Thus, we presented in previous sections a structured body of
knowledge of the MCR practice, which is helpful for both researchers and
practitioners. In this section, we discuss further insights derived from our
analysis.

###  7.1 Historical Developments

Given the primary studies analyzed in this systematic review, we provide a
historical perspective of the developments in the field of MCR in Figure 8. We
summarize the total number of papers published per year, as well as papers of
each type per year. We can observe an increase in research work on MCR in
recent years. This trend indicates the importance and timeliness of the topic.
Nevertheless, in the last two years (i.e., 2018 and 2019) we can observe a
decrease in the total number of papers. This variation is mainly due to a
decreased number of proposals. Nevertheless, this decline in the number of
papers in the last couple of years does not provide enough evidence to claim
any medium- and long-term trend.

Figure 8: Distribution of primary studies by type per year of publication.

###  7.2 Taxonomy of Research Work on MCR

Our analysis of the primary studies of MCR selected for being investigated in
this SLR leads us to identify key topics in this research area. We categorized
the topics investigated in foundational studies, proposals of approaches to
support the practice, and types of evaluation studies in the previous
sections. We now summarize and propose a taxonomy of aspects of the research
on MCR in Figure 9, which complements the mentioned categories.

In our taxonomy, we first highlight the _goals_ of the research work on MCR,
which can be related to: (i) a better understanding of one or more code review
aspects, e.g. challenges faced by practitioners and factors influencing an
outcome; (ii) novel approaches that provide support to practitioners; and
(iii) evaluation techniques or tools, which aim to assess the effectiveness of
MCR-supporting approaches. Works on MCR target a particular _scope_ , which
can be the MCR process as a whole or a particular task. Studies or proposals
generally rely on collected or existing data, which can be mined from
repositories or obtained from human subjects. These data can be from a
combination of locations, e.g. OSS or academia. This is captured by the
_source of data_ facet. Finally, the last facet refers to the _output_
associated with the contribution of the work (which can be assessed metrics
and/or a particular type of contribution).

Figure 9: A taxonomy of MCR research based on primary studies of our SLR.

###  7.3 Actionable Implications and Future Directions

The findings of our systematic literature review revealed that multiple
aspects of MCR have been addressed in the last years. This section discusses
the implications of our findings and issues that remain unaddressed.

##### MCR process improvement

Empirical studies of MCR demonstrate the feasibility of extracting information
from the history of review activity stored in tools, such as Gerrit. For
practitioners, these findings suggest how to improve internal outcomes, such
as that the submission of small code changes increases reviewers’
participation and reduces the review duration. Moreover, the findings related
to the CodeFlow Analytics from Microsoft [42] demonstrate that developers can
use review data to improve MCR. Although CodeFlow Analytics is an internal
platform of Microsoft, there is an opportunity for researchers to study how
developers can use this data to improve the MCR process in their own projects
and what outcomes can support strategic decision making to improve this
process.

##### Code Improvement

Differently from inspections, MCR has as one of its main benefits source code
improvement. Nevertheless, changes based on comments are made in an individual
basis. Consequently, the same comments might be made in different code
reviews. Similar code improvements in reviewed requests have been identified
[95] and common themes emerged from the manual analyses of review feedback [3,
40]. Moreover, there is evidence that authors repeatedly introduce the same
types of problems despite the reviewer feedback [93]. For practitioners, this
implies more attention before submitting a request, disseminating and checking
for issues that were earlier addressed, and reducing the review cost. For
researchers, these findings suggest that natural language processing (NLP)
techniques can be used to extract recurrent bad practices in projects based on
comments to avoid them to occur again by means of knowledge dissemination.

##### Exploration of Non-technical MCR Benefits

Differently from code inspections, whose primary focus was bug detection, MCR
brings various other benefits, such as knowledge transfer, collective code
ownership, and learning. However, these non-technical benefits have been
little explored in foundational studies and existing approaches do not focus
on improving them.

##### User studies on MCR

Only few experiments involving human subjects have been conducted in the
context of MCR. In our set of 139 primary studies, 19 reported an experiment
(13.67%). Including other methodological approaches involving participants,
i.e., experiments, interviews, and surveys, 47 of 139 (33.81%) papers present
a study with practitioners or students. MCR is essentially a human-based
activity, supported by tools. Consequently, further user studies must be done.
Most of the evaluations of code reviewer recommenders rely only on accuracy
metrics. However, as known in the recommender systems research area, other
aspects, such as novelty and transparency, are key for the adoption of
recommenders. Recently, Kovalenko et al. [159] observed this issue and
conducted an _in vivo_ performance evaluation of a reviewer recommender. As a
key finding, the researchers indicate the need for more user-centric
approaches to designing and evaluating the recommenders. Therefore, user
studies can help researchers to understand how approaches to support MCR are
used and perceived and what are the barriers faced by practitioners.

##### Studies in small and medium-sized companies

From the 86 foundational studies, 55 studies selected data from 70 different
open-source projects. The most frequent projects have been Qt, OpenStack, and
Android with 23, 19, and 13 occurrences, respectively. In 17 of 55 studies,
there is an overlap in which data from two or three of these projects were
used in the same study, e.g., Qt and OpenStack are both data source projects
in 13 studies. Considering research in the industry, our systematic review
identified 17 papers involving data and participants from companies, a small
number compared to studies involving open-source projects. From these studies
in the industry, 6 are projects conducted at Microsoft. While the variety of
open-source projects might suggest that the existing evidence of MCR spans
multiple contexts, we observe as an implication the opportunity of future
research to explore code review in other industrial contexts, such as small
and medium-sized companies, to understand whether the existing knowledge might
be generalized to other scenarios.

###  7.4 Research Limitations

The goal of this study is to identify the state of the art on MCR, providing a
structured overview of the research done in this field. We thus performed a
_systematic_ review in order to minimize the research bias, mitigating the
influence of the researchers’ expectations during the selection and analysis
of a large number of primary studies.

To mitigate the limitations of our SLR in the review planning phase, we
selected widely used digital libraries as sources and specified keywords as
search string that cover the studied theme, assuming that the selected
strategy would retrieve the largest number of relevant studies. Three of our
selected databases—ACM Digital Library, IEEE Xplore, SpringerLink—were also
used in other secondary studies on MCR [8, 7, 10]. While these other studies
also searched the Scopus database, we selected ScienceDirect as a source.
However, both Scopus and ScienceDirect are provided by Elsevier.

Although this search is large, it is not complete and, therefore, our review
does not cover all existing work on MCR. An example of a study that was not
retrieved in our search is that of Rigby et al. [160] because it uses the term
peer review, which is not covered by our search string. In this case, however,
the findings of this study were used for comparison purposes by Rigby and Bird
[4], which is a primary study in our review. As discussed when we introduced
our search string, some authors used _peer review_ to refer to the MCR
practice, but this term is also used in other contexts, such as peer review of
scientific papers. Our search identified papers that used the term _peer code
review_ , but not solely _peer review_. The advantage of systematic reviews is
that they can be further extended in future reviews.

Moreover, to identify further studies, a snowballing approach could have been
used. This approach has not been followed in the present work because a
preliminary analysis of the obtained results indicated that the most relevant
studies had been retrieved by searching our selected digital libraries.
Nevertheless, we estimated whether and how the lack of snowballing could have
influenced our review. We randomly selected 20 papers from the set of primary
studies, creating our referencing list. We then conducted a backward
snowballing [161] using this reference list to identify additional primary
studies. We followed the same selection process of our SLR in this backward
snowballing, using the same selection criteria. We checked 619 new results
from our reference list and identified 8 papers that should be included in our
review but were not identified by our procedure. From these eight papers, we
checked 240 other results, including 2 papers. Then, we checked 26 results
from those 2 papers, but we did not find more entries. Our procedure to
estimate whether and how the lack of snowballing could have influenced our
review identified 10 papers that were not identified by our search. The papers
identified in this procedure used specific terms to refer to MCR practice,
such as “PR review” [162] and “patch review” [163]. Our estimation illustrates
the amount of work required to perform backward snowballing. As our focus was
conducting a systematic literature review, not a systematic mapping study, we
suggest snowballing as future work, in which both backward and forward
snowballing can be conducted as a complement to our review. In addition, our
SLR in the present form already includes several papers. Including the
snowballing approach could lead to a lengthy paper, which might be difficult
to read.

The selection and classification of primary studies might also be considered a
threat to validity. To avoid bias, we followed systematic procedures for the
search, selection, data extraction, information labeling, and data analysis.
Moreover, these tasks were conducted by the first author of this paper, and
then the outcomes were reviewed and discussed with the second author. A single
researcher analyzing the primary studies in an SLR is a practice observed in
other studies, e.g. in the work of Paiva et al. [164]. However, these are
subjective tasks, and other studies can result in different categorizations
and summarizations of the research on MCR.

##  8 Conclusion

Code review is a well-known practice of quality assurance in software
development that evolved from a structured and rigid form (i.e. software
inspection) to a flexible, tool-based, and asynchronous process, namely modern
code review (MCR). As MCR gained increasing popularity in recent years, the
practice has been largely investigated in academia. Therefore, to have a
comprehensive view of what has been done in this field, we presented in this
paper the results of a systematic literature review on MCR, which includes 139
primary studies.

We identified three main categories of studies, namely foundational studies,
proposals of novel approaches to support the practice, and evaluations of
proposed approaches. Each paper category was systematically analyzed observing
aspects relevant for each type of study. Most of the investigated work
consists of foundational studies that have been conducted to better understand
the motivations for the adoption of MCR, its challenges and benefits, and
analysis of which influence factors lead to which MCR outcomes. From the
proposals of novel approaches to support MCR, the most common are those to
help code checking and to recommend reviewers. Evaluations of MCR-supporting
approaches have been done mostly offline and few studies involving human
subjects have been conducted.

We performed a _systematic_ literature review in order to reduce the bias in
the selection and analysis of a large number of primary studies on MCR.
Although this search is large, it is not complete and, therefore, our review
may have left out other existing studies on MCR. To mitigate the limitations
of SLRs, we selected widely used digital libraries as sources, assuming that
they would contain the largest number of relevant studies. Future SLRs on MCR
may target other digital libraries or gray literature as well as cover future
years given that the research on MCR has been active to a great extent in the
recent years.

## Acknowledgements

Ingrid Nunes thanks for CNPq grants ref. 313357/2018-8 and ref. 428157/2018-1.
This study was financed in part by the Coordenação de Aperfeiçoamento de
Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001.

## References

  * Baum et al. [2016] T. Baum, O. Liskin, K. Niklas, K. Schneider,  A faceted classification scheme for change-based industrial code review processes,  in: 2016 IEEE International Conference on Software Quality, Reliability and Security (QRS), 2016, pp. 74–85. doi:10.1109/QRS.2016.19.
  * Fagan [1976] M. Fagan,  Design and code inspections to reduce errors in program development,  IBM Syst. J. 15 (1976) 182–211.
  * Bacchelli and Bird [2013] A. Bacchelli, C. Bird,  Expectations, outcomes, and challenges of modern code review,  in: Proceedings of the 2013 International Conference on Software Engineering, ICSE’13, IEEE Press, 2013, pp. 712–721.
  * Rigby and Bird [2013] P. Rigby, C. Bird,  Convergent contemporary software peer review practices,  in: Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2013, ACM, New York, NY, USA, 2013, pp. 202–212. URL: http://doi.acm.org/10.1145/2491411.2491444. doi:10.1145/2491411.2491444.
  * Thongtanunam et al. [2015] P. Thongtanunam, C. Tantithamthavorn, R. G. Kula, N. Yoshida, H. Iida, K. ichi Matsumoto,  Who should review my code? a file location-based code-reviewer recommendation approach for modern code review,  in: 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), IEEE, 2015, pp. 141–150. doi:10.1109/SANER.2015.7081824.
  * Ouni et al. [2016] A. Ouni, R. G. Kula, K. Inoue,  Search-based peer reviewers recommendation in modern code review,  in: 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME), IEEE, 2016, pp. 367–377. doi:10.1109/ICSME.2016.65.
  * Fronza et al. [2020] I. Fronza, A. Hellas, P. Ihantola, T. Mikkonen,  Code reviews, software inspections, and code walkthroughs: Systematic mapping study of research topics,  in: D. Winkler, S. Biffl, D. Mendez, J. Bergsmann (Eds.), Software Quality: Quality Intelligence in Software and Systems Engineering, Springer International Publishing, Cham, 2020, pp. 121–133.
  * Badampudi et al. [2019] D. Badampudi, R. Britto, M. Unterkalmsteiner,  Modern code reviews - preliminary results of a systematic mapping study,  in: Proceedings of the Evaluation and Assessment on Software Engineering, EASE’19, ACM, New York, NY, USA, 2019, pp. 340–345. doi:10.1145/3319008.3319354.
  * Coelho et al. [2019] F. Coelho, T. Massoni, E. L. Alves,  Refactoring-aware code review: a systematic mapping study,  in: 2019 IEEE/ACM 3rd International Workshop on Refactoring (IWoR), IEEE, 2019, pp. 63–66.
  * Nazir et al. [2020] S. Nazir, N. Fatima, S. Chuprat,  Modern code review benefits-primary findings of a systematic literature review,  in: Proceedings of the 3rd International Conference on Software Engineering and Information Management, ICSIM ’20, Association for Computing Machinery, New York, NY, USA, 2020, p. 210–215. doi:10.1145/3378936.3378954.
  * Fatima et al. [2019a] N. Fatima, S. Nazir, S. Chuprat,  Knowledge sharing, a key sustainable practice is on risk: An insight from modern code review,  in: 2019 IEEE 6th International Conference on Engineering Technologies and Applied Sciences (ICETAS), IEEE, 2019a, pp. 1–6.
  * Fatima et al. [2019b] N. Fatima, S. Nazir, S. Chuprat,  Understanding the impact of feedback on knowledge sharing in modern code review,  in: 2019 IEEE 6th International Conference on Engineering Technologies and Applied Sciences (ICETAS), IEEE, 2019b, pp. 1–5.
  * Nazir et al. [2019a] S. Nazir, N. Fatima, S. Chuprat,  Situational factors affecting software engineers sustainability: A vision of modern code review,  in: 2019 IEEE 6th International Conference on Engineering Technologies and Applied Sciences (ICETAS), IEEE, 2019a, pp. 1–6.
  * Nazir et al. [2019b] S. Nazir, N. Fatima, S. Chuprat,  Does project associated situational factors have impact on sustainability of modern code review workforce?,  in: 2019 IEEE 6th International Conference on Engineering Technologies and Applied Sciences (ICETAS), IEEE, 2019b, pp. 1–5.
  * Doğan et al. [2019] E. Doğan, E. Tüzün, K. A. Tecimer, H. A. Güvenir,  Investigating the validity of ground truth in code reviewer recommendation studies,  in: 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), IEEE, 2019, pp. 1–6.
  * Cohen [2010] J. Cohen,  Modern code review,  Making Software: What Really Works, and Why We Believe It (2010) 329–336.
  * Sadowski et al. [2018] C. Sadowski, E. Söderberg, L. Church, M. Sipko, A. Bacchelli,  Modern code review: A case study at google,  in: Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice, ICSE-SEIP ’18, ACM, New York, NY, USA, 2018, pp. 181–190. URL: http://doi.acm.org/10.1145/3183519.3183525. doi:10.1145/3183519.3183525.
  * Santos and Nunes [2018] E. W. d. Santos, I. Nunes,  Investigating the effectiveness of peer code review in distributed software development based on objective and subjective data,  Journal of Software Engineering Research and Development 6 (2018) 14.
  * Jiang et al. [2017] J. Jiang, Y. Yang, J. He, X. Blanc, L. Zhang,  Who should comment on this pull request? analyzing attributes for more accurate commenter recommendation in pull-based development,  Information and Software Technology 84 (2017) 48 – 62.
  * Kitchenham and Charters [2007] B. Kitchenham, S. Charters, Guidelines for performing systematic literature reviews in software engineering, Technical Report EBSE-2007-01, MKeele University, 2007.
  * Rahman et al. [2017] M. M. Rahman, C. K. Roy, R. G. Kula,  Predicting usefulness of code review comments using textual features and developer experience,  in: 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR), IEEE Press, 2017, pp. 215–226. doi:10.1109/MSR.2017.17.
  * Thongtanunam et al. [2014] P. Thongtanunam, R. G. Kula, A. E. C. Cruz, N. Yoshida, H. Iida,  Improving code review effectiveness through reviewer recommendations,  in: Proceedings of the 7th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE 2014, ACM, New York, NY, USA, 2014, pp. 119–122. URL: http://doi.acm.org/10.1145/2593702.2593705. doi:10.1145/2593702.2593705.
  * Hundhausen et al. [2011] C. D. Hundhausen, P. Agarwal, M. Trevisan,  Online vs. face-to-face pedagogical code reviews: An empirical comparison,  in: Proceedings of the 42Nd ACM Technical Symposium on Computer Science Education, SIGCSE ’11, ACM, New York, NY, USA, 2011, pp. 117–122. doi:10.1145/1953163.1953201.
  * Petersen and Zingaro [2018] A. Petersen, D. Zingaro,  Code reviews in large, first-year courses,  in: Proceedings of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education, ITiCSE 2018, ACM, New York, NY, USA, 2018, pp. 354–355. doi:10.1145/3197091.3205832.
  * Kawahara [2016] A. E. B. . H. A. . M. Kawahara,  Examination of coding violations focusing on their change patterns over releases,  in: 2016 23rd Asia-Pacific Software Engineering Conference (APSEC), IEEE, Hamilton, New Zealand, 2016, pp. 121–128. doi:10.1109/APSEC.2016.027.
  * Singh et al. [2017] D. Singh, V. R. Sekar, K. T. Stolee, B. Johnson,  Evaluating how static analysis tools can reduce code review effort,  in: 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), IEEE, 2017, pp. 101–105. doi:10.1109/VLHCC.2017.8103456.
  * Müller [2005] M. M. Müller,  Two controlled experiments concerning the comparison of pair programming to peer review,  Journal of Systems and Software 78 (2005) 166 – 179.
  * McIntosh et al. [2014] S. McIntosh, Y. Kamei, B. Adams, A. E. Hassan,  The impact of code review coverage and code review participation on software quality: A case study of the qt, vtk, and itk projects,  in: Proceedings of the 11th Working Conference on Mining Software Repositories, MSR 2014, ACM, New York, NY, USA, 2014, pp. 192–201. URL: http://doi.acm.org/10.1145/2597073.2597076. doi:10.1145/2597073.2597076.
  * Shimagaki et al. [2016] J. Shimagaki, Y. Kamei, S. McIntosh, A. E. Hassan, N. Ubayashi,  A study of the quality-impacting practices of modern code review at sony mobile,  in: Proceedings of the 38th International Conference on Software Engineering Companion, ICSE ’16, ACM, New York, NY, USA, 2016, pp. 212–221. URL: http://doi.acm.org/10.1145/2889160.2889243. doi:10.1145/2889160.2889243.
  * Albayrak and Davenport [2010] O. Albayrak, D. Davenport,  Impact of maintainability defects on code inspections,  in: Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM ’10, ACM, New York, NY, USA, 2010, pp. 50:1–50:4. URL: http://doi.acm.org/10.1145/1852786.1852850. doi:10.1145/1852786.1852850.
  * Beller et al. [2014] M. Beller, A. Bacchelli, A. Zaidman, E. Juergens,  Modern code reviews in open-source projects: Which problems do they fix?,  in: Proceedings of the 11th Working Conference on Mining Software Repositories, MSR 2014, ACM, New York, NY, USA, 2014, pp. 202–211. URL: http://doi.acm.org/10.1145/2597073.2597082. doi:10.1145/2597073.2597082.
  * German et al. [2018] D. M. German, G. Robles, G. Poo-Caamaño, X. Yang, H. Iida, K. Inoue,  ”was my contribution fairly reviewed?”: A framework to study the perception of fairness in modern code reviews,  in: Proceedings of the 40th International Conference on Software Engineering, ICSE ’18, ACM, New York, NY, USA, 2018, pp. 523–534. URL: http://doi.acm.org/10.1145/3180155.3180217. doi:10.1145/3180155.3180217.
  * Rahman and Roy [2017] M. M. Rahman, C. K. Roy,  Impact of continuous integration on code reviews,  in: Proceedings of the 14th International Conference on Mining Software Repositories, MSR ’17, IEEE Press, 2017, pp. 499–502. URL: https://doi.org/10.1109/MSR.2017.39. doi:10.1109/MSR.2017.39.
  * Bosu et al. [2015] A. Bosu, M. Greiler, C. Bird,  Characteristics of useful code reviews: An empirical study at microsoft,  in: Proceedings of the 12th Working Conference on Mining Software Repositories, MSR ’15, IEEE Press, 2015, pp. 146–156. URL: http://dl.acm.org/citation.cfm?id=2820518.2820538.
  * Thompson and Wagner [2017] C. Thompson, D. Wagner,  A large-scale study of modern code review and security in open source projects,  in: Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering, PROMISE, ACM, New York, NY, USA, 2017, pp. 83–92. URL: http://doi.acm.org/10.1145/3127005.3127014. doi:10.1145/3127005.3127014.
  * Yang et al. [2017] C. Yang, X. Zhang, L. Zeng, Q. Fan, G. Yin, H. Wang,  An empirical study of reviewer recommendation in pull-based development model,  in: Proceedings of the 9th Asia-Pacific Symposium on Internetware, Internetware’17, ACM, New York, NY, USA, 2017, pp. 14:1–14:6. URL: http://doi.acm.org/10.1145/3131704.3131718. doi:10.1145/3131704.3131718.
  * Kitagawa et al. [2016] N. Kitagawa, H. Hata, A. Ihara, K. Kogiso, K. Matsumoto,  Code review participation: Game theoretical modeling of reviewers in gerrit datasets,  in: Proceedings of the 9th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE ’16, ACM, New York, NY, USA, 2016, pp. 64–67. URL: http://doi.acm.org/10.1145/2897586.2897605. doi:10.1145/2897586.2897605.
  * Izquierdo-Cortazar et al. [2017] D. Izquierdo-Cortazar, N. Sekitoleko, J. M. Gonzalez-Barahona, L. Kurth,  Using metrics to track code review performance,  in: Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering, EASE’17, ACM, New York, NY, USA, 2017, pp. 214–223. URL: http://doi.acm.org/10.1145/3084226.3084247. doi:10.1145/3084226.3084247.
  * Bosu and Carver [2012] A. Bosu, J. C. Carver,  Peer code review in open source communitiesusing reviewboard,  in: Proceedings of the ACM 4th Annual Workshop on Evaluation and Usability of Programming Languages and Tools, PLATEAU ’12, ACM, New York, NY, USA, 2012, pp. 17–24. URL: http://doi.acm.org/10.1145/2414721.2414726. doi:10.1145/2414721.2414726.
  * Spadini et al. [2018] D. Spadini, M. Aniche, M.-A. Storey, M. Bruntink, A. Bacchelli,  When testing meets code review: Why and how developers review tests,  in: Proceedings of the 40th International Conference on Software Engineering, ICSE ’18, ACM, New York, NY, USA, 2018, pp. 677–687. URL: http://doi.acm.org/10.1145/3180155.3180192. doi:10.1145/3180155.3180192.
  * Baum et al. [2016] T. Baum, O. Liskin, K. Niklas, K. Schneider,  Factors influencing code review processes in industry,  in: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2016, ACM, New York, NY, USA, 2016, pp. 85–96. URL: http://doi.acm.org/10.1145/2950290.2950323. doi:10.1145/2950290.2950323.
  * Bird et al. [2015] C. Bird, T. Carnahan, M. Greiler,  Lessons learned from building and deploying a code review analytics platform,  in: 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories, MSR ’15, IEEE Press, 2015, pp. 191–201. doi:10.1109/MSR.2015.25.
  * Kononenko et al. [2016] O. Kononenko, O. Baysal, M. W. Godfrey,  Code review quality: How developers see it,  in: Proceedings of the 38th International Conference on Software Engineering, ICSE ’16, ACM, New York, NY, USA, 2016, pp. 1028–1038. URL: http://doi.acm.org/10.1145/2884781.2884840. doi:10.1145/2884781.2884840.
  * Bosu and Carver [2014] A. Bosu, J. C. Carver,  Impact of developer reputation on code review outcomes in oss projects: An empirical investigation,  in: Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM ’14, ACM, New York, NY, USA, 2014, pp. 33:1–33:10. URL: http://doi.acm.org/10.1145/2652524.2652544. doi:10.1145/2652524.2652544.
  * Thongtanunam et al. [2015] P. Thongtanunam, S. McIntosh, A. E. Hassan, H. Iida,  Investigating code review practices in defective files: An empirical study of the qt system,  in: 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories, MSR ’15, IEEE Press, 2015, pp. 168–179. doi:10.1109/MSR.2015.23.
  * Kovalenko and Bacchelli [2018] V. Kovalenko, A. Bacchelli,  Code review for newcomers: Is it different?,  in: Proceedings of the 11th International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE ’18, ACM, New York, NY, USA, 2018, pp. 29–32. URL: http://doi.acm.org/10.1145/3195836.3195842. doi:10.1145/3195836.3195842.
  * Efstathiou and Spinellis [2018] V. Efstathiou, D. Spinellis,  Code review comments: Language matters,  in: Proceedings of the 40th International Conference on Software Engineering: New Ideas and Emerging Results, ICSE-NIER ’18, ACM, New York, NY, USA, 2018, pp. 69–72. URL: http://doi.acm.org/10.1145/3183399.3183411. doi:10.1145/3183399.3183411.
  * Bosu et al. [2014] A. Bosu, J. C. Carver, M. Hafiz, P. Hilley, D. Janni,  Identifying the characteristics of vulnerable code changes: An empirical study,  in: Proceedings of the 22Nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2014, ACM, New York, NY, USA, 2014, pp. 257–268. URL: http://doi.acm.org/10.1145/2635868.2635880. doi:10.1145/2635868.2635880.
  * Begel and Vrzakova [2018] A. Begel, H. Vrzakova,  Eye movements in code review,  in: Proceedings of the Workshop on Eye Movements in Programming, EMIP ’18, ACM, New York, NY, USA, 2018, pp. 5:1–5:5. URL: http://doi.acm.org/10.1145/3216723.3216727. doi:10.1145/3216723.3216727.
  * Dunsmore et al. [2000] A. Dunsmore, M. Roper, M. Wood,  Object-oriented inspection in the face of delocalisation,  in: Proceedings of the 22Nd International Conference on Software Engineering, ICSE ’00, ACM, 2000, pp. 467–476. URL: http://doi.acm.org/10.1145/337180.337343. doi:10.1145/337180.337343.
  * Hirao et al. [2015] T. Hirao, A. Ihara, K.-i. Matsumoto,  Pilot study of collective decision-making in the code review process,  in: Proceedings of the 25th Annual International Conference on Computer Science and Software Engineering, CASCON ’15, IBM Corp., Riverton, NJ, USA, 2015, pp. 248–251. URL: http://dl.acm.org/citation.cfm?id=2886444.2886485.
  * Thongtanunam et al. [2016] P. Thongtanunam, S. McIntosh, A. E. Hassan, H. Iida,  Revisiting code ownership and its relationship with software quality in the scope of modern code review,  in: Proceedings of the 38th International Conference on Software Engineering, ICSE ’16, ACM, New York, NY, USA, 2016, pp. 1039–1050. URL: http://doi.acm.org/10.1145/2884781.2884852. doi:10.1145/2884781.2884852.
  * Floyd et al. [2017] B. Floyd, T. Santander, W. Weimer,  Decoding the representation of code in the brain: An fmri study of code review and expertise,  in: Proceedings of the 39th International Conference on Software Engineering, ICSE ’17, IEEE Press, 2017, pp. 175–186. URL: https://doi.org/10.1109/ICSE.2017.24. doi:10.1109/ICSE.2017.24.
  * Uwano et al. [2006] H. Uwano, M. Nakamura, A. Monden, K.-i. Matsumoto,  Analyzing individual performance of source code review using reviewers’ eye movement,  in: Proceedings of the 2006 Symposium on Eye Tracking Research &Amp; Applications, ETRA ’06, ACM, New York, NY, USA, 2006, pp. 133–140. URL: http://doi.acm.org/10.1145/1117309.1117357. doi:10.1145/1117309.1117357.
  * Lee and Carver [2017] A. Lee, J. C. Carver,  Are one-time contributors different?: A comparison to core and periphery developers in floss repositories,  in: 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), IEEE Press, 2017, pp. 1–10. doi:10.1109/ESEM.2017.7.
  * Meneely et al. [2014] A. Meneely, A. C. R. Tejeda, B. Spates, S. Trudeau, D. Neuberger, K. Whitlock, C. Ketant, K. Davis,  An empirical investigation of socio-technical code review metrics and security vulnerabilities,  in: Proceedings of the 6th International Workshop on Social Software Engineering, SSE 2014, ACM, New York, NY, USA, 2014, pp. 37–44. URL: http://doi.acm.org/10.1145/2661685.2661687. doi:10.1145/2661685.2661687.
  * Bosu et al. [2017] A. Bosu, J. C. Carver, C. Bird, J. Orbeck, C. Chockley,  Process aspects and social dynamics of contemporary code review: Insights from open source development and industrial practice at microsoft,  IEEE Transactions on Software Engineering 43 (2017) 56–75.
  * Sutherland and Venolia [2009] A. Sutherland, G. Venolia,  Can peer code reviews be exploited for later information needs?,  in: 2009 31st International Conference on Software Engineering - Companion Volume, IEEE, 2009, pp. 259–262. doi:10.1109/ICSE-COMPANION.2009.5070996.
  * Chandrika et al. [2017] K. R. Chandrika, J. Amudha, S. D. Sudarsan,  Recognizing eye tracking traits for source code review,  in: 2017 22nd IEEE International Conference on Emerging Technologies and Factory Automation (ETFA), IEEE, 2017, pp. 1–8. doi:10.1109/ETFA.2017.8247637.
  * Asundi and Jayant [2007] J. Asundi, R. Jayant,  Patch review processes in open source software development communities: A comparative case study,  in: Proceedings of the 40th Annual Hawaii International Conference on System Sciences, HICSS’07, IEEE Computer Society, USA, 2007, p. 166c. URL: https://doi.org/10.1109/HICSS.2007.426. doi:10.1109/HICSS.2007.426.
  * MacLeod et al. [2018] L. MacLeod, M. Greiler, M.-A. Storey, C. Bird, J. Czerwonka,  Code reviewing in the trenches: Challenges and best practices,  IEEE Software 35 (2018) 34–42.
  * Ueda et al. [2017] Y. Ueda, A. Ihara, T. Hirao, T. Ishio, K. Matsumoto,  How is if statement fixed through code review? a case study of qt project,  in: 2017 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), IEEE Press, 2017, pp. 207–213. doi:10.1109/ISSREW.2017.32.
  * Armstrong et al. [2017] F. Armstrong, F. Khomh, B. Adams,  Broadcast vs. unicast review technology: Does it matter?,  in: 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST), IEEE, 2017, pp. 219–229. doi:10.1109/ICST.2017.27.
  * Ebert et al. [2017] F. Ebert, F. Castor, N. Novielli, A. Serebrenik,  Confusion detection in code reviews,  in: 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME), IEEE, 2017, pp. 549–553. doi:10.1109/ICSME.2017.40.
  * Kononenko et al. [2015] O. Kononenko, O. Baysal, L. Guerrouj, Y. Cao, M. W. Godfrey,  Investigating code review quality: Do people and participation matter?,  in: 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME), IEEE Computer Society, 2015, pp. 111–120. doi:10.1109/ICSM.2015.7332457.
  * Bavota and Russo [2015] G. Bavota, B. Russo,  Four eyes are better than two: On the impact of code reviews on software quality,  in: 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME), IEEE, 2015, pp. 81–90. doi:10.1109/ICSM.2015.7332454.
  * Runeson and Andrews [2003] P. Runeson, A. Andrews,  Detection or isolation of defects? an experimental comparison of unit testing and code inspection,  in: 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003., IEEE Computer Society, 2003, pp. 3–13. doi:10.1109/ISSRE.2003.1251026.
  * Bosu and Carver [2013] A. Bosu, J. C. Carver,  Impact of peer code review on peer impression formation: A survey,  in: 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement, IEEE, 2013, pp. 133–142. doi:10.1109/ESEM.2013.23.
  * Liang and Mizuno [2011] J. Liang, O. Mizuno,  Analyzing involvements of reviewers through mining a code review repository,  in: 2011 Joint Conference of the 21st International Workshop on Software Measurement and the 6th International Conference on Software Process and Product Measurement, IEEE, 2011, pp. 126–132. doi:10.1109/IWSM-MENSURA.2011.33.
  * di Biase et al. [2016] M. di Biase, M. Bruntink, A. Bacchelli,  A security perspective on code review: The case of chromium,  in: 2016 IEEE 16th International Working Conference on Source Code Analysis and Manipulation (SCAM), IEEE, 2016, pp. 21–30. doi:10.1109/SCAM.2016.30.
  * Panichella et al. [2015] S. Panichella, V. Arnaoudova, M. D. Penta, G. Antoniol,  Would static analysis tools help developers with code reviews?,  in: 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), IEEE, 2015, pp. 161–170. doi:10.1109/SANER.2015.7081826.
  * Swamidurai et al. [2014] R. Swamidurai, B. Dennis, U. Kannan,  Investigating the impact of peer code review and pair programming on test-driven development,  in: IEEE SOUTHEASTCON 2014, IEEE, 2014, pp. 1–5. doi:10.1109/SECON.2014.6950664.
  * Morales et al. [2015] R. Morales, S. McIntosh, F. Khomh,  Do code review practices impact design quality? a case study of the qt, vtk, and itk projects,  in: 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), IEEE Press, 2015, pp. 171–180. doi:10.1109/SANER.2015.7081827.
  * Baysal et al. [2012] O. Baysal, O. Kononenko, R. Holmes, M. W. Godfrey,  The secret life of patches: A firefox case study,  in: 2012 19th Working Conference on Reverse Engineering, IEEE Computer Society, 2012, pp. 447–455. doi:10.1109/WCRE.2012.54.
  * Rigby et al. [2012] P. Rigby, B. Cleary, F. Painchaud, M.-A. Storey, D. German,  Contemporary peer review in action: Lessons from open source development,  IEEE Software 29 (2012) 56–61.
  * Bernhart and Grechenig [2013] M. Bernhart, T. Grechenig,  On the understanding of programs with continuous code reviews,  in: 2013 21st International Conference on Program Comprehension (ICPC), 2013, pp. 192–198. doi:10.1109/ICPC.2013.6613847.
  * Duraes et al. [2016] J. Duraes, H. Madeira, J. Castelhano, C. Duarte, M. C. Branco,  Wap: Understanding the brain at software debugging,  in: 2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE), IEEE, 2016, pp. 87–92. doi:10.1109/ISSRE.2016.53.
  * Murakami et al. [2017] Y. Murakami, M. Tsunoda, H. Uwano,  Wap: Does reviewer age affect code review performance?,  in: 2017 IEEE 28th International Symposium on Software Reliability Engineering (ISSRE), IEEE, 2017, pp. 164–169. doi:10.1109/ISSRE.2017.37.
  * Li et al. [2017] Z.-X. Li, Y. Yu, G. Yin, T. Wang, H.-M. Wang,  What are they talking about? analyzing code reviews in pull-based development model,  Journal of Computer Science and Technology 32 (2017) 1060–1075.
  * Thongtanunam et al. [2017] P. Thongtanunam, S. McIntosh, A. E. Hassan, H. Iida,  Review participation in modern code review,  Empirical Software Engineering 22 (2017) 768–817.
  * Baum et al. [2017] T. Baum, H. Leßmann, K. Schneider,  The choice of code review process: A survey on the state of the practice,  in: M. Felderer, D. Méndez Fernández, B. Turhan, M. Kalinowski, F. Sarro, D. Winkler (Eds.), Product-Focused Software Process Improvement, Springer International Publishing, Cham, 2017, pp. 111–127.
  * McIntosh et al. [2016] S. McIntosh, Y. Kamei, B. Adams, A. E. Hassan,  An empirical study of the impact of modern code review practices on software quality,  Empirical Software Engineering 21 (2016) 2146–2189.
  * Baysal et al. [2016] O. Baysal, O. Kononenko, R. Holmes, M. W. Godfrey,  Investigating technical and non-technical factors influencing modern code review,  Empirical Software Engineering 21 (2016) 932–959.
  * Hirao et al. [2016] T. Hirao, A. Ihara, Y. Ueda, P. Phannachitta, K.-i. Matsumoto,  The impact of a low level of agreement among reviewers in a code review process,  in: K. Crowston, I. Hammouda, B. Lundell, G. Robles, J. Gamalielsson, J. Lindman (Eds.), Open Source Systems: Integrating Communities, Springer International Publishing, Cham, 2016, pp. 97–110.
  * Alami et al. [2019] A. Alami, M. L. Cohn, A. Wasowski,  Why does code review work for open source software communities?,  in: 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), 2019, pp. 1073–1083.
  * Pascarella et al. [2018] L. Pascarella, D. Spadini, F. Palomba, M. Bruntink, A. Bacchelli,  Information needs in contemporary code review,  Proc. ACM Hum.-Comput. Interact. 2 (2018).
  * Spadini et al. [2019] D. Spadini, F. Palomba, T. Baum, S. Hanenberg, M. Bruntink, A. Bacchelli,  Test-driven code review: An empirical study,  in: 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), 2019, pp. 1061–1072.
  * Zanaty et al. [2018] F. E. Zanaty, T. Hirao, S. McIntosh, A. Ihara, K. Matsumoto,  An empirical study of design discussions in code review,  in: Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM ’18, Association for Computing Machinery, New York, NY, USA, 2018. URL: https://doi.org/10.1145/3239235.3239525. doi:10.1145/3239235.3239525.
  * Hirao et al. [2019] T. Hirao, S. McIntosh, A. Ihara, K. Matsumoto,  The review linkage graph for code review analytics: A recovery approach and empirical study,  in: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019, Association for Computing Machinery, New York, NY, USA, 2019, p. 578–589. URL: https://doi.org/10.1145/3338906.3338949. doi:10.1145/3338906.3338949.
  * Ram et al. [2018] A. Ram, A. A. Sawant, M. Castelluccio, A. Bacchelli,  What makes a code change easier to review: An empirical investigation on code change reviewability,  in: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2018, Association for Computing Machinery, New York, NY, USA, 2018, p. 201–212. URL: https://doi.org/10.1145/3236024.3236080. doi:10.1145/3236024.3236080.
  * Ebert et al. [2018] F. Ebert, F. Castor, N. Novielli, A. Serebrenik,  Communicative intention in code review questions,  in: 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME), IEEE, 2018, pp. 519–523.
  * An et al. [2018] L. An, F. Khomh, S. Mcintosh, M. Castelluccio,  Why did this reviewed code crash? an empirical study of mozilla firefox,  in: 2018 25th Asia-Pacific Software Engineering Conference (APSEC), IEEE, 2018, pp. 396–405.
  * Ueda et al. [2018] Y. Ueda, A. Ihara, T. Ishio, K. Matsumoto,  Impact of coding style checker on code review - a case study on the openstack projects,  in: 2018 9th International Workshop on Empirical Software Engineering in Practice (IWESEP), 2018, pp. 31–36.
  * Norikane et al. [2018] T. Norikane, A. Ihara, K. Matsumoto,  Do review feedbacks influence to a contributor’s time spent on oss projects?,  in: 2018 IEEE International Conference on Big Data, Cloud Computing, Data Science & Engineering (BCD), IEEE, 2018, pp. 109–113.
  * Ueda et al. [2019] Y. Ueda, T. Ishio, A. Ihara, K. Matsumoto,  Mining source code improvement patterns from similar code review works,  in: 2019 IEEE 13th International Workshop on Software Clones (IWSC), IEEE, 2019, pp. 13–19.
  * Zampetti et al. [2019] F. Zampetti, G. Bavota, G. Canfora, M. Di Penta,  A study on the interplay between pull request review and continuous integration builds,  in: 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER), IEEE, 2019, pp. 38–48.
  * Ebert et al. [2019] F. Ebert, F. Castor, N. Novielli, A. Serebrenik,  Confusion in code reviews: Reasons, impacts, and coping strategies,  in: 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER), 2019, pp. 49–60.
  * Jiang et al. [2019] J. Jiang, A. Mohamed, L. Zhang,  What are the characteristics of reopened pull requests? a case study on open source projects in github,  IEEE Access 7 (2019) 102751–102761.
  * Paixao and Maia [2019] M. Paixao, P. H. Maia,  Rebasing in code review considered harmful: A large-scale empirical investigation,  in: 2019 19th International Working Conference on Source Code Analysis and Manipulation (SCAM), IEEE, 2019, pp. 45–55.
  * Paul et al. [2019] R. Paul, A. Bosu, K. Z. Sultana,  Expressions of sentiments during code reviews: Male vs. female,  in: 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER), IEEE, 2019, pp. 26–37.
  * Paixao et al. [2019] M. Paixao, J. Krinke, D. Han, C. Ragkhitwetsagul, M. Harman,  The impact of code review on architectural changes,  IEEE Transactions on Software Engineering (2019).
  * El Asri et al. [2019] I. El Asri, N. Kerzazi, G. Uddin, F. Khomh, M. J. Idrissi,  An empirical study of sentiments in code reviews,  Information and Software Technology 114 (2019) 37–54.
  * Wang et al. [2019] Q. Wang, X. Xia, D. Lo, S. Li,  Why is my code change abandoned?,  Information and Software Technology 110 (2019) 108–120.
  * Baum et al. [2019] T. Baum, K. Schneider, A. Bacchelli,  Associating working memory capacity and code change ordering with code review performance,  Empirical Softw. Engg. 24 (2019) 1762–1798.
  * Ruangwan et al. [2019] S. Ruangwan, P. Thongtanunam, A. Ihara, K. Matsumoto,  The impact of human factors on the participation decision of reviewers in modern code review,  Empirical Software Engineering 24 (2019) 973–1016.
  * Yu et al. [2016] Y. Yu, H. Wang, G. Yin, T. Wang,  Reviewer recommendation for pull-requests in github: What can we learn from code review and bug assignment?,  Information and Software Technology 74 (2016) 204 – 218.
  * Müller et al. [2012] S. Müller, M. Würsch, T. Fritz, H. C. Gall,  An approach for collaborative code reviews using multi-touch technology,  in: 2012 5th International Workshop on Co-operative and Human Aspects of Software Engineering (CHASE), IEEE Press, 2012, pp. 93–99. doi:10.1109/CHASE.2012.6223031.
  * Balachandran [2013] V. Balachandran,  Reducing human effort and improving quality in peer code reviews using automatic static analysis and reviewer recommendation,  in: Proceedings of the 2013 International Conference on Software Engineering, ICSE ’13, IEEE Press, 2013, pp. 931–940.
  * Barnett et al. [2015] M. Barnett, C. Bird, J. a. Brunet, S. K. Lahiri,  Helping developers help themselves: Automatic decomposition of code review changesets,  in: Proceedings of the 37th International Conference on Software Engineering - Volume 1, ICSE ’15, IEEE Press, 2015, pp. 134–144.
  * Rahman et al. [2016] M. M. Rahman, C. K. Roy, J. A. Collins,  Correct: Code reviewer recommendation in github based on cross-project and technology experience,  in: 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C), IEEE Press, 2016, pp. 222–231.
  * Zhang et al. [2015] T. Zhang, M. Song, J. Pinedo, M. Kim,  Interactive code review for systematic changes,  in: 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, volume 1 of ICSE ’15, IEEE Press, 2015, pp. 111–122. doi:10.1109/ICSE.2015.33.
  * Sripada et al. [2016] S. K. Sripada, Y. R. Reddy, S. Khandelwal,  Architecting an extensible framework for gamifying software engineering concepts,  in: Proceedings of the 9th India Software Engineering Conference, ISEC ’16, ACM, New York, NY, USA, 2016, pp. 119–130. URL: http://doi.acm.org/10.1145/2856636.2856649. doi:10.1145/2856636.2856649.
  * Hao et al. [2013] Y. Hao, G. Li, L. Mou, L. Zhang, Z. Jin,  Mct: A tool for commenting programs by multimedia comments,  in: Proceedings of the 2013 International Conference on Software Engineering, ICSE ’13, IEEE Press, 2013, pp. 1339–1342. URL: http://dl.acm.org/citation.cfm?id=2486788.2487000.
  * Tao and Kim [2015] Y. Tao, S. Kim,  Partitioning composite code changes to facilitate code review,  in: 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories, MSR ’15, IEEE Press, 2015, pp. 180–190. doi:10.1109/MSR.2015.24.
  * Priest and Plimmer [2006] R. Priest, B. Plimmer,  Rca: Experiences with an ide annotation tool,  in: Proceedings of the 7th ACM SIGCHI New Zealand Chapter’s International Conference on Computer-human Interaction: Design Centered HCI, CHINZ ’06, ACM, New York, NY, USA, 2006, pp. 53–60. URL: http://doi.acm.org/10.1145/1152760.1152767. doi:10.1145/1152760.1152767.
  * Soltanifar et al. [2016] B. Soltanifar, A. Erdem, A. Bener,  Predicting defectiveness of software patches,  in: Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM ’16, ACM, New York, NY, USA, 2016, pp. 22:1–22:10. URL: http://doi.acm.org/10.1145/2961111.2962601. doi:10.1145/2961111.2962601.
  * Duley et al. [2010] A. Duley, C. Spandikow, M. Kim,  A program differencing algorithm for verilog hdl,  in: Proceedings of the IEEE/ACM International Conference on Automated Software Engineering, ASE ’10, ACM, New York, NY, USA, 2010, pp. 477–486. URL: http://doi.acm.org/10.1145/1858996.1859093. doi:10.1145/1858996.1859093.
  * Kalyan et al. [2016] A. Kalyan, M. Chiam, J. Sun, S. Manoharan,  A collaborative code review platform for github,  in: 2016 21st International Conference on Engineering of Complex Computer Systems (ICECCS), IEEE, 2016, pp. 191–196. doi:10.1109/ICECCS.2016.032.
  * Ge et al. [2017] X. Ge, S. Sarkar, J. Witschey, E. Murphy-Hill,  Refactoring-aware code review,  in: 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), IEEE, 2017, pp. 71–79. doi:10.1109/VLHCC.2017.8103453.
  * Zanjani et al. [2016] M. B. Zanjani, H. Kagdi, C. Bird,  Automatically recommending peer reviewers in modern code review,  IEEE Transactions on Software Engineering 42 (2016) 530–543.
  * Tymchuk et al. [2015] Y. Tymchuk, A. Mocci, M. Lanza,  Code review: Veni, vidi, vici,  in: 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), IEEE, 2015, pp. 151–160. doi:10.1109/SANER.2015.7081825.
  * Ahmed et al. [2017] T. Ahmed, A. Bosu, A. Iqbal, S. Rahimi,  Senticr: A customized sentiment analysis tool for code review interactions,  in: Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering, ASE 2017, IEEE Press, 2017, pp. 106–111.
  * Baum et al. [2017] T. Baum, K. Schneider, A. Bacchelli,  On the optimal order of reading source code changes for review,  in: 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME), IEEE, Shanghai, China, 2017, pp. 329–340. doi:10.1109/ICSME.2017.28.
  * Nagoya et al. [2005] F. Nagoya, S. Liu, Y. Chen,  A tool and case study for specification-based program review,  in: 29th Annual International Computer Software and Applications Conference (COMPSAC’05), volume 1, IEEE Press, 2005, pp. 375–380 Vol. 2. doi:10.1109/COMPSAC.2005.36.
  * Lanubile and Mallardo [2002] F. Lanubile, T. Mallardo,  Tool support for distributed inspection,  in: Proceedings of the 26th International Computer Software and Applications Conference on Prolonging Software Life: Development and Redevelopment, COMPSAC ’02, IEEE Computer Society, 2002, pp. 1071–1076. URL: http://dl.acm.org/citation.cfm?id=645984.675578.
  * Harel and Kantorowitz [2005] A. Harel, E. Kantorowitz,  Estimating the number of faults remaining in software code documents inspected with iterative code reviews,  in: IEEE International Conference on Software - Science, Technology Engineering (SwSTE’05), IEEE Computer Society, 2005, pp. 151–160. doi:10.1109/SWSTE.2005.1.
  * Pangsakulyanont et al. [2014] T. Pangsakulyanont, P. Thongtanunam, D. Port, H. Iida,  Assessing mcr discussion usefulness using semantic similarity,  in: 2014 6th International Workshop on Empirical Software Engineering in Practice, 2014, pp. 49–54. doi:10.1109/IWESEP.2014.11.
  * Xia et al. [2017] Z. Xia, H. Sun, J. Jiang, X. Wang, X. Liu,  A hybrid approach to code reviewer recommendation with collaborative filtering,  in: 2017 6th International Workshop on Software Mining, IEEE, 2017, pp. 24–31. doi:10.1109/SOFTWAREMINING.2017.8100850.
  * Zhang et al. [2011] X. Zhang, B. Dorn, W. Jester, J. V. Pelt, G. Gaeta, D. Firpo,  Design and implementation of java sniper: A community-based software code review web solution,  in: 2011 44th Hawaii International Conference on System Sciences, IEEE Computer Society, 2011, pp. 1–10. doi:10.1109/HICSS.2011.145.
  * Mishra and Sureka [2014] R. Mishra, A. Sureka,  Mining peer code review system for computing effort and contribution metrics for patch reviewers,  in: 2014 IEEE 4th Workshop on Mining Unstructured Data, IEEE Computer Society, 2014, pp. 11–15. doi:10.1109/MUD.2014.11.
  * Menarini et al. [2017] M. Menarini, Y. Yan, W. G. Griswold,  Semantics-assisted code review: An efficient tool chain and a user study,  in: 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE), IEEE Press, 2017, pp. 554–565. doi:10.1109/ASE.2017.8115666.
  * Perry et al. [2002] D. E. Perry, A. Porter, M. W. Wade, L. G. Votta, J. Perpich,  Reducing inspection interval in large-scale software development,  IEEE Transactions on Software Engineering 28 (2002) 695–705.
  * Wang et al. [2017] C. Wang, X. Xie, P. Liang, J. Xuan,  Multi-perspective visualization to assist code change review,  in: 2017 24th Asia-Pacific Software Engineering Conference (APSEC), IEEE Press, 2017, pp. 564–569. doi:10.1109/APSEC.2017.66.
  * Xia et al. [2015] X. Xia, D. Lo, X. Wang, X. Yang,  Who should review this change?: Putting text and file location analyses together for more accurate recommendations,  in: 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME), IEEE Press, 2015, pp. 261–270. doi:10.1109/ICSM.2015.7332472.
  * Aman [2013] H. Aman,  0-1 programming model-based method for planning code review using bug fix history,  2013 20th Asia-Pacific Software Engineering Conference (APSEC) 2 (2013) 37–42.
  * Fejzer et al. [2018] M. Fejzer, P. Przymus, K. Stencel,  Profile based recommendation of code reviewers,  Journal of Intelligent Information Systems 50 (2018) 597–619.
  * Fan et al. [2018] Y. Fan, X. Xia, D. Lo, S. Li,  Early prediction of merged code changes to prioritize reviewing tasks,  Empirical Software Engineering 23 (2018) 3346–3393.
  * Luna Freire et al. [2018] V. d. C. Luna Freire, J. Brunet, J. C. A. de Figueiredo,  Automatic decomposition of java open source pull requests: A replication study,  in: A. M. Tjoa, L. Bellatreche, S. Biffl, J. van Leeuwen, J. Wiedermann (Eds.), SOFSEM 2018: Theory and Practice of Computer Science, Springer International Publishing, Cham, 2018, pp. 255–268.
  * Baum and Schneider [2016] T. Baum, K. Schneider,  On the need for a new generation of code review tools,  in: P. Abrahamsson, A. Jedlitschka, A. Nguyen Duc, M. Felderer, S. Amasaki, T. Mikkonen (Eds.), Product-Focused Software Process Improvement, Springer International Publishing, Cham, 2016, pp. 301–308.
  * Asthana et al. [2019] S. Asthana, R. Kumar, R. Bhagwan, C. Bird, C. Bansal, C. Maddila, S. Mehta, B. Ashok,  Whodo: Automating reviewer suggestions at scale,  in: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019, Association for Computing Machinery, New York, NY, USA, 2019, pp. 937–945. URL: https://doi.org/10.1145/3338906.3340449. doi:10.1145/3338906.3340449.
  * Huang et al. [2018] Y. Huang, N. Jia, X. Chen, K. Hong, Z. Zheng,  Salient-class location: Help developers understand code change in code review,  in: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2018, Association for Computing Machinery, New York, NY, USA, 2018, p. 770–774. URL: https://doi.org/10.1145/3236024.3264841. doi:10.1145/3236024.3264841.
  * Wang et al. [2019] M. Wang, Z. Lin, Y. Zou, B. Xie,  Cora: Decomposing and describing tangled code changes for reviewer,  in: Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering, ASE ’19, IEEE Press, 2019, p. 1050–1061. URL: https://doi.org/10.1109/ASE.2019.00101. doi:10.1109/ASE.2019.00101.
  * Huang et al. [2018] K. Huang, B. Chen, X. Peng, D. Zhou, Y. Wang, Y. Liu, W. Zhao,  Cldiff: Generating concise linked code differences,  in: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ASE 2018, Association for Computing Machinery, New York, NY, USA, 2018, p. 679–690. URL: https://doi.org/10.1145/3238147.3238219. doi:10.1145/3238147.3238219.
  * Wen et al. [2018] R. Wen, D. Gilbert, M. G. Roche, S. McIntosh,  Blimp tracer: Integrating build impact analysis with code review,  in: 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2018, pp. 685–694.
  * Hanam et al. [2019] Q. Hanam, A. Mesbah, R. Holmes,  Aiding code change understanding with semantic change impact analysis,  in: 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2019, pp. 202–212.
  * Liao et al. [2019] Z. Liao, Z. Wu, J. Wu, Y. Zhang, J. Liu, J. Long,  Tirr: A code reviewer recommendation algorithm with topic model and reviewer influence,  in: 2019 IEEE Global Communications Conference (GLOBECOM), 2019, pp. 1–6. doi:10.1109/GLOBECOM38437.2019.9014249.
  * Jiang et al. [2019] J. Jiang, D. Lo, J. Zheng, X. Xia, Y. Yang, L. Zhang,  Who should make decision on this pull request? analyzing time-decaying relationships and file similarities for integrator prediction,  Journal of Systems and Software 154 (2019) 196 – 210.
  * Guo et al. [2019] B. Guo, Y.-W. Kwon, M. Song,  Decomposing composite changes for code review and regression test selection in evolving software,  Journal of Computer Science and Technology 34 (2019) 416–436.
  * Khandelwal et al. [2017] S. Khandelwal, S. K. Sripada, Y. R. Reddy,  Impact of gamification on code review process: An experimental study,  in: Proceedings of the 10th Innovations in Software Engineering Conference, ISEC ’17, ACM, New York, NY, USA, 2017, pp. 122–126. URL: http://doi.acm.org/10.1145/3021460.3021474. doi:10.1145/3021460.3021474.
  * Baum et al. [2016] T. Baum, F. Kortum, K. Schneider, A. Brack, J. Schauder,  Comparing pre commit reviews and post commit reviews using process simulation,  in: Proceedings of the International Conference on Software and Systems Process, ICSSP ’16, ACM, New York, NY, USA, 2016, pp. 26–35. URL: http://doi.acm.org/10.1145/2904354.2904362. doi:10.1145/2904354.2904362.
  * Peng et al. [2018] Z. Peng, J. Yoo, M. Xia, S. Kim, X. Ma,  Exploring how software developers work with mention bot in github,  in: Proceedings of the Sixth International Symposium of Chinese CHI, ChineseCHI ’18, ACM, New York, NY, USA, 2018, pp. 152–155. URL: http://doi.acm.org/10.1145/3202667.3202694. doi:10.1145/3202667.3202694.
  * Hannebauer et al. [2016] C. Hannebauer, M. Patalas, S. Stünkel, V. Gruhn,  Automatically recommending code reviewers based on their expertise: An empirical comparison,  in: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, ASE 2016, ACM, New York, NY, USA, 2016, pp. 99–110. URL: http://doi.acm.org/10.1145/2970276.2970306. doi:10.1145/2970276.2970306.
  * Mizuno and Liang [2015] O. Mizuno, J. Liang, Does a Code Review Tool Evolve as the Developer Intended?, Springer International Publishing, Cham, 2015, pp. 59–74. doi:10.1007/978-3-319-11265-7_5.
  * Runeson and Wohlin [1998] P. Runeson, C. Wohlin,  An experimental evaluation of an experience-based capture-recapture method in software code inspections,  Empirical Software Engineering 3 (1998) 381–406.
  * Gibbs [2007] G. R. Gibbs,  Thematic coding and categorizing,  in: Analyzing Qualitative Data, SAGE Publications Ltd., London, 2007\.
  * Kerzazi and El Asri [2016] N. Kerzazi, I. El Asri,  Who can help to review this piece of code?,  in: H. Afsarmanesh, L. M. Camarinha-Matos, A. Lucas Soares (Eds.), Collaboration in a Hyperconnected World, Springer International Publishing, Cham, 2016, pp. 289–301.
  * Kagdi et al. [2008] H. Kagdi, M. Hammad, J. I. Maletic,  Who can help me with this source code change?,  in: 2008 IEEE International Conference on Software Maintenance, 2008, pp. 157–166. doi:10.1109/ICSM.2008.4658064.
  * Jiang et al. [2015] J. Jiang, J.-H. He, X.-Y. Chen,  Coredevrec: Automatic core member recommendation for contribution evaluation,  Journal of Computer Science and Technology 30 (2015) 998–1016.
  * Kovalenko et al. [2020] V. Kovalenko, N. Tintarev, E. Pasynkov, C. Bird, A. Bacchelli,  Does Reviewer Recommendation Help Developers?,  IEEE Transactions on Software Engineering 46 (2020) 710–731.
  * Rigby et al. [2008] P. C. Rigby, D. M. German, M.-A. Storey,  Open source software peer review practices: A case study of the apache server,  in: Proceedings of the 30th International Conference on Software Engineering, ICSE ’08, Association for Computing Machinery, New York, NY, USA, 2008, p. 541–550. URL: https://doi.org/10.1145/1368088.1368162. doi:10.1145/1368088.1368162.
  * Wohlin [2014] C. Wohlin,  Guidelines for snowballing in systematic literature studies and a replication in software engineering,  in: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, EASE ’14, Association for Computing Machinery, New York, NY, USA, 2014. URL: https://doi.org/10.1145/2601248.2601268. doi:10.1145/2601248.2601268.
  * Ying et al. [2016] H. Ying, L. Chen, T. Liang, J. Wu,  Earec: Leveraging expertise and authority for pull-request reviewer recommendation in github,  in: 2016 IEEE/ACM 3rd International Workshop on CrowdSourcing in Software Engineering (CSI-SE), 2016, pp. 29–35. doi:10.1109/CSI-SE.2016.013.
  * Nurolahzade et al. [2009] M. Nurolahzade, S. M. Nasehi, S. H. Khandkar, S. Rawal,  The role of patch review in software evolution: An analysis of the mozilla firefox,  in: Proceedings of the Joint International and Annual ERCIM Workshops on Principles of Software Evolution (IWPSE) and Software Evolution (Evol) Workshops, IWPSE-Evol ’09, Association for Computing Machinery, New York, NY, USA, 2009, p. 9–18. URL: https://doi.org/10.1145/1595808.1595813. doi:10.1145/1595808.1595813.
  * Paiva et al. [2021] D. M. B. Paiva, A. P. Freire, R. P. de Mattos Fortes,  Accessibility and software engineering processes: A systematic literature review,  Journal of Systems and Software 171 (2021) 110819.