# A Universal Machine Learning Model for Elemental Grain Boundary Energies

Weike Ye  Hui Zheng  Chi Chen  Shyue Ping Ong<EMAIL_ADDRESS>Department
of NanoEngineering, University of California San Diego, 9500 Gilman Dr, Mail
Code 0448, La Jolla, CA 92093-0448, United States

###### Abstract

The grain boundary (GB) energy has a profound influence on the grain growth
and properties of polycrystalline metals. Here, we show that the energy of a
GB, normalized by the bulk cohesive energy, can be described purely by four
geometric features. By machine learning on a large computed database of 361
small $\Sigma$ ($\Sigma<10$) GBs of more than 50 metals, we develop a model
that can predict the grain boundary energies to within a mean absolute error
of 0.13 J m-2 . More importantly, this universal GB energy model can be
extrapolated to the energies of high $\Sigma$ GBs without loss in accuracy.
These results highlight the importance of capturing fundamental scaling
physics and domain knowledge in the design of interpretable, extrapolatable
machine learning models for materials science.

###### keywords:

Grain boundary energy , Modeling , Density Functional Theory (DFT) , Machine
learning

Grain boundaries (GBs) play an important role in determining the strength,
toughness, and corrosion resistance of materials[1, 2]. A key property of a GB
is its energy, which determines grain growth and the GB distribution. While
the GB energy can be accurately calculated using electronic structure methods
such as density functional theory (DFT) calculations, the requirement for
large supercells to model the inherently low symmetry GB structure limits such
computationally intensive approaches to relatively small $\Sigma$ GBs.
Nevertheless, substantial databases of GB energies and other properties have
been developed using high-throughput DFT. For example, the GB database
(GBDB)[3] developed by the present authors contains the calculated GB energies
and work of separation of more than 50 elemental metals for both tilt and
twist GBs up to $\Sigma=9$.

Alternatively, machine learning (ML) techniques have emerged as a means to
develop models that can directly predict the GB energy from compositional and
structural features.[4, 5, 6, 7, 8] However, existing ML models targeting
elemental GBs are limited in scope by chemistry or structure type, such as
face-centered cubic (fcc) Cu [4], Ni [5, 6], Al [7], or body-centered cubic
(bcc) Fe [8] systems. These limitations are primarily a result of the choice
of data source; these prior works have been developed using data sets computed
using embedded atom method (EAM) potentials. While much less computationally
intensive than DFT methods, EAM calculations are far less accurate, especially
for non-fcc metals,[3] and EAM potentials are available for only a limited
subset of elements. Furthermore, the majority of these prior works rely on
featurization approaches such as the Smooth Overlap of Atomic Positions
(SOAP)[5], [6] and the pair-correlation function (PCF)[7] that generate a
large number of features which do not provide direct interpretability using
commonly-used GB descriptions.

In this letter, we outline a physics-informed approach to develop a universal
ML model for the GB energy of all metals, rather than for a subset of metals.
We will demonstrate that the energy of small $\Sigma$ GBs of metals can be
predicted to within a mean absolute error (MAE) of 0.13 J m-2 using a gradient
boosting regression (GBR)[9] model of the cohesive energy and four geometric
GB features only. More critically, the same model can be extrapolated, without
retraining, to predict the energies of high $\Sigma$ GBs with a comparable MAE
of 0.12 J m-2 . This work provides not only a means to rapidly predict the GB
energies of any element, but also highlights the importance of choosing
appropriate target normalization and features for the development of
interpretable and extrapolatable ML.

The critical starting point is in re-evaluating the choice of target for our
ML GB model. While prior works have attempted to directly predict the absolute
GB energy, we do not believe this to be an optimal choice of target. The GB
energy $E_{GB}$ is the excess energy of the GB compared to the bulk per unit
area, which can be obtained from computational models as:

$E_{GB}=\frac{E_{GB,supercell}-n\cdot E^{atom}_{bulk}}{2A}$ (1)

where $E_{GB,supercell}$ is the energy of the supercell GB model, $n$ is the
number of atoms in the GB model, $E^{atom}_{bulk}$ is the energy per atom of
the bulk, $A$ is the area of the GB and the factor of 2 accounts for the fact
that there are two GBs per supercell model. The physical interpretation is
that $E_{GB}$ is related to the energy necessary to break or stretch bonds at
the GB from their bulk equilibrium configuration. This energy to stretch or
break bonds scales with the cohesive energy of the metal $E_{coh}$ (see Figure
LABEL:fig:ecoh),[10] which ranges from $\sim 1.1$ eV atom-1 for the alkali
metals to $\sim 8.9$ eV atom-1 for tungsten. To remove this chemical scaling
effect, we have elected to use the normalized GB energy
$\hat{E_{GB}}=E_{GB}/E_{coh}$ as our choice of target.

Based on the coincident-site-lattice (CSL) theory [11, 12], the GB can be
specified at a macroscopic level by five degrees of freedom (DOF): two DOFs to
define the plane normal of the GB (or alternatively the Miller indices
$(hkl)$), two DOFs to define the rotation axis ($[uvw]$) and one DOF to define
the misorientation angle ($\theta$). Miller indices, which are defined to be
integers by convention, are non-optimal for a regression task. As such, the
$(hkl)$ and $[uvw]$ were converted to the inter-planar distances of the GB
plane ($d_{GB}$) and the normal plane to the rotation axis ($d_{rot}$),
respectively. The cosine of the misorientation angle ($\cos\theta$) was used
instead of the misorientation angle itself.

In addition to these geometric GB features, we included three additional
features related to bond stretching and breaking at the GB, which were
partially inspired by prior works in the literature[13]. To describe the bond
deformation, we used both the average bond length in the GB supercell,
$\bar{BL}=\sum_{i=1}^{n}(BL_{GB}^{i})/n$, and the average change in bond
lengths between the GB supercell and its bulk conventional lattice,
$\bar{\Delta BL}=\sum_{i=1}^{n}(BL_{GB}^{i}-BL_{0})/n$, where $BL_{GB}^{i}$ is
the bond length of the $i$th bond in the GB supercell, $BL_{0}$ is the bond
length in the corresponding bulk conventional structure, and $n$ is the number
of bonds counted in the GB supercell. Here, the bonds are identified by
performing a local environment analysis via a Voronoi tessellation-based
algorithm implemented in the Python Materials Genomics (pymatgen) package[14].
A positive (negative) $\bar{\Delta BL}$ indicates overall bond stretching
(compressing) at the GB. According to the Read-Shockley dislocation model[15],
$E_{GB}$ of GBs with small misorientation angles is proportional to the shear
modulus $G$. Ratanaphan et al.[10] have also shown previously that the GB
energies of bcc Mo and Fe are related to $G$. The multi-linear regression
models developed by Zheng et al. [3] extended this conclusion to more bcc,
fcc, and hexagonal closest packed (hcp) metals. Therefore, we include the DFT
Voigt-Reuss-Hill shear modulus $G$ from the Materials Project as the final
feature. Figure 1(a) summarizes the preliminary set of six features considered
in work.

An initial dataset of GB energies was obtained from the GBDB[3], which
contains the energies of 316 GBs of 53 elements in fcc, bcc, hcp and double-
hcp (dhcp) structures, after excluding Lu, Eu, and Hg due to the lack of the
bulk elastic data. The $\Sigma$s of the GBs range from 3 to 9. The maximum
Miller index (MMI) for the rotation axis and the grain boundary plane are 1
and 3, respectively. In this work, we extended the MMI for the grain boundary
plane to 4 for 5 elements (As, Nb, Pt, Cu, and Ir), which added 5 more GBs to
the initial data. Interested readers are referred to ref 3 for the details on
the GB structure generation and computational methods. For the model
development, the 321 GBs with $\Sigma\leq 9$ were divided into training (258
GBs) and test (63 GBs) sets using stratified random sampling. The training
data comprises 80% of the GBs from each element with more than one GB, and the
single GB from the remaining elements.

A preliminary model selection process was performed in an automatic fashion
utilizing a tree-based pipeline optimization tool (TPOT)[16] with the six
initial features (refer to Supplementary Materials for details). The suggested
pipeline from TPOT is a decision-tree-based GBR model preceded by a polynomial
feature transformation step. We recognize that there is a risk of redundancy
in our selected features, e.g., $G$ has a direct relationship with $E_{coh}$,
which was used to normalize the GB energy. Therefore, a comprehensive search
for the “best subset” of the six initial features was performed using the
suggested pipeline. As shown in Figure 1(b), it was found that the model’s
test MAE converges with only four features, ($d_{GB}$, $\cos\theta$,
$\bar{\Delta BL}$, $\bar{BL}$) being the optimal set with the lowest test MAE.
Further increase in the number of features leads to small increases in test
MAE and decreases in training MAE, i.e., evidence of overfitting. It is worth
noting that the only non-geometric feature, the shear modulus $G$, is not
within the optimal subset, indicating that our proposed normalization with the
cohesive energy has effectively addressed the chemical dependence of $E_{GB}$.

As illustrated in Figure 2(a), the final optimal ML pipeline starts from the
GB initial structure and the corresponding bulk conventional structure and
executes the following steps: (i) featurizes the input GB structure, (ii)
applies a polynomial transformation of the features up to degree 2, (iii)
makes a prediction based on the trained decision-tree-based GBR model.
Following this pipeline, we achieved the MAEs for $\hat{E_{GB}}$ at 1.21
$\times$ 10-7 and 3.38 $\times$ 10-7 mol m-2 for the training and the test
data, respectively (Figure 2(b)), which translate to MAEs for $E_{GB}$ at 0.04
and 0.13 J m-2 (Figure 2(c)). The distributions of the absolute errors for
each element show that 48 out of the 55 elements have MAEs($E_{GB}$) less than
0.1 J m-2 (Figure LABEL:fig:error(b)). Actinide and lanthanide elements such
as Th, Ac, Yb, and Ce display the highest errors in $E_{GB}$, which may be a
result of less accurate DFT calculations without proper Hubbard U values
applied to account for the strong self-interaction of the f electrons. Some
transition metals as Cr and Fe also exhibit higher MAEs, which we attribute to
the uncertainty in the ground state magnetic ordering at the GBs in the DFT
calculations.

Figure 1: Feature Engineering. (a) Initial feature candidates based on the
macroscopic geometry, microscopic bonding environment in the GB supercell, and
the shear modulus of the elemental bulk system. (b) Best subset selection of
features. There are a total of 63 possible subsets of 6 features. The
swarmplot summarizes the performances of all the subsets, categorized by the
number of features ($n_{f}$). The blue diamond point refers to the global
optimal subset with the lowest test MAE in the predicted normalized grain
boundary energy $\hat{E_{GB}}$. Figure 2: The machine learning pipeline and
the performance. (a) The schematic illustration of the pipeline developed in
this work. (b) and (c) are parity plots demonstrating the pipeline’s
performances on $\hat{E_{GB}}$ and $E_{GB}$, respectively.

Figure 3 shows the permutation importance of the four input features, which is
obtained by randomly shuffling the values of one feature and calculating the
decrease in the MAE of the ML pipeline.[17] The feature with highest
importance is $\bar{BL}$, the average bond length, while the remaining three
GB features ($d_{GB}$, $\cos\theta$ and $\bar{\Delta BL}$) have similar
feature importances. This is consistent with previous observations by
Guziewski et al.[18] where the grain boundary energies of silicon carbide are
better described by microscopic descriptors than the macroscopic ones.
Furthermore, the fact that the $d_{GB}$ has higher importance than
$\cos\theta$ echos with the conclusion drawn by Rohrer et al. [19], who found
that variations in the grain boundary plane induce a greater change in the
energy than the variations in the misorientation.

Figure 3: Permutation feature importance of the four geometric input features.

Thus far, the optimal model presented has been trained only on low $\Sigma$
GBs that can be readily computed using standard DFT computations. If it were
limited to low $\Sigma$ GBs, such a model would be of limited utility. To
demonstrate its ability to extrapolate to high $\Sigma$ GBs, which require far
more expensive DFT computations, an additional 40 calculations on Ta, Pd, Cu,
Pt and Li GBs with $\Sigma$ from 17 to 66 with MMI of the grain boundary plane
$\leq$ 8 and MMI of the rotation axis $\leq$ 1 were performed (Figure
LABEL:fig:data). The model, without further retraining, achieved an
MAE($E_{GB}$) of 0.12 J m-2 on this data set (Figure 4(b)), commensurate with
the test error of the small $\Sigma$ GB test set.

Figure 4: Performance of GB model on unseen high $\sigma$ GBs. Parity plot of
predicted versus DFT (a) normalized $\hat{E_{GB}}$, (b) absolute $E_{GB}$.

To summarize, we have developed a physics-informed ML pipeline that predicts
GB energies for more than 50 metals to within 0.13 J $m^{-2}$. This ML model
can be applied to high $\Sigma$ GBs without loss in accuracy. A key innovation
of our approach is to normalize the GB energy by the elemental cohesive energy
to remove chemical scaling effects, resulting in a target that can be modelled
purely using geometric and structural features. We believe this conceptual
approach is general and is key to the development of ML models that are
interpretable and extrapolatable. We have not attempted to model non-elemental
GBs in this work given the lack of a sufficiently large dataset for model
training. Nevertheless, we can speculate on the applicability of the same
approach to GBs of non-elements, e.g., alloys and ceramics. Non-elemental GBs
are much more complex, given that there may be compositional differences
between the GB and the bulk region, e.g., preferential segregation of certain
elements, etc [20]. While we believe some form of target normalization with an
averaged bond energy descriptor, e.g., formation energies per atom, etc.,
would still be useful to remove large energetic scaling effects, it is likely
be less effective than in the elemental case. Furthermore, a purely geometric
/ structural descriptor set would not be sufficient and compositional degrees
of freedom would need to be included as well. Regardless of these limitations,
we believe the conceptual framework developed in this work is sound and should
be extended to other properties that have a well-defined scaling relationship
with bond strength, e.g., elastic constants, etc.

All GB data and models have been made available in the Github repository of
the open-source Materials Machine Learning (maml) package at
https://github.com/materialsvirtuallab/maml.

## Acknowledgement

This work is supported by the Materials Project, funded by the U.S. Department
of Energy, Office of Science, Office of Basic Energy Sciences, Materials
Sciences and Engineering Division under Contract no. DE-AC02-05-CH11231:
Materials Project program KC23MP. The authors also acknowledge computational
resources provided by the National Energy Research Scientific Computing Centre
(NERSC), the Triton Shared Computing Cluster (TSCC) at the University of
California, San Diego, and the Extreme Science and Engineering Discovery
Environment (XSEDE) supported by National Science Foundation under grant no.
ACI-1053575.

## References

  * Tan et al. [2008] L. Tan, K. Sridharan, T. R. Allen, R. K. Nanstad, D. A. McClintock, Journal of Nuclear Materials 374 (2008) 270–280. doi:10.1016/j.jnucmat.2007.08.015.
  * Shimada et al. [2002] M. Shimada, H. Kokawa, Z. J. Wang, Y. S. Sato, I. Karibe, Acta Materialia 50 (2002) 2331–2341. doi:10.1016/S1359-6454(02)00064-2.
  * Zheng et al. [2020] H. Zheng, X.-G. Li, R. Tran, C. Chen, M. Horton, D. Winston, K. A. Persson, S. P. Ong, Acta Materialia 186 (2020) 40–49. doi:10.1016/j.actamat.2019.12.030.
  * Kiyohara et al. [2016] S. Kiyohara, H. Oda, T. Miyata, T. Mizoguchi, Science Advances 2 (2016) e1600746. doi:10.1126/sciadv.1600746.
  * Rosenbrock et al. [2017] C. W. Rosenbrock, E. R. Homer, G. Csányi, G. L. W. Hart, npj Computational Materials 3 (2017) 1–7. doi:10.1038/s41524-017-0027-x.
  * Snow et al. [2019] B. D. Snow, D. D. Doty, O. K. Johnson, Front. Mater. 6 (2019). doi:10.3389/fmats.2019.00120.
  * Gomberg et al. [2017] J. A. Gomberg, A. J. Medford, S. R. Kalidindi, Acta Materialia 133 (2017) 100–108. doi:10.1016/j.actamat.2017.05.009.
  * Echeverri Restrepo et al. [2014] S. Echeverri Restrepo, S. Tamayo Giraldo, B. J. Thijsse, Computational Materials Science 86 (2014) 170–173. doi:10.1016/j.commatsci.2014.01.039.
  * Friedman [2001] J. H. Friedman, The Annals of Statistics 29 (2001) 1189–1232. doi:10.1214/aos/1013203451.
  * Ratanaphan et al. [2015] S. Ratanaphan, D. L. Olmsted, V. V. Bulatov, E. A. Holm, A. D. Rollett, G. S. Rohrer, Acta Materialia 88 (2015) 346–354. doi:10.1016/j.actamat.2015.01.069.
  * Grimmer et al. [1974] H. Grimmer, W. Bollmann, D. H. Warrington, Acta Cryst A 30 (1974) 197–207. doi:10.1107/S056773947400043X.
  * Wolf and Lutsko [1989] D. Wolf, J. F. Lutsko, Zeitschrift für Kristallographie - Crystalline Materials 189 (1989) 239–262. doi:10.1524/zkri.1989.189.14.239.
  * Gibson and Schuh [2016] M. A. Gibson, C. A. Schuh, Scripta Materialia 113 (2016) 55–58. doi:10.1016/j.scriptamat.2015.09.041.
  * Ong et al. [2013] S. P. Ong, W. D. Richards, A. Jain, G. Hautier, M. Kocher, S. Cholia, D. Gunter, V. L. Chevrier, K. A. Persson, G. Ceder, Computational Materials Science 68 (2013) 314–319. doi:10.1016/j.commatsci.2012.10.028.
  * Read and Shockley [1950] W. T. Read, W. Shockley, Phys. Rev. 78 (1950) 275–289. doi:10.1103/PhysRev.78.275.
  * Olson et al. [2016] R. S. Olson, N. Bartley, R. J. Urbanowicz, J. H. Moore, in: Proceedings of the Genetic and Evolutionary Computation Conference 2016, GECCO ’16, Association for Computing Machinery, New York, NY, USA, 2016, pp. 485–492. doi:10.1145/2908812.2908918.
  * Breiman [2001] L. Breiman, Machine Learning 45 (2001) 5–32. doi:10.1023/A:1010933404324.
  * Guziewski et al. [2021] M. Guziewski, D. Montes de Oca Zapiain, R. Dingreville, S. P. Coleman, ACS Appl. Mater. Interfaces 13 (2021) 3311–3324. doi:10.1021/acsami.0c15980.
  * Rohrer [2011] G. S. Rohrer, J Mater Sci 46 (2011) 5881–5895. doi:10.1007/s10853-011-5677-3.
  * Li et al. [2020] X.-G. Li, C. Chen, H. Zheng, Y. Zuo, S. P. Ong, npj Comput Mater 6 (2020) 1–10. doi:10.1038/s41524-020-0339-0.


# Deduplicating and Ranking Solution Programs for Suggesting Reference
Solutions

Atsushi Shirafuji  Graduate Department of Computer
and Information Systems
University of AizuAizu-WakamatsuJapan<EMAIL_ADDRESS>and  Yutaka
Watanobe  Department of Computer Science
and Engineering
University of AizuAizu-WakamatsuJapan<EMAIL_ADDRESS>
###### Abstract.

Referring to solution programs written by other users is helpful for learners
in programming education. However, current online judge systems just list all
solution programs submitted by users for references, and the programs are
sorted based on the submission date and time, execution time, or user rating,
ignoring to what extent the programs can be helpful to be referenced. In
addition, users struggle to refer to a variety of solution approaches since
there are too many duplicated and near-duplicated programs.

To motivate learners to refer to various solutions to learn better solution
approaches, in this paper, we propose an approach to deduplicate and rank
common solution programs in each programming problem. Inspired by the nature
that the many-duplicated program adopts a more common approach and can be a
general reference, we remove the near-duplicated solution programs and rank
the unique programs based on the duplicate count.

The experiments on the solution programs submitted to a real-world online
judge system demonstrate that the number of programs is reduced by 60.20%,
whereas the baseline only reduces by 29.59% after the deduplication, meaning
that users only need to refer to 39.80% of programs on average. Furthermore,
our analysis shows that top-10 ranked programs cover 29.95% of programs on
average, indicating that users can grasp 29.95% of solution approaches by
referring to only 10 programs. The proposed approach shows the potential of
reducing the learners’ burden of referring to too many solutions and
motivating them to learn a variety of solution approaches.

solution programs, near-duplicates, deduplication, online judge systems,
programming education

††ccs: Applied computing E-learning††ccs: Applied computing Learning
management systems

##  1\. Introduction

Programming novices learn correct and better approaches to solving programming
problems by referring to various solution programs written by expert
programmers or instructors. Among Japanese programmers, _shakyō-style
learning_ (originally meaning the sutra copying, hand-copying of Buddhist
sutras), which is manually typing the reference source code, is considered one
of the ways of learning programming for novices.

Using e-learning systems in programming education is becoming more popular as
online judge systems can automatically and immediately evaluate users’
submitted programs (Wasik et al., 2018; Watanobe et al., 2022). As some online
judge systems publicly show the submitted solution programs, such as Aizu
Online Judge (AOJ)111https://onlinejudge.u-aizu.ac.jp/. and
AtCoder222https://atcoder.jp/., users can refer to other users’ solution
approaches.

Referring to solution programs written by other users is important for
learners in programming education. However, current online judge systems just
list all solution programs submitted by users for references, and the programs
are ranked based on several criteria, such as the submission date and time,
execution time or memory, and user ratings, which ignore to what extent the
program can be a reference. The effective (e.g., faster and less computational
cost) programs can be unreadable since experienced programmers may write
complicated programs for effectiveness, leveraging the advanced algorithms or
data structures. Similarly, higher-rating users can write more complicated
programs for effectiveness. Therefore, the performance of programs and the
user ratings cannot be good metrics for sorting solution programs for novice
programmers to use as references.

Moreover, users struggle to refer to a variety of solution approaches since
there are too many duplicated and near-duplicated programs. For example, a set
of introductory programming problems, ITP1, provided on AOJ, each programming
problem has 7,430 solution programs on average in Python. It is clear that
novice programmers struggle to find the standard solution approaches from the
thousands of solution programs when they want to refer to better solution
approaches after solving or giving up on solving the problem.

To motivate the learners to refer to various reference solutions to learn
better solution approaches, in this paper, we propose an approach to
deduplicate and rank solution programs submitted by users in each programming
problem. The approach is inspired by the nature that a many-duplicated program
adopts a more common approach and can be a reference. We remove near-
duplicated programs and rank the unique (deduplicated) programs based on the
duplicate count.

The advantages of the proposed approach are as follows.

  * •

Deduplication makes it easy to browse a variety of solution approaches.

  * •

The suggested programs are ensured to be correct.

  * •

Through the code formatter, programs with consistent readability are stably
suggested.

The quantitative evaluation of the proposed approach shows that it
significantly reduces the number of programs after deduplication, with an
average reduction of 60.20%. This means that users only need to refer to
39.80% of the programs on average. Furthermore, the top-10 ranked programs
cover an average of 29.95% of the solution programs, indicating that users can
grasp nearly 30% of the solution approaches by referring to only 10 programs.
Although the top-ranked programs do not necessarily indicate that they are the
most helpful programs, these results demonstrate the effectiveness of the
proposed approach in reducing the learners’ burden of referring to too many
solutions and improving the coverage of solution programs.

The contributions of this work are as follows.

  * •

The proposal of an approach to deduplicate and rank solution programs in
programming education, which helps learners find common reference solutions.

  * •

The evaluation of the proposed approach using a real-world dataset of
introductory programming problems shows its effectiveness in reducing the
number of programs and improving the coverage of solution programs.

  * •

Identifying limitations and future directions for the proposed approach,
including the code format and the need for a sufficient number of solution
programs for reliable suggestions.

##  2\. Related Work

Figure 1. Illustration of the proposed approach.

There are several prior works that are related to our work on deduplicating
and ranking solution programs for suggesting reference solutions. We divide
the related work into three categories: source code deduplication, online
judge systems, and code-to-code generation.

###  2.1. Source Code Deduplication

Source code deduplication is a task to identify and remove duplicate or near-
duplicate source code, which is semantically or syntactically similar source
code. This task is usually performed in creating large-scale source code
datasets (Puri et al., 2021; Kocetkov et al., 2023) or training machine
learning models on source code (Fried et al., 2023; Allal et al., 2023) to
avoid redundancy, overfitting, and memorization (Allamanis, 2019; Lee et al.,
2022; Kandpal et al., 2022). Not only based on the exact match but also the
code clone detection task can be applied for deduplication. Code clone
detection algorithms typically consider the syntactic or semantic similarity
between code snippets and apply various techniques such as tokenization and
abstract syntax tree (AST) analysis. Several tools have been developed for
this task, such as CloneDR (Baxter et al., 1998), CCFinder (Kamiya et al.,
2002), and DECKARD (Jiang et al., 2007). While these techniques are useful for
identifying duplicate or near-duplicate programs, they are not directly
applicable to our task of suggesting reference solutions to programming
problems because our goal is to restore the complete working program among the
near-duplicate programs.

###  2.2. Online Judge Systems

Another related area of research is online judge systems, which are platforms
that allow users to submit solution programs to a set of programming problems
and automatically evaluate the submitted programs (Wasik et al., 2018). Many
online judge systems, such as AOJ and AtCoder, make the submitted solutions
publicly available, allowing other users to refer to them for educational and
reference purposes. This allows learners to study and learn from existing
solution programs and explore different approaches to solving the same
problem. However, the existing online judge systems do not provide effective
mechanisms for deduplicating and ranking the solution programs. They often
sort the programs based on criteria such as submission date and time,
execution time, or user ratings333AtCoder’s Rating System,
https://www.dropbox.com/s/ixci4amralioaif/rating.pdf.444Ranking Criteria -
Aizu Online Judge, https://judge.u-aizu.ac.jp/onlinejudge/rating_note.jsp.,
which may not reflect the quality or popularity of the programs. Our work aims
to address this limitation and enhance the usefulness of these solution
programs by proposing an approach to deduplicate and rank the solution
programs based on their popularity.

###  2.3. Code-to-Code Generation

Code-to-code generation is a task to generate code based on a given input
code. This task is closely related to our work as we aim to generate reference
solution programs from the user-submitted solution programs. It is important
to clearly distinguish our work from code repair or error correction tasks
(Rahman et al., 2021; Matsumoto et al., 2021), generating correct programs
from incorrect programs, whereas our work only focuses on correct programs in
both input and output. There has been some research on code-to-code generation
tasks, such as code recommendation and code refactoring (Luan et al., 2019;
Silavong et al., 2022). These tasks involve generating code similar to the
given input code but improved in some way, such as being more efficient,
readable, or maintainable. For example, the work by Madaan et al. (Madaan et
al., 2023a) demonstrated the use of large language models (i.e., Codex (Chen
et al., 2021) and CodeGen (Nijkamp et al., 2023)) for improving the execution
time of given programs. Similarly, Madaan et al. (Madaan et al., 2023b)
proposed improving code readability by iteratively refining code generation.
While these approaches are similar to ours in modifying existing code to
improve it, they focus on improving individual programs rather than
deduplicating and ranking solution programs for suggesting reference
solutions. In our work, we perform deduplication and ranking based on the
duplicate count, leveraging the AST to capture the structural similarity of
the solution programs. Furthermore, our approach does not require any input
programs from the user, as we aim to suggest existing solution programs rather
than generating new ones.

In summary, while there has been prior work on source code deduplication,
online judge systems, and code-to-code generation, our work focuses on the
specific problem of deduplicating and ranking solution programs in programming
education. We propose an approach that considers the semantic equivalence and
popularity of the programs to provide helpful reference solutions for
learners.

##  3\. Methodology

The proposed approach is primarily inspired by the nature that the more
duplicated programs are more common and can be references for learners. To
suggest the more common and helpful reference solution programs to users, we
propose an approach to remove duplicated programs and re-rank the user-
submitted solution programs based on their popularity (i.e., duplicated
count). The proposed approach is illustrated in Figure 1. The approach can be
divided into three phases: normalization, deduplication, and ranking.

###  3.1. Normalization

Figure 2. Illustration of normalization, which is divided into four phases:
tokenization, anonymization, detokenization, and formatting.

In the normalization phase, to better capture the syntactic equality of the
programs in the deduplication phase, we normalize the superficial
representations of the programs. In more detail, we ignore the white lines,
white spaces, comments, docstrings, and identifier names (e.g., variable,
function, and class names). The normalization can be further divided into four
phases: tokenization, anonymization, detokenization, and formatting. Figure 2
illustrates the normalization.

Although several prior works used AST to capture the more abstract structure
of the programs (Yamaguchi et al., 2012; Yoshizawa and Watanobe, 2018), using
source code rather than AST is more suitable in this work since we need to
reconstruct the source code from AST to suggest the complete working source
code for users. However, we also use the AST in the anonymization phase to
decide which name should be anonymized (e.g., functions defined in the
standard library should not be anonymized, whereas the user-defined functions
should be anonymized).

####  3.1.1. Tokenization

For tokenization, we develop a lightweight tokenizer using the standard
tokenize555https://docs.python.org/3/library/tokenize.html. library in Python
to remove comments and docstrings completely. In general, docstrings cannot be
removed in tokenization because a docstring can be considered a string literal
not assigned to any variables. Similarly, comments using string literals
cannot be removed. Therefore, we develop a tokenizer to support removing the
comments and docstrings by removing the string literals not assigned to any
variables.

####  3.1.2. Anonymization

To normalize the identifier names (e.g., variable, function, and class names),
we employ anonymization. The anonymization is inspired by the normalization
phase of prior works (Bhoopchand et al., 2016; Terada and Watanobe, 2021),
leveraging the ast666https://docs.python.org/3/library/ast.html. and
astor777https://github.com/berkerpeksag/astor. libraries.

Although the normalized programs only use the anonymized variable names, we
also collect the original ones. We create a map of anonymized to original
identifier names (hereinafter, identifier map) to restore the original
variable names from the anonymized ones, called identifier-mapping. This
identifier map will be used in the final ranking phase in Section 3.3 to
restore the most popular variable names from the duplicated programs.

####  3.1.3. Detokenization & Formatting

Detokenization is to restore the source code from the normalized tokens. This
reconstruction is a process to back into human-readable source code after
normalization, performed at the token level.

After the detokenization, we apply a code formatter to format the detokenized
source code. In the detokenization phase, concatenating each token with a
space can make the source code unreadable because of excessive spaces.
Although this does not pose a semantic error, suggesting source code that
adopts an uncommon format could be harmful to learners. We employ the
yapf888https://github.com/google/yapf. library as the code formatter.

###  3.2. Deduplication

After the normalization, we deduplicate the normalized programs based on the
exact match, which considers the programs to be the same if the normalized
programs are exactly the same.

The duplicated programs are removed in this phase. We refer to the programs
after the deduplication as unique programs. For each unique program, we count
the number of duplicated programs and collect the identifier maps for the next
ranking phase.

###  3.3. Ranking

The ranking phase has two steps to make a list of common reference programs:
(1) program ranking and (2) identifier ranking. Program ranking is to make a
list of common programs by sorting the unique programs for each problem based
on the duplicate count of the program. Identifier ranking is to make a list of
common identifier names by sorting the identifier maps for each unique program
based on the duplicate count of the identifier map. Both rankings consider the
popularity based on duplicate count.

In the identifier ranking, we count the duplicated identifier names using the
identifier maps. To keep the consistency of using the identifiers and to avoid
altering the original code, the identifier map is only considered a duplicate
if all the identifiers are the same. For instance, identifier maps such as
$\\{\text{VAR01}=x,\text{VAR02}=y\\}$, and
$\\{\text{VAR01}=x,\text{VAR02}=s\\}$ are considered different due to the
difference in the variable VAR02. If we count duplicates for individual
variables, we may select $\\{\text{VAR01}=s,\text{VAR02}=s\\}$, which does not
appear in any solution programs and breaks the behavior. We apply this method
as it eliminates the risk of selecting inconsistent identifier names and
guarantees to suggest a complete working program.

##  4\. Experiment

We use introductory programming problems and apply the proposed methodology to
demonstrate how it contributes to suggesting better solution programs with
less user burden in referring to too many programs.

###  4.1. Dataset

For the target programming problems, we use a set of introductory programming
problems provided on AOJ (Watanobe, 2004). AOJ is one of the popular online
judge systems where users submit programs to solve given programming problems
(Wasik et al., 2018). AOJ currently stores approximately 8 million programs
submitted by 100 thousand users for 3 thousand programming problems (Watanobe
et al., 2022).

We target a course named Introduction to Programming I
(ITP1)999https://onlinejudge.u-aizu.ac.jp/courses/lesson/2/ITP1/all.. The
course has 44 introductory programming problems, ranging from standard
input/output to class definition.

The submitted programs include not only correct programs but also wrong or
incomplete programs. Each submission has a verdict, an evaluation result of
executing hidden test cases on the judge system. For instance, Accepted (AC)
indicates a correct program, Runtime Error (RE) indicates an incorrect program
failed in execution, and Wrong Answer (WA) also indicates an incorrect program
that the output result is wrong. There are more
verdicts101010https://onlinejudge.u-aizu.ac.jp/judges_replies.. However, we
only use the submissions judged as AC in this work. In addition, we target
Python 3 programs.

Moreover, we only use public programs, which users set their code policy to
make the submitted programs publicly available, following the Terms of
Use111111https://onlinejudge.u-aizu.ac.jp/term_of_use. of AOJ. The public
programs can be used for research or educational purposes and are collected in
popular datasets, such as CodeNet (Puri et al., 2021) and CodeContests (Li et
al., 2022).

Table 1. Dataset statistics of 44 problems. Solutions indicates correct programs, Available indicates publicly available correct programs, and Outliers indicates programs failed in normalization. We use Valid in the following sections. | #Submissions | #Solutions | #Available | #Outliers | #Valid
---|---|---|---|---|---
Total | 868,143 | 326,933 | 150,497 | 14 | 150,483
Mean | 19,731 | 7,430 | 3,420 | 0 | 3,420
Std | 13,462 | 5,204 | 2,191 | 1 | 2,191

Table 1 shows the number of programs used in this work. ITP1 had 868,143
submissions at the time. After filtering only AC programs, the number of
solution programs is 326,933. Still, it contains private programs and programs
that are not archived yet. After filtering only public and archived programs,
the number of available solution programs is 150,497. Fourteen programs are
further filtered out as outliers because they fail in conversion into AST.
Finally, the number of valid solution programs is 150,483. On average, each
programming problem has 3,420 solution programs.

Table 2. Decrease of the number of programs after deduplication on 44 problems and their average. Baseline is based on the exact match deduplication without normalization. #Unique indicates the number of unique programs after deduplication, and ratio indicates the ratio of unique programs out of solution programs. #Unique and ratio are smaller the better. Problem | #Solutions | Baseline | Ours
---|---|---|---
|  | #Unique | Ratio | #Unique | Ratio
ITP1_1_A | 10,699 | 380 | 3.55% | 81 | 0.76%
ITP1_1_B | 9,044 | 1,954 | 21.61% | 350 | 3.87%
ITP1_1_C | 8,004 | 4,649 | 58.08% | 1,446 | 18.07%
ITP1_1_D | 6,782 | 5,154 | 76.00% | 2,223 | 32.78%
ITP1_2_A | 5,954 | 4,000 | 67.18% | 1,011 | 16.98%
ITP1_2_B | 5,240 | 3,104 | 59.24% | 661 | 12.61%
ITP1_2_C | 5,434 | 4,254 | 78.28% | 2,035 | 37.45%
ITP1_2_D | 4,768 | 4,130 | 86.62% | 1,937 | 40.63%
ITP1_3_A | 5,191 | 1,541 | 29.69% | 234 | 4.51%
ITP1_3_B | 4,784 | 3,905 | 81.63% | 1,797 | 37.56%
ITP1_3_C | 4,727 | 3,983 | 84.26% | 2,331 | 49.31%
ITP1_3_D | 4,207 | 3,604 | 85.67% | 1,185 | 28.17%
ITP1_4_A | 4,246 | 3,470 | 81.72% | 1,866 | 43.95%
ITP1_4_B | 3,927 | 3,511 | 89.41% | 2,338 | 59.54%
ITP1_4_C | 3,999 | 3,462 | 86.57% | 2,009 | 50.24%
ITP1_4_D | 4,036 | 3,199 | 79.26% | 1,340 | 33.20%
ITP1_5_A | 3,770 | 3,195 | 84.75% | 1,576 | 41.80%
ITP1_5_B | 3,326 | 3,037 | 91.31% | 1,929 | 58.00%
ITP1_5_C | 3,204 | 2,923 | 91.23% | 1,979 | 61.77%
ITP1_5_D | 2,432 | 1,973 | 81.13% | 1,478 | 60.77%
ITP1_6_A | 3,379 | 2,754 | 81.50% | 1,495 | 44.24%
ITP1_6_B | 3,070 | 2,707 | 88.18% | 2,190 | 71.34%
ITP1_6_C | 2,551 | 2,227 | 87.30% | 1,847 | 72.40%
ITP1_6_D | 2,250 | 2,014 | 89.51% | 1,487 | 66.09%
ITP1_7_A | 2,379 | 2,133 | 89.66% | 1,658 | 69.69%
ITP1_7_B | 3,633 | 3,224 | 88.74% | 2,360 | 64.96%
ITP1_7_C | 2,032 | 1,822 | 89.67% | 1,574 | 77.46%
ITP1_7_D | 1,831 | 1,597 | 87.22% | 1,331 | 72.69%
ITP1_8_A | 2,166 | 1,252 | 57.80% | 570 | 26.32%
ITP1_8_B | 2,218 | 1,995 | 89.95% | 1,089 | 49.10%
ITP1_8_C | 2,082 | 1,753 | 84.20% | 1,435 | 68.92%
ITP1_8_D | 1,986 | 1,677 | 84.44% | 857 | 43.15%
ITP1_9_A | 1,920 | 1,699 | 88.49% | 1,269 | 66.09%
ITP1_9_B | 1,720 | 1,522 | 88.49% | 854 | 49.65%
ITP1_9_C | 1,696 | 1,553 | 91.57% | 1,089 | 64.21%
ITP1_9_D | 1,514 | 1,376 | 90.89% | 1,196 | 79.00%
ITP1_10_A | 1,773 | 1,542 | 86.97% | 862 | 48.62%
ITP1_10_B | 1,543 | 1,418 | 91.90% | 1,306 | 84.64%
ITP1_10_C | 1,524 | 1,404 | 92.13% | 1,229 | 80.64%
ITP1_10_D | 1,417 | 1,277 | 90.12% | 1,165 | 82.22%
ITP1_11_A | 1,292 | 1,173 | 90.79% | 1,025 | 79.33%
ITP1_11_B | 904 | 835 | 92.37% | 750 | 82.96%
ITP1_11_C | 938 | 815 | 86.89% | 741 | 79.00%
ITP1_11_D | 891 | 759 | 85.19% | 683 | 76.66%
Total | 150,483 | 105,956 | — | 59,868 | —
Mean | 3,420 | 2,408 | 70.41% | 1,361 | 39.80%

###  4.2. Results

####  4.2.1. Deduplication

Table 2 shows the decrease in the number of programs after deduplication,
compared with the baseline approach. The baseline skips the normalization
phase and directly deduplicates based on the exact match of the raw source
code. Compared with the baseline, our proposed approach can remove more near-
duplicated programs, resulting in fewer unique programs. On average, the
proposed approach reduces the number of programs from 3,420 to 1,361, which
reduces by 60.20%, whereas the baseline approach only reduces from 3,420 to
2,408, which reduces by 29.59%. Our approach reduces the number of programs by
43.48% compared with the baseline. This suggests that users can grasp more
solution approaches by referring to fewer programs.

Figure 3. The comparison between the baseline and our proposed approaches of
the ratio of unique programs out of solution programs. The smaller the better.

As shown in Figure 3, we observe that the initial problems have a higher rate
of reduction (i.e., a lower ratio of unique programs) in the number of
programs. This is because, in ITP1, the easier problems are located at the
beginning and tend to have simple solution approaches. Conversely, the
reduction is significantly low for more challenging problems like ITP1_11,
where a class definition is required to solve. We find that the proposed
approach is highly effective on easier programming problems, whereas it is
less effective on more complex programming problems with more diversity in
solution approaches.

####  4.2.2. Ranking

Figure 4. Average solution coverage on top-$n$ ranked programs on 44 problems.
The higher the better.

Figure 4 shows the coverage of solution programs by top-$n$ ranked unique
programs, comparing with the baseline approach based on the exact match
deduplication without normalization. The average coverage of 13.16% for the
top-1 program indicates that the users can grasp 13.16% of the solution
programs by referring to that one program. Similarly, the coverage of 29.95%
for the top-10 programs indicates that the users can grasp approximately 30%
of the solution programs by referring to only 10 programs. This shows the
contribution of the proposed approach for users to refer to fewer programs to
grasp solution approaches used to solve a programming problem.

However, the increase in coverage with an increase in $N$ does not necessarily
show a good result. Although a higher coverage indicates that users can grasp
more solution approaches, an increase in $N$ implies an increase in the user
burden of referring to $N$ solutions. Therefore, this is a trade-off
relationship. In future work, we expect higher coverage with smaller $N$.

####  4.2.3. Qualitative Evaluation

As the qualitative evaluation, we manually evaluate the deduplicated
(suggested) programs. In each problem, the system suggests top-5 programs with
top-3 identifiers for each program.

Figure 5. Examples of top-5 programs for ITP1_3_A.

##### Helpfulness:

Figure 5 shows an example of top-5 programs for ITP1_3_A, which is to output
“Hello World” 1,000 times. Overall, the suggested programs are mostly readable
and can be helpful references. In the example, users can instantly grasp that
the programming problem can be solved using a for-loop or while-loop with a
print function.

However, some programs can still be considered near-duplicates as some use the
same approaches with a slight difference. In addition, we find that the top-
ranked programs do not necessarily indicate that they are the most helpful
programs. In the example, the most helpful solution is not the 1st-ranked
program, as the 3rd-ranked program defines the unused loop variable $i$ as an
anonymous variable _ (underscore). Similarly, the 4th-ranked program is better
than the 2nd-ranked program as it uses an augmented assignment operator += to
update the variable $i$.

While there is no guarantee that the best solution approaches will be
suggested, it can provide common and helpful solution programs with a certain
degree of reliability.

##### Code Readability:

For the readability of the suggested programs, we find that the normalization
phase negatively impacted the code format, although we used a code formatter
in the final phase. This is primarily due to removing tokens such as comments,
white lines, and white spaces in the normalization. Normalization is necessary
to better detect the near-duplicated programs. Although the code formatter
added white lines and spaces if required, we find that the original programs
are more readable than the suggested programs. Furthermore, removing comments
and docstrings leaves the code without helpful explanations of functions and
variables, making it less comprehensible for other users. Although it would be
beneficial to incorporate comments back into the code, directly using user-
written comments can be challenging. Further improvement of the approach can
be delegated to the task of generating natural language comments for the given
program.

##  5\. Limitation

Although the proposed approach has various advantages described in Section 1,
it still has the following limitations.

First, it requires a certain amount of solution programs already submitted by
users for reliable suggestions. The number of user solutions influences the
ranking based on the number of duplicates. Therefore, it cannot be applied to,
or the suggestions can be less helpful in, recently released programming
problems. However, this also indicates that the reliability can be increased
as the number of solutions increases.

Second, it currently supports Python programming language only because of the
dependency on the programming language in the normalization phase (e.g.,
tokenizer and formatter). However, the fundamental idea of the proposed
approach is versatile enough to be applied to any programming language if the
normalization can be supported for the language,

To the best of our knowledge, this work is the first investigation on
deduplicating solution programs for suggesting common solution programs as
references. These limitations are expected to be mitigated in future work.

##  6\. Conclusion

In this paper, we proposed an approach to deduplicate and rank solution
programs in programming education, with the aim of suggesting common and
helpful reference solutions to learners. We performed experiments using a
dataset of introductory programming problems from Aizu Online Judge to
evaluate the effectiveness of the approach.

The experimental results showed that the approach significantly reduced the
number of programs after deduplication compared to a baseline approach, with
an average reduction of 60.20%. This means that users only need to refer to
39.80% of the programs to grasp all the solution approaches on average.
Furthermore, we found that the top-10 ranked programs covered an average of
29.95% of the programs. This demonstrates that users can grasp nearly 30% of
the solution approaches by referring to only 10 programs.

The qualitative evaluation of the suggested programs also confirmed that they
were mostly readable and could serve as helpful reference solutions. However,
we identified some limitations of the approach, such as the code format and
the need for a sufficient number of solution programs for reliable
suggestions.

In conclusion, our proposed approach shows promise in reducing the burden on
learners to refer to too many solutions and motivating them to learn a variety
of better approaches. Future work includes incorporating the proposed approach
into a real-world online judge system and evaluating its effectiveness in an
actual educational setting.

###### Acknowledgements.

This work was supported by the Japan Society for the Promotion of Science
(JSPS) KAKENHI Grant Number JP23H03508.

## References

  * (1)
  * Allal et al. (2023) Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, et al. 2023\.  SantaCoder: don’t reach for the stars!  (2023).  arXiv:2301.03988
  * Allamanis (2019) Miltiadis Allamanis. 2019\.  The Adverse Effects of Code Duplication in Machine Learning Models of Code. In _Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software_ _(Onward! 2019)_. 143–153.  https://doi.org/10.1145/3359591.3359735
  * Baxter et al. (1998) Ira D. Baxter, Andrew Yahin, Leonardo Moura, Marcelo Sant’Anna, and Leorraine Bier. 1998\.  Clone detection using abstract syntax trees. In _Proceedings of the International Conference on Software Maintenance_ _(ICSM)_. 368–377.  https://doi.org/10.1109/ICSM.1998.738528
  * Bhoopchand et al. (2016) Avishkar Bhoopchand, Tim Rocktäschel, Earl Barr, and Sebastian Riedel. 2016.  Learning Python Code Suggestion with a Sparse Pointer Network.  (2016).  arXiv:1611.08307
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, et al. 2021\.  Evaluating Large Language Models Trained on Code.  (2021).  arXiv:2107.03374
  * Fried et al. (2023) Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, et al. 2023\.  InCoder: A Generative Model for Code Infilling and Synthesis. In _Proceedings of the 11th International Conference on Learning Representations_ _(ICLR)_.  https://openreview.net/forum?id=hQwb-lbM6EL
  * Jiang et al. (2007) Lingxiao Jiang, Ghassan Misherghi, Zhendong Su, and Stephane Glondu. 2007.  DECKARD: Scalable and Accurate Tree-Based Detection of Code Clones. In _Proceedings of the 29th International Conference on Software Engineering_ _(ICSE)_. 96–105.  https://doi.org/10.1109/ICSE.2007.30
  * Kamiya et al. (2002) Toshihiro Kamiya, Shinji Kusumoto, and Katsuro Inoue. 2002\.  CCFinder: a multilinguistic token-based code clone detection system for large scale source code.  _IEEE Transactions on Software Engineering_ 28, 7 (2002), 654–670.  https://doi.org/10.1109/TSE.2002.1019480
  * Kandpal et al. (2022) Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022\.  Deduplicating Training Data Mitigates Privacy Risks in Language Models. In _Proceedings of the 39th International Conference on Machine Learning_ _(ICML, Vol. 162)_. 10697–10707.  https://proceedings.mlr.press/v162/kandpal22a.html
  * Kocetkov et al. (2023) Denis Kocetkov, Raymond Li, Loubna Ben allal, Jia LI, Chenghao Mou, et al. 2023\.  The Stack: 3 TB of permissively licensed source code.  _Transactions on Machine Learning Research_ (2023).  https://openreview.net/forum?id=pxpbTdUEpD
  * Lee et al. (2022) Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, et al. 2022\.  Deduplicating Training Data Makes Language Models Better. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 8424–8445.  https://doi.org/10.18653/v1/2022.acl-long.577
  * Li et al. (2022) Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, et al. 2022\.  Competition-level code generation with AlphaCode.  _Science_ 378, 6624 (2022), 1092–1097.  https://doi.org/10.1126/science.abq1158
  * Luan et al. (2019) Sifei Luan, Di Yang, Celeste Barnaby, Koushik Sen, and Satish Chandra. 2019.  Aroma: Code Recommendation via Structural Code Search.  _Proc. ACM Program. Lang._ 3, OOPSLA, Article 152 (2019), 28 pages.  https://doi.org/10.1145/3360578
  * Madaan et al. (2023a) Aman Madaan, Alexander Shypula, Uri Alon, Milad Hashemi, Parthasarathy Ranganathan, et al. 2023a.  Learning Performance-Improving Code Edits.  (2023).  arXiv:2302.07867
  * Madaan et al. (2023b) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, et al. 2023b.  Self-Refine: Iterative Refinement with Self-Feedback.  (2023).  arXiv:2303.17651
  * Matsumoto et al. (2021) Taku Matsumoto, Yutaka Watanobe, and Keita Nakamura. 2021\.  A Model with Iterative Trials for Correcting Logic Errors in Source Code.  _Applied Sciences_ 11, 11 (2021).  https://doi.org/10.3390/app11114755
  * Nijkamp et al. (2023) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, et al. 2023\.  CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. In _Proceedings of the 11th International Conference on Learning Representations_ _(ICLR)_.  https://openreview.net/forum?id=iaYcJKpY2B_
  * Puri et al. (2021) Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, et al. 2021\.  CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks. In _Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks (Round 2)_.  https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/a5bfc9e07964f8dddeb95fc584cd965d-Abstract-round2.html
  * Rahman et al. (2021) Md. Mostafizer Rahman, Yutaka Watanobe, and Keita Nakamura. 2021\.  A Bidirectional LSTM Language Model for Code Evaluation and Repair.  _Symmetry_ 13, 2 (2021).  https://doi.org/10.3390/sym13020247
  * Silavong et al. (2022) Fran Silavong, Sean Moran, Antonios Georgiadis, Rohan Saphal, and Robert Otter. 2022\.  Senatus: A Fast and Accurate Code-to-Code Recommendation Engine. In _Proceedings of the 19th International Conference on Mining Software Repositories_ _(MSR)_. 511–523.  https://doi.org/10.1145/3524842.3527947
  * Terada and Watanobe (2021) Kenta Terada and Yutaka Watanobe. 2021.  Code completion for programming education based on deep learning.  _International Journal of Computational Intelligence Studies_ 10, 2-3 (2021), 78–98.  https://doi.org/10.1504/IJCISTUDIES.2021.115424
  * Wasik et al. (2018) Szymon Wasik, Maciej Antczak, Jan Badura, Artur Laskowski, and Tomasz Sternal. 2018.  A Survey on Online Judge Systems and Their Applications.  _ACM Comput. Surv._ 51, 1, Article 3 (2018), 34 pages.  https://doi.org/10.1145/3143560
  * Watanobe (2004) Yutaka Watanobe. 2004\.  Aizu Online Judge.  https://onlinejudge.u-aizu.ac.jp
  * Watanobe et al. (2022) Yutaka Watanobe, Md. Mostafizer Rahman, Taku Matsumoto, Uday Kiran Rage, and Penugonda Ravikumar. 2022.  Online Judge System: Requirements, Architecture, and Experiences.  _International Journal of Software Engineering and Knowledge Engineering_ 32, 4 (2022), 1–30.  https://doi.org/10.1142/S0218194022500346
  * Yamaguchi et al. (2012) Fabian Yamaguchi, Markus Lottmann, and Konrad Rieck. 2012\.  Generalized Vulnerability Extrapolation Using Abstract Syntax Trees. In _Proceedings of the 28th Annual Computer Security Applications Conference_ _(ACSAC)_. 359–368.  https://doi.org/10.1145/2420950.2421003
  * Yoshizawa and Watanobe (2018) Yuto Yoshizawa and Yutaka Watanobe. 2018.  Logic Error Detection Algorithm for Novice Programmers based on Structure Pattern and Error Degree. In _Proceedings of the 2018 9th International Conference on Awareness Science and Technology_ _(iCAST)_. 297–301.  https://doi.org/10.1109/ICAwST.2018.8517171