# Exploring Automated Code Evaluation Systems and Resources for Code Analysis:
A Comprehensive Survey

Md. Mostafizer Rahman<EMAIL_ADDRESS>0000-0001-9368-7638 Dhaka
University of Engineering & Technology, GazipurBangladesh1707 The University
of Aizu, JapanAizuwakamatsuJapan ,  Yutaka Watanobe  The University of
AizuAizuwakamatsuJapan<EMAIL_ADDRESS>,  Atsushi Shirafuji  The
University of AizuAizuwakamatsuJapan<EMAIL_ADDRESS>and  Mohamed
Hamada  The University of AizuAizuwakamatsuJapan<EMAIL_ADDRESS>
(2018)

###### Abstract.

The automated code evaluation system (AES) is mainly designed to reliably
assess user-submitted code. The code is compiled and then tested in a unified
environment with predefined input and output test cases. Due to their
extensive range of applications and the accumulation of valuable resources,
AESs are becoming increasingly popular. Research on the application of AES and
their real-world resource exploration for diverse coding tasks is still
lacking. In this study, we conducted a comprehensive survey on AESs and their
resources. This survey explores the application areas of AESs, available
resources, and resource utilization for coding tasks. AESs are categorized
into programming contests, programming learning and education, recruitment,
online compilers, and additional modules, depending on their application. We
explore the available datasets and other resources of these systems for
research, analysis, and coding tasks. The success of machine learning models
for inference procedures depends primarily on the purity of the data, where
the accumulated real-life data (e.g., codes and submission logs) from AESs can
be a valuable treasure. Moreover, we provide an overview of machine learning-
driven coding tasks, such as bug detection, code review, comprehension,
refactoring, search, representation, and repair. These tasks are performed
using real-life datasets. In addition, we briefly discuss the Aizu Online
Judge platform as a real example of an AES from the perspectives of system
design (hardware and software), operation (competition and education), and
research. This is due to the scalability of the AOJ platform (programming
education, competitions, and practice), open internal features (hardware and
software), attention from the research community, open source data (e.g.,
solution codes and submission documents), and transparency. We also analyze
the overall performance of this system and the perceived challenges over the
years.

Resources of Online Judge, datasets, Machine Learning, Aizu Online Judge

††copyright: acmcopyright††journalyear: 2018††doi: XXXXXXX.XXXXXXX††journal:
JACM††journalvolume: 37††journalnumber: 4††article: 111††publicationmonth:
8††ccs: General and reference Code Assessment††ccs: Evaluation platform Online
Judge††ccs: Code Analysis and dataset Machine Learnig††ccs: Networks Online
Computing

##  1\. Introduction

Competitive programming contests (CPCs) are usually held online or offline
(local area network), where participants try to solve programming problems
according to given specifications. CPCs, also called programming sports, are
recognized worldwide. Many educational institutions and multinational tech
giants (e.g., Google, Facebook, etc.) foster these programming events.
Generally, CPC organizers provide a set of mathematical or logical problems
for participants. Typically, problems are related to categories such as data
structures, string analysis, graph theory, combinatorics, computational
geometry, number theory, and game theory. In addition, artificial intelligence
(AI) and constraint programming are also included in certain competitions.
Participants write code to solve those problems. The problem-solving process
can be divided into two main steps: ($i$) development of an efficient
algorithm and ($ii$) implementation of the algorithm in an appropriate
programming language. The correctness of the submitted solution code is judged
in a special environment called an online judge (OJ) or automated code
evaluation system (AES). Numerous factors are considered in the judging,
including output quality, program size, memory usage, and CPU time. In most
contests, the submitted solution code is automatically evaluated by the host
computer (judge server), where each solution code is judged against a set of
test cases. A solution code is _accepted_ only if it passes all test cases,
otherwise, it is _rejected_. However, in some contests, partial judging may
occur depending on the quality of the results, the number of test cases
passed, or other criteria.

In the last 30 years, we have observed a significant rise in the popularity of
programming computing events, exemplified by the International Collegiate
Programming Contest (ICPC). This competition holds the distinction of being
the largest, oldest, and most fiercely CPC for university students worldwide.
The ICPC serves as a platform where students can engage with one another,
enhance their programming abilities, develop algorithmic thinking, teamwork,
and problem-solving skills. It not only provides a valuable opportunity for
academic, community, and industry collaboration but also holds the overarching
goal of nurturing the next generation of skilled computing professionals. The
inaugural edition of the ICPC took place at Texas A&M University in 1970
(Wasik et al., 2018). Over the years, the ICPC has evolved into one of the
most esteemed programming competitions worldwide. A testament to its
widespread appeal can be observed through the participation numbers. For
instance, the event attracted over 52,709 students from 3,233 universities
across 110 countries in 2018, a substantial increase compared to the
participation of 840 teams from 560 universities in 1997. Following the
success of the first ICPC final, numerous algorithmic competitions
subsequently adopted similar AES. The International Olympiad in Informatics
(IOI) is one of them, which integrated AES into its evaluation process
starting in 1989. Contrasting the ICPC, the IOI targets specifically to high
school students, with participants tackling problems individually instead of
working in teams. Notably, participants in the IOI receive scores that can be
full or partial based on the merit of their submitted solutions, rather than
solely binary (correct or incorrect) scores like ICPC. Furthermore, big
companies such as Google and Facebook regularly organize CPCs (Li et al.,
2022a). The popular Codeforces CPC platform hosts weekly contests that attract
a substantial number of participants, reaching tens of thousands (Mirzayanov,
2020). Impressively, this platform has over 500,000 active users. In a study
conducted by Cormack et al. (Cormack et al., 2006), the fundamental
regulations, evaluation procedures, and scoring mechanisms employed by
prominent CPCs such as the ICPC and IOI were elucidated.

The CPC environment encompasses a distinct system designed to automatically
verify the correctness of submitted solution code relying on predefined
input/output test cases. Moreover, the system calculates various resources
such as time and memory usage limits during the evaluation process of the
submitted solution code. Such a specialized system is commonly referred to as
an AES111The terms automated code evaluation system (AES), automated code
assessment system (AAS), and online judge system (OJ) are employed
interchangeably to refer to the same concept. or OJ system. The concept of an
OJ system was first introduced by Kurnia and collaborators (Kurnia et al.,
2001) in 2001, enabling the automated and real-time evaluation of solution
code. However, the development, implementation, and maintenance of an OJ
system is a complex endeavor, as various essential factors must be carefully
considered to ensure its security and seamless operation. Typically, an AES
assesses submitted codes using either local or cloud-based infrastructures and
should be well-equipped to handle various threats that may arise during the
evaluation process. For instance, submitted solution codes can take the form
of executable files (_.exe_) or malicious source code that have the potential
to alter the test environment, prolong compilation time, or manipulate
restricted resources such as memory and time. A comprehensive exploration of
potential threat types and corresponding countermeasures for CPCs can be found
in a study (Forisek, 2007). To mitigate such threats and ensure secure
execution of the submitted solution code, recent advancements have introduced
specialized sandboxes (Yi et al., 2014), virtualization techniques,
containerization (Felter et al., 2015), and the utilization of Docker
frameworks (Merkel, 2014). These approaches hold promise in enhancing the
security and reliability of AES.

Apart from CPC, AESs offer a range of auxiliary functionalities that find
application in diverse domains such as programming education, online
compilers, data mining, recruitment, and software development. Furthermore,
the resources generated by AESs, including evaluation results, solution codes,
and logs, are highly regarded as valuable assets for both educational and
industrial research purposes. Over the years, numerous academic research
endeavors have yielded significant findings by leveraging the resources
provided by AESs, which are extensively utilized for programming tutoring
across various academic institutions (Rahman et al., 2021b, 2022). These
results have shed light on submission patterns, verdict statistics, problem-
solving capabilities for constraint-oriented tasks, and prevalent solution
patterns among different student groups. Educators are capitalizing on the
insights gained from such analytical research to enhance their lesson plans
and teaching methodologies. Moreover, the amassed data resources within AESs
are recognized as one of the largest real-world code repositories. This has
paved the way for software engineering research in areas such as software code
analysis, vulnerability prediction, source code mining, suggestions for class
and method names, and software refactoring (Aggarwal, 2020; Abunadi and
Alenezi, 2015; M. Mostafizer et al., 2021; Allamanis and Sutton, 2013; M.
Mostafizer et al., 2020; Aniche et al., 2022; Allamanis et al., 2015; Rahman
et al., 2020).

In recent times, the field of AI, particularly machine learning (ML) and deep
learning (DL), has witnessed remarkable advancements in text (Abdeljaber et
al., 2017; Lee et al., 2017), speech (Sainath et al., 2015; Greff et al.,
2017; Graves et al., 2013), and image (Krizhevsky et al., 2017; Szegedy et
al., 2015) processing. These breakthroughs in ML and DL have been facilitated
by the proliferation of open-source codes and the rapid development of
computational hardware. This progress has inspired both practitioners and
researchers to explore source code and software engineering challenges
(Allamanis et al., 2018; Alsolai and Roper, 2020; Le et al., 2020; Wan et al.,
2021; Zhang and Tsai, 2003). Within the domain of source code analysis,
researchers and professionals have embraced the utilization of DL models for
various code-related tasks. These tasks encompass code representation
(Allamanis et al., 2018; Hellendoorn and Devanbu, 2017), program synthesis (Le
et al., 2020; Wan et al., 2021; Zhang and Tsai, 2003; Hellendoorn and Devanbu,
2017; Lima et al., 2020; Omri and Sinz, 2020; Zhang et al., 2022; Yahav,
2018), code completion (M. Mostafizer et al., 2020; Liu et al., 2021), code
testing (Lima et al., 2020; Omri and Sinz, 2020; Zhang et al., 2022), code
summarization (Allamanis et al., 2015; Liu et al., 2021; LeClair et al.,
2019), code refactoring (Aniche et al., 2022), code repair (M. Mostafizer et
al., 2021), and analysis of source code quality and vulnerability (Alsolai and
Roper, 2020; Azeem et al., 2019; Shabtai et al., 2009; Shen and Chen, 2020;
Ucci et al., 2019). The adoption of DL for code analysis is on the rise,
accompanied by a growing array of methods, techniques, resources, tools, and
datasets. Consequently, researchers face the challenge of comprehending the
expansive landscape of available resources and research directions in this
domain. However, numerous endeavors have been made to consolidate application-
specific research through survey publications (Wasik et al., 2018; Sharma et
al., 2021b; Xiaomeng et al., 2018). In addition to the state-of-the-art (SOTA)
surveys, our survey offers several advantages to researchers and professionals
engaged in code analysis tasks. Firstly, it encompasses a comprehensive
summary of a substantial number of SOTA surveys. Secondly, it provides a
literature review categorized by specific areas such as quality assessment,
testing, code repair, and program synthesis within the field of software
engineering. Lastly, an exploration of the existing tools and datasets
available for code analysis.

In the past few years, the resources provided by AESs have emerged as valuable
datasets for a variety of code analysis tasks. One notable dataset is CodeNet
(Puri et al., 2021), a comprehensive collection encompassing 14 million real-
world solution codes and approximately 500 million lines of code spanning 55
programming languages. These codes have been sourced from the Aizu Online
Judge (AOJ) (Watanobe, 2018) and AtCoder (Naohiro, 2012) OJ systems. CodeNet
is a meticulously curated dataset specifically designed for code analysis
applications, including code classification and code similarity measurement.
Another large dataset, CodeContests (Li et al., 2022a), is tailored for ML
applications and is extensively employed in the training of models like
Alphacode. CodeContests incorporates datasets sourced from diverse CPC
platforms, including AOJ, Atcoder, CodeChef, Codeforces, and HackerEarth.
After implementing the necessary filtering procedures, the CodeContests
dataset contains a total of 715.1 GB of code. POJ-104 (Li et al., 2022a) and
GCJ (Ullah et al., 2019a) are extensively utilized benchmark datasets that
originated from a pedagogical AES and the Google Code Jam competition held
between 2008 and 2020, respectively. Notably, GCJ-297 (Google, 2021) stands as
a benchmark dataset comprising 297 problems and approximately 208,000 solution
codes. For an in-depth exploration of code analysis tasks and their associated
benchmark datasets, a comprehensive discussion can be found in CodeXGLUE (Lu
et al., 2021).

The objective of this paper is as follows: To the best of our knowledge, there
is a lack of existing surveys on AESs and their available resources for code
analysis tasks, our aim is to bridge this gap. Initially, we provide an
overview of the current landscape of AESs and their potential application
domains. Considering the diverse nature of these systems, a classification
based on their specific applications holds significant value for both
researchers and professionals. We summarized the resources, including
datasets, generated by AESs and other similar platforms. These accumulated
datasets can serve as valuable assets for practitioners and researchers
engaging in further research and analysis. Moreover, we provide a
comprehensive review of recent studies focusing on code analysis tasks
employing ML techniques. We also present an in-depth investigation of a real-
world AES, exploring aspects such as system design (software and hardware),
operation, maintenance, and research opportunities.

The remainder of this paper is organized as follows: Section 2 introduces the
evaluation methodology and scoring of AESs, Section 3 classifies AES based on
their use, Section 4 examines the available resources for AESs, Section 5
summarizes the ML-based coding analysis tasks, Section 6 explores aspects of a
real-world AES, including system design (software and hardware), operation,
maintenance and research opportunities, and finally, Section 7 concludes the
study.

##  2\. Automated Code Evaluation Systems

###  2.1. Evaluation Method

AES is a reliable, secure, and continuous online evaluation system for
algorithms submitted by users around the world. For better understanding, the
AES evaluation method can be defined as follows:

#### Evaluation Method:

The evaluation method consists of three main steps. ($i$) code submission,
($ii$) code evaluation with test datasets, and ($iii$) evaluation score.

In the code submission phase, the submitted code is compiled and verified
whether the code is executable in a homogeneous evaluation environment or not.
If the verification is successful, each solution code is reliably evaluated on
a coherent evaluation infrastructure using problem-specific test cases. The
evaluation of the test cases determines for each submission: ($i$) the code
executes without errors, ($ii$) the resource constraints (time and memory)
have not been exceeded for a given problem, ($iii$) the obtained result
satisfies all problem definitions.

Code evaluation is basically a complex process in which the submitted code
must be compiled considering each test instance. We can formally define the
test instance as follows.

#### Test Instance:

Let, $\Omega$ be an alphabet containing input and output data. A test instance
$\tau_{i}\in\mathcal{T}$, where $\mathcal{T}$ is the set of all test instances
of a particular problem. Here, $\tau_{i}$ can be defined as a triple
$\tau_{i}=(g_{i},h_{i},p_{i})$, where $g_{i}=\Omega^{*}$ and
$h_{i}=\Omega^{*}$ are the input data and reference output, respectively, and
$p_{i}=\Omega^{*}$ is the set of parameters.

The parameter set $p_{i}$ represents the resource limits, such as the maximum
usage of primary memory (RAM), CPU time, etc., that cannot be exceeded during
evaluation for a given instance. The parameter set $p_{i}$ can be empty
($p_{i}=\emptyset$) if the configuration of the evaluation engine is set to
default resource limits. In code evaluation, the solution is compiled as an
output of the submission phase. Here, the solution can be defined as follows.

#### Solution:

A solution is a function, denoted $b(g_{i},p_{i}^{\prime})\longrightarrow
z_{i}^{\prime}$, that computes output data $z_{i}^{\prime}$ based on input
data $g_{i}$ and parameters $p_{i}^{\prime}$ given by the evaluation engine.
It represents the binary form of submission.

Here, the execution parameter set $p_{i}^{\prime}$ provided by the evaluation
engine can be the same as the original parameter set $p_{i}$ defined as part
of the respective test instance (i.e., $p_{i}^{\prime}=p_{i}$). However, some
variations may also occur, e.g., $p_{i}^{\prime}$ can be a subset of $p_{i}$
($p_{i}^{\prime}\subset p_{i}$), $p_{i}^{\prime}$ can be completely different
from $p_{i}$ (i.e., $p_{i}^{\prime}\not\subset p_{i}$), or $p_{i}^{\prime}$
can be an empty set (i.e., $p_{i}^{\prime}=\emptyset$). In contrast to the
parameter set $p_{i}$, $p_{i}^{\prime}$ is usually set to empty because the
parameters affecting the evaluation are hidden from the solution.

The evaluation engine can be formally defined as follows.

#### Evaluation Engine:

The evaluation engine is a function that executes a binary file ($b$) with a
test instance $\tau_{i}$, denoted
$e(b,\tau_{i})\longrightarrow(v_{i},s_{i},l_{i})$. This function returns the
evaluation verdict $v_{i}$, the evaluation score $s_{i}$
($s_{i}\in\mathbb{R}$), and a list of statistics $l_{i}$ for the entire
execution process.

The evaluation verdict $v_{i}$ can be one of the following: Accepted (AC),
Time Limit Exceeded (TLE), Memory Limit Exceeded (MLE), Wrong Answer (WA),
Runtime Error (RE), Presentation Error (PE), and Output Limit Exceeded (OLE).
Details of these evaluation verdicts can be found in (Watanobe et al., 2022b).
Many OJ systems evaluate submissions according to the ICPC rule, which
evaluates the submissions on each test instance $\tau_{i}$ in a binary way,
i.e., correct or incorrect. In this case, the evaluation score $s_{i}$ is
always 0 (i.e., $s_{i}=0$) because $s_{i}$ is not used in evaluation according
to ICPC rules (Wasik et al., 2018). In addition, $s_{i}$ characterizes the
evaluation even if users have received the verdict ”Accepted” (i.e.
$v_{i}=AC$) for a particular instance. In general, the test instances
$\tau_{i}$ can be characterized by the different values of $s_{i}$, but there
are also more complex scoring methods that can be applied. Some evaluation
scoring methods are presented in Section 2.2. Finally, the statistics $l_{i}$
collects information about the maximum utilization of resources, such as
memory and time during the execution. If OJ does not share this information
with the user, $l_{i}$ can be empty, $l_{i}=\emptyset$.

###  2.2. Evaluation Score

Evaluation score is basically the aggregated evaluation of $s_{i}$ and $v_{i}$
to calculate the final score $s$ and verdict $v$ of each user submission. The
final score of a submission is used to rank the solutions to the problem.

A solution receives an AC verdict if and only if it passes all test instances
(Eq. 1). Otherwise, the solution receives another verdict that is different
from AC (e.g., WA, TLE) (Eq. 2).

(1) $v=AC\Longleftrightarrow\forall_{i}v_{i}=AC$

(2) $v=v_{j}\Longleftrightarrow(\forall_{i<j}v_{i}=AC)\wedge(v_{j}\neq AC)$

If the problem is not an optimization problem, the total evaluation score $s$
is calculated by the following two ways. (1) According to the ICPC evaluation
rules, it is a binary evaluation of the submission, and thus, $s=0$. (2)
Otherwise, it is the sum of the evaluation scores for all correctly solved
instances, denoted as Eq. 3.

(3) $s=\sum_{i=1}^{|\mathcal{T}|}\begin{cases}s_{i},&\text{$if$ $v_{i}=AC$}\\\
0,&\text{$otherwise$}\end{cases}$

When calculating the score for the optimization problems, the best solution
among all the submissions submitted by the participants is taken into account.
In most cases, Eq. 4 is used to calculate the evaluation score $s$ for the
optimization problems where the objective function is maximized (Wasik et al.,
2018). Here, $b_{i}$ denotes the score of the best solution at the $i^{th}$
instance. Usually, Eq. 4 is used to rank all submitted solutions to the
problems.

(4)
$s=\frac{100}{|\mathcal{T}|}\sum_{i=1}^{|\mathcal{T}|}\begin{cases}\frac{s_{i}}{b_{i}},&\text{$if$
$v_{i}=AC$}\\\ 0,&\text{$otherwise$}\end{cases}$

In addition, more customized and complex evaluation procedures can be used for
scoring. For example, the Polish Olympiad in Informatics uses Eq. 5, which
penalizes the solution if it takes more than half of the time limit (Wasik et
al., 2018), where $\Upsilon_{i}$ denotes the maximum point that can be
assigned to a test instance $\tau_{i}$, $\Lambda_{i}$ is the maximum time
limit for a solution, and $\lambda_{i}$ is the processing time required for
the solution to produce output for an instance $\tau_{i}$.

(5)
$s=\Upsilon_{i}~{}.~{}min\Biggl{(}1.0,2.0.\frac{\Lambda_{i}-\lambda_{i}}{\Lambda_{i}}\Biggl{)}$

##  3\. Classification of Automated Code Evaluation Systems

In this section, we present the classification of AESs on the basis of
application domains. Considering the programming contests, there are few
literature reviews on the classification of contests organized using AES. Pohl
(Pohl, 2006) was the first to propose a classification of programming contests
based on criteria such as contest style, duration, submission and evaluation
methods, scoring, and entrance. Furthermore, classifications based on
programming contests, types of programming exercises, and characteristics of
the AES system have also been discussed in studies (Combéfis and Wautelet,
2014; NÉMETH and ZSAKÓ, 2015). However, most of these classifications are
limited to a single application such as education or programming contests.
There is no clear or explicit classification of AESs based on their potential
applications that can be useful for users. Therefore, we decided to classify
AESs based on their applications as follows.

###  3.1. Competitive Programming Contest

OJ systems have a wide range of applications for competitive programming. Many
educational institutions use this platform to prepare their students to
participate in competitive programming contests. Competitive programming
contests are also held by various organizations and have gained popularity.
The first OJ is UVa (Revilla et al., 2008), which received great attention
worldwide. It was founded in 1995 at the University of Valladolid in Spain.
Based on the collected UVa dataset, Skiena and Revilla (Skiena and Revilla,
2003) wrote the book ”Programming Challenges: The Programming Contest Training
Manual” to help students in programming contests. Judge0 (Došilović and
Mekterović, 2020) is an open-source222https://github.com/judge0/judge0,
scalable, and powerful online code execution tool that can be used in a wide
range of programming-related applications such as programming competitions,
e-learning platforms, recruitment, and online code editors and IDEs. A partial
list of OJ systems is given in Table 1.

Table 1. OJ systems used for the competitive programming contests Name |  In Operation |  Language |  # Problems |  # Users |  Founded
---|---|---|---|---|---
UVa Online Judge |  Yes |  Eng |  4,300 |  100,000 |  1995
Aizu Online Judge (AOJ) |  Yes |  Eng, Jpn |  3,000 |  130,000 |  2004
National Tsing Hua University Online Judge |  Yes |  Eng |  10,000 |  - |  2015
National Taiwan University Online Judge |  Yes |  Chi |  2600 |  600 |  2016
Sphere Online Judge (SPOJ) |  Yes |  Eng, Pol, Por, Vie |  20,000 |  315,000 |  2004
PKU Judge Online (POJ) |  Yes |  Eng, Chi |  3,000 |  250,000 |  2003
Topcoder |  Yes |  Eng |  5,200 |  4,000 |  2001
Codeforces |  Yes |  Eng, Rus |  3,000 |  600,000 |  2010
AtCoder |  Yes |  Eng, Jpn |  5,900 |  400,000 |  2012
Google Code Jam (GCJ) |  Yes |  Eng |  450 |  670,000 |  2003
Facebook Hacker Cup |  Yes |  Eng |  - |  80,000 |  2011
HUSTOJ |  Yes |  Eng, Chi |  650 |  26,000 |  2014
Timus Online Judge |  Yes |  Eng |  1,000 |  110,000 |  2000
IEEEXtreme |  Yes |  Eng |  - |  - |  2006
ICPC |  Yes |  Eng |  - |  - |  1970
IOI |  Yes |  Eng |  - |  - |  1989
Judge0 |  Yes |  Eng |  - |  - |  2017
LeetCode |  Yes |  Eng |  - |  - |  2015

Furthermore, data science, AI and ML experts compete against each other on
various online platforms, adding a new dynamic to competitive programming. In
particular, AI game competitions are known as bot programming competitions or
AI programming competitions. Many online platforms are used for AI, ML, and
data science competitions, including the Kaggle platform, which is mainly used
for data science and ML competitions. Google AI Challenge is a biennial
competition for students; Battlecode is one of the longest-running AI
programming competitions hosted by the Massachusetts Institute of Technology;
Lux AI is an AI programming competition held on the Kaggle platform. There are
also numerous platforms and AI/ML challenges such as Halite, CodinGame, AI
Coliseum, Russian AI Cup, AWS DeepRacer, SamurAI, CodeCup, Coder One, Terminal
by Correlation One, SIGNATE, and so on.

###  3.2. Academic Tool

Recently, OJ systems have emerged as academic tools for programming learning,
programming assignments, and assessment. Teachers/instructors at many
educational institutions automatically grade students’ assignments using OJ.
The benefits of using OJ system in education are innumerable. For example, the
submitted solution codes are checked with higher accuracy, no wrong solutions
are accepted, students can get their result immediately, and the teacher can
take action to improve students’ programming skills based on the result. A
successful application of the OJ system for algorithms and data structures and
analytical investigation based on the collected data is presented in studies
(Rahman et al., 2021b, 2022; Cheang et al., 2003). Peking University has
integrated POJ (Wen-xin and Wei, 2005) as an essential tool for programming
education. POJ is being utilized in various programming courses, such as
Introduction to Computing, Problem Solving and Programming, and Data
Structures and Algorithms. Ala-Mutka (Ala-Mutka, 2005) gave a detailed review
of the application of OJ systems in education. A review (Ihantola et al.,
2010) addresses the available software for automatic assessment of programming
tasks that may be of interest for learning programming. Fonte et al. (Fonte et
al., 2013) presented the advanced version of OJ system that can be used to
provide valuable feedback to programmers to help them understand where the
problem lies and how to improve the solution code. Additionally, literature
(Cosma and Joy, 2008; Pu et al., 2016; Rivers and Koedinger, 2013; Singh et
al., 2013) explores topics such as code plagiarism detection, automated code
correction, and personalized feedback through OJ systems.

In contrast, the limited scope of black box testing and oversimplified
feedback (correct or incorrect) restricts its widespread application in
programming education. To overcome this constraint, Zhou et al. (Zhou et al.,
2018) introduced a novel OJ system framework that enables effective
programming education and provides comprehensive assessment of submitted
solutions. The new OJ system comprises four modules, including code quality
checking, similarity checking, personalized feedback for students, and
advising on teaching adjustments. This OJ system has been implemented in a
programming in C course for over ten years and has demonstrated substantial
improvements in students’ programming abilities. Lu et al. (Lu et al., 2017)
showed the positive influence of the OJ system, which increases the
performance level and also arouses students’ interest in programming
throughout the year. Teachers are increasingly incorporating OJ systems into
their daily programming teaching activities. Although the performance of OJ
systems is impressive, the contributions and effects of teachers are
invaluable and irreplaceable. OJ systems can serve as excellent tools to
assist teachers in programming education. By providing reliable, high-
efficiency, and objective evaluations, as well as fair and accurate scoring,
OJ systems can greatly reduce teachers’ workload (Zhou et al., 2018). Teachers
can utilize OJ systems to assign programming tasks, analyze student
performance through reports and statistics, identify code similarities, and
more. For example, in a recursive algorithm lesson, students may be given a
problem that can be solved using either brute force or a recursive algorithm.
If the teacher finds that most students used the brute force algorithm (which
is relatively easy than recursive algorithm), they may reinforce the use of
the recursive algorithm to improve students’ algorithmic skills. Table 2 shows
a partial list of OJ systems that have been incorporated into programming
education.

Table 2. OJ systems used for the Education Name |  In Operation |  Language |  # Problems |  Founded
---|---|---|---|---
UVa Online Judge |  Yes |  Eng |  4,300 |  1995
Aizu Online Judge |  Yes |  Eng, Jpn |  3,000 |  2004
Jutge.org |  Yes |  Eng, Esp, Fre, Cat, Dut |  4,843 |  2006
POJ |  Yes |  Eng, Chi |  3,000 |  2003
CodeChef |  Yes |  Eng |  3,000 |  2009
CodeHunt |  Yes |  Eng |  8,300 |  2014
Codecademy |  Yes |  Eng |  - |  2011
CodeWars |  Yes |  Eng |  1,200 |  2012
URI Online Judge |  Yes |  Eng, Spa, Por |  1,100 |  2011
Hanghou Dianzi University (HDU) OJ |  Yes |  Eng, Chi |  6,000 |  -
Judge0 |  Yes |  Eng |  - |  2017
LeetCode |  Yes |  Eng |  - |  2015

###  3.3. Recruitment and Selection

In recent years, the use of technologies such as big data analytics, AI, and
OJ in the recruitment process is becoming more common among companies and
organizations (Oksanen, 2018). These technologies provide rigorous, effective,
and cost-efficient ways to find suitable candidates. One study (Oksanen, 2018)
explores how prescreening applications, practical skills testing, and
candidate communication can be handled using AI. Furthermore, technology in
the different phases of the recruitment and selection process is presented in
the study (Nikolaou, 2021). The phases are attracting, screening, selection,
and on-boarding and socialization. There are numerous platforms that use OJ
systems to support the recruiting process. These platforms are mainly
commercial and automatically evaluate the submitted codes and rank the
programmers. We present some OJ systems that are used for recruitment. For
example, _HackerEarth_ , _HackerRank_ , _Qualified_ , _CodeEval_ , _Codility_
, _Track Test_ , and so on. _HackerEarth_ is an online platform dedicated to
hiring talented developers, hosting crowdsourcing-based ideas, and organizing
hackathons. _Codility_ helps hiring managers to find the best developers from
a large pool of skilled programmers in the shortest possible time. A list of
OJ systems applicable to the recruitment and selection process can be found in
(Wasik et al., 2018).

###  3.4. Online Compilers

Another category of OJ systems are online compilers, where the codes developed
in different languages by user solutions can be compiled and executed remotely
through a browser. _Codeanywhere_ is one of the most feature-rich online
compilers that offers a dedicated custom development environment and real-time
collaboration. It allows users to automatically connect to GitHub, Amazon
Cloud, FTP servers and Bitbucket. _Coding Ground_ offers a full-featured IDE
that allows users to edit, compile and run their projects in the cloud. Codio
is a cloud-based IDE that supports a large number of programming languages,
can be integrated into e-learning platforms, and can be used to detect
plagiarism. Furthermore, there are many online compilers whose goal is to
provide support for code compilation without any physical configuration.
CodeSkulptor, for example, is an online compiler for the Python programming
language and also supports the learning process of programmers. The Codepad
compiler allows users to share their code with other collaborators via URLs.
In addition, C++ Shell, Web compiler, OnlineGDB C, Tutorialspoint, Replit,
Rextester, myCompiler, OneCompiler, Online Compiler CodeChef, and
Techiedelight are compilers used for online code compilation.

###  3.5. Development Platforms

In contrast, many OJ development platforms are available to host programming
competitions or educational activities in local infrastructure. _DOMjudge_ is
a well-known OJ development platform that can be easily installed to host
programming contests. It allows users to prepare and run programming contests
according to ACM ICPC rules. Mooshak (Leal and Silva, 2003) is an automatic
judge and full-fledged contest manager. It is an open system that can be
installed on a Linux operating system with Apache HTTP server. The
architecture of Mooshak is suitable not only for small competitions with one
server, but also for competitions with multiple sites. CloudCoder, a web-based
platform inspired by CodingBat (Spacco et al., 2015), is designed to assess
students’ submitted code assigned by the instructor. At Tsinghua University,
an online assessment system was developed for university-level code evaluation
(Zheng et al., 2015), allowing users to add new problems. However, the code
resources of this system are not open source. Xavier and collaborators
described an OJ platform developed at the University of Porto, which drew
inspiration from DOMjudge and Moodle (Xavier et al., 2011). They reviewed the
OJ development platforms. Furthermore, various plug-ins for the Moodle
platform have been created for code evaluation, such as Code Runner, Virtual
Programming Lab, and Online Judge plug-in for Moodle (Rodríguez-del Pino et
al., 2012; Zhigang et al., 2001). Table 3 provides a list of online compilers
and development platforms.

Table 3. OJ systems are used for online compilers and development platforms Name |  In Operation |  Founded |  Compiler |  Development Platform
---|---|---|---|---
Codeanywhere |  Yes |  2013 |  $\checkmark$ |  $\times$
Coding Ground |  Yes |  2006 |  $\checkmark$ |  $\times$
DOMJudge |  Yes |  2004 |  $\times$ |  $\checkmark$
C ++ Shell |  Yes |  2014 |  $\checkmark$ |  $\times$
Mooshak |  2015 |  2005 |  $\times$ |  $\checkmark$
SIO2 |  Yes |  2012 |  $\times$ |  $\checkmark$
Ideone |  Yes |  2009 |  $\checkmark$ |  $\times$
CodeChef |  Yes |  - |  $\checkmark$ |  $\times$
Virtual Programming Lab |  Yes |  2012 |  $\times$ |  $\checkmark$
Online Compiler |  Yes |  2009 |  $\checkmark$ |  $\times$
A+ |  Yes |  2017 |  $\times$ |  $\checkmark$
Codio |  Yes |  2013 |  $\checkmark$ |  $\times$
TestMyCode |  Yes |  2013 |  $\times$ |  $\checkmark$
Programiz |  Yes |  - |  $\checkmark$ |  $\times$
xLx |  Yes |  2001 |  $\times$ |  $\checkmark$
Web-CAT |  Yes |  2003 |  $\times$ |  $\checkmark$
Web Compiler |  Yes |  2014 |  $\checkmark$ |  $\times$
Codepad |  Yes |  2008 |  $\checkmark$ |  $\times$
BOSS |  Yes |  2012 |  $\times$ |  $\checkmark$
CodeSkulptor |  Yes |  2012 |  $\checkmark$ |  $\times$
CloudCoder |  Yes |  2012 |  $\times$ |  $\checkmark$
Tsinghua Online Judge |  Yes |  2012 |  $\times$ |  $\checkmark$
paiza.io |  Yes |  2014 |  $\checkmark$ |  $\times$
Judge0 |  Yes |  2017 |  $\checkmark$ |  $\checkmark$

##  4\. Resources of Automated Code Evaluation Systems

In this section, we present a consolidate overview of AESs datasets and tools.
Since AESs are regularly used for a variety of purposes, including
recruitment, programming education and contests, data mining, and research, a
huge number of data resources (codes, problems, analysis results, scores,
submission logs, etc.) are generated. Kaggle is an online platform for data
scientists and ML programmers (Goldbloom, 2010). It is also used to organize
data mining and ML competitions. Kaggle allows users to collaborate with
others, publish datasets, find datasets of others, access GPU-integrated
notebooks, and compete against each other to solve data science problems.
There are many data science and ML codes and datasets (e.g., sports, health,
travel, software, and food) that are available (open source) and mostly
reliable. DREAM Challenges is another platform for solving biological
problems, especially scientific challenges, and is therefore targeted at
researchers only (Meyer and Saez-Rodriguez, 2021; Costello and Stolovitzky,
2013; Saez-Rodriguez et al., 2016). OpenML (Van Rijn et al., 2013) is an open
platform for sharing datasets, code, algorithms, and experimental results. It
has a rich and diverse data repository that stores more than 5,000 datasets
and supports Python, Java, REST API and many other scientific programming
languages. Recently, Hugging Face 333https://huggingface.co/. has gained
attention as the open-source platform for sharing ML models, datasets,
evaluation metrics, and demo applications for open science. Since the platform
is a Git-based code repository similar to GitHub, researchers can easily share
their own models and datasets. In addition, users can easily try the public
models and datasets through Python libraries.

Furthermore, numerous solution codes and logs are available on various
platforms (AOJ, AtCoder, etc.) and can be used for education and programming
research. These real-world and rich data resources have become attractive for
coding, education, learning analytics, ML, and data mining research (Rahman et
al., 2022, 2021c). In a study (Rahman et al., 2021b), a comprehensive data-
driven analysis based on OJ data was conducted. The experimental results show
the shortcomings of students’ programming and the scope for possible
improvements. Hsiao et al. (Hsiao et al., 2020) leveraged an educational
learning analysis tool called ”Web Programming Grading Assistant (WPGA)” to
study the effectiveness of students’ programming learning. Rahman et al.
(Rahman et al., 2022) conducted educational data mining using OJ data to
support programming learning. However, benchmark datasets have had a
significant impact on the growth of coding-related research using ML. Here we
present some benchmark datasets for code intelligence research. Code
representation is a fundamental activity for preparing source code that is
compatible with ML models. It involves converting code into a numerical
representation to feed ML models to solve specific tasks (e.g., pattern
recognition, method name prediction, and comment classification). code2vec
(Alon et al., 2019) and GHTorrent (Gousios, 2013) can be useful datasets for
code representation tasks. Code quality assessment is a broad category of
coding tasks with subcategories such as clone detection, smell detection,
quality assessment and prediction. BigCloneBench, Multi-label Smells, DL
Smells, QScored, and ML for software refactoring datasets can be useful for
code quality assessment (Aniche et al., 2022; Guggulothu and Moiz, 2020;
Sharma et al., 2021a; Sharma and Kessentini, 2021; Svajlenko et al., 2014a).
Vulnerability analysis is another important area of code analysis that is used
to determine whether the code is vulnerable or not. Datasets such as Draper
VDISC, Genome, TRL, and Project-KB (Lin et al., 2018; Ponta et al., 2019;
Russell et al., 2018; Zhou and Jiang, 2012) are used to identify
vulnerabilities in code. Code testing datasets such as Defect4j, BugDetection,
DeepBugs, and DTLDP (Chen et al., 2020a; Just et al., 2014; Li et al., 2019a;
Pradel and Sen, 2018a) can be useful for the testing task.

In addition, Code search (Gu et al., 2018; Husain et al., 2019; Premtoon et
al., 2020) is important when programmers want to use other codes. This system
automatically finds semantically similar codes based on a natural language
query. The code completion system (M. Mostafizer et al., 2020; Bielik et al.,
2016; Raychev et al., 2014; Svyatkovskiy et al., 2020, 2019) can help
programmers automatically complete their code. Also, the code-to-code
translation system (Karaivanov et al., 2014; Roziere et al., 2020; Nguyen et
al., 2015) assists programmers translate their code from one language to
another (e.g., Python to Java and Java to Python). As the use of real datasets
for coding-related research increases, therefore, we present a list of
available datasets from platforms such as OJ, contest platforms, Stack
Overflow, and GitHub in Table 4.

Table 4. A list of datasets and their application in coding analysis  Sl. |  Dataset Name |  Coding Tasks |  Size
---|---|---|---
1 |  CodeNet (Puri et al., 2021) |  code classification, code similarity |  14 million
2 |  Aizu (Watanobe, 2018) |  code classification, code-to-code translation, code completion, refactoring, summarization |  8 million
3 |  AtCoder (Naohiro, 2012) |  code classification, code-to-code translation, code completion, refactoring, summarization |  7.5 million
4 |  BigCloneBench (Svajlenko et al., 2014b) |  code clone |  6 million (true clone pair) and 260,000 (false clone pair)
5 |  POJ-104 (Ben-Nun et al., 2018) |  code classification, code similarity, code clone |  52,000
6 |  GCJ (Ullah et al., 2019b) |  code plagiarism detection |  2.4 million
7 |  PY150 (Raychev et al., 2016) |  code completion |  150,000
8 |  Devign (Zhou et al., 2019) |  fault detection |  27,318
9 |  QuixBugs (Lin et al., 2017) |  code repair |  40
10 |  Bugs2Fix (Tufano et al., 2019) |  code repair |  122,000
11 |  CodeSearchNet (Husain et al., 2019) |  code summarization |  1.1 million
12 |  CodeContests (Li et al., 2022a) |  text-to-code generation |  4.5 million
13 |  APPS (Hendrycks et al., 2021) |  text-to-code generation |  10,000
14 |  CONCODE (Iyer et al., 2018) |  text-to-code generation |  104,000
15 |  MBPP (Austin et al., 2021) |  text-to-code generation |  974
16 |  MathQA-Python (Austin et al., 2021) |  text-to-code generation |  23,914
17 |  HumanEval (Chen et al., 2021) |  text-to-code generation |  164
18 |  HumanEval-X444https://huggingface.co/datasets/THUDM/humaneval-x. |  text-to-code generation |  820
19 |  HumanEval Infilling (Bavarian et al., 2022) |  code infilling |  8,488
20 |  code-docstring-corpus (Miceli Barone and Sennrich, 2017) |  text-to-code generation, code summarization |  150,370
21 |  CoNaLa (Yin et al., 2018a) |  text-to-code generation, code summarization |  2,879 (annotated) and 598,237 (not annotated)
22 |  MCoNaLa (Wang et al., 2023) |  text-to-code generation, code summarization |  896
23 |  MBXP (Athiwaratkun et al., 2022) |  text-to-code generation, code summarization, code translation, code infilling, prompt robustness |  —
24 |  CodeXGLUE (Lu et al., 2021) |  clone detection, defect detection, cloze test, code completion, code translation, code search, code repair, code summarization, text-to-code generation, document translation |  —
25 |  StaQC (Yao et al., 2018) |  code repair |  148K Python and 120K SQL question-code pairs from StackOverflow
26 |  IntroClass (Le Goues et al., 2015) |  code repair |  —

##  5\. Code Analysis Using Machine Learning

According to Evans Data
Corporation555https://evansdata.com/press/viewRelease.php?pressID=278, there
were approximately 23.9 million professional developers in 2019, and that
number is expected to reach approximately 28.7 million by the end of 2024 (Lu
et al., 2021). The ML-based code intelligence can be used to improve the
productivity of a growing number of professional programmers. At the same
time, benchmark datasets have a significant impact on the growth of applied ML
research. Recently, researchers have begun to utilize statistical models such
as neural networks for code analysis tasks. Also, the application of pre-
trained models that learn from large programming data (e.g., code), such as
BERT (Devlin et al., 2019)-based models (Liu et al., 2019b; Feng et al., 2020;
Kanade et al., 2020; Wang et al., 2021; Jiang et al., 2021; Guo et al., 2021)
and GPT (Radford et al., 2018, 2019; Brown et al., 2020)-based models (Chen et
al., 2021; Ouyang et al., 2022), have achieved great success in a wide range
of coding tasks. With the growth of resources, datasets, tools, and methods,
code analysis research is also expanding. Therefore, Table 5 summarizes recent
attempts of code analysis tasks using ML.

Table 5. Machine learning approaches for coding analysis tasks  Sl. |  Code Analysis Task |  Article
---|---|---
1 |  Defect Detection |  (Zhou et al., 2019; Li et al., 2019b; Pradel and Sen, 2018b; Ray et al., 2016; Wang et al., 2016a, b; Qasem et al., 2020; Butgereit, 2019; Chen et al., 2020b)
2 |  Clone Detection |  (Svajlenko et al., 2014b; Büch and Andrzejak, 2019; Mou et al., 2016; Bandara and Wijayarathna, 2011; Zhang and Khoo, 2021; Wang et al., 2020a; Mostaeen et al., 2020; White et al., 2016a; Mostaeen et al., 2018; Ye et al., 2020)
3 |  Code Completion |  (M. Mostafizer et al., 2020; Liu et al., 2021; Bielik et al., 2016; Raychev et al., 2014; Svyatkovskiy et al., 2020, 2019; Liu et al., 2020; Li et al., 2018)
4 |  Code Repair |  (M. Mostafizer et al., 2021; Tufano et al., 2019; Goues et al., 2019; Gupta et al., 2017; Amorim et al., 2018; Liu et al., 2016; Gopinath et al., 2014; Vasic et al., 2019; Le et al., 2015)
5 |  Code Search |  (Gu et al., 2018; Husain et al., 2019; Wang et al., 2020b; Sachdev et al., 2018; Wan et al., 2020; Shuai et al., 2020)
6 |  Code Summarization |  (Allamanis et al., 2016; Clement et al., 2020; Fernandes et al., 2018; Hu et al., 2018a; Iyer et al., 2016; Wan et al., 2018; Wei et al., 2019)
7 |  Text-to-Code Generation |  (Clement et al., 2020; Wei et al., 2019; Guo et al., 2019; Iyer et al., 2019; Yin et al., 2018b; Yin and Neubig, 2017)
8 |  Code Classification |  (M. Mostafizer et al., 2021, 2020; Rahman et al., 2020; Watanobe et al., 2022a; Ma et al., 2018; Ugurel et al., 2002; Ji et al., 2018)

A brief description of the coding tasks with ML is as follows. The purpose of
the task _defect detection_ is to identify errors/defects within the body of
the source code. Classification of codes as buggy or not is based on the
identification of errors in the code. Compiling the code is not enough to
detect all errors in the code, because logic errors may exist in the code
after compilation. Sometimes, the entire code needs to be manually inspected
to detect logic errors, which is time-consuming and costly. ML models are able
to detect these logic errors in the code based on a large number of correct
code corpus. Ghassan (Samara, 2017) proposed a practical approach to detect
logic errors in code for object-oriented environments such as C# .Net
Framework. To avoid logic errors, the proposed environment includes some
predefined behaviors using Xceed, Alsing, and Mind Fusion Components. Al-
Ashwal et al. (Al-Ashwal et al., 2018) presented a tool for identifying logic
errors in Java code using both static and dynamic methods. Song et al. (Song
et al., 2019) proposed an automatic logic error detection method for
functional programming tasks using correct reference solutions for each
counter assignment. This method was very efficient than manually identifying
logic errors in code. The BiLSTM model is used to detect both logical and
syntactic errors in code (M. Mostafizer et al., 2021).

_Clone detection_ is used to identify semantically similar code. It has two
subtasks: searching for similar codes and classification. Code cloning is a
well-known approach of reusing source code in software development. Ain et al.
(Ain et al., 2019) provided a comprehensive review of the latest tools,
trends, and techniques in the field of code clone detection. Code clones pose
a challenge to software maintenance because verification is required for
similar segments when a bug is identified in one segment. Feng and
collaborators (Li et al., 2017) proposed a token-based approach to clone
detection using a DL model called CCLEARNER. A classifier is trained using
known method-level code clones and non-clones and then used the trained
classifier to detect code clones of a given code. Fang et al. (Fang et al.,
2020) proposed a supervised DL model for function-level code clone detection.
They proposed a joint code representation that uses fusion embedding
techniques to learn the unseen semantic and syntactic features from the source
code. The proposed fusion learning approach outperformed state-of-the-art
models in detecting function-level code clones on a large C++ program dataset.
White et al. (White et al., 2016b) introduced a learning-based code clone
detection technique using DL. Moreover, a systematic review of DL applications
for code clone detection has been conducted in the study (Lei et al., 2022).

_Code completion_ is another task that helps programmer complete code
correctly. Programmers sometimes get confused what to write next, and in such
cases, code completion helps them to complete the code. Code completion can be
done in two ways ($i$) token-level completion and ($ii$) line-level
completion. _Code repair_ task identifies bugs and automatically fix them.
Typically, they identify bugs in the code and fix the code according to the
context of the code. Rahman et al. (M. Mostafizer et al., 2021) proposed a
BiLSTM la nguage model for code evaluation and repair. The model is trained on
a large number of correct codes and used for code evaluation and repair. The
model also proposes correct words as replacements for error words in the code
Li et al. (Li et al., 2022b) proposed a DL model called DEAR that can
repair/fix common errors that require dependent changes in the subsequent one
or multiple segments in the code. They leveraged a tree-based LSTM model and a
divide-and-conquer strategy to understand the actual context of the code.

_Code search_ is a task that measures the semantic relevance between the code
and natural language. This is the activity of searching the code based on a
natural language query. Gu et al. (Gu et al., 2018) proposed a novel DL model
called CODEnn for code search. The model embeds code and its natural
description together in a high-dimensional vector space. Code snippets can be
retrieved for a natural language query based on their high-dimensional vector
space, which ensures a more relevant code search. Ling et al. (Ling et al.,
2020) proposed an adaptive model for deep code search (AdaCS), which is
trained once and applied to different code bases. AdaCS divides the learning
procedure into two main steps: embedding domain-specific words and identifying
common syntactic patterns for matching. Experimental results show that AdaCS
exhibits improved resilience and outperforms existing state-of-the-art deep
code search techniques when applied to unseen software projects that are not
included in the training dataset.

_Code summarization_ provides abstracted summary of the code and helps to
better understand the program code and software maintenance. Therefore,
producing high-quality annotations is a major challenge for researchers. Liu
et al. (Liu et al., 2019c) proposed a novel code summarization method based on
neural networks using call dependency information of the source code. All call
dependencies were extracted and converted into a sequence of tokens for the
Seq2Seq model for code summarization. The experimental results demonstrated
that the model understands the call dependency during the code summarization
task. _Text-to-code_ task is used to generate codes based on the input of the
natural language description. _Code classification_ can be done in many ways,
such as classifying source code based on programming language, application,
and error.

To perform coding tasks, researchers have developed various ML models and
tools and have had great success in the tasks of code representation, code
search, code understanding, code summarization, quality assessment,
vulnerability assessment, testing, and code synthesis. Here we present a list
of tools and ML models used to analyze code, as shown in Table 6. Each
model/tool is listed with the actual reference, citation, year of first
appearance online, and a short description. All metadata for these
models/tools was collected manually by searching author websites, electronic
libraries, and publisher websites.

Table 6. A list of ML models and tools used to analyze code  Tasks |  Tools/ML Models | #Citations |  Year of Pub. |  Description
---|---|---|---|---
Code Search |  Deep Code Search (Gu et al., 2018) | 418 |  2018 |  Code embedding technique is used to search code
Code Representation |  Graph-based code modeling (Allamanis et al., 2017) | 702 |  2018 |  Code modeling with graph
Code2vec (Alon et al., 2019) | 929 |  2019 |  Generate distributed representation of the code
Vocabulary learning on code (Cvitkovic et al., 2019) | 40 |  2019 |  Generate AST from Java code
CC2vec (Hoang et al., 2020) | 110 |  2020 |  Use the representation of code changes
Quality Assessment |  SVF (Sui and Xue, 2016) | 269 |  2016 |  Interprocedural static value-flow analysis in LLVM
ML for software refactoring (Aniche et al., 2022) | 47 |  2020 |  Explores the effectiveness of ML in predicting software refactoring
CREC (Yue et al., 2018) | 32 |  2018 |  Automated code clone recommendation for refactoring
SMAD (Barbez et al., 2020) | 30 |  2020 |  SMAD is used to detect two renowned anti-patterns: God Class and Feature Envy
DL smells (Sharma et al., 2021a) | 32 |  2021 |  Detects code smells by using DL and transfer learning
Vulnerability Assessment |  VCCFinder (Perl et al., 2015) | 219 |  2015 |  Detects vulnerable codes in repositories
SWAN (Piskachev et al., 2019) | 15 |  2019 |  Detects vulnerable codes
WAP (Medeiros et al., 2013) | 9 |  2013 |  Detects and corrects vulnerable codes
Code Understanding |  CodeNN (Iyer et al., 2016) | 607 |  2016 |  Neural attention based model is used to summarize the code
ChangeScribe (Cortés-Coy et al., 2014) | 171 |  2014 |  Automatically generate commit messages based on code changes
DeepSim (Zhao and Huang, 2018) | 192 |  2018 |  DL is used to find code similarity
NeuralCodeSum (Ahmad et al., 2020) | 226 |  2020 |  Source code summarization using Transformer
CODES (Panichella et al., 2012) | 118 |  2012 |  Source descriptions are mined from programmers’ communication
Rencos (Zhang et al., 2020) | 132 |  2020 |  Code summarization based on neural network
Testing |  AppFlow (Hu et al., 2018b) | 61 |  2018 |  Using ML to synthesize robust reusable UI tests
DeepFuzz (Liu et al., 2019a) | 57 |  2019 |  Automatic generation of syntax for Fuzz testing

##  6\. The AOJ: an example of an Automated Code Evaluation System

The Aizu Online Judge (AOJ) (Watanobe, 2018) is an AES that has been in
operation for more than 11 years. It has a wide range of applications such as
academic, programming contests, regular practice and research. AOJ has over
130,000 registered users who perform programming activities regularly. It has
already evaluated more than eight million program codes. The University of
Aizu, Japan, officially uses the AOJ platform for several programming courses
as an active academic tool. In addition, AOJ has API services to access its
resources for research purposes666http://developers.u-aizu.ac.jp/., which has
led to a large number of research in education, coding analysis, data mining,
and datasets based on its data (Rahman et al., 2021b, 2022; M. Mostafizer et
al., 2021, 2020; Rahman et al., 2020; Puri et al., 2021; Watanobe et al.,
2022a). Considering the scalability and impact of the AOJ system on practical
applications, we present the AOJ system as an example of AES in this study.

###  6.1. Hardware Architecture

The AOJ system has a complex component architecture with multiple dedicated
components that work together to provide a smooth and uninterrupted
evaluation. Fig. 1 shows the component diagram of the AOJ system, in which the
web server, database server, judge server, broadcaster, and load balancer are
interconnected. The web server is the interface used to communicate between
the external system and the OJ core system. Basic tasks such as browsing,
registration, authentication, and solution submission are implemented in the
web server. It also communicates with the database server and load balancer to
obtain the evaluation/judgment results of each submission. The database server
may consist of one or more servers that store all information related to the
evaluation, such as problem set, evaluation results, solution codes,
statistical data, and user information. The database servers communicate with
the web server and load balancer via private APIs. The judge master is a key
server responsible for managing the judge data for the judge cluster. It
provides refined judge data for all judge servers and builds the necessary
_checker_ and _reactive_ programs for all judge servers. It also contains a
non-stop integration mechanism and is considered one of the special components
of the OJ system. Similarly, the load balancer is a core component of the OJ
system to interact between the web server and the judge servers. It also
communicates with the broadcaster and the database server as well.

Figure 1. Component diagram of the AOJ system

The load balancer is primarily responsible for scheduling submissions for
judging. It receives the evaluation results from the judge servers and the
notification is sent to the web server, database server and broadcaster. The
broadcaster is basically used to inform the users about the status of the
submissions asynchronously. Finally, the main role of the judge servers is to
perform the assigned tasks based on the judge data. The judge servers should
be isolated and only allow authorized processes to connect. It is also
important to avoid data loss due to malicious or unexpected operations. More
details about these hardware components of the AOJ system can be found in
(Watanobe et al., 2022b). We have presented the system information of AOJ in
Table 7. Table 7 shows the AOJ
servers777https://onlinejudge.u-aizu.ac.jp/system_info., the configurations of
each server, and the application of the servers for programming languages.
This is a list of servers, including spare/unavailable servers, for quick
replacement/update for seamless operation.

Table 7. System information of judge servers and corresponding application for programming languages  Judge Server |  Machine |  Processor and RAM |  Using for
---|---|---|---
#0 |  IBM ThinkPad |  Pentium (R) M 1.30 GHz and 1 GB |  C, C++
#1 |  Dell PowerEdge R300 |  Dual Core Intel(R) Xeon(R) E3113 3.0 GHz and 1 GB |  C, C++
#2 |  Dell PowerEdge R300 |  Dual Core Intel(R) Xeon(R) E3113 3.0 GHz and 1 GB |  JAVA
#3 |  Dell PowerEdge R210 |  Intel(R) Xeon(R) Processor E3-1270 3.40 GHz and 2 GB |  Ruby, Python, Python3, PHP, JavaScript
#4 |  Dell PowerEdge R210 II |  Intel(R) Xeon(R) Processor E3-1270 3.40 GHz and 4 GB |  C, C++
#5 |  Dell PowerEdge R210 II |  Intel(R) Xeon(R) Processor E3-1240v2 3.40 GHz and 8 GB |  C++11, JAVA, C#, D
#6 |  Dell PowerEdge R210 II |  Intel(R) Xeon(R) Processor E3-1280 v2 3.6 GHz, Turbo 4C/8T and 8 GB |  Ruby, Python, Python3, PHP, JavaScript
#7 |  Dell PowerEdge R220 |  Intel(R) Xeon(R) Processor E3-1286 v3 3.7 GHz, Turbo 4C/8T and 8 GB |  C++11, JAVA, Scala, Haskell, OCaml, C#, D
#8 |  Dell PowerEdge R220 |  Intel(R) Xeon(R) Processor E3-1286 v3 3.7 GHz, Turbo 4C/8T and 8 GB |  C, C++, C++14
#9 |  Dell PowerEdge R210 II |  Intel(R) Xeon(R) Processor E3-1280 v2 3.6 GHz, Turbo 4C/8T and 8 GB |  Ruby, Python, Python3, PHP, JavaScript, Rust, Go, Kotlin
#10 |  Dell PowerEdge R240 |  Intel(R) Xeon(R) Processor E-2286G 4.0 GHz, Turbo 6C/12T and 8 GB |  C++14, C++17, Rust
#11 |  Dell PowerEdge R240 |  Intel(R) Xeon(R) Processor E-2286G 4.0 GHz, Turbo 6C/12T and 8 GB |  C++11, JAVA, C#, PHP, Kotlin
#12 |  Dell PowerEdge R250 |  Intel(R) Xeon(R) Processor E-2378G 2.8 GHz, Turbo 8C/16T and 8 GB |  PyPy3
#13 |  Dell PowerEdge R250 |  Intel(R) Xeon(R) Processor E-2378G 2.8 GHz, Turbo 8C/16T and 8 GB |

###  6.2. Software Architecture

In this section, we briefly describe the software architecture of the AOJ
system, in particular the core components such as load balancer, broadcaster,
and judge server. The load balancer is a core component of any OJ system. It
receives submissions from the web server and distributes them to the judge
server. Finally, it reports judging results using a variety of channels.
Multithreaded processes are used to organize the components of the load
balancer, which includes three instances such as _SubmissionReceiver_ ,
_SubmissionSender_ , and _SubmissionProvider_. The software architecture,
submission, and evaluation in the load balancer are shown in Fig. 2. In
addition, the details of the operation and flow of the load balancer are
described in (Watanobe et al., 2022b).

Figure 2. Internal software architecture of load balancer

Fig. 3 shows the software architecture of the broadcaster, representing the
communication between the broadcaster, load balancer, and external systems.
Since the broadcaster is an asynchronous communication, WebSocket is used for
push-type communication. First, the external system establishes a connection
to send a request to the broadcaster via WebSocket in order to access the
broadcaster. Next, the load balancer sends the change status to the
Broadcaster through a TCP connection. The Broadcaster then receives the status
using its internal TCP server. Finally, the Broadcaster broadcasts the status
to all connected external systems through a pre-established WebSocket
connection.

Figure 3. Software architecture of broadcaster

The internal software architecture of the Judge server is shown in Fig. 4. An
operating system is installed on the Judge server and there are five
processes: Controller, Observer, Executor, Launcher, and Judge. The Observer
is an administrator that monitors all submission processes. If there is
something bad in the submissions, it reacts and terminates them. A Controller
is a communicator between the load balancer and Judge server. It receives
submissions from the load balancer and provides the appropriate judging
results. Launcher is used to prepare the submissions for the executor, then
the executor is activated by the launcher to execute the program code.
Finally, the judge evaluates the program code based on the judge data.
However, problems can occur in the Launcher, Executor, and Judge phases when
executing and evaluating program code. If the processes such as code.exe in
the Executor phase and _checker_ and _reactor_ in the Judge phase take more
time than usual, these processes are terminated by the Observer.

Figure 4. Software architecture of judge server

In addition, the entire operation of the AOJ system relies on a variety of
software technologies (Watanobe et al., 2022b). The web server is implemented
with the Spring framework (2018 and onwards) and Apache Tomcat (before 2018).
Scala is used for the broadcaster and the load balancer runs entirely on a
Java application. In contrast, judge servers are run on scripting languages.

###  6.3. Operational Availability

Since the AOJ system is active in a variety of programming activities, here
are some brief operation-oriented statistics. AOJ has successfully held 100
on-site programming contests and over 3,000 virtual contests. AOJ is used as
an academic tool in programming courses at the University of Aizu and has
evaluated over 8 million submitted solution codes through its long-term
service. The AOJ system has been in operation around the clock for more than a
decade to learn and practice programming. We have calculated the availability
of the AOJ system in hours from 2011 to 2022. The total possible hours (TPH)
can be calculated using Eq. 6.

(6) $TPH=years_{operation}\times(365\times 24)$

In each year, the services of the AOJ system are suspended for planned
maintenance. Therefore, the total operating hours (TOH) can be calculated
using Eq. 7, where $y$ is the year and $MH_{y}$ stands for the total number of
maintenance hours in a year $y$.

(7) $TOH_{y}=(365\times 24)-MH_{y}$

Assuming $MH_{2011}$ is 71, $TOH_{2011}$ would be 8,689 hours according to the
above formula. We consider an hour as an actual working hour (AWH) if the AOJ
system evaluates at least one submission in that hour, otherwise, it is
considered an idle hour (IH). Fig. 5 shows the $TOH_{y}$, $MH_{y}$, $AWH_{y}$,
and $IH_{y}$ of the AOJ system during 2011-2022.

20112012201320142015201620172018201920202021$8{,}200$$8{,}300$$8{,}400$$8{,}500$$8{,}600$$8{,}700$YearhoursTOHAWH
(a)

20112012201320142015201620172018201920202021$0$$50$$150$$250$$350$$450$YearhoursMHIH
(b)

Figure 5. The availability of the AOJ system during 2011-2022

From Fig. 5, some observations can be made: ($i$) $AWH$ is relatively low in
2011 and 2012; ($ii$) in last few years, $AWH$ is close to $TOH$; ($iii$) $IH$
was also found to be very low in 2020, 2021, and 2022; ($iv$) $MH$ also
decreases year by year. Furthermore, the $TPH_{(2011-2022)}$ for the past 12
years was 105,120 hours, in which the AOJ system was in operation (available)
for about 104,656 hours, which is 99.56% of the $TPH_{(2011-2022)}$ excluding
scheduled $MH$. In addition, the average $AWH$ is 98.74% of $TOH$ over the
past 12 years. This statistic demonstrates the high operational availability
of the AOJ system throughout the years.

###  6.4. Computational Performance

Over the years, the AOJ system has been used in a variety of areas, including
competitions, academics, and practice. As a result, the AOJ system has
received millions of submissions for evaluation each year.
LABEL:{submission_received_1} provides the statistics of submissions received
by the AOJ system over the years, showing that the AOJ received approximately
1,059,756 and 1,096,846 submissions in 2021 and 2022, respectively. Moreover,
the number of submissions appears to be increasing each year. In addition,
LABEL:{day_wise_submission_1} shows a further breakdown of submissions for
October 2022, where we randomly selected the month.
LABEL:{day_wise_submission_1} demonstrates that the AOJ received a large
number of submissions on each day.

201120122013201420152016201720182019202020212022$0.2$$0.4$$0.6$$0.8$$1$$1.2$$\cdot
10^{6}$Year#Submissions (a)

12345678910111213141516171819202122232425262728293031Submission/day#
Submissions# submissions/day (b)

Figure 6. The statistics of submissions received by the AOJ system over the
years

The average waiting time for the AOJ system to evaluate a submission was
calculated. Fig. 7 shows the waiting time for the AOJ to evaluate submissions
by year. The results show that most submissions were processed within one
second (approximately 79.11%). The waiting time has tended to decrease in
recent years (see Fig. 7), which means that the effectiveness of the AOJ in
evaluating the submitted code is increasing. However, the number of
submissions has increased significantly in recent years. These statistics
demonstrate that the AOJ system can be a good candidate for a state-of-the-art
AES system.

20122013201420152016201720182019202020212022$0.5$$1$$1.5$$2$$2.5$$3$Yearwaiting
time (Sec)Average Figure 7. Year-wise waiting time to evaluate submissions by
AOJ

###  6.5. Research

The real-world resources (e.g., source codes, logs, and educational data) of
the AOJ system have become important for research in learning analytics,
educational data mining, coding analysis, and programming support. The data
resources of the AOJ system are freely available for
research888http://developers.u-aizu.ac.jp/index. Meanwhile, the solution code
of the AOJ system has been used by IBM for their CodeNET project (Puri et al.,
2021) and by Google for their AlphaCode project (Li et al., 2022a). Table 8
shows the list of published research papers using AOJ data resources. It can
be seen that the published research papers using AOJ data resources have
attracted the attention of researchers and received numerous citations. It is
noted that the citations are counted based on Google scholar. In addition,
other research papers can be found on the AOJ
website999https://onlinejudge.u-aizu.ac.jp/papers.

Table 8. A list of published research using AOJ data Category of Study |  Title | #Citations | Year of Pub.
---|---|---|---
Learning Analytics and Data Mining |  Impact of Practical Skills on Academic Performance: A Data-Driven Analysis (Rahman et al., 2021b) | 12 | 2021
Learning Path Recommendation System for Programming Education Based on Neural Networks (Saito and Watanobe, 2020) | 73 | 2020
A Novel Rule-Based Online Judge Recommender System to Promote Computer Programming Education (Rahman et al., 2021c) | 8 | 2021
Data Analysis and Code Assessment Using Machine Learning Techniques for Programming Activities (Rahman, 2022) | 1 | 2022
Educational Data Mining to Support Programming Learning Using Problem-Solving Data (Rahman et al., 2022) | 12 | 2022
Categorization of frequent errors in solution codes created by novice programmers (Rahman et al., 2021a) | 3 | 2021
Dataset |  Project CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks (Puri et al., 2021) | 44 | 2021
Competition-Level Code Generation with AlphaCode (Li et al., 2022a) | 155 | 2022
Programming Support |  A Model with Iterative Trials for Correcting Logic Errors in Source Code (Matsumoto et al., 2021) | 5 | 2021
A Bidirectional LSTM Language Model for Code Evaluation and Repair (M. Mostafizer et al., 2021) | 50 | 2021
Prompt Sensitivity of Language Model for Solving Programming Problems (Shirafuji et al., 2022) | - | 2022
Source Code Assessment and Classification Based on Estimated Error Probability Using Attentive LSTM Language Model and Its Application in Programming Education (Rahman et al., 2020) | 43 | 2020
Algorithm to Determine Extended Edit Distance between Program Codes (Anzai and Watanobe, 2019) | 5 | 2019
Code Completion for Programming Education based on Deep Learning (Terada and Watanobe, 2021) | 6 | 2021
Identifying Algorithm in Program Code based on Structural Features Using CNN Classification Model (Watanobe et al., 2022a) | 6 | 2022
Automatic Generation of Fill-in-the-Blank Programming Problems (Terada and Watanobe, 2019) | 14 | 2019
Logic Error Detection System Based On Structure Pattern And Error Degree (Yoshizawa and Watanobe, 2019) | 7 | 2019
OJ System |  Online Judge System: Requirements, Architecture, and Experiences (Watanobe et al., 2022b) | 9 | 2022

##  7\. Conclusion

The extensive utilization of automated code evaluation systems (AESs) across
various domains is undeniably significant, and the reach of these systems
continues to expand. Consequently, the classification of AESs according to
their specific applications holds immense value for users. Additionally, the
data resources generated by AESs offer considerable utility for diverse
research and development tasks. In this survey paper, we have presented a
thorough and concise examination of AESs and their associated data resources.
Our initial emphasis was on categorizing AESs according to their respective
application domains. Subsequently, we provided an overview of the available
resources and datasets within AESs, facilitating users in their research and
development endeavors. Furthermore, we summarized the application of machine
learning in various coding analysis tasks aimed at problem-solving in
programming. Lastly, we briefly discussed the Aizu Online Judge System as an
illustrative real-world example of an AES, considering factors such as
hardware and software development, operation, maintenance, and research
aspects.

###### Acknowledgements.

This research was supported by the Japan Society for the Promotion of Science
(JSPS) KAKENHI (https://kaken.nii.ac.jp/en/grant/KAKENHI-PROJECT-23H03508/
accessed on 15 June 2023). Grant Number: 23H03508.

## References

  * (1)
  * Abdeljaber et al. (2017) Osama Abdeljaber, Onur Avci, Serkan Kiranyaz, Moncef Gabbouj, and Daniel J. Inman. 2017.  Real-time vibration-based structural damage detection using one-dimensional convolutional neural networks.  _Journal of Sound and Vibration_ 388 (2017), 154–170.  https://doi.org/10.1016/j.jsv.2016.10.043
  * Abunadi and Alenezi (2015) Ibrahim Abunadi and Mamdouh Alenezi. 2015.  Towards Cross Project Vulnerability Prediction in Open Source Web Applications. In _Proceedings of The International Conference on Engineering’ MIS 2015_ (Istanbul, Turkey) _(ICEMIS ’15)_. Association for Computing Machinery, New York, NY, USA, Article 42, 5 pages.  https://doi.org/10.1145/2832987.2833051
  * Aggarwal (2020) Simran Aggarwal. 2020\.  Software Code Analysis Using Ensemble Learning Techniques _(AISS ’19)_. Association for Computing Machinery, New York, NY, USA, Article 9, 7 pages.  https://doi.org/10.1145/3373477.3373486
  * Ahmad et al. (2020) Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2020.  A transformer-based approach for source code summarization.  _arXiv preprint arXiv:2005.00653_ (2020).
  * Ain et al. (2019) Qurat Ul Ain, Wasi Haider Butt, Muhammad Waseem Anwar, Farooque Azam, and Bilal Maqbool. 2019\.  A Systematic Review on Code Clone Detection.  _IEEE Access_ 7 (2019), 86121–86144.  https://doi.org/10.1109/ACCESS.2019.2918202
  * Al-Ashwal et al. (2018) Deena Al-Ashwal, Eman Zaid Al-Sewari, and Asma Abdulghani Al-Shargabi. 2018\.  A CASE tool for JAVA programs logical errors detection: Static and dynamic testing. In _2018 International Arab Conference on Information Technology (ACIT)_. IEEE, 1–6.
  * Ala-Mutka (2005) Kirsti M Ala-Mutka. 2005\.  A survey of automated assessment approaches for programming assignments.  _Computer science education_ 15, 2 (2005), 83–102.
  * Allamanis et al. (2015) Miltiadis Allamanis, Earl T. Barr, Christian Bird, and Charles Sutton. 2015.  Suggesting Accurate Method and Class Names. In _Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering_ (Bergamo, Italy) _(ESEC/FSE 2015)_. Association for Computing Machinery, New York, NY, USA, 38–49.  https://doi.org/10.1145/2786805.2786849
  * Allamanis et al. (2018) Miltiadis Allamanis, Earl T. Barr, Premkumar Devanbu, and Charles Sutton. 2018.  A Survey of Machine Learning for Big Code and Naturalness.  _ACM Comput. Surv._ 51, 4, Article 81 (jul 2018), 37 pages.  https://doi.org/10.1145/3212695
  * Allamanis et al. (2017) Miltiadis Allamanis, Marc Brockschmidt, and Mahmoud Khademi. 2017\.  Learning to represent programs with graphs.  _arXiv preprint arXiv:1711.00740_ (2017).
  * Allamanis et al. (2016) Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016\.  A convolutional attention network for extreme summarization of source code. In _International conference on machine learning_. PMLR, 2091–2100.
  * Allamanis and Sutton (2013) Miltiadis Allamanis and Charles Sutton. 2013.  Mining Source Code Repositories at Massive Scale Using Language Modeling. In _Proceedings of the 10th Working Conference on Mining Software Repositories_ (San Francisco, CA, USA) _(MSR ’13)_. IEEE Press, 207–216.
  * Alon et al. (2019) Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. 2019.  code2vec: Learning distributed representations of code.  _Proceedings of the ACM on Programming Languages_ 3, POPL (2019), 1–29.
  * Alsolai and Roper (2020) Hadeel Alsolai and Marc Roper. 2020.  A systematic literature review of machine learning techniques for software maintainability prediction.  _Information and Software Technology_ 119 (2020), 106214.  https://doi.org/10.1016/j.infsof.2019.106214
  * Amorim et al. (2018) Leonardo Afonso Amorim, Mateus F. Freitas, Altino Dantas, Eduardo F. de Souza, Celso G. Camilo-Junior, and Wellington S. Martins. 2018\.  A New Word Embedding Approach to Evaluate Potential Fixes for Automated Program Repair. In _2018 International Joint Conference on Neural Networks (IJCNN)_. 1–8.  https://doi.org/10.1109/IJCNN.2018.8489079
  * Aniche et al. (2022) Maurício Aniche, Erick Maziero, Rafael Durelli, and Vinicius H. S. Durelli. 2022.  The Effectiveness of Supervised Machine Learning Algorithms in Predicting Software Refactoring.  _IEEE Transactions on Software Engineering_ 48, 4 (2022), 1432–1450.  https://doi.org/10.1109/TSE.2020.3021736
  * Anzai and Watanobe (2019) Kazuki Anzai and Yutaka Watanobe. 2019.  Algorithm to determine extended edit distance between program codes. In _2019 IEEE 13th International symposium on embedded multicore/many-core systems-on-chip (MCSoC)_. IEEE, 180–186.
  * Athiwaratkun et al. (2022) Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ramanathan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta, Dan Roth, and Bing Xiang. 2022\.  Multi-lingual Evaluation of Code Generation Models.  arXiv:2210.14868 [cs.LG]
  * Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021.  Program Synthesis with Large Language Models.  arXiv:2108.07732 [cs.PL]
  * Azeem et al. (2019) Muhammad Ilyas Azeem, Fabio Palomba, Lin Shi, and Qing Wang. 2019.  Machine learning techniques for code smell detection: A systematic literature review and meta-analysis.  _Information and Software Technology_ 108 (2019), 115–138.
  * Bandara and Wijayarathna (2011) Upul Bandara and Gamini Wijayarathna. 2011.  A machine learning based tool for source code plagiarism detection.  _International Journal of Machine Learning and Computing_ 1, 4 (2011), 337\.
  * Barbez et al. (2020) Antoine Barbez, Foutse Khomh, and Yann-Gaël Guéhéneuc. 2020\.  A machine-learning based ensemble method for anti-patterns detection.  _Journal of Systems and Software_ 161 (2020), 110486.
  * Bavarian et al. (2022) Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. 2022.  Efficient Training of Language Models to Fill in the Middle.  arXiv:2207.14255 [cs.CL]
  * Ben-Nun et al. (2018) Tal Ben-Nun, Alice Shoshana Jakobovits, and Torsten Hoefler. 2018.  Neural Code Comprehension: A Learnable Representation of Code Semantics. In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ _(NIPS’18)_. Curran Associates Inc., Red Hook, NY, USA, 3589–3601.
  * Bielik et al. (2016) Pavol Bielik, Veselin Raychev, and Martin Vechev. 2016\.  PHOG: probabilistic model for code. In _International conference on machine learning_. PMLR, 2933–2942.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners. In _Proceedings of Advances in Neural Information Processing Systems (NeurIPS)_ , Vol. 33. 1877–1901.
  * Butgereit (2019) Laurie Butgereit. 2019\.  Using Machine Learning to Prioritize Automated Testing in an Agile Environment. In _2019 Conference on Information Communications Technology and Society (ICTAS)_. 1–6.  https://doi.org/10.1109/ICTAS.2019.8703639
  * Büch and Andrzejak (2019) Lutz Büch and Artur Andrzejak. 2019.  Learning-Based Recursive Aggregation of Abstract Syntax Trees for Code Clone Detection. In _2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER)_. 95–104.  https://doi.org/10.1109/SANER.2019.8668039
  * Cheang et al. (2003) Brenda Cheang, Andy Kurnia, Andrew Lim, and Wee-Chong Oon. 2003.  On Automated Grading of Programming Assignments in an Academic Institution.  _Comput. Educ._ 41, 2 (sep 2003), 121–131.  https://doi.org/10.1016/S0360-1315(03)00030-7
  * Chen et al. (2020a) Jinyin Chen, Keke Hu, Yue Yu, Zhuangzhi Chen, Qi Xuan, Yi Liu, and Vladimir Filkov. 2020a.  Software visualization and deep transfer learning for effective software defect prediction. In _Proceedings of the ACM/IEEE 42nd international conference on software engineering_. 578–589.
  * Chen et al. (2020b) Jinyin Chen, Keke Hu, Yue Yu, Zhuangzhi Chen, Qi Xuan, Yi Liu, and Vladimir Filkov. 2020b.  Software Visualization and Deep Transfer Learning for Effective Software Defect Prediction. In _2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE)_. 578–589.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021\.  Evaluating Large Language Models Trained on Code.  _arXiv preprint_ arXiv:2107.03374 (2021).  https://doi.org/10.48550/arXiv.2107.03374
  * Clement et al. (2020) Colin B Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, and Neel Sundaresan. 2020\.  PyMT5: multi-mode translation of natural language and Python code with transformers.  _arXiv preprint arXiv:2010.03150_ (2020).
  * Combéfis and Wautelet (2014) Sébastien Combéfis and Jérémy Wautelet. 2014.  Programming Trainings and Informatics Teaching Through Online Contests.  _Olympiads in Informatics_ 8 (2014).
  * Cormack et al. (2006) Gordon Cormack, Ian Munro, Troy Vasiga, and Graeme Kemkes. 2006.  Structure, Scoring and Purpose of Computing Competitions.  _Informatics in Education_ 5, 1 (jan 2006), 15–36.
  * Cortés-Coy et al. (2014) Luis Fernando Cortés-Coy, Mario Linares-Vásquez, Jairo Aponte, and Denys Poshyvanyk. 2014.  On automatically generating commit messages via summarization of source code changes. In _2014 IEEE 14th International Working Conference on Source Code Analysis and Manipulation_. IEEE, 275–284.
  * Cosma and Joy (2008) Georgina Cosma and Mike Joy. 2008.  Towards a definition of source-code plagiarism.  _IEEE Transactions on Education_ 51, 2 (2008), 195–200.
  * Costello and Stolovitzky (2013) JC Costello and G Stolovitzky. 2013.  Seeking the wisdom of crowds through challenge-based competitions in biomedical research.  _Clinical Pharmacology & Therapeutics_ 93, 5 (2013), 396–398.
  * Cvitkovic et al. (2019) Milan Cvitkovic, Badal Singh, and Animashree Anandkumar. 2019\.  Open vocabulary learning on source code with a graph-structured cache. In _International Conference on Machine Learning_. PMLR, 1475–1485.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186.  https://doi.org/10.18653/v1/N19-1423
  * Došilović and Mekterović (2020) Herman Zvonimir Došilović and Igor Mekterović. 2020.  Robust and Scalable Online Code Execution System. In _2020 43rd International Convention on Information, Communication and Electronic Technology (MIPRO)_. 1627–1632.  https://doi.org/10.23919/MIPRO48935.2020.9245310
  * Fang et al. (2020) Chunrong Fang, Zixi Liu, Yangyang Shi, Jeff Huang, and Qingkai Shi. 2020.  Functional Code Clone Detection with Syntax and Semantics Fusion Learning. In _Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis_ (Virtual Event, USA) _(ISSTA 2020)_. Association for Computing Machinery, New York, NY, USA, 516–527.  https://doi.org/10.1145/3395363.3397362
  * Felter et al. (2015) Wes Felter, Alexandre Ferreira, Ram Rajamony, and Juan Rubio. 2015.  An updated performance comparison of virtual machines and Linux containers. In _2015 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)_. 171–172.  https://doi.org/10.1109/ISPASS.2015.7095802
  * Feng et al. (2020) Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020.  CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. Association for Computational Linguistics, Online, 1536–1547.  https://doi.org/10.18653/v1/2020.findings-emnlp.139
  * Fernandes et al. (2018) Patrick Fernandes, Miltiadis Allamanis, and Marc Brockschmidt. 2018.  Structured neural summarization.  _arXiv preprint arXiv:1811.01824_ (2018).
  * Fonte et al. (2013) Daniela Fonte, Daniela da Cruz, Alda Lopes Gançarski, and Pedro Rangel Henriques. 2013.  A flexible dynamic system for automatic grading of programming exercises.  (2013).
  * Forisek (2007) Michal Forisek. 2007\.  Security of Programming Contest Systems.
  * Goldbloom (2010) Anthony Goldbloom. 2010\.  Data Prediction Competitions – Far More than Just a Bit of Fun. In _2010 IEEE International Conference on Data Mining Workshops_. 1385–1386.  https://doi.org/10.1109/ICDMW.2010.56
  * Google (2021) Google. 2021.  gcj-dataset.  Available: https://openreview.net/attachment?id=AZ4vmLoJft&name=supplementary_material.
  * Gopinath et al. (2014) Divya Gopinath, Sarfraz Khurshid, Diptikalyan Saha, and Satish Chandra. 2014.  Data-guided repair of selection statements. In _Proceedings of the 36th International Conference on Software Engineering_. 243–253.
  * Goues et al. (2019) Claire Le Goues, Michael Pradel, and Abhik Roychoudhury. 2019\.  Automated program repair.  _Commun. ACM_ 62, 12 (2019), 56–65.
  * Gousios (2013) Georgios Gousios. 2013\.  The GHTorent dataset and tool suite. In _2013 10th Working Conference on Mining Software Repositories (MSR)_. IEEE, 233–236.
  * Graves et al. (2013) Alex Graves, Navdeep Jaitly, and Abdel-rahman Mohamed. 2013\.  Hybrid speech recognition with Deep Bidirectional LSTM. In _2013 IEEE Workshop on Automatic Speech Recognition and Understanding_. 273–278.  https://doi.org/10.1109/ASRU.2013.6707742
  * Greff et al. (2017) Klaus Greff, Rupesh K. Srivastava, Jan Koutník, Bas R. Steunebrink, and Jürgen Schmidhuber. 2017\.  LSTM: A Search Space Odyssey.  _IEEE Transactions on Neural Networks and Learning Systems_ 28, 10 (2017), 2222–2232.  https://doi.org/10.1109/TNNLS.2016.2582924
  * Gu et al. (2018) Xiaodong Gu, Hongyu Zhang, and Sunghun Kim. 2018\.  Deep Code Search. In _2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE)_. 933–944.  https://doi.org/10.1145/3180155.3180167
  * Guggulothu and Moiz (2020) Thirupathi Guggulothu and Salman Abdul Moiz. 2020.  Code smell detection using multi-label classification approach.  _Software Quality Journal_ 28 (2020), 1063–1086.
  * Guo et al. (2021) Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie LIU, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021\.  GraphCodeBERT: Pre-training Code Representations with Data Flow. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=jLoC4ez43PZ
  * Guo et al. (2019) Daya Guo, Duyu Tang, Nan Duan, Ming Zhou, and Jian Yin. 2019.  Coupling retrieval and meta-learning for context-dependent semantic parsing.  _arXiv preprint arXiv:1906.07108_ (2019).
  * Gupta et al. (2017) Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish Shevade. 2017\.  DeepFix: Fixing Common C Language Errors by Deep Learning. In _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence_ (San Francisco, California, USA) _(AAAI’17)_. AAAI Press, 1345–1351.
  * Hellendoorn and Devanbu (2017) Vincent J. Hellendoorn and Premkumar Devanbu. 2017.  Are Deep Neural Networks the Best Choice for Modeling Source Code?. In _Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering_ (Paderborn, Germany) _(ESEC/FSE 2017)_. Association for Computing Machinery, New York, NY, USA, 763–773.  https://doi.org/10.1145/3106237.3106290
  * Hendrycks et al. (2021) Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021.  Measuring Coding Challenge Competence With APPS.  _In: Proceedings of Conference on Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks (Round 2)_ (2021).
  * Hoang et al. (2020) Thong Hoang, Hong Jin Kang, David Lo, and Julia Lawall. 2020\.  CC2Vec: Distributed representations of code changes. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_. 518–529.
  * Hsiao et al. (2020) I-Han Hsiao, Po-Kai Huang, and Hannah Murphy. 2020\.  Integrating Programming Learning Analytics Across Physical and Digital Space.  _IEEE Transactions on Emerging Topics in Computing_ 8, 1 (2020), 206–217.  https://doi.org/10.1109/TETC.2017.2701201
  * Hu et al. (2018b) Gang Hu, Linjie Zhu, and Junfeng Yang. 2018b.  AppFlow: using machine learning to synthesize robust, reusable UI tests. In _Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 269–282.
  * Hu et al. (2018a) Xing Hu, Ge Li, Xin Xia, David Lo, Shuai Lu, and Zhi Jin. 2018a.  Summarizing Source Code with Transferred API Knowledge. In _Proceedings of the 27th International Joint Conference on Artificial Intelligence_ (Stockholm, Sweden) _(IJCAI’18)_. AAAI Press, 2269–2275.
  * Husain et al. (2019) Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019.  Codesearchnet challenge: Evaluating the state of semantic code search.  _arXiv preprint arXiv:1909.09436_ (2019).
  * Ihantola et al. (2010) Petri Ihantola, Tuukka Ahoniemi, Ville Karavirta, and Otto Seppälä. 2010.  Review of Recent Systems for Automatic Assessment of Programming Assignments. In _Proceedings of the 10th Koli Calling International Conference on Computing Education Research_ (Koli, Finland) _(Koli Calling ’10)_. Association for Computing Machinery, New York, NY, USA, 86–93.  https://doi.org/10.1145/1930464.1930480
  * Iyer et al. (2019) Srinivasan Iyer, Alvin Cheung, and Luke Zettlemoyer. 2019\.  Learning programmatic idioms for scalable semantic parsing.  _arXiv preprint arXiv:1904.09086_ (2019).
  * Iyer et al. (2016) Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2016.  Summarizing source code using a neural attention model. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 2073–2083.
  * Iyer et al. (2018) Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018.  Mapping Language to Code in Programmatic Context. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Brussels, Belgium, 1643–1652.  https://doi.org/10.18653/v1/D18-1192
  * Ji et al. (2018) Tao Ji, Jinkun Pan, Liqian Chen, and Xiaoguang Mao. 2018\.  Identifying Supplementary Bug-fix Commits. In _2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC)_ , Vol. 01. 184–193.  https://doi.org/10.1109/COMPSAC.2018.00031
  * Jiang et al. (2021) Xue Jiang, Zhuoran Zheng, Chen Lyu, Liang Li, and Lei Lyu. 2021.  TreeBERT: A tree-based pre-trained model for programming language. In _Uncertainty in Artificial Intelligence_. PMLR, 54–63.
  * Just et al. (2014) René Just, Darioush Jalali, and Michael D Ernst. 2014\.  Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In _Proceedings of the 2014 international symposium on software testing and analysis_. 437–440.
  * Kanade et al. (2020) Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2020.  Learning and Evaluating Contextual Embedding of Source Code. In _Proceedings of the 37th International Conference on Machine Learning_ _(ICML’20)_. JMLR.org, Article 474, 12 pages.
  * Karaivanov et al. (2014) Svetoslav Karaivanov, Veselin Raychev, and Martin Vechev. 2014\.  Phrase-Based Statistical Translation of Programming Languages. In _Proceedings of the 2014 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software_ (Portland, Oregon, USA) _(Onward! 2014)_. Association for Computing Machinery, New York, NY, USA, 173–184.  https://doi.org/10.1145/2661136.2661148
  * Krizhevsky et al. (2017) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2017\.  ImageNet Classification with Deep Convolutional Neural Networks.  _Commun. ACM_ 60, 6 (may 2017), 84–90.  https://doi.org/10.1145/3065386
  * Kurnia et al. (2001) Andy Kurnia, Andrew Lim, and Brenda Cheang. 2001.  Online Judge.  _Computers & Education_ 36, 4 (2001), 299–315.  https://doi.org/10.1016/S0360-1315(01)00018-5
  * Le et al. (2020) Triet H. M. Le, Hao Chen, and Muhammad Ali Babar. 2020.  Deep Learning for Source Code Modeling and Generation: Models, Applications, and Challenges.  _ACM Comput. Surv._ 53, 3, Article 62 (jun 2020), 38 pages.  https://doi.org/10.1145/3383458
  * Le et al. (2015) Xuan-Bach D. Le, Tien-Duy B. Le, and David Lo. 2015\.  Should fixing these failures be delegated to automated program repair?. In _2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE)_. 427–437.  https://doi.org/10.1109/ISSRE.2015.7381836
  * Le Goues et al. (2015) Claire Le Goues, Neal Holtschulte, Edward K Smith, Yuriy Brun, Premkumar Devanbu, Stephanie Forrest, and Westley Weimer. 2015.  The ManyBugs and IntroClass benchmarks for automated repair of C programs.  _IEEE Transactions on Software Engineering_ 41, 12 (2015), 1236–1256.
  * Leal and Silva (2003) José Paulo Leal and Fernando Silva. 2003.  Mooshak: A Web-based multi-site programming contest system.  _Software: Practice and Experience_ 33, 6 (2003), 567–581.
  * LeClair et al. (2019) Alexander LeClair, Siyuan Jiang, and Collin McMillan. 2019\.  A Neural Model for Generating Natural Language Summaries of Program Subroutines. In _Proceedings of the 41st International Conference on Software Engineering_ (Montreal, Quebec, Canada) _(ICSE ’19)_. IEEE Press, 795–806.  https://doi.org/10.1109/ICSE.2019.00087
  * Lee et al. (2017) Song-Mi Lee, Sang Min Yoon, and Heeryon Cho. 2017.  Human activity recognition from accelerometer data using Convolutional Neural Network. In _2017 IEEE International Conference on Big Data and Smart Computing (BigComp)_. 131–134.  https://doi.org/10.1109/BIGCOMP.2017.7881728
  * Lei et al. (2022) Maggie Lei, Hao Li, Ji Li, Namrata Aundhkar, and Dae-Kyoo Kim. 2022.  Deep learning application on code clone detection: A review of current knowledge.  _Journal of Systems and Software_ 184 (2022), 111141.  https://doi.org/10.1016/j.jss.2021.111141
  * Li et al. (2018) Jian Li, Yue Wang, Michael R. Lyu, and Irwin King. 2018\.  Code Completion with Neural Attention and Pointer Networks. In _Proceedings of the 27th International Joint Conference on Artificial Intelligence_ (Stockholm, Sweden) _(IJCAI’18)_. AAAI Press, 4159–25.
  * Li et al. (2017) Liuqing Li, He Feng, Wenjie Zhuang, Na Meng, and Barbara Ryder. 2017.  CCLearner: A Deep Learning-Based Clone Detection Approach. In _2017 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. 249–260.  https://doi.org/10.1109/ICSME.2017.46
  * Li et al. (2022a) Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. 2022a.  Competition-level code generation with AlphaCode.  _Science_ 378, 6624 (2022), 1092–1097.  https://doi.org/10.1126/science.abq1158
  * Li et al. (2022b) Yi Li, Shaohua Wang, and Tien N. Nguyen. 2022b.  DEAR: A Novel Deep Learning-Based Approach for Automated Program Repair. In _Proceedings of the 44th International Conference on Software Engineering_ (Pittsburgh, Pennsylvania) _(ICSE ’22)_. Association for Computing Machinery, New York, NY, USA, 511–523.  https://doi.org/10.1145/3510003.3510177
  * Li et al. (2019a) Yi Li, Shaohua Wang, Tien N Nguyen, and Son Van Nguyen. 2019a.  Improving bug detection via context-based code representation learning and attention-based neural networks.  _Proceedings of the ACM on Programming Languages_ 3, OOPSLA (2019), 1–30.
  * Li et al. (2019b) Yi Li, Shaohua Wang, Tien N. Nguyen, and Son Van Nguyen. 2019b.  Improving Bug Detection via Context-Based Code Representation Learning and Attention-Based Neural Networks.  _Proc. ACM Program. Lang._ 3, OOPSLA, Article 162 (oct 2019), 30 pages.  https://doi.org/10.1145/3360588
  * Lima et al. (2020) Rui Lima, António Miguel Rosado da Cruz, and Jorge Ribeiro. 2020\.  Artificial Intelligence Applied to Software Testing: A Literature Review. In _2020 15th Iberian Conference on Information Systems and Technologies (CISTI)_. 1–6.  https://doi.org/10.23919/CISTI49556.2020.9141124
  * Lin et al. (2017) Derrick Lin, James Koppel, Angela Chen, and Armando Solar-Lezama. 2017.  QuixBugs: A Multi-Lingual Program Repair Benchmark Set Based on the Quixey Challenge. In _Proceedings Companion of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity_ (Vancouver, BC, Canada) _(SPLASH Companion 2017)_. Association for Computing Machinery, New York, NY, USA, 55–56.  https://doi.org/10.1145/3135932.3135941
  * Lin et al. (2018) Guanjun Lin, Jun Zhang, Wei Luo, Lei Pan, Yang Xiang, Olivier De Vel, and Paul Montague. 2018.  Cross-project transfer representation learning for vulnerable function discovery.  _IEEE Transactions on Industrial Informatics_ 14, 7 (2018), 3289–3297.
  * Ling et al. (2020) Chunyang Ling, Zeqi Lin, Yanzhen Zou, and Bing Xie. 2020\.  Adaptive Deep Code Search. In _Proceedings of the 28th International Conference on Program Comprehension_ (Seoul, Republic of Korea) _(ICPC ’20)_. Association for Computing Machinery, New York, NY, USA, 48–59.  https://doi.org/10.1145/3387904.3389278
  * Liu et al. (2019c) Bohong Liu, Tao Wang, Xunhui Zhang, Qiang Fan, Gang Yin, and Jinsheng Deng. 2019c.  A Neural-Network Based Code Summarization Approach by Using Source Code and Its Call Dependencies. In _Proceedings of the 11th Asia-Pacific Symposium on Internetware_ (Fukuoka, Japan) _(Internetware ’19)_. Association for Computing Machinery, New York, NY, USA, Article 12, 10 pages.  https://doi.org/10.1145/3361242.3362774
  * Liu et al. (2016) Chang Liu, Xinyun Chen, Eui Chul Shin, Mingcheng Chen, and Dawn Song. 2016.  Latent attention for if-then program synthesis.  _Advances in Neural Information Processing Systems_ 29 (2016).
  * Liu et al. (2020) Fang Liu, Ge Li, Bolin Wei, Xin Xia, Zhiyi Fu, and Zhi Jin. 2020\.  A self-attentional neural architecture for code completion with multi-task learning. In _Proceedings of the 28th International Conference on Program Comprehension_. 37–47.
  * Liu et al. (2021) Fang Liu, Ge Li, Yunfei Zhao, and Zhi Jin. 2021\.  Multi-Task Learning Based Pre-Trained Language Model for Code Completion. In _Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering_ (Virtual Event, Australia) _(ASE ’20)_. Association for Computing Machinery, New York, NY, USA, 473–485.  https://doi.org/10.1145/3324884.3416591
  * Liu et al. (2019a) Xiao Liu, Xiaoting Li, Rupesh Prajapati, and Dinghao Wu. 2019a.  Deepfuzz: Automatic generation of syntax valid c programs for fuzz testing. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 1044–1051.
  * Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _arXiv preprint_ arXiv:1907.11692 (2019).  https://doi.org/10.48550/arXiv.1907.11692
  * Lu et al. (2021) Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021\.  Codexglue: A machine learning benchmark dataset for code understanding and generation.  _arXiv preprint arXiv:2102.04664_ (2021).
  * Lu et al. (2017) Xudong Lu, Dongyu Zheng, and Lei Liu. 2017.  Data Driven Analysis on the Effect of Online Judge System. In _2017 IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData)_. 573–577.  https://doi.org/10.1109/iThings-GreenCom-CPSCom-SmartData.2017.90
  * M. Mostafizer et al. (2020) Rahman M. Mostafizer, Yutaka Watanobe, and Keita Nakamura. 2020\.  A Neural Network Based Intelligent Support Model for Program Code Completion.  _Scientific Programming_ 2020 (2020).  https://doi.org/10.1155/2020/7426461
  * M. Mostafizer et al. (2021) Rahman M. Mostafizer, Yutaka Watanobe, and Keita Nakamura. 2021\.  A Bidirectional LSTM Language Model for Code Evaluation and Repair.  _Symmetry_ 13, 2 (2021).  https://doi.org/10.3390/sym13020247
  * Ma et al. (2018) Yuzhan Ma, Sarah Fakhoury, Michael Christensen, Venera Arnaoudova, Waleed Zogaan, and Mehdi Mirakhorli. 2018.  Automatic Classification of Software Artifacts in Open-Source Applications. In _Proceedings of the 15th International Conference on Mining Software Repositories_ (Gothenburg, Sweden) _(MSR ’18)_. Association for Computing Machinery, New York, NY, USA, 414–425.  https://doi.org/10.1145/3196398.3196446
  * Matsumoto et al. (2021) Taku Matsumoto, Yutaka Watanobe, and Keita Nakamura. 2021\.  A model with iterative trials for correcting logic errors in source code.  _Applied Sciences_ 11, 11 (2021), 4755.
  * Medeiros et al. (2013) Ibéria Medeiros, Nuno F Neves, and Miguel Correia. 2013\.  Securing energy metering software with automatic source code correction. In _2013 11th IEEE International Conference on Industrial Informatics (INDIN)_. IEEE, 701–706.
  * Merkel (2014) Dirk Merkel. 2014\.  Docker: Lightweight Linux Containers for Consistent Development and Deployment.  _Linux J._ 2014, 239, Article 2 (mar 2014).
  * Meyer and Saez-Rodriguez (2021) Pablo Meyer and Julio Saez-Rodriguez. 2021.  Advances in systems biology modeling: 10 years of crowdsourcing DREAM challenges.  _Cell Systems_ 12, 6 (2021), 636–653.
  * Miceli Barone and Sennrich (2017) Antonio Valerio Miceli Barone and Rico Sennrich. 2017.  A Parallel Corpus of Python Functions and Documentation Strings for Automated Code Documentation and Code Generation. In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_. Asian Federation of Natural Language Processing, Taipei, Taiwan, 314–319.  https://aclanthology.org/I17-2053
  * Mirzayanov (2020) Mike Mirzayanov. 2020\.  Codeforces: Results of 2020.  https://codeforces.com/blog/entry/89502.  Accessed: 2023-05-23.
  * Mostaeen et al. (2020) Golam Mostaeen, Banani Roy, Chanchal K Roy, Kevin Schneider, and Jeffrey Svajlenko. 2020.  A machine learning based framework for code clone validation.  _Journal of Systems and Software_ 169 (2020), 110686.
  * Mostaeen et al. (2018) Golam Mostaeen, Jeffrey Svajlenko, Banani Roy, Chanchal K. Roy, and Kevin A. Schneider. 2018.  On the Use of Machine Learning Techniques Towards the Design of Cloud Based Automatic Code Clone Validation Tools. In _2018 IEEE 18th International Working Conference on Source Code Analysis and Manipulation (SCAM)_. 155–164.  https://doi.org/10.1109/SCAM.2018.00025
  * Mou et al. (2016) Lili Mou, Ge Li, Lu Zhang, Tao Wang, and Zhi Jin. 2016.  Convolutional Neural Networks over Tree Structures for Programming Language Processing. In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_ (Phoenix, Arizona) _(AAAI’16)_. AAAI Press, 1287–1293.
  * Naohiro (2012) Takahashi Naohiro. 2012\.  AtCoder Inc.  Available: https://atcoder.jp/.
  * NÉMETH and ZSAKÓ (2015) Ágnes Erdősné NÉMETH and László ZSAKÓ. 2015\.  Online training and contests for informatics contestants of secondary school age.  _Edukacja-Technika-Informatyka_ 6, 1 (2015), 273–280.
  * Nguyen et al. (2015) Anh Tuan Nguyen, Tung Thanh Nguyen, and Tien N. Nguyen. 2015\.  Divide-and-Conquer Approach for Multi-phase Statistical Migration for Source Code (T). In _2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. 585–596.  https://doi.org/10.1109/ASE.2015.74
  * Nikolaou (2021) Ioannis Nikolaou. 2021\.  What is the Role of Technology in Recruitment and Selection?  _The Spanish journal of psychology_ 24 (2021), e2.
  * Oksanen (2018) Reija Oksanen. 2018\.  _New technology-based recruitment methods_.  Master’s thesis.
  * Omri and Sinz (2020) Safa Omri and Carsten Sinz. 2020.  Deep Learning for Software Defect Prediction: A Survey. In _Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops_ (Seoul, Republic of Korea) _(ICSEW’20)_. Association for Computing Machinery, New York, NY, USA, 209–214.  https://doi.org/10.1145/3387940.3391463
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022\.  Training language models to follow instructions with human feedback. In _Advances in Neural Information Processing Systems_ , Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (Eds.).  https://openreview.net/forum?id=TG8KACxEON
  * Panichella et al. (2012) Sebastiano Panichella, Jairo Aponte, Massimiliano Di Penta, Andrian Marcus, and Gerardo Canfora. 2012\.  Mining source code descriptions from developer communications. In _2012 20th IEEE International Conference on Program Comprehension (ICPC)_. IEEE, 63–72.
  * Perl et al. (2015) Henning Perl, Sergej Dechand, Matthew Smith, Daniel Arp, Fabian Yamaguchi, Konrad Rieck, Sascha Fahl, and Yasemin Acar. 2015\.  Vccfinder: Finding potential vulnerabilities in open-source projects to assist code audits. In _Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security_. 426–437.
  * Piskachev et al. (2019) Goran Piskachev, Lisa Nguyen Quang Do, and Eric Bodden. 2019\.  Codebase-adaptive detection of security-relevant methods. In _Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis_. 181–191.
  * Pohl (2006) Wolfgang Pohl. 2006\.  Computer Science Contests for Secondary School Students: Approaches to Classification.  _Informatics in Education_ 5, 1 (jan 2006), 125–132.
  * Ponta et al. (2019) Serena Elisa Ponta, Henrik Plate, Antonino Sabetta, Michele Bezzi, and Cédric Dangremont. 2019\.  A manually-curated dataset of fixes to vulnerabilities of open-source software. In _2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR)_. IEEE, 383–387.
  * Pradel and Sen (2018a) Michael Pradel and Koushik Sen. 2018a.  Deepbugs: A learning approach to name-based bug detection.  _Proceedings of the ACM on Programming Languages_ 2, OOPSLA (2018), 1–25.
  * Pradel and Sen (2018b) Michael Pradel and Koushik Sen. 2018b.  DeepBugs: A Learning Approach to Name-Based Bug Detection.  _Proc. ACM Program. Lang._ 2, OOPSLA, Article 147 (oct 2018), 25 pages.  https://doi.org/10.1145/3276517
  * Premtoon et al. (2020) Varot Premtoon, James Koppel, and Armando Solar-Lezama. 2020\.  Semantic Code Search via Equational Reasoning. In _Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation_ (London, UK) _(PLDI 2020)_. Association for Computing Machinery, New York, NY, USA, 1066–1082.  https://doi.org/10.1145/3385412.3386001
  * Pu et al. (2016) Yewen Pu, Karthik Narasimhan, Armando Solar-Lezama, and Regina Barzilay. 2016.  sk_p: a neural program corrector for MOOCs. In _Companion Proceedings of the 2016 ACM SIGPLAN International Conference on Systems, Programming, Languages and Applications: Software for Humanity_. 39–40.
  * Puri et al. (2021) Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladmir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, et al. 2021\.  Project codenet: A large-scale ai for code dataset for learning a diversity of coding tasks.  _arXiv preprint arXiv:2105.12655_ 1035 (2021).
  * Qasem et al. (2020) Osama Al Qasem, Mohammed Akour, and Mamdouh Alenezi. 2020\.  The Influence of Deep Learning Algorithms Factors in Software Fault Prediction.  _IEEE Access_ 8 (2020), 63945–63960.  https://doi.org/10.1109/ACCESS.2020.2985290
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.  (2018).
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019\.  Language Models are Unsupervised Multitask Learners.  (2019).
  * Rahman (2022) Md Mostafizer Rahman. 2022\.  Data Analysis and Code Assessment Using Machine Learning Techniques for Programming Activities.  https://doi.org/10.15016/00000215
  * Rahman et al. (2021a) Md Mostafizer Rahman, Shunsuke Kawabayashi, and Yutaka Watanobe. 2021a.  Categorization of frequent errors in solution codes created by novice programmers. In _SHS Web of Conferences_ , Vol. 102. EDP Sciences, 04014\.
  * Rahman et al. (2021b) Md. Mostafizer Rahman, Yutaka Watanobe, Rage Uday Kiran, Truong Cong Thang, and Incheon Paik. 2021b.  Impact of Practical Skills on Academic Performance: A Data-Driven Analysis.  _IEEE Access_ 9 (2021), 139975–139993.  https://doi.org/10.1109/ACCESS.2021.3119145
  * Rahman et al. (2022) Md. Mostafizer Rahman, Yutaka Watanobe, Taku Matsumoto, Rage Uday Kiran, and Keita Nakamura. 2022\.  Educational Data Mining to Support Programming Learning Using Problem-Solving Data.  _IEEE Access_ 10 (2022), 26186–26202.  https://doi.org/10.1109/ACCESS.2022.3157288
  * Rahman et al. (2020) Md. Mostafizer Rahman, Yutaka Watanobe, and Keita Nakamura. 2020\.  Source Code Assessment and Classification Based on Estimated Error Probability Using Attentive LSTM Language Model and Its Application in Programming Education.  _Applied Sciences_ 10, 8 (2020).  https://doi.org/10.3390/app10082973
  * Rahman et al. (2021c) Md Mostafizer Rahman, Yutaka Watanobe, Uday Kiran Rage, and Keita Nakamura. 2021c.  A novel rule-based online judge recommender system to promote computer programming education. In _Advances and Trends in Artificial Intelligence. From Theory to Practice: 34th International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, IEA/AIE 2021, Kuala Lumpur, Malaysia, July 26–29, 2021, Proceedings, Part II 34_. Springer, 15–27.
  * Ray et al. (2016) Baishakhi Ray, Vincent Hellendoorn, Saheel Godhane, Zhaopeng Tu, Alberto Bacchelli, and Premkumar Devanbu. 2016.  On the ”Naturalness” of Buggy Code. In _Proceedings of the 38th International Conference on Software Engineering_ (Austin, Texas) _(ICSE ’16)_. Association for Computing Machinery, New York, NY, USA, 428–439.  https://doi.org/10.1145/2884781.2884848
  * Raychev et al. (2016) Veselin Raychev, Pavol Bielik, and Martin Vechev. 2016\.  Probabilistic Model for Code with Decision Trees.  _SIGPLAN Not._ 51, 10 (oct 2016), 731–747.  https://doi.org/10.1145/3022671.2984041
  * Raychev et al. (2014) Veselin Raychev, Martin Vechev, and Eran Yahav. 2014\.  Code Completion with Statistical Language Models. In _Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation_ (Edinburgh, United Kingdom) _(PLDI ’14)_. Association for Computing Machinery, New York, NY, USA, 419–428.  https://doi.org/10.1145/2594291.2594321
  * Revilla et al. (2008) Miguel A Revilla, Shahriar Manzoor, and Rujia Liu. 2008\.  Competitive learning in informatics: The UVa online judge experience.  _Olympiads in Informatics_ 2, 10 (2008), 131–148.
  * Rivers and Koedinger (2013) Kelly Rivers and Kenneth R Koedinger. 2013.  Automatic generation of programming feedback: A data-driven approach. In _The First Workshop on AI-supported Education for Computer Science (AIEDCS 2013)_ , Vol. 50. 50–59.
  * Rodríguez-del Pino et al. (2012) Juan Carlos Rodríguez-del Pino, Enrique Rubio Royo, and Zenón Hernández Figueroa. 2012.  A Virtual Programming Lab for Moodle with automatic assessment and anti-plagiarism features.  (2012).
  * Roziere et al. (2020) Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. 2020.  Unsupervised Translation of Programming Languages. In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ (Vancouver, BC, Canada) _(NIPS’20)_. Curran Associates Inc., Red Hook, NY, USA, Article 1730, 11 pages.
  * Russell et al. (2018) Rebecca Russell, Louis Kim, Lei Hamilton, Tomo Lazovich, Jacob Harer, Onur Ozdemir, Paul Ellingwood, and Marc McConley. 2018\.  Automated Vulnerability Detection in Source Code Using Deep Representation Learning. In _2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA)_. 757–762.  https://doi.org/10.1109/ICMLA.2018.00120
  * Sachdev et al. (2018) Saksham Sachdev, Hongyu Li, Sifei Luan, Seohyun Kim, Koushik Sen, and Satish Chandra. 2018\.  Retrieval on Source Code: A Neural Code Search. In _Proceedings of the 2nd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages_ (Philadelphia, PA, USA) _(MAPL 2018)_. Association for Computing Machinery, New York, NY, USA, 31–41.  https://doi.org/10.1145/3211346.3211353
  * Saez-Rodriguez et al. (2016) Julio Saez-Rodriguez, James C Costello, Stephen H Friend, Michael R Kellen, Lara Mangravite, Pablo Meyer, Thea Norman, and Gustavo Stolovitzky. 2016.  Crowdsourcing biomedical research: leveraging communities as innovation engines.  _Nature Reviews Genetics_ 17, 8 (2016), 470–486.
  * Sainath et al. (2015) Tara N. Sainath, Brian Kingsbury, George Saon, Hagen Soltau, Abdel rahman Mohamed, George Dahl, and Bhuvana Ramabhadran. 2015.  Deep Convolutional Neural Networks for Large-scale Speech Tasks.  _Neural Networks_ 64 (2015), 39–48.  https://doi.org/10.1016/j.neunet.2014.08.005 Special Issue on “Deep Learning of Representations”.
  * Saito and Watanobe (2020) Tomohiro Saito and Yutaka Watanobe. 2020.  Learning Path Recommendation System for Programming Education Based on Neural Networks.  _Int. J. Distance Educ. Technol._ 18, 1 (jan 2020), 36–64.  https://doi.org/10.4018/IJDET.2020010103
  * Samara (2017) Ghassan Samara. 2017\.  A practical approach for detecting logical error in object oriented environment.  _arXiv preprint arXiv:1712.04189_ (2017).
  * Shabtai et al. (2009) Asaf Shabtai, Robert Moskovitch, Yuval Elovici, and Chanan Glezer. 2009.  Detection of malicious code by applying machine learning classifiers on static features: A state-of-the-art survey.  _information security technical report_ 14, 1 (2009), 16–29.
  * Sharma et al. (2021a) Tushar Sharma, Vasiliki Efstathiou, Panos Louridas, and Diomidis Spinellis. 2021a.  Code smell detection by deep direct-learning and transfer-learning.  _Journal of Systems and Software_ 176 (2021), 110936.
  * Sharma et al. (2021b) Tushar Sharma, Maria Kechagia, Stefanos Georgiou, Rohit Tiwari, Indira Vats, Hadi Moazen, and Federica Sarro. 2021b.  A survey on machine learning techniques for source code analysis.  _arXiv preprint arXiv:2110.09610_ (2021).
  * Sharma and Kessentini (2021) Tushar Sharma and Marouane Kessentini. 2021.  Qscored: A large dataset of code smells and quality metrics. In _2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)_. IEEE, 590–594.
  * Shen and Chen (2020) Zhidong Shen and Si Chen. 2020.  A survey of automatic software vulnerability detection, program repair, and defect prediction techniques.  _Security and Communication Networks_ 2020 (2020), 1–16.
  * Shirafuji et al. (2022) Atsushi Shirafuji, Takumi Ito, Makoto Morishita, Yuki Nakamura, Yusuke Oda, Jun Suzuki, and Yutaka Watanobe. 2022.  Prompt Sensitivity of Language Model for Solving Programming Problems.  (2022), 346–359.  https://doi.org/10.3233/FAIA220264
  * Shuai et al. (2020) Jianhang Shuai, Ling Xu, Chao Liu, Meng Yan, Xin Xia, and Yan Lei. 2020.  Improving Code Search with Co-Attentive Representation Learning. In _Proceedings of the 28th International Conference on Program Comprehension_ (Seoul, Republic of Korea) _(ICPC ’20)_. Association for Computing Machinery, New York, NY, USA, 196–207.  https://doi.org/10.1145/3387904.3389269
  * Singh et al. (2013) Rishabh Singh, Sumit Gulwani, and Armando Solar-Lezama. 2013\.  Automated feedback generation for introductory programming assignments. In _Proceedings of the 34th ACM SIGPLAN conference on Programming language design and implementation_. 15–26.
  * Skiena and Revilla (2003) Steven S Skiena and Miguel A Revilla. 2003.  Programming challenges: The programming contest training manual.  _Acm SIGACT News_ 34, 3 (2003), 68–74.
  * Song et al. (2019) Dowon Song, Myungho Lee, and Hakjoo Oh. 2019.  Automatic and scalable detection of logical errors in functional programming assignments.  _Proceedings of the ACM on Programming Languages_ 3, OOPSLA (2019), 1–30.
  * Spacco et al. (2015) Jaime Spacco, Paul Denny, Brad Richards, David Babcock, David Hovemeyer, James Moscola, and Robert Duvall. 2015.  Analyzing student work patterns using programming exercise data. In _Proceedings of the 46th ACM Technical Symposium on Computer Science Education_. 18–23.
  * Sui and Xue (2016) Yulei Sui and Jingling Xue. 2016.  SVF: interprocedural static value-flow analysis in LLVM. In _Proceedings of the 25th international conference on compiler construction_. 265–266.
  * Svajlenko et al. (2014a) Jeffrey Svajlenko, Judith F Islam, Iman Keivanloo, Chanchal K Roy, and Mohammad Mamun Mia. 2014a.  Towards a big data curated benchmark of inter-project code clones. In _2014 IEEE International Conference on Software Maintenance and Evolution_. IEEE, 476–480.
  * Svajlenko et al. (2014b) Jeffrey Svajlenko, Judith F. Islam, Iman Keivanloo, Chanchal K. Roy, and Mohammad Mamun Mia. 2014b.  Towards a Big Data Curated Benchmark of Inter-project Code Clones. In _2014 IEEE International Conference on Software Maintenance and Evolution_. 476–480.  https://doi.org/10.1109/ICSME.2014.77
  * Svyatkovskiy et al. (2020) Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020.  IntelliCode Compose: Code Generation Using Transformer. In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Virtual Event, USA) _(ESEC/FSE 2020)_. Association for Computing Machinery, New York, NY, USA, 1433–1443.  https://doi.org/10.1145/3368089.3417058
  * Svyatkovskiy et al. (2019) Alexey Svyatkovskiy, Ying Zhao, Shengyu Fu, and Neel Sundaresan. 2019.  Pythia: AI-Assisted Code Completion System. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_ (Anchorage, AK, USA) _(KDD ’19)_. Association for Computing Machinery, New York, NY, USA, 2727–2735.  https://doi.org/10.1145/3292500.3330699
  * Szegedy et al. (2015) Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015.  Going deeper with convolutions. In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 1–9.  https://doi.org/10.1109/CVPR.2015.7298594
  * Terada and Watanobe (2019) Kenta Terada and Yutaka Watanobe. 2019.  Automatic generation of fill-in-the-blank programming problems. In _2019 IEEE 13th International Symposium on Embedded Multicore/Many-core Systems-on-Chip (MCSoC)_. IEEE, 187–193.
  * Terada and Watanobe (2021) Kenta Terada and Yutaka Watanobe. 2021.  Code completion for programming education based on deep learning.  _International Journal of Computational Intelligence Studies_ 10, 2-3 (2021), 78–98.
  * Tufano et al. (2019) Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. 2019.  An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation.  _ACM Trans. Softw. Eng. Methodol._ 28, 4, Article 19 (sep 2019), 29 pages.  https://doi.org/10.1145/3340544
  * Ucci et al. (2019) Daniele Ucci, Leonardo Aniello, and Roberto Baldoni. 2019\.  Survey of machine learning techniques for malware analysis.  _Computers & Security_ 81 (2019), 123–147.
  * Ugurel et al. (2002) Secil Ugurel, Robert Krovetz, and C. Lee Giles. 2002\.  What’s the Code? Automatic Classification of Source Code Archives. In _Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ (Edmonton, Alberta, Canada) _(KDD ’02)_. Association for Computing Machinery, New York, NY, USA, 632–638.  https://doi.org/10.1145/775047.775141
  * Ullah et al. (2019a) Farhan Ullah, Hamad Naeem, Sohail Jabbar, Shehzad Khalid, Muhammad Ahsan Latif, Fadi Al-turjman, and Leonardo Mostarda. 2019a.  Cyber Security Threats Detection in Internet of Things Using Deep Learning Approach.  _IEEE Access_ 7 (2019), 124379–124389.  https://doi.org/10.1109/ACCESS.2019.2937347
  * Ullah et al. (2019b) Farhan Ullah, Hamad Naeem, Sohail Jabbar, Shehzad Khalid, Muhammad Ahsan Latif, Fadi Al-turjman, and Leonardo Mostarda. 2019b.  Cyber Security Threats Detection in Internet of Things Using Deep Learning Approach.  _IEEE Access_ 7 (2019), 124379–124389.  https://doi.org/10.1109/ACCESS.2019.2937347
  * Van Rijn et al. (2013) Jan N Van Rijn, Bernd Bischl, Luis Torgo, Bo Gao, Venkatesh Umaashankar, Simon Fischer, Patrick Winter, Bernd Wiswedel, Michael R Berthold, and Joaquin Vanschoren. 2013.  OpenML: A collaborative science platform. In _Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2013, Prague, Czech Republic, September 23-27, 2013, Proceedings, Part III 13_. Springer, 645–649.
  * Vasic et al. (2019) Marko Vasic, Aditya Kanade, Petros Maniatis, David Bieber, and Rishabh Singh. 2019.  Neural program repair by jointly learning to localize and repair.  _arXiv preprint arXiv:1904.01720_ (2019).
  * Wan et al. (2020) Yao Wan, Jingdong Shu, Yulei Sui, Guandong Xu, Zhou Zhao, Jian Wu, and Philip S. Yu. 2020.  Multi-Modal Attention Network Learning for Semantic Source Code Retrieval. In _Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering_ (San Diego, California) _(ASE ’19)_. IEEE Press, 13–25.  https://doi.org/10.1109/ASE.2019.00012
  * Wan et al. (2018) Yao Wan, Zhou Zhao, Min Yang, Guandong Xu, Haochao Ying, Jian Wu, and Philip S. Yu. 2018.  Improving Automatic Source Code Summarization via Deep Reinforcement Learning. In _Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering_ (Montpellier, France) _(ASE ’18)_. Association for Computing Machinery, New York, NY, USA, 397–407.  https://doi.org/10.1145/3238147.3238206
  * Wan et al. (2021) Zhiyuan Wan, Xin Xia, David Lo, and Gail C. Murphy. 2021\.  How does Machine Learning Change Software Development Practices?  _IEEE Transactions on Software Engineering_ 47, 9 (2021), 1857–1871.  https://doi.org/10.1109/TSE.2019.2937083
  * Wang et al. (2016a) Song Wang, Devin Chollak, Dana Movshovitz-Attias, and Lin Tan. 2016a.  Bugram: Bug detection with n-gram language models. In _2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE)_. 708–719.
  * Wang et al. (2016b) Song Wang, Taiyue Liu, and Lin Tan. 2016b.  Automatically Learning Semantic Features for Defect Prediction. In _Proceedings of the 38th International Conference on Software Engineering_ (Austin, Texas) _(ICSE ’16)_. Association for Computing Machinery, New York, NY, USA, 297–308.  https://doi.org/10.1145/2884781.2884804
  * Wang et al. (2020a) Wenhan Wang, Ge Li, Bo Ma, Xin Xia, and Zhi Jin. 2020a.  Detecting code clones with graph neural network and flow-augmented abstract syntax tree. In _2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER)_. IEEE, 261–271.