checkmark

# Survey of Natural Language Processing for Education: Taxonomy, Systematic
Review, and Future Trends

Yunshi Lan<EMAIL_ADDRESS>Xinyuan Li  Hanyue Du  Xuesong Lu  Ming Gao
Weining Qian  Aoying Zhou  School of Data Engineering and Science, East China
Normal University, Zhongshan Rd, Shanghai, 200062, China

###### Abstract

Natural Language Processing (NLP) aims to analyze text or speech via
techniques in the computer science field. It serves the applications in
domains of healthcare, commerce, education and so on. Particularly, NLP has
been applied to the education domain to help teaching and learning. In this
survey, we review recent advances in NLP with the focus on solving problems
relevant to the education domain. In detail, we begin with introducing the
related background. Then, we present a taxonomy of NLP in the education
domain. Next, we illustrate the task definition, challenges, and corresponding
techniques based on the above taxonomy. After that, we showcase some off-the-
shelf demonstrations in this domain and conclude with future directions.

###### keywords:

natural language processing, educational NLP, educational applications,
survey.

##  1 Introduction

Natural Language Processing (NLP) refers to the branch of computer science,
that empowers computers the ability to understand text and spoken words as the
way human beings can. It serves real-world scenarios. For example, NLP can
help in healthcare applications. A wide range of NLP techniques are being
utilized to anticipate the treatment process of the patients. Question
answering [1, 2] and dialogue generation [3] help conduct medical diagnosis,
administrative assistants for appointments and billing. Besides, NLP can help
in commerce applications. There is a growing trend of incorporating NLP
techniques into analytics and customer management platforms to help learn more
about the relationship between the consumers and items via sentiment analysis
[4]. In the education domain, NLP techniques can automate the tutoring and
grading process. These techniques are able to analyze students performance,
allowing for more flexible learning, thus freeing up valuable time for
teachers. For example, given a passage in the textbook as the context and a
concept as the correct answer, NLP techniques can quickly construct a
multiple-choice question associated with the test question and the
distractors. This could facilitate the teachers to make quick quizzes for
classes.

We notice there are a few surveys related to NLP for education. However, they
either focus on a certain task like question answering, grammatical error
correction [5, 6, 7, 8] or summarize in the shallow perspectives of concepts
and definition [9, 10, 11]. In comparison, this survey provides a
comprehensive overview of NLP for education from a technical point of view.
(1) New taxonomy: we present a novel taxonomy for educational NLP and identify
representative NLP tasks that can be applied for learning and comprehension as
well as writing and assessment, and then discuss them in detail; (2)
Comprehensive review: we collect recent papers published in well-recognized
computer science venues and review their methods for solving the challenges of
corresponding tasks; (3) Rich resources: we discuss the publicly-available
datasets and demonstrations with their various attributes, which makes it easy
for readers to get started for their research.

For the rest of the manuscript, we first present the taxonomy of NLP in the
education domain, which includes eight representative tasks. For each task, we
review the datasets and techniques for the identified tasks, pointing out the
major challenges and corresponding solutions. Then, we summarize the available
demonstrations. At last, we conclude the survey with future directions.

Figure 1: An overview of NLP for education. The left section displays the
systematic frame of relevant NLP techniques, including data source, processing
and analysis techniques, and applications. The right section demonstrates the
outcomes of diverse techniques.

##  2 Taxonomy of NLP in Education Domain

Systems built on the basis of NLP applications and equipped with NLP
techniques could perform as empowered AI tutoring, which enhances the teaching
and learning experience of students, teachers, parents, or tutors. With the
systems, students are able to get access to private tutoring and extra support
outside the classroom, which avoids increasing the workload of their teachers.

We first demonstrate an overview of NLP for education in Figure 1, where the
systematic frame of relevant NLP techniques and their corresponding
instantiations are displayed. The data sources are significant as the
fundamental supports for subsequent analyses and applications. Generally, we
could obtain educational data from textbooks, syllabus, exams, and curricula
materials, which may contain behavioral data of students, test questions and
contexts, and guidance signals of error correction and so on. From the
perspective of teaching subjects, the data resources cover subjects like
chemistry, math, linguistics, computer science and so on, which could support
applications in different subjects. Given the raw data resources, we could
conduct processing and analysis (e.g., knowledge graph construction [12],
information extraction [13], information retrieval [14], knowledge alignment
[15]) to extract the key information from massive data and organize them in
structured knowledge graphs (e.g., EduKG111https://edukg.cn/team), databases
(e.g., WordNet222https://wordnet.princeton.edu/documentation) and lists, which
could be applied to various downstream applications. Upon the well-organized
data, NLP tasks are applied (e.g., question answering, question construction,
automated assessment, and error correction) to form high-level applications to
serve teachers and students in real-world scenarios.
Grammarly333https://app.grammarly.com, Gauthmath444https://www.gauthmath.com,
CodeSubmit555https://codesubmit.io, AutoQG 666https://autoqg.net, and
EduChat777https://www.educhat.top/ are possible educational products and
demonstrations developed based on these NLP applications.

This survey mainly focuses on the application-level. We highlight the most
representative NLP applications with the taxonomy shown in Figure 2.
Considering that NLP techniques play significant roles in different phases of
education, on the one hand, NLP could serve as a teaching assistant for
learning and comprehension. When students learn the curriculum, the following
NLP tasks could simplify the difficulties of learning.

  * •

Question Answering (QA), as a typical application, could not only provide
tutorials to the homework for students but also save time for overloaded
educators by providing feedback on questions. Based on the different goals of
question answering, textbook question answering and math word problem solving
are presented as the two typical subcategories of question answering. These
two tasks both aim to generate answers from the given contexts automatically.
They can be widespread in the education of science subjects.

  * •

Question Construction (QC) is to generate questions as well as the distractors
given the correct answers, which helps the teachers create a wide range of
test questions using pre-existing learning resources. Generally, a good test
question may consist of a question and multiple candidate answers. Therefore,
question generation and distractor construction are two key sub-tasks to be
reviewed.

On the other hand, NLP improves writing and assessment for students. When
given constructive criticism and writing prompts, students are likely to go
deep into the topic. However, teachers sometimes have difficulties responding
thoughtfully to students due to time limitation. But they will benefit from
the following tasks.

Figure 2: The taxonomy of NLP for Education. The applicable subjects of the
NLP tasks for teaching and learning are shown in the brackets.

  * •

Automated Assessment (AA) aims to grade assignments for students
automatically. Language grading is a representative scenario of AA, which
includes automated essay scoring for natural language learning and automated
code scoring for program language learning. For essay grading, grammar,
spelling, and sentence structure are key aspects that need to be considered.
For code grading, the correctness of the code, style aspects, code quality are
metrics to be viewed.

  * •

Error Correction (EC) for language are useful to improve language learning via
explicating the correct edits. Still, taking language learning as an example,
grammatical error correction and code error correction are canonical sub-tasks
for EC. The methods are not simply required to provide error-free outputs
based on the ill-posed inputs, more detailed instructions should be provided
to shed light on the reasons or knowledge points to the errors.

Please be aware that there are some other NLP tasks that are related to
education. For example, tasks like text rewriting [16] and controllable text
generation [17] can also help teachers to prepare suitable teaching materials.
In this survey, we mainly focus on the highlighted tasks, which are widely
explored NLP tasks in education domain. Our discussion is carried out based on
the above taxonomy.

for tree= forked edges, grow’=0, draw, rounded corners, node
options=align=left, calign=edge midpoint, , [NLP Applications for Education,
text width=1.7cm, for tree=fill=white!20 [Textbook Question Answering, text
width=1.8cm, for tree=fill=white!20 [Comprehend the multi-modal contexts and
complex questions, text width=2.8cm, for tree=fill=white!20 [ deep learning-
based methods combining RNN and CNN [18, 19]; spatial attention [20, 21, 22,
23]; compositional strategies [24]; bilinear pooling schemes [25, 26]., text
width=5.5cm, node options=align=left ] ] [Supplement domain-specific
knowledge, text width=2.8cm, for tree=fill=white!20 [ pre-train models with
domain knowledge [27, 28]; diagrams parsing enhancement [29, 30, 31]., text
width=5.5cm, node options=align=left ] ] [Strengthen zero-shot capabilities,
text width=2.8cm, for tree=fill=white!20 [ collaborate LLMs with textbooks [1,
32, 33, 34]; develop LLMs-based multi-agents [35, 2]., text width=5.5cm, node
options=align=left ] ] ] [Math Word Problem Solving, text width=1.8cm, for
tree=fill=white!20 [Understand the gap between natural language and abstract
expression, text width=2.8cm, for tree=fill=white!20 [propose seq2seq [36, 37,
38, 39] framework enriched with structured encoders and decoders [40, 41];
reinforcement learning [37, 42]; knowledge distillation [43]., text
width=5.5cm, node options=align=left ] ] [Solve MWPs under zero/few-shot
setting, text width=2.8cm, for tree=fill=white!20 [ propose LLMs-enhanced MWP
solvers [44, 45, 46, 47, 35]; develop LLMs-based multi-agents [48]., text
width=5.5cm, node options=align=left ] ] ] [Question Generation, text
width=1.8cm, for tree=fill=white!20 [Identify the key statement from the
context and generate the corresponding question, text width=2.8cm, for
tree=fill=white!20 [ conduct rule-matching [49]; develop deep learning-based
methods with attention [50]; lingtuistic features [51, 52, 53]; multi-modal
fusion [54]; multi-task learning [55]; reinforcement learning [56]., text
width=5.5cm, node options=align=left ] ] [Generate more flexible questions for
adaptive education, text width=2.8cm, for tree=fill=white!20 [ propose
difficulty-control QG methods [57, 58, 59]; syllabus-aligned methods [60, 61];
knowledge-tracking QG methods [62, 63]., text width=5.5cm, node
options=align=left ] ] ] [Distractor Generation, text width=1.8cm, for
tree=fill=white!20 [Design context-consistent and deceptive false answers,
text width=2.8cm, for tree=fill=white!20 [ propose rule-based methods [64];
feature-based methods [65]; generation-based methods [66, 67]; ranking-based
methods [68, 69, 70]., text width=5.5cm, node options=align=left ] ] ]
[Automated Essay Scoring, text width=1.8cm, for tree=fill=white!20 [Model
different facets of the essay, text width=2.8cm, for tree=fill=white!20 [
design various features (e.g. length-based features [71]; lexical features
[72, 71]; prompt-relevant features [73]; readability features [71]; syntactic
features [71, 74]) and solve via regression and ranking [75] with enhanced
training procedure [76]., text width=5.5cm, node options=align=left ] ]
[Sparse annotation of essays, text width=2.8cm, for tree=fill=white!20 [
decouple the domain-dependent and domain-independent features and aggregate
heuristic supervision signals [77, 78, 79]; design enriched annotations [80,
81]., text width=5.5cm, node options=align=left ] ] ] [Automated Code Scoring,
text width=1.8cm, for tree=fill=white!20 [Understanding of the long-dependency
of code, text width=2.8cm, for tree=fill=white!20 [ conduct dynamic analysis
to assess the execution of the code [82, 83, 84, 85]; static analysis to
predict scores with the designed features [86, 87, 88, 89]; hybrid methods
combine both [90]., text width=5.5cm, node options=align=left ] ] ] [Language
Error Correction, text width=1.8cm, for tree=fill=white!20 [Identify and
correct grammar errors accurately, text width=2.8cm, for tree=fill=white!20
[design manual rules and classification-based models [91, 92]; neural network
models like seq2seq models [93, 94, 95, 96, 97] and seq2edit models [98, 99,
100, 101]., text width=5.5cm, node options=align=left ] ] ] [Code Error
Correction, text width=1.8cm, for tree=fill=white!20 [Identify and correct
errors in the long-distance dependency of the code, text width=2.8cm, for
tree=fill=white!20 [develop methods with abstract syntax tree [102, 103];
denoising strategies [104]; use identifier tagging and comment information
[105]; grammar constraints [106]., text width=5.5cm, node options=align=left ]
] ] ]

Figure 3: The detailed techniques of NLP applications. The hierarchical
structure is arranged with: NLP applications $\rightarrow$ Challenges
$\rightarrow$ Solutions.

##  3 Review of Techniques

To help readers catch up with cutting-edge research on these highlighted
tasks, in this section, we provide a comprehensive review of the evaluated
resources and techniques for these tasks, showing emphasis on the development
of techniques and solutions to the key challenges. The review is arranged in a
tree in Figure 3 for a quick lookup.

###  3.1 Question Answering

Figure 4: The examples of Question Answering in education domain. The red box
denotes the given context and the green box denotes the required prediction
for the tasks. The displayed examples are extracted from SCIENCEQA and GSM8K,
respectively.

Even though QA has been explored as a popular NLP task for decades.
Educational QA requires a system to answer the domain-specific questions,
related to natural science, social science, language science and so on [107].
These systems act as assistants to help answer difficult questions such that
the students could learn from them and boost their performance. Question
answering in different subjects may be presented in different formats. In
Figure 4, we display two typical examples of question answering in the
education domain. To answer the displayed questions, the systems need domain-
specific knowledge like mathematics, and science. Furthermore, the contexts of
the questions are presented as text and images. Moreover, the output of the
questions can presented as an entity or a math expression. In this section, we
will present cutting-edge techniques for Textbook QA and MWP solving.

####  3.1.1 Textbook Question Answering

Textbook Question Answering (TQA) is a task that requires a system to
comprehensively understand the multi-modal information from the textbook
curriculum, spreading across text documents, images, and diagrams. The major
challenge of textbook question answering is to comprehend the multi-modal
domain-specific contexts as well as the questions, and then identify the key
information to the questions.

Datasets. Kembhavi et al. (2017) introduced TQA dataset, which aims to
evaluate a system that contains multi-modal contexts and diverse topics in the
scientific domain. Similar datasets such as AI2D [29], DVQA [31], and VLQA
[109] have been constructed to support researches on multi-modal reasoning in
the scientific domain. However, these datasets lack of annotated explanations
for the answers in the form of supporting facts. SCIENCEQA [1] is a large-
scale textbook question answering dataset with annotated lectures and
explanations. The dataset is collected from elementary and high school science
curricula, which contain diverse science topics including natural science,
social science, and language science. A recently released TheoremQA dataset
further covers textbook questions at the university level [2]. Besides the
scientific domain, there are some datasets focusing on the medical domain.
MEDQA [110] and MedMCQA [111] are two medical QA datasets encompassing a broad
spectrum of healthcare topics deriving from both real-world scenarios and
simulated exams.

Methods. From the technical perspective, textbook question answering is visual
question answering in nature [112, 113, 114]. Conventional VQA studies employ
RNN to encode the question and CNN to encode the image [18, 19], respectively.
Then the multi-modal information is fused together to comprehend the
questions. Besides, other methods employing spatial attention [20, 21, 22,
23], compositional strategies [24], and bilinear pooling schemes [25, 26] are
proposed to boost the performance of VQA.

While there are significant similarities between VQA and textbook question
answering, textbook question answering differs in the domain-specific
knowledge required for the accompanying context and the innovative inclusion
of diagrams and tables. To bridge the gap, Ram et al. (2021) proposed a pre-
training schema tailored for question answering. Specifically, by masking
recurring span selection and selecting the correct span in the passage, their
method boosts the performance for textbook question answering when only a
hundred examples are available in the specific domains. Adversarial training
framework is also adapted to domain generalization [28], which facilitates the
question answering models to learn domain-invariant features. Xu et al. (2022)
introduced a novel Pre-trained Machine Reader as a retrofit of pre-trained
Masked Language Models (MLMs), which resolves the discrepancy between model
pre-training and down-stream fine-tuning of specific domain MLMs. To
understand the diagrams and tables, graph-based parsing methods are developed
to extract the concepts from diagrams [29] by converting a diagram to a
diagram parse graph. Optical Character Recognition (OCR) is employed to
identify the chart-specific answers from the charts, which is further aligned
with the questions [30, 31].

Recently, Large Language Models (LLMs) have shown impressive capabilities in
various NLP tasks, researchers start applying LLMs to textbook question
answering. Lu et al. (2022) first applied GPT-3 to solve the textbook question
answering. To apply the full advantage of LLMs to image modality, they use
caption models to translate visual information into the language modality and
prompt LLMs to perform Chain-of-Thought. This inspires the follow-up study
[32], distilling the training signals from the LLMs to a smaller model. To
inject general knowledge from the LLMs to the specific domains, Wang et al.
(2023b) augmented queries for LLMs on textbook question answering by
retrieving evidence from textbooks, such that the response of the LLMs would
likely to be authorized by the textbooks. Liu et al. (2023) extended the
instruction tuning to a multi-modal field, which results in a multi-modal
model that connects a vision encoder and a LLM for general-purpose visual and
language understanding. More recent studies [35] develop compositional
reasoning with LLMs as a planner that assembles a sequence of tools (e.g.,
Hugging Face as image captioner, Github as text detector, Web Search as bing
search, and Python as program verifier.) to execute and answer the textbook
questions. Instead of solving the questions with given contexts, Chen et al.
(2023) introduced a system, that first retrieves the theories from the open
domain and then answers the question based on the retrieved theories.

Dataset | Science subject | Source level | Context | #Q
---|---|---|---|---
TQA | life/earth/physical | grade 6-8 | image/ | $26$K
science | science curricula | diagram
Geometry3K | geometry | grade 6-12 digital libraries | image/diagram |  $3$K
AI2D | science | grade 1-6 | image/diagram |  $5$K
SCIENCEQA | natural/social/ | grade 1-12 | image/diagram | $21$K
language science | science curriculum | text
MedQA | medicine | professional medical | text | $40$K
board exam
MedMCQA | medicine | simulated exams | text |  $200$K
TheoremQA | math/physics/EE/CS | university-level exam | text | $800$
Dolphin-18K | math | grade school | text |  $18$K
DRAW-1K | math | algebra | text |  $1$K
Math23K | math | grade school math | text |  $23$K
MathQA | math | grade school math | text |  $37$K
ASDiv | math | grade school math | text |  $2$K
GSM8K | math | grade school math | text |  $8$K
IconQA | math | K-12 math | text/image |  $107$K
TABMWP | math | grade school math | text/table |  $38$K

Table 1: Comparison of textbook question answering and MWP solving datasets.
The first section includes datasets of Textbook QA. The second section
includes datasets of MWP solving. “#Q” denotes the number of questions.

####  3.1.2 Math Word Problem solving

Math Word Problem (MWP) solving is a specific type of question answering,
aiming to convert a narrative description to an abstract expression. This task
is challenging because there remains a wide semantic gap to parse human-
readable words into machine-understandable logic.

Datasets. Dolphin-$18$K [116] is an early large-scale MWP solving dataset,
which contains over $18,000$ annotated math word problems of elementary
mathematics. DRAW-$1$K [117] is introduced as a testbed to algebra word
problem solvers. Math$23$K [36] is a Chinese MWP solving dataset.
Subsequently, MathQA [118] and ASDiv [119] further extend the scale and
diversity of the existing datasets, which makes it particularly suited for
usage in training deep learning models to solve MWP problems. Cobbe et al.
(2021) introduced a more challenging dataset GSM$8$K, which is complicated in
high linguistic diversity. IconQA [121] extends MWP solving to abstract
diagram understanding and comprehensive cognitive reasoning by involving
diagrams in problems. Moreover, TABMWP [122] includes tables to enhance the
difficulty of the MWPs, which requires the table parsing and understanding
capabilities [122]. As we can see, there is a set of MWP datasets that examine
various mathematics capabilities. With the development of this task, the
evaluated datasets become increasingly complicated in linguistic diversity and
multi-modalities.

Methods. The investigation of MWP solving can be traced back to $1960$s [123].
Early work leverages manually crafted rules and schemas for pattern matching,
where human intervention is heavily relied on. Subsequently, various
strategies of feature engineering and statistical learning are proposed to
boost the performance. With the development of techniques and the emergence of
large-scale datasets, deep learning-based methods have been widely applied to
MWP solving. The majority of the methods follow an encoder-decoder framework
[36], which first encodes the narratives into dense vector space, then
generates the mathematical expression token as the output sequence. To improve
the generation of complex solution expressions, some methods are proposed to
formulate a mathematical expression as an abstract tree [37, 38, 39, 43]. To
improve the understanding to the narratives, some studies focus on improving
the encoding of the narratives by re-organizing the textual contexts with
structure [40, 41]. Due to the weak supervision caused by the lack of
annotated expressions, multiple studies [37, 42] employ reinforcement learning
to solve MWPs. Also, some researchers [124] apply knowledge distillation to
MWP solving tasks, which learns a smaller model from a large pre-trained
generic model.

As the MWP solving task is the desirable testbed for evaluating the reasoning
capabilities, a series of studies have been developed to discuss the abilities
of LLMs for solving MWPs. Researchers find that prompting the LLMs with
effective planning strategies could activate the potential of LLMs, which
results in impressive results on MWP solving under the zero/few-shot
scenarios. Chain-of-thought is a typical promoting strategy that is meant to
decompose a complicated MWP to sub-questions such that LLMs could conduct
reasoning step by step [125]. Follow-up methods improve it by designing more
interactive prompting or constructing more efficient demonstrations [44, 45,
46, 47, 35]. With the emergence of the idea of leveraging LLMs as an agent
[126], researchers [48] delegate the computation to a program interpreter,
which can be conducted by the LLMs and executed by a Python interpreter. In
this case, the reasoning step can be verified in an iteration manner.

###  3.2 Question Construction

Question Construction (QC) aims to automatically construct questions from a
given context, which performs a significant role in education [127]. Usually,
Multiple Choice Questions (MCQ) are common question types for a test quiz. The
construction of MCQ consists of two parts: Question Generation (QG) and
Distractor Generation (DG). The former generates questions based on a given
context, while the latter generates distractors to complement the correct
ones. In Figure 5, we display a typical example of QG and DG in the education
domain. In this section, we introduce the relevant datasets and methods for QG
and DG in detail.

Figure 5: The examples of Question Construction. The displayed examples are
extracted from SciQ.

####  3.2.1 Question Generation

The goal of QG is to automatically generate a question from a given sentence
or paragraph, with the assistance of answer information when available. The
challenge lies in identifying the key statement based on the context with or
without the utilization of the answer (namely answer-aware QG or answer-
agnostic QG [128]) and generating a question based on the statement.

Datasets. QG and QA are dual tasks [129]. They both require reasoning between
questions and answers. As a result, some MCQ datasets initially designed for
QA tasks, including SciQ [130], RACE [131], and FairytaleQA [132], are also
leveraged for researches on QG [133, 134, 135, 136]. Besides, there are some
datasets particularly constructed for QG, such as LearningQ [137], KHANQ
[138], and EduQG [139]. These datasets extend across various subjects (e.g.,
science, medicine) and various education levels (e.g., preliminary school,
middle school, and university.), the details of which are displayed in Table
2. Regarding these datasets, LearningQ is collected from instructor-crafted
learning questions, which contain cognitively demanding documents for QG that
requires reasoning, but the absence of answers limits its application to
answer-aware QG. KHANQ provides (context, prompt, question) triples collected
from questions about the online courses raised by students. The prompt, which
indicates the background knowledge of students or presupposition of the
questions, provides hints on the information that should be brought into focus
during question generation. EduQG is a high-quality MCQ dataset generated by
educational experts and questions are linked to their cognitive complexity,
which makes this dataset move closer to reality.

Dataset | Science subject | Source | Question type | Answer type | #Q
---|---|---|---|---|---
SciQ | science | online science | normal | MC& | $13.7$K
textbooks | extractive
RACE | linguistic | secondary school | normal& | MC& | $100$K
English exams | cloze | extractive
FairytaleQA | literature | grade K-8 | normal | extractive | $10$K
narrative stories
LearningQ | computing/ | online education | normal | - | $231$K
science/business/ | platform
humanities/math
KHANQ | science | online education | normal | extractive | $1$K
platform
EduQG | politics/physiology/ | online textbooks | normal& | MC& | $3$K
biology/business/ | cloze | extractive
law/sociology/
psychology/history
MCQL | biology/physics/ | website | normal | MC | $7.1$K
chemistry
Televic | math/health/ | online education | normal | MC | $62$K
history/geography/ | platform
science

Table 2: Comparison of QG and DG datasets. The format of the question is
categorized as either normal (e.g., interrogative sentence.) or cloze (e.g.,
fill-in-the-blank.), and the answer is categorized as either multiple-choice
(MC) or extractive (parts of the text are considered as the answer) [139].

Methods. Early methods for general QG primarily relied on rule matching [49].
Du and Cardie (2017) first applied the Seq2Seq model with attention to
automatic QG. Built upon this framework, methods such as integrating
linguistic features [51, 52, 53], training a multi-modal QG model [54],
leveraging multi-task learning [55] and reinforcement learning [56] have also
been augmented to optimize the QG models. With the advent of language models,
the BERT model and LLM GPT-3 have been applied to QG [140, 141]. Besides,
regarding answer information in the QG system, researchers proposed to encode
the answer together with the context [142], utilize the position information
of the answer in the context [143, 144], and achieve answer-aware QG, while
using summaries of the texts as input can serve as a strategy for answer-
agnostic QG as it can help the modeling of topic salience, which denotes the
importance of specific words or phrases in context [128].

Importantly, there are some unique notes when we adopt QG to the education
domain. First, controlling the difficulty levels of the generated questions is
vital for improving the efficiency of education. Gao et al. (2018) proposed a
method for difficulty-controllable QG, and the difficulty of a question is
judged according to whether a well-trained QA model is able to answer it
correctly. Then Cheng et al. (2021) redefined the difficulty level as the
number of inference steps required for a question to answer. Recently, Uto et
al. (2023) considered the difficulty associated with learners’ ability, which
is measured by the item response theory [145]. Second, it is important to
align generated questions to the syllabus, which benefits to identify the
focus of the test quiz. Hence, multiple studies [60, 61] train binary
classifiers or ranking models to judge whether the generated question is
related to certain entities and concepts or pedagogical contents with the
objective of maximizing both relevance and diversity. Third, customized
education is in high demand and we notice there are a few studies focusing on
generating personalized questions for students. Srivastava and Goodman (2021)
developed a knowledge tracking model that uses students’ answer histories to
predict their performance on new questions, fine-tuning it with an
autoregressive language model. Subsequently, Wang and Li (2023) introduced the
few-shot knowledge tracking model to predict question difficulty by
incorporating sequences of student states and questions.

####  3.2.2 Distractor Generation

As we mentioned above, DG is another component for question construction,
which aims to generate distractors with the given context, question, and
correct answer. A good distractor should be relevant to the context,
grammatically coherent to the interrogative of the question, and deceptive as
a false answer [130].

Datasets. The aforementioned MCQs for QG tasks such as SciQ and RACE can also
be leveraged for DG tasks. Specifically, the distractors in SciQ are designed
by crowd workers with reference [130]. Instead of providing entities and
concepts as distractors, the distractors of RACE are at sentence-level [146],
which are collected from the secondary school English exams. Alongside the
datasets mentioned before, there are other MCQ datasets especially designed
for DG. The data of MCQL [68] is crawled from the web and provides distractors
that are either phrases or sentences, covering multi-subjects. Besides,
Televic [147] is collected through Televic Education’s platform with
distractors manually crafted by experts, featuring distractors across multiple
languages and domains.

Methods. Early studies for DG are rule-based and feature-based, which rely on
linguistic rules [64] and specific features like word frequency [65] or
diverse similarity measurement between distractors and correct answers [68].
With the rapid development of deep learning, the construction of distractors
can be divided into two lines of mainstream: generation-based and ranking-
based categories [70]. Generation-based methods automatically generate
distractors token by token and ranking-based methods regard DG as a ranking
task for a pre-defined distractor candidate set. Amongst the generation-based
methods, Gao et al. (2019) proposed a hierarchical model with dual attention
for sentence and word relevance and irrelevant information filtering to
generate distractors that are semantically consistent with and traceable in
the context. Besides, Bitew et al. (2023) employed LLMs for DG. They feed
relevant questions as in-context examples to generate more plausible
distractors. As for the ranking-based method, Liang et al. (2018) proposed a
two-stage framework: the first stage trains a model as a filter to reduce the
candidate size, and the second stage trains another model for ranking the
candidate. However, the two-stage framework encounters error accumulation, and
some studies have discarded the first stage to address this issue, but led to
reduced efficiency [69]. As follow-up studies, Wang et al. (2023) proposed an
end-to-end model for retrieval in the distractor candidates with a balance of
efficiency and high performance.

###  3.3 Automated Assessment

Figure 6: The examples of AES and ACS tasks. The displayed examples are
extracted from ASAP dataset and paper [148], respectively.

Automated assessment is a widely-investigated task in the education domain,
which is helpful in reducing the burden of teachers in various writing tasks.
Regarding different objects to be scored, different criterion is considered.
For instance, in Figure 6, scoring an essay in a language curriculum requires
the measurement of particular essay quality such as persuasiveness and
organization. For a piece of code in the program curriculum, it is more
important to measure the correctness of the code, the completeness of the
comments, and quality issues. Therefore, we discuss the techniques for
Automated Essay Scoring (AES) and Automated Code Scoring (ACS) in the
following subsections.

####  3.3.1 Automated Essay Scoring

AES aims to develop a system to score an essay automatically, where the
written text is given as the input and the system summarizes its quality as a
score. The challenge of this task lies in understanding the content of the
essay by modeling the different facets of the text. Until now, a series of
methods have been developed to automatically score the essays.

Dataset | Essay Types | Source | Language | Exp | #Q
---|---|---|---|---|---
CLC-FCE | A/N/C/S/L | non-native: ESOL test | English | holistic |  $1$K
ASAP | A/R/N | native: grades 7-10 exam | English | holistic |  $17$K
TOEFL 11 | A | non-native: TOEFL test | English | holistic |  $1$K
ICLE | A | non-native: | English | organization/clarity/ |  $4$K
undergraduate exam | adherence/persuasiveness |
HSK | A/N | non-native: HSK test | Chinese | holistic |  $10$K

Table 3: Comparison of AES datasets. The essay types contain: argumentative
(A), response (R), narrative (N), comment (C), suggestion (S) and letter (L).
“Exp” denotes the type of scores annotated. “#Q” denotes the number of
questions.

Datasets. The Cambridge Learner Corpus-First Certificate in English exam (CLC-
FCE) [149] is a widely used dataset for AES tasks. It provides both the
holistic score and the manually tagged linguistic error types such that it
makes possible to evaluate the instructional feedback of the developed AES
systems. It is worth noting the dataset is annotated with the prompt of the
essay, which indicates the topic of the essay. The Automated Student
Assessment Prize (ASAP) is a Kaggle competition released for AES. A similar
dataset TOEFL 11 corpus [150] contains essays from TOEFL exams, and it comes
with a three-level proficiency annotation, which facilitates the AES research
with the consideration of the difficulty level. Instead of holistic scores,
fine-grained instructional annotation is needed for dimension-specific essay
scoring research. International Corpus of Learner English (ICLE) and Argument
Annotated Essays (AAE) [151] are two datasets containing essays annotated with
the perspectives of structure organization, thesis clarity, prompt adherence,
and argument persuasiveness. There are a few AES datasets, which are designed
for other languages such as Chinese [152]. We display the details of the above
datasets in Table 3.

Methods. Due to the availability of the annotation and wide applicability of
the commercial value, the majority of the studies usually predict a single
score for an essay. The early AES systems can be formulated as a regression
task, a classification task, or a ranking task. Specifically, a series of
features such as length-based features [71], lexical features [72, 71],
prompt-relevant features [73], readability features [71], syntactic features
[71, 74] and so on, are first extracted from the essay, then off-the-shelf
learning algorithms are leveraged to fit the annotated scores [153]. For
regression, linear regression, support vector regression, and sequential
minimal optimization are employed. For classification, SMO, logistic
regression, and Bayesian networks are utilized. For ranking, SVM ranking can
be applied. Subsequently, researchers propose neural networks with different
architectures, including CNN, LSTM, and attention mechanism, to encode the
essay and eventually predict the scores. Particularly, the document structure
can be captured via hierarchically encoding the sentences from the word level
to the sentence level. The supervision signals can be formulated with the
combination of regression and ranking [75]. Hybrid methods are proposed to
integrate the handcrafted features in deep learning-based methods [154]. Zeng
et al. (2023) applied curriculum learning to the AES training procedure.

Besides, there are some other challenges existing in the AES tasks. Due to the
sparse annotation of the essays in the target domains, multiple studies [155,
156, 157] design neural networks decoupling the domain-dependent and domain-
independent features, to facilitate the models to learn knowledge from
different domains. Other methods aggregate multiple heuristic signals as
supervision [79], enhance the features with capsule networks [78], and conduct
interpretable essay scoring systems [77]. Regarding the evaluation of the
essays, multiple dimensions are considered in a few studies such as cohesion,
syntax, vocabulary, phraseology, grammar, and conventions[80, 81], which is
able to construct more diverse evaluation metrics.

####  3.3.2 Automated Code Scoring

Similar to AES, ACS grades the score of a code snippet on various dimensions.
Comparing with essay grading, code grading is more complicated in
understanding the long-dependency inside a code snippet and the relationship
between identifiers.

Datasets. We notice that there are few publicly-available ACS datasets due to
the privacy issue. The source data contain submissions in programming courses
that can be collected on the academy platforms such as JetBrains Academy,
MOOC, tutoring sessions and so on [85, 158, 148].

Methods.

Code correctness is a vital factor in judging the quality of a code snippet,
which can be quickly evaluated using compilers and interpreters. Therefore,
dynamic analyses evaluate a code snippet by its performance of execution. By
executing on a set of test cases, the execution results are supposed to be the
same as the excepted results. In this way, many dynamic analysis tools for
detecting code quality issues are developed, such as AutoStyle [82], WebTA
[83], Refactor Tutor [84] and Hyperstyle [85]. On the contrary, some
researchers apply static analysis approaches to assess a code snippet, where
static features are extracted from the code without executing it. The static
features include complexity, comments, number of lines, number of operators,
number of operands and so on. The static analysis approaches leverage these
features to identify the quality of code [86]. Machine learning algorithms
also have been applied to ACS. Clustering, breadth-first search, and neural
network approaches are utilized to track the logic errors in code [87, 88,
89]. Besides, Queirós and Leal (2012) proposed hybrid approaches combining
static and dynamic approaches.

Instead of directly scoring a code snippet based on the extracted features,
there are a series of studies focusing on evaluating the contextual embedding
of a code snippet, which benefits to measure the distance between the
evaluated code and the correct code for ACS [159]. The embedding of variable
and method identifiers utilizes local and global context[160], abstract syntax
trees [161], and AST paths [162, 163]. Later, Feng et al. (2020) and Kanade et
al. (2020) proposed CodeBERT and CuBERT, respectively. They model code beyond
the simple tokenization.

###  3.4 Error Correction

Compared with AA, EC requires systems to provide explicit edits and
corresponding revisions to the errors. Similarly, language learning, as an
important educational scenario, contains applications of Grammatical Error
Correction (GEC) and Code Error Correction (CEC) for natural language learning
and machine language learning. In terms of the techniques of GEC and CEC
tasks, the CEC differs from GEC due to their different syntax. And we display
two typical examples of them in Figure 7.

Figure 7: The examples of GEC and CEC tasks. The examples are extracted from
BEA-2019 and Bugs2Fix, respectively.

####  3.4.1 Grammatical Error Correction

GEC tasks are designed to identify and correct grammatical errors in text,
including spelling errors caused by phonological confusion and visual
confusion, as well as grammatical errors caused by incorrect use of
grammatical rules. For the GEC task, how to correctly identify and correct
grammar errors is difficult, and how to avoid over-correction is challenging.

Datasets. Lang-8 is one of the most widely used datasets for language error
correction, covering 80 languages, but its distribution of languages is very
skewed, with Japanese and English being its most prevalent languages [166].
CLANG-8 [166] is a clean dataset based on Lang-8, fixing some mismatches
between data sources and targets in Lang-8. CoNLL-2014 [167] and BEA-2019
[168] are both commonly used English language error correction datasets that
refine different types of syntax errors. In addition, with the popularity of
GEC tasks, more and more non-English datasets have been proposed and widely
used by researchers. SIGHAN [169] is an early proposed data set for Chinese
spelling check, which only contains grammatical errors such as pronunciation
and shape errors of characters. The CTC [170] dataset not only contains
spelling errors, but also introduces grammatical error and semantic error,
which further improves the error types of the Chinese grammatical error
dataset. The FCGEC [171] and FlaCGEC [172] datasets have added more linguistic
annotations, introduced more granular grammatical rules, and covered 28 and
210 error types, respectively, presenting new challenges to the task of
language error correction. At the same time, there are many GEC datasets for
low-resource languages. There are dataset GECCC [173] developed in Czech,
dataset RULEC-GEC [174] developed in Russian, dataset Falko-Merlin [175]
developed in German, dataset COWS-L2H [176] developed in Spanish, dataset UA-
GEC [177] developed in Ukrainian and dataset RONACC [178] developed in
Romanian.

Methods. Early work to handle language error correction tasks is mostly
achieved by manually defining rules or building classification-based models,
with modest performance [91, 92]. With the widespread use of neural networks,
Seq2seq [179] model has become the mainstream method to solve the task. It
treats grammatical error correction as a monolingual machine translation task,
in which source sentences containing grammatical errors are tagged one by one
to be translated into the correct target sentences. The Seq2seq methods are
mainly built based on RNN, CNN, or Transformer framework. BART [93], T5 [94],
EBGEC [95], SynGEC [96] and CSynGEC [97] are good models in practice. After
that, Sequence-to-editing (Seq2edit) [180] methods have gradually emerged. It
treats the GEC task as a sequential labeling problem, marking the text spans
with appropriate error tags, leaving the rest of the text unchanged, and
generating one edit operation per prediction. The commonly used efficient
Seq2edit models include PIE [98], EditNTS [99], LaserTagger [100], GECToR
[101] and so on. Recently, the TemplateGEC [181] model proposed by Li et al.
integrates the two frameworks of Seq2seq and Seq2edit, making use of their
ability in error detection and correction, alleviating the problem of over-
correction of seq2seq model to a certain extent, enhancing the effectiveness
and robustness of the model. As a result, it gains impressive results in GEC
tasks. Even though LLMs are able to solve GEC tasks in nature, researchers
found that they encounter severe issues of over-correction [182].

Dataset | Language | Source | Exp | #Q
---|---|---|---|---
LANG-8 | Multi | LANG-8 Website | ✗ | 1M
(80 languages)
CLANG-8 | En, GE, RU | LANG-8 | ✗ | 2.6M
CoNLL-2014 | En |  NUCLE [183] | ✓ | 58K
BEA-2019 | En | NUCLE, FCE, W&I, | ✓ | 686K
LOCNESS, LANG-8
SIGHAN | Zh | - | ✗ | 12K
CTC | Zh | Texts written by native speakers | ✓ | 218K
FCGEC | Zh | Chinese examinations | ✓ | 41K
FlaCGEC | Zh |  HSK [184] | ✓ | 13K
GECCC | CS | Essays, website texts | ✓ | 83K
RULEC-GEC | RU | - | ✓ | 12K
Falko-MERLIN | GE | MERLIN, FalkoEssayL2, | ✓ | 24K
FalkoEssayWhig
COWS-L2H | ES | Essays | ✓ | 12K
UA-GEC | UK | - | ✓ | 20K
RONACC | RO | TV and radio shows | ✓ | 10K
Defects4J | Java | Open source programs | - | 357
ManyBugs | C | Open source programs | - | 185
IntroClass | C | Student programs | - | 998
QuixBugs | Java, Python |  Quixey Challenge 888https://gigaom.com/2012/01/19/quixey-challenge/ | - | 40
Bugs2Fix | Java | Github | - | 2.3M
CodeReview | Multi | Github | - | 642K
(9 languages)
CodeReview-New | Multi | CodeReview, code reviews | - | 15K
(16 languages)
Table 4: Comparison of GEC and CEC datasets. “Exp” denotes whether the
annotation contains the explicit error types. “#Q” denotes the number of
sentences in the dataset.

####  3.4.2 Code Error Correction

Code Error Correction (CEC) fixes a buggy code snippet to make it error-free
and more coherent. It is a challenging task as it requires the model to have a
sufficient understanding of the code in long-distance dependency and make
proper corrections to the erroneous code spans.

Datasets. Defects4J [185], ManyBugs [186] and IntroClass [186] are some of the
earlier datasets proposed for CEC tasks, focusing on Java and C languages
respectively, but with small amounts of data. Lin et al.(2017) proposed the
first multi-lingual parallel corpus of CEC benchmarks, namely QuixBugs[187],
which contains $40$ buggy code snippets translated into both Java and Python.
Xia et al. (2022) combined the aforementioned Defects4J, ManyBugs, and
QuixBugs datasets into a single CEC evaluation dataset to test the effect of
PLM on the CEC task. Nowadays, the widespread use of collaborative software
development platforms such as Github999https://github.com/ makes it possible
to collect and analyze code review data from open source projects. GitHub’s
Forge provides a large amount of history-changing and bug-fixing commits from
software projects, which can be the resource of CEC datasets in nature.
Bugs2Fix [189] is one such dataset mined from GitHub bug-fixing commits,
containing 2.3M bug-fix pairs (BFPS). CodeReview [190] is also a large-scale
dataset of code changes and code reviews collected from open source projects,
involving nine of the most popular programming languages. Because CodeReview
has some low-quality code review data, and the data distribution in its test
set is very similar to the data distribution in the pre-training and fine-
tuning datasets, there is a potential risk when training with a model such as
ChatGPT, which can affect the evaluation of the model’s generalization
ability. Therefore, Guo et al. (2023) create a new code review dataset
CodeReview-New [191] based on it. By designing strict filtering rules to
screen out low-quality data in CodeReview, they ended up retaining data from
467 repositories out of an initial 2,029.

Methods. The CEC and GEC tasks share similar principles and we can formulate
both of them as sequence modeling, neural machine translation, and language
modeling. Nevertheless, there are still some fundamental differences in their
technical implementations. GEC tasks are typically solved by representing text
as word embeddings or other forms of semantic vectors. While CEC tasks often
require models to have a better understanding of long contexts, existing
techniques capture the structural and semantic information of code by
representing it as a sequence of symbols or an Abstract Syntax Tree (AST)
[192]. Therefore, several pre-trained language models are trained using a
large corpus of programming languages. CuBERT [193] is a pre-trained model for
CEC tasks, which models only the Python language and is mainly applied to
discriminative tasks of code understanding. CodeBERT [194] proposed in the
same period is also modified on the basis of the BERT model, which is a
multilingual programming language model that can handle dual-modal data
(programming language (PL) and natural language (NL)). It supports modeling in
6 programming languages (Python, Java, JavaScript, PHP, Ruby, and Go) and
combines pre-training tasks for replacement tag detection with hybrid
objective functions, enabling the model to handle training using NL-PL data
pairs. Both CuBERT and CodeBERT ignore the structural information of code. In
response to this issue, Guo et al.(2020) proposed the GraphCodeBert model to
represent source code information based on AST data flow, so as to reflect
deeper code information. As AST has a tree structure and cannot be directly
used as the input of Transformer models, GraphCodeBERT allows the model to
learn the AST data flow at the cost of losing a considerable part of
information. In order to make more effective and comprehensive use of the code
structure information provided by AST, UniXcoder [103] improves on the basis
of GraphCodeBERT. They construct a one-to-one mapping function to convert an
AST into a sequence structure, and then encode it together with the code text.
In addition, some models use other ways rather than ASTs to better understand
and adapt to the structural characteristics of a code snippet, and perform
well. PLBART [104] adds additional denoising strategies to the BART model to
correlate PL and NL text. It generates grammatically and logically valid code
via large-scale sequence-to-sequence pre-training that incorporates the
denoising strategies to better understand program grammars and logical flows.
CodeT5 [105] uses the T5 model architecture and makes use of the identifier
tagging information and comment information to better model the structural
characteristics of code. CodePAD [106] designs an approach based on pushdown
automaton (PDA), which takes into account the grammar constraints of
programming languages. It ensures the availability of generated code and the
correctness of grammar, and avoids the time-consuming and labor-intensive
disadvantages of AST-based approaches.

##  4 Demonstration Systems

To facilitate applications on education scenarios, in this section, we present
some off-the-shelf demo cases with brief illustrations of their usages for
each task mentioned in Section 3. Information on these demos will be presented
in Table 5.

For QA, OpenVINO101010https://software.intel.com/enus/openvino-toolkit is an
open-source toolkit that can realize automatic QA, while PrimeQA [195] is a
public repository that enables researchers to train state-of-the-art models
for multilingual QA. For MWP, MWPToolkit [196] offers a holistic and
extensible framework with deep learning-based solvers and popular datasets,
and Symbolab111111https://www.symbolab.com/ is an online math platform that
can be applied for MWP tasks. For QC, AutoQG [197] is an online multilingual
QG service and Questimator [198] offers MCQs generation for educational
assessment with a certain topic. For AES, LinggleWrite [199] is a writing
coach tool for essay assessment and scoring, while
IntelliMetric121212https://www.intellimetric.com/direct/ is an online AES
platform that supports multi-language. For ACS,
Qualified131313https://www.qualified.io/ and
CoderPad141414https://coderpad.io/ are online programming platforms that offer
code assessment. For GEC, ALLECS [200] and
Grammarly151515https://www.grammarly.com/ are web applications that provide
online GEC service. For CEC, CodeGeeX [201] is an open-source and multilingual
CEC model that can be attached as IDE extensions, while
FavTutor161616https://aitools.favtutor.com/ is an online code assistant tool
that can be applied for CEC tasks.

Demo | Task | Form | URL
---|---|---|---
OpenVINO | QA | toolkit | https://software.intel.com/enus/openvino-toolkit
PrimeQA | QA | toolkit | https://github.com/primeqa
MWPToolkit | MWP | toolkit | https://github.com/LYH-YF/MWPToolkit
Symbolab | MWP | platform | https://www.symbolab.com/
AutoQG | QC | application | https://autoqg.net/
Questimator | QC | application | https://github.com/methodical-software/questimator
LinggleWrite | AES | application | https://f.linggle.com/
IntelliMetric | AES | platform | https://www.intellimetric.com/direct/
Qualified | ACS | platform | https://www.qualified.io/
CoderPad | ACS | platform | https://coderpad.io/
ALLECS | GEC | application | https://sterling8.d2.comp.nus.edu.sg/allecs/
Grammarly | GEC | application | https://www.grammarly.com/
CodeGeeX | CEC | extension | https://codegeex.cn/
FavTutor | CEC | platform | https://aitools.favtutor.com/
Table 5: Information on demos that can be applied in education domain.

##  5 Future Trends

According to the taxonomy of NLP in the education domain, we discuss several
promising future directions for different tasks.

More datasets in education domain. We have summarized the commonly used
datasets for educational NLP, which range from question answering to error
correction, there is still a demand for datasets fitting in more diverse
education scenarios. For example, more AES and GEC datasets in non-English
languages should be annotated to support the research on languages in low-
scenarios. Similarly, public ACS datasets are in high demand as the testbed
for evaluating the ACS systems.

The usage of LLMs. Due to the advancement of LLMs, recent studies have started
to incorporate LLMs with various educational NLP tasks. Researchers have
explored to prompt LLMs for solving zero/few-shot QA, generating questions
over knowledge bases [202], and correcting grammatical errors [203] but there
remain some disadvantages such as LLMs perform poor in the specific domains
and over-correction issues for solving GEC tasks. Therefore, more studies
focusing on easing the above drawbacks should be developed.

Intervention of difficulty-level control. Most existing NLP techniques for
education applications neglect difficulty-level control, which is an important
factor in the procedure of learning, comprehension, and assessment at
different levels. Up to now, even though there are multiple studies [57, 58,
59] trying to model the difficulty level in the QG task, they fail to align
the level to the difficulty level of syllabus.

Interpretable educational NLP. Existing studies are able to obtain good
results on various tasks like question answering, question construction,
automated assessment, and error correction. However, there are limited
techniques developed with the perspective of interpretability. For example,
interpretable GEC methods should not only correct the erroneous sentences but
also provide reasons for the correction. However this has been ignored by most
studies.

Integrated systems for education. Even though an individual task can be well-
formulated and solved via specific techniques, there is a demand of integrated
systems for education. Baladn et al. (2023) tuned open-source LLMs for
generating teacher responses in BEA 2023 Shared Task. EduChat [205] is a more
general LLM in the general education domain, which is able to automatically
assess essays, provide emotional support, and conduct Socratic teaching. In
the future, more tasks should be integrated to empower the system, making it
perform as an efficient teaching assistant.

##  6 Conclusions

This survey attempted to provide a comprehensive overview of NLP in the
education domain, including background, taxonomy of NLP tasks in the education
domain, task definition, challenges as well as corresponding solutions for
various tasks. In the last section, we suggest some future directions for NLP
in the education domain. The demand for educational applications is increasing
and educational NLP will thrive to be a promising research area. We hope this
survey will give a comprehensive picture of educational NLP and we encourage
more contribution in this field.

## References

  *   * Lu et al. [2022] Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., Kalyan, A.: Learn to explain: Multimodal reasoning via thought chains for science question answering. In: NeurIPS (2022)
  * Chen et al. [2023] Chen, W., Yin, M., Ku, M., Wan, E., Ma, X., Xu, J., Xia, T., Wang, X., Lu, P.: Theoremqa: A theorem-driven question answering dataset. arXiv preprint arXiv:2305.12524 (2023)
  * Wu et al. [2020] Wu, S., Li, Y., Zhang, D., Zhou, Y., Wu, Z.: Diverse and informative dialogue generation with context-specific commonsense knowledge awareness. In: ACL (2020)
  * Yu and Jiang [2016] Yu, J., Jiang, J.: Learning sentence embeddings with auxiliary tasks for cross-domain sentiment classification. In: EMNLP (2016)
  * Agarwal et al. [2019] Agarwal, A., Sachdeva, N., Yadav, R.K., Udandarao, V., Mittal, V., Gupta, A., Mathur, A.: Eduqa: Educational domain question answering system using conceptual network mapping. In: ICASSP (2019)
  * Bryant et al. [2023] Bryant, C., Yuan, Z., Muhammad, R., Qorib, Q., Cao, H., Ng, H., Briscoe, T.: Grammatical error correction: A survey of the state of the art. CL 49(3) (2023)
  * Zhang et al. [2019] Zhang, D., Wang, L., Zhang, L., Dai, B.T., Shen, H.T.: The gap of semantic parsing: A survey on automatic math word problem solvers. TPAMI 42(9) (2019)
  * Messer et al. [2023] Messer, M., Brown, N.C.C., Kölling, M., Shi, M.: Automated grading and feedback tools for programming education: A systematic review. TOCE (2023)
  * Litman [2016] Litman, D.: Natural language processing for enhancing teaching and learning. In: AAAI (2016)
  * Qadir [2023] Qadir, J.: Engineering education in the era of chatgpt: Promise and pitfalls of generative ai for education. In: EDUCON (2023)
  * Kurni et al. [2023] Kurni, M., Mohammed, M.S., Srinivasa, K.: A Beginner’s Guide to Introduce Artificial Intelligence in Teaching and Learning. Springer, ??? (2023)
  * Wright et al. [2022] Wright, D., Gentile, A.L., Faux, N., Beck, K.L.: Bioact: Biomedical knowledge base construction using active learning. bioRxiv (2022)
  * Khodra et al. [2011] Khodra, M.L., Widyantoro, D.H., Aziz, E., Bambang, R.T.: Information extraction from scientific paper using rhetorical classifier. In: ICEEI (2011)
  * Schmitt et al. [2022] Schmitt, A., Wambsganss, T., Leimeister, J.M.: Conversational agents for information retrieval in the education domain: A user-centered design investigation. Proc. ACM Hum.-Comput. Interact. (2022)
  * Li et al. [2023] Li, Z., Cheng, L., Zhang, C., Zhu, X., Zhao, H.: Multi-source education knowledge graph construction and fusion for college curricula. In: ICALT (2023)
  * Cripwell et al. [2023] Cripwell, L., Legrand, J., Gardent, C.: Context-aware document simplification. In: ACL Finding (2023)
  * Li et al. [2022] Li, X., Thickstun, J., Gulrajani, I., Liang, P.S., Hashimoto, T.B.: Diffusion-lm improves controllable text generation. Advances in Neural Information Processing Systems (2022)
  * Antol et al. [2015] Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D.: Vqa: Visual question answering. In: ICCV (2015)
  * Malinowski et al. [2015] Malinowski, M., Rohrbach, M., Fritz, M.: Ask your neurons: A neural-based approach to answering questions about images. In: ICCV (2015)
  * Lu et al. [2016] Lu, J., Yang, J., Batra, D., Parikh, D.: Hierarchical question-image co-attention for visual question answering. NeurIPS (2016)
  * Noh and Han [2016] Noh, H., Han, B.: Training recurrent answering units with joint loss minimization for vqa. arXiv preprint arXiv:1606.03647 (2016)
  * Xu et al. [2015] Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R., Zemel, R., Bengio, Y.: Show, attend and tell: Neural image caption generation with visual attention. In: ICML (2015)
  * Yang et al. [2016] Yang, Z., He, X., Gao, J., Deng, L., Smola, A.: Stacked attention networks for image question answering. In: CVPR (2016)
  * Andreas et al. [2016] Andreas, J., Rohrbach, M., Darrell, T., Klein, D.: Learning to compose neural networks for question answering. arXiv preprint arXiv:1601.01705 (2016)
  * Fukui et al. [2016] Fukui, A., Park, D.H., Yang, D., Rohrbach, A., Darrell, T., Rohrbach, M.: Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv preprint arXiv:1606.01847 (2016)
  * Kim et al. [2016] Kim, J.-H., On, K.-W., Lim, W., Kim, J., Ha, J.-W., Zhang, B.-T.: Hadamard product for low-rank bilinear pooling. arXiv preprint arXiv:1610.04325 (2016)
  * Ram et al. [2021] Ram, O., Kirstain, Y., Berant, J., Globerson, A., Levy, O.: Few-Shot question answering by pretraining span selection (2021)
  * Lee et al. [2019] Lee, S., Kim, D., Park, J.: Domain-agnostic question-answering with adversarial training. (2019)
  * Kembhavi et al. [2016] Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., Farhadi, A.: A diagram is worth a dozen images. In: ECCV (2016)
  * Poco and Heer [2017] Poco, J., Heer, J.: Reverse-engineering visualizations: Recovering visual encodings from chart images. CGF (2017)
  * Kafle et al. [2018] Kafle, K., Price, B., Cohen, S., Kanan, C.: Dvqa: Understanding data visualizations via question answering. In: CVPR (2018)
  * Wang et al. [2023a] Wang, L., Hu, Y., He, J., Xu, X., Liu, N., Liu, H., Shen, H.T.: T-SciQ: Teaching multimodal chain-of-thought reasoning via large language model signals for science question answering (2023)
  * Wang et al. [2023b] Wang, Y., Ma, X., Chen, W.: Augmenting black-box llms with medical textbooks for clinical question answering. arXiv preprint arXiv:2309.02233 (2023)
  * Liu et al. [2023] Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning (2023)
  * Lu et al. [2023] Lu, P., Peng, B., Cheng, H., Galley, M., Chang, K.-W., Wu, Y.N., Zhu, S.-C., Gao, J.: Chameleon: Plug-and-play compositional reasoning with large language models (2023)
  * Wang et al. [2017] Wang, Y., Liu, X., Shi, S.: Deep neural solver for math word problems. In: EMNLP (2017)
  * Wang et al. [2018] Wang, L., Wang, Y., Cai, D., Zhang, D., Liu, X.: Translating a math word problem to a expression tree. (2018)
  * Chiang and Chen [2019] Chiang, T.-R., Chen, Y.-N.: Semantically-aligned equation generation for solving and reasoning math word problems. arXiv preprint arXiv:1811.00720 (2019)
  * Liu et al. [2019] Liu, Q., Guan, W., Li, S., Kawahara, D.: Tree-structured decoding for solving math word problems. In: EMNLP-IJCNLP (2019)
  * Zhang et al. [2020] Zhang, J., Wang, L., Lee, R.K.-W., Bin, Y., Wang, Y., Shao, J., Lim, E.-P.: Graph-to-tree learning for solving math word problems. In: ACL (2020)
  * Lin et al. [2021] Lin, X., Huang, Z., Zhao, H., Chen, E., Liu, Q., Wang, H., Wang, S.: Hms: A hierarchical solver with dpendency-enhanced understanding for math word problem. AAAI (2021)
  * Hong et al. [2021] Hong, Y., Li, Q., Ciao, D., Huang, S., Zhu, S.-C.: Learning by fixing: Solving math word problems with weak supervision. AAAI (2021)
  * Xie and Sun [2019] Xie, Z., Sun, S.: A goal-driven tree-structured neural model for math word problems. In: Ijcai (2019)
  * Wang et al. [2023] Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R.K.-W., Lim, E.-P.: Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. arXiv preprint arXiv:2305.04091 (2023)
  * Zhang et al. [2022] Zhang, Z., Zhang, A., Li, M., Smola, A.: Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493 (2022)
  * Zhou et al. [2022] Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Bousquet, O., Le, Q., Chi, E.: Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625 (2022)
  * Zheng et al. [2023] Zheng, C., Liu, Z., Xie, E., Li, Z., Li, Y.: Progressive-hint prompting improves reasoning in large language models. arXiv preprint arXiv:2304.09797 (2023)
  * Chen et al. [2023] Chen, W., Ma, X., Wang, X., Cohen, W.W.: Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. TMLR (2023)
  * Kunichika et al. [2004] Kunichika, H., Katayama, T., Hirashima, T., Takeuchi, A.: Automated question generation methods for intelligent english learning systems and its evaluation. In: ICCE, vol. 670 (2004)
  * Du and Cardie [2017] Du, X., Cardie, C.: Identifying where to focus in reading comprehension for neural question generation. In: EMNLP (2017)
  * Harrison and Walker [2018] Harrison, V., Walker, M.: Neural generation of diverse questions using answer focus, contextual and linguistic features. arXiv preprint arXiv:1809.02637 (2018)
  * Zhou et al. [2018] Zhou, Q., Yang, N., Wei, F., Tan, C., Bao, H., Zhou, M.: Neural question generation from text: A preliminary study. In: NLPCC (2018)
  * Naeiji [2022] Naeiji, A.: Question generation using sequence-to-sequence model with semantic role labels (2022)
  * Wang and Baraniuk [2023] Wang, Z., Baraniuk, R.: Multiqg-ti: Towards question generation from multi-modal sources. arXiv preprint arXiv:2307.04643 (2023)
  * Zhou et al. [2019] Zhou, W., Zhang, M., Wu, Y.: Multi-task learning with language modeling for question generation. arXiv preprint arXiv:1908.11813 (2019)
  * Chen et al. [2019] Chen, Y., Wu, L., Zaki, M.J.: Natural question generation with reinforcement learning based graph-to-sequence model. arXiv preprint arXiv:1910.08832 (2019)
  * Gao et al. [2018] Gao, Y., Bing, L., Chen, W., Lyu, M.R., King, I.: Difficulty controllable generation of reading comprehension questions. arXiv preprint arXiv:1807.03586 (2018)
  * Cheng et al. [2021] Cheng, Y., Li, S., Liu, B., Zhao, R., Li, S., Lin, C., Zheng, Y.: Guiding the growth: Difficulty-controllable question generation through step-by-step rewriting. arXiv preprint arXiv:2105.11698 (2021)
  * Uto et al. [2023] Uto, M., Tomikawa, Y., Suzuki, A.: Difficulty-controllable neural question generation for reading comprehension using item response theory. In: BEA (2023)
  * Steuer et al. [2021] Steuer, T., Filighera, A., Meuser, T., Rensing, C.: I do not understand what i cannot define: Automatic question generation with pedagogically-driven content selection. arXiv preprint arXiv:2110.04123 (2021)
  * Hadifar et al. [2023] Hadifar, A., Bitew, S.K., Deleu, J., Hoste, V., Develder, C., Demeester, T.: Diverse content selection for educational question generation. In: EACL (2023)
  * Srivastava and Goodman [2021] Srivastava, M., Goodman, N.: Question generation for adaptive education. arXiv preprint arXiv:2106.04262 (2021)
  * Wang and Li [2023] Wang, Y., Li, L.: Difficulty-controlled question generation in adaptive education for few-shot learning. In: ADMA (2023)
  * Mitkov et al. [2003] Mitkov, R., et al.: Computer-aided generation of multiple-choice tests. In: HLT-NAAC (2003)
  * Brown et al. [2005] Brown, J., Frishkoff, G., Eskenazi, M.: Automatic question generation for vocabulary assessment. In: HLT/EMNLP (2005)
  * Gao et al. [2019] Gao, Y., Bing, L., Li, P., King, I., Lyu, M.R.: Generating distractors for reading comprehension questions from real examinations. In: AAAI, vol. 33 (2019)
  * Bitew et al. [2023] Bitew, S.K., Deleu, J., Develder, C., Demeester, T.: Distractor generation for multiple-choice questions with predictive prompting and large language models. arXiv preprint arXiv:2307.16338 (2023)
  * Liang et al. [2018] Liang, C., Yang, X., Dave, N., Wham, D., Pursel, B., Giles, C.L.: Distractor generation for multiple choice questions using learning to rank. In: BEA (2018)
  * Sinha et al. [2020] Sinha, M., Dasgupta, T., Mandav, J.: Ranking multiple choice question distractors using semantically informed neural networks. In: CIKM (2020)
  * Wang et al. [2023] Wang, J., Rong, W., Bai, J., Sun, Z., Ouyang, Y., Xiong, Z.: Multi-source soft labeling and hard negative sampling for retrieval distractor ranking. TLT (2023)
  * Zesch et al. [2015] Zesch, T., Wojatzki, M., Scholten-Akoun, D.: Task-independent features for automated essay grading. In: BEA (2015)
  * Phandi et al. [2015] Phandi, P., Chai, K.M.A., Ng, H.T.: Flexible domain adaptation for automated essay scoring using correlated linear regression. In: EMNLP (2015)
  * Louis and Higgins [2010] Louis, A., Higgins, D.: Off-topic essay detection using short prompt texts. In: HLT-NAACL (2010)
  * Chen and He [2013] Chen, H., He, B.: Automated essay scoring by maximizing human-machine agreement. In: EMNLP (2013)
  * Yang et al. [2020] Yang, R., Cao, J., Wen, Z., Wu, Y., He, X.: Enhancing automated essay scoring performance via fine-tuning pre-trained language models with combination of regression and ranking. In: EMNLP (2020)
  * Zeng et al. [2023] Zeng, Z., Gasevic, D., Chen, G.: On the effectiveness of curriculum learning in educational text scoring. AAAI (2023)
  * Yupei and Renfen [2021] Yupei, W., Renfen, H.: A prompt-independent and interpretable automated essay scoring method for Chinese second language writing. In: CCL (2021)
  * Sharma et al. [2021] Sharma, A., Kabra, A., Kapoor, R.: Feature enhanced capsule networks for robust automatic essay scoring. In: Machine Learning and Knowledge Discovery in Databases. (2021)
  * Wang et al. [2023] Wang, C., Jiang, Z., Yin, Y., Cheng, Z., Ge, S., Gu, Q.: Aggregating multiple heuristic signals as supervision for unsupervised automated essay scoring. In: ACL (2023)
  * Persing and Ng [2013] Persing, I., Ng, V.: Modeling thesis clarity in student essays. In: ACL (2013)
  * Persing and Ng [2014] Persing, I., Ng, V.: Modeling prompt adherence in student essays. In: ACL (2014)
  * Roy Choudhury et al. [2016] Roy Choudhury, R., Yin, H., Fox, A.: Scale-driven automatic hint generation for coding style. In: ITS (2016)
  * Ureel II and Wallace [2019] Ureel II, L.C., Wallace, C.: Automated critique of early programming antipatterns. In: SIGCSE, pp. 738–744 (2019)
  * Keuning et al. [2021] Keuning, H., Heeren, B., Jeuring, J.: A tutoring system to learn code refactoring. In: SIGCSE (2021)
  * Birillo et al. [2022] Birillo, A., Vlasov, I., Burylov, A., Selishchev, V., Goncharov, A., Tikhomirova, E., Vyahhi, N., Bryksin, T.: Hyperstyle: A tool for assessing the code quality of solutions to programming assignments. In: SIGCSE (2022)
  * Lajis et al. [2018] Lajis, A., Nasir, H.M., Aziz, N.A.: Proposed assessment framework based on bloom taxonomy cognitive competency: Introduction to programming. In: ICSCA (2018)
  * Glassman et al. [2015] Glassman, E.L., Scott, J., Singh, R., Guo, P.J., Miller, R.C.: Overcode: Visualizing vriation in student solutions to programming problems at scale. TOCHI 22(2) (2015)
  * Talbot et al. [2020] Talbot, M., Geldreich, K., Sommer, J., Hubwieser, P.: Re-use of programming patterns or problem solving? representation of scratch programs by tgraphs to support static code analysis. In: WiPSCE (2020)
  * Von Wangenheim et al. [2018] Von Wangenheim, C.G., Hauck, J.C., Demetrio, M.F., Pelle, R., Cruz Alves, N., Barbosa, H., Azevedo, L.F.: Codemaster–automatic assessment and grading of app inventor and snap! programs. INFEDU 17(1) (2018)
  * Queirós and Leal [2012] Queirós, R.A.P., Leal, J.P.: Petcha: a programming exercises teaching assistant. In: ITiCSE (2012)
  * Knight and Chander [1994] Knight, K., Chander, I.: Automated postediting of documents. AAAI (1994)
  * Rozovskaya et al. [2014] Rozovskaya, A., Chang, K.-W., Sammons, M., Roth, D., Habash, N.: The illinois-columbia system in the conll-2014 shared task. In: CoNLL (2014)
  * Lewis et al. [2019] Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., Zettlemoyer, L.: Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 (2019)
  * Raffel et al. [2020] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR 21(1) (2020)
  * [95] Kaneko, M., Takase, S., Niwa, A., Okazaki, N.: Interpretability for language learners using example-based grammatical error correction. arXiv preprint arXiv:2203.07085
  * Zhang et al. [2022] Zhang, Y., Zhang, B., Li, Z., Bao, Z., Li, C., Zhang, M.: Syngec: Syntax-enhanced grammatical error correction with a tailored gec-oriented parser. arXiv preprint arXiv:2210.12484 (2022)
  * Zhang and Li [2022] Zhang, Y., Li, Z.: Csyngec: Incorporating constituent-based syntax for grammatical error correction with a tailored gec-oriented parser. arXiv preprint arXiv:2211.08158 (2022)
  * Awasthi et al. [2019] Awasthi, A., Sarawagi, S., Goyal, R., Ghosh, S., Piratla, V.: Parallel iterative edit models for local sequence transduction. (2019)
  * Dong et al. [2019] Dong, Y., Li, Z., Rezagholizadeh, M., Cheung, J.C.K.: EditNTS: An neural programmer-interpreter model for sentence simplification through explicit editing. (2019)
  * Malmi et al. [2019] Malmi, E., Krause, S., Rothe, S., Mirylenka, D., Severyn, A.: Encode, tag, realize: High-precision text editing. arXiv preprint arXiv:1909.01187 (2019)
  * Omelianchuk et al. [2020] Omelianchuk, K., Atrasevych, V., Chernodub, A., Skurzhanskyi, O.: Gector – grammatical error correction: Tag, not rewrite. (2020)
  * Guo et al. [2020] Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., Zhou, L., Duan, N., Svyatkovskiy, A., Shengyu, F., Tufano, M., Deng, S., Clement, C., Drain, D., Sundaresan, N., Yin, J., Jiang, D., Zhou, M.: Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366 (2020)
  * [103] Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., Yin, J.: Unixcoder: Unified cross-modal pre-training for code representation. arXiv preprint arXiv:2203.03850
  * Ahmad et al. [2021] Ahmad, W., Chakraborty, S., Ray, B.e.a.: Unified pre-training for program understanding and generation. In: NAACL (2021)
  * Wang et al. [2021] Wang, Y., Wang, W., Joty, S., Hoi, S.C.H.: CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In: Moens, M.-F., Huang, X., Specia, L., Yih, S.W.-t. (eds.) EMNLP (2021)
  * Dong et al. [2022] Dong, Y., Jiang, X., Liu, Y., Li, G., Jin, Z.: Codepad: Sequence-based code generation with pushdown automaton (2022)
  * Soares et al. [2021] Soares, T.G., Azhari, A., Rokhman, N., Wonarko, E.: Education question answering systems: a survey. In: IMECS (2021)
  * Kembhavi et al. [2017] Kembhavi, A., Seo, M., Schwenk, D., Choi, J., Farhadi, A., Hajishirzi, H.: Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In: CVPR (2017)
  * Sampat et al. [2020] Sampat, S.K., Yang, Y., Baral, C.: Visuo-linguistic question answering (vlqa) challenge. arXiv preprint arXiv:2005.00330 (2020)
  * Jin et al. [2021] Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., Szolovits, P.: What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences (2021)
  * Pal et al. [2022] Pal, A., Umapathi, L.K., Sankarasubbu, M.: Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In: CHIL (2022)
  * Dosovitskiy et al. [2020] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * Gao et al. [2019] Gao, P., Jiang, Z., You, H., Lu, P., Hoi, S.C., Wang, X., Li, H.: Dynamic fusion with intra-and inter-modality attention flow for visual question answering. In: CVPR (2019)
  * Gao et al. [2022] Gao, F., Ping, Q., Thattai, G., Reganti, A., Wu, Y.N., Natarajan, P.: Transform-retrieve-generate: Natural language-centric outside-knowledge visual question answering. In: CVPR (2022)
  * Xu et al. [2022] Xu, W., Li, X., Zhang, W., Zhou, M., Bing, L., Lam, W., Si, L.: From clozing to comprehending: Retrofitting pre-trained language model to pre-trained machine reader. arXiv preprint arXiv:2212.04755 (2022)
  * Huang et al. [2016] Huang, D., Shi, S., Lin, C.-Y., Yin, J., Ma, W.-Y.: How well do computers solve math word problems? large-scale dataset construction and evaluation. In: ACL (2016)
  * Upadhyay and Chang [2016] Upadhyay, S., Chang, M.-W.: Annotating derivations: A new evaluation strategy and dataset for algebra word problems. arXiv preprint arXiv:1609.07197 (2016)
  * Amini et al. [2019] Amini, A., Gabriel, S., Lin, S., Koncel-Kedziorski, R., Choi, Y., Hajishirzi, H.: MathQA: Towards interpretable math word problem solving with operation-based formalisms. (2019)
  * Miao et al. [2020] Miao, S.-y., Liang, C.-C., Su, K.-Y.: A diverse corpus for evaluating and developing English math word problem solvers. (2020)
  * Cobbe et al. [2021] Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., Schulman, J.: Training verifiers to solve math word problems (2021)
  * Lu et al. [2021] Lu, P., Qiu, L., Chen, J., Xia, T., Zhao, Y., Zhang, W., Yu, Z., Liang, X., Zhu, S.-C.: Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. (2021)
  * Lu et al. [2022] Lu, P., Qiu, L., Chang, K.-W., Wu, Y.N., Zhu, S.-C., Rajpurohit, T., Clark, P., Kalyan, A.: Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209.14610 (2022)
  * Zhang et al. [2020a] Zhang, D., Wang, L., Zhang, L., Dai, B.T., Shen, H.T.: The gap of semantic parsing: A survey on automatic math word problem solvers. TPAMI (2020)
  * Zhang et al. [2020b] Zhang, J., Lee, R.K.-W., Lim, E.-P., Qin, W., Wang, L., Shao, J., Sun, Q.: Teacher-student networks with multiple decoders for solving math word problem. In: IJCAI (2020)
  * Wei et al. [2023] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., Zhou, D.: Chain-of-thought prompting elicits reasoning in large language models (2023)
  * Gou et al. [2023] Gou, Z., Shao, Z., Gong, Y., Yang, Y., Huang, M., Duan, N., Chen, W., et al.: Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint arXiv:2309.17452 (2023)
  * Das et al. [2021] Das, B., Majumder, M., Phadikar, S., Sekh, A.A.: Automatic question generation and answer assessment: A survey. RPTEL (2021)
  * Dugan et al. [2022] Dugan, L., Miltsakaki, E., Upadhyay, S., Ginsberg, E., Gonzalez, H., Choi, D., Yuan, C., Callison-Burch, C.: A feasibility study of answer-agnostic question generation for education. arXiv preprint arXiv:2203.08685 (2022)
  * Tang et al. [2017] Tang, D., Duan, N., Qin, T., Yan, Z., Zhou, M.: Question answering and question generation as dual tasks. arXiv preprint arXiv:1706.02027 (2017)
  * Welbl et al. [2017] Welbl, J., Liu, N.F., Gardner, M.: Crowdsourcing multiple choice science questions. arXiv preprint arXiv:1707.06209 (2017)
  * Lai et al. [2017] Lai, G., Xie, Q., Liu, H., Yang, Y., Hovy, E.: Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683 (2017)
  * Xu et al. [2022] Xu, Y., Wang, D., Yu, M., Ritchie, D., Yao, B., Wu, T., Zhang, Z., Li, T.J.-J., Bradford, N., Sun, B., et al.: Fantastic questions and where to find them: Fairytaleqa–an authentic dataset for narrative comprehension. arXiv preprint arXiv:2203.13947 (2022)
  * Wu et al. [2023] Wu, Y., Nouri, J., Megyesi, B., Henriksson, A., Duneld, M., Li, X.: Towards data-effective educational question generation with prompt-based learning. In: SAI (2023)
  * Jia et al. [2021] Jia, X., Zhou, W., Sun, X., Wu, Y.: Eqg-race: Examination-type question generation. In: AAAI, vol. 35 (2021)
  * Steuer et al. [2022] Steuer, T., Filighera, A., Tregel, T.: Investigating educational and noneducational answer selection for educational question generation. IEEE Access (2022)
  * Zhao et al. [2022] Zhao, Z., Hou, Y., Wang, D., Yu, M., Liu, C., Ma, X.: Educational question generation of children storybooks via question type distribution learning and event-centric summarization. arXiv preprint arXiv:2203.14187 (2022)
  * Chen et al. [2018] Chen, G., Yang, J., Hauff, C., Houben, G.-J.: Learningq: A large-scale dataset for educational question generation. In: ICWSM, vol. 12 (2018)
  * Gong et al. [2022] Gong, H., Pan, L., Hu, H.: Khanq: A dataset for generating deep questions in education. In: ICCL (2022)
  * Hadifar et al. [2023] Hadifar, A., Bitew, S.K., Deleu, J., Develder, C., Demeester, T.: Eduqg: A multi-format multiple-choice dataset for the educational domain. IEEE Access (2023)
  * Chan and Fan [2019] Chan, Y.-H., Fan, Y.-C.: A recurrent bert-based model for question generation. In: MRQA (2019)
  * Wang et al. [2022] Wang, Z., Valdez, J., Basu Mallick, D., Baraniuk, R.G.: Towards human-like educational question generation with large language models. In: AIED (2022)
  * Yuan et al. [2017] Yuan, X., Wang, T., Gulcehre, C., Sordoni, A., Bachman, P., Subramanian, S., Zhang, S., Trischler, A.: Machine comprehension by text-to-text neural question generation. arXiv preprint arXiv:1705.02012 (2017)
  * Sun et al. [2018] Sun, X., Liu, J., Lyu, Y., He, W., Ma, Y., Wang, S.: Answer-focused and position-aware neural question generation. In: EMNLP (2018)
  * Ma et al. [2020] Ma, X., Zhu, Q., Zhou, Y., Li, X.: Improving question generation with sentence-level semantic matching and answer position inferring. In: AAAI, vol. 34 (2020)
  * Lord [2012] Lord, F.M.: Applications of Item Response Theory to Practical Testing Problems. Routledge, ??? (2012)
  * Qiu et al. [2020] Qiu, Z., Wu, X., Fan, W.: Automatic distractor generation for multiple choice questions in standard tests. arXiv preprint arXiv:2011.13100 (2020)
  * Bitew et al. [2022] Bitew, S.K., Hadifar, A., Sterckx, L., Deleu, J., Develder, C., Demeester, T.: Learning to reuse distractors to support multiple choice question generation in education. IEEE-TLT (2022)
  * Tigina et al. [2023] Tigina, M., Birillo, A., Golubev, Y., Keuning, H., Vyahhi, N., Bryksin, T.: Analyzing the quality of submissions in online programming courses. arXiv preprint arXiv:2301.11158 (2023)
  * Yannakoudakis et al. [2011] Yannakoudakis, H., Briscoe, T., Medlock, B.: A new dataset and method for automatically grading ESOL texts. In: ACL (2011)
  * Blanchard et al. [2013] Blanchard, D., Tetreault, J., Higgins, D., Cahill, A., Chodorow, M.: Toefl 11: A corpus of non-native english. ETS Research Report Series 2013(2) (2013)
  * Stab and Gurevych [2014] Stab, C., Gurevych, I.: Annotating argument components and relations in persuasive essays. In: COLING (2014)
  * Wang and Hu [2021] Wang, Y., Hu, R.: A prompt-independent and interpretable automated essay scoring method for chinese second language writing. In: CCL (2021)
  * Yamaura et al. [2023] Yamaura, M., Fukuda, I., Uto, M.: Neural automated essay scoring considering logical structure. In: AIED (2023)
  * Uto et al. [2020] Uto, M., Xie, Y., Ueno, M.: Neural automated essay scoring incorporating handcrafted features. In: ICCL (2020)
  * Ridley et al. [2008] Ridley, R., He, L., Dai, X., Huang, S., Chen, J.: Prompt agnostic essay scorer: A domain generalization approach to cross-prompt automated essay scoring. arXiv preprint arXiv:2008.01441 (2008)
  * Jiang et al. [2023] Jiang, Z., Gao, T., Yin, Y., Liu, M., Yu, H., Cheng, Z., Gu, Q.: Improving domain generalization for prompt-aware essay scoring via disentangled representation learning. In: ACL (2023)
  * Jiang et al. [2021] Jiang, Z., Liu, M., Yin, Y., Yu, H., Cheng, Z., Gu, Q.: Learning from graph propagation via ordinal distillation for one-shot automated essay scoring. In: WWW (2021)
  * Birillo et al. [2023] Birillo, A., Artser, E., Golubev, Y., Tigina, M., Keuning, H., Vyahhi, N., Bryksin, T.: Detecting code quality issues in pre-written templates of programming tasks in online courses. arXiv preprint arXiv:2304.12376 (2023)
  * Wang et al. [2023] Wang, D., Zhang, E., Lu, X.: Automatic grading of student code with similarity measurement. In: ECML PKDD (2023)
  * Allamanis et al. [2015] Allamanis, M., Barr, E.T., Bird, C., Sutton, C.: Suggesting accurate method and class names. In: FSE (2015)
  * Zhang et al. [2019] Zhang, J., Wang, X., Zhang, H., Sun, H., Wang, K., Liu, X.: A novel neural source code representation based on abstract syntax tree. In: ICSE (2019)
  * Alon et al. [2019] Alon, U., Zilberstein, M., Levy, O., Yahav, E.: Code2vec: Learning distributed representations of code. PACMPL (2019)
  * Hellendoorn et al. [2019] Hellendoorn, V.J., Sutton, C., Singh, R., Maniatis, P., Bieber, D.: Global relational models of source code. In: ICLR (2019)
  * Feng et al. [2020] Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al.: Codebert: A pre-trained model for programming and natural languages. arXiv preprint arXiv:2002.08155 (2020)
  * Kanade et al. [2020] Kanade, A., Maniatis, P., Balakrishnan, G., Shi, K.: Learning and evaluating contextual embedding of source code. In: ICML (2020)
  * Rothe et al. [2021] Rothe, S., Mallinson, J., Malmi, E., Krause, S., Severyn, A.: A simple recipe for multilingual grammatical error correction. (2021)
  * Ng et al. [2014] Ng, H.T., Wu, S.M., Briscoe, T., Hadiwinoto, C., Susanto, R.H., Bryant, C.: The conll-2014 shared task on grammatical error correction. In: CoNLL (2014)
  * Bryant et al. [2019] Bryant, C., Felice, M., Andersen, Ø.E., Briscoe, T.: The bea-2019 shared task on grammatical error correction. In: BEA (2019)
  * Tseng et al. [2015] Tseng, Y.-H., Lee, L.-H., Chang, L.-P., Chen, H.-H.: Introduction to sighan 2015 bake-off for chinese spelling check. In: SIGHAN (2015)
  * Zhao et al. [2022] Zhao, H., Wang, B., Wu, D., Che, W., Chen, Z., Wang, S.: Overview of ctc 2021: Chinese text correction for native speakers. arXiv preprint arXiv:2208.05681 (2022)
  * Xu et al. [2022] Xu, L., Wu, J., Peng, J., Fu, J., Cai, M.: Fcgec: Fine-grained corpus for chinese grammatical error correction. arXiv preprint arXiv:2210.12364 (2022)
  * Du et al. [2023] Du, H., Zhao, Y., Tian, Q., Wang, J., Wang, L., Lan, Y., Lu, X.: Flacgec: A chinese grammatical error correction dataset with fine-grained linguistic annotation. In: CIKM (2023)
  * Náplava et al. [2022] Náplava, J., Straka, M., Straková, J., Rosen, A.: Czech grammar error correction with a large and diverse corpus. TACL (2022)
  * Rozovskaya and Roth [2019] Rozovskaya, A., Roth, D.: Grammar error correction in morphologically rich languages: The case of russian. TACL (2019)
  * Grundkiewicz and Junczys-Dowmunt [2014] Grundkiewicz, R., Junczys-Dowmunt, M.: The wiked error corpus: A corpus of corrective wikipedia edits and its application to grammatical error correction. In: LNCS (2014)
  * Davidson et al. [2020] Davidson, S., Yamada, A., Mira, P., Carando, A., Sánchez-Gutiérrez, C., Sagae, K.: Developing nlp tools with a new corpus of learner spanish. LRE (2020)
  * Syvokon and Nahorna [2021] Syvokon, O., Nahorna, O.: Ua-gec: Grammatical error correction and fluency corpus for the ukrainian language. arxiv (2021)
  * Cotet et al. [2020] Cotet, T.-M., Ruseti, S., Dascalu, M.: Neural grammatical error correction for romanian. In: ICTAI (2020)
  * Sutskever et al. [2014] Sutskever, I., Vinyals, O., Le, Q.: Sequence to sequence learning with neural networks. arXiv: Computation and Language,arXiv: Computation and Language (2014)
  * Stahlberg and Kumar [2020] Stahlberg, F., Kumar, S.: Seq2Edits: Sequence transduction using span-level edit operations. In: EMNLP (2020)
  * Li et al. [2023] Li, Y., Liu, X., Wang, S., Gong, P., Wong, D.F., Gao, Y., Huang, H., Zhang, M.: TemplateGEC: Improving grammatical error correction with detection template. In: Rogers, A., Boyd-Graber, J., Okazaki, N. (eds.) Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2023)
  * Fang et al. [2023] Fang, T., Yang, S., Lan, K., Wong, D.F., Hu, J., Chao, L.S., Zhang, Y.: Is chatgpt a highly fluent grammatical error correction system? a comprehensive evaluation. arXiv preprint arXiv:2304.01746 (2023)
  * Dahlmeier et al. [2013] Dahlmeier, D., Ng, H., Wu, S.: Building a large annotated corpus of learner english: The nus corpus of learner english. BEA (2013)
  * Baoli [2011] Baoli, Z.: The principles for building the “international corpus of learner chinese”. Applied Linguistics (2011)
  * Just et al. [2014] Just, R., Jalali, D., Ernst, M.D.: Defects4j: a database of existing faults to enable controlled testing studies for java programs. In: ISSTA (2014)
  * Le Goues et al. [2015] Le Goues, C., Holtschulte, N., Smith, E.K., Brun, Y., Devanbu, P., Forrest, S., Weimer, W.: The manybugs and introclass benchmarks for automated repair of c programs. TSE (2015)
  * Lin et al. [2017] Lin, D., Koppel, J., Chen, A., Solar-Lezama, A.: Quixbugs: a multi-lingual program repair benchmark set based on the quixey challenge. In: SPLASH Companion (2017)
  * Xia et al. [2022] Xia, C., Wei, Y., Zhang, L.: Practical program repair in the era of large pre-trained language models. arXiv preprint arXiv:2210.14179 (2022)
  * Tufano et al. [2018] Tufano, M., Watson, C., Bavota, G., Penta, M., White, M., Poshyvanyk, D.: An empirical study on learning bug-fixing patches in the wild via neural machine translation. TOSEM (2018)
  * Li et al. [2022] Li, Z., Lu, S., Guo, D., Duan, N., Jannu, S., Jenks, G., Majumder, D., Green, J., Svyatkovskiy, A., Fu, S., Sundaresan, N.: Automating code review activities by large-scale pre-training. In: ESEC/FSE (2022)
  * Guo et al. [2023] Guo, Q., Cao, J., Xie, X., Liu, S., Li, X., Chen, B., Peng, X.: Exploring the potential of ChatGPT in automated code refinement: An empirical study (2023)
  * Falleri et al. [2014] Falleri, J.-R., Morandat, F., Blanc, X., Martinez, M., Monperrus, M.: Fine-grained and accurate source code differencing. In: ASE (2014)
  * Kanade et al. [2019] Kanade, A., Maniatis, P., Balakrishnan, G., Shi, K.: Learning and evaluating contextual embedding of source code. arxiv (2019)
  * Feng et al. [2020] Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., Zhou, M.: Codebert: A pre-trained model for programming and natural languages. In: EMNLP (2020)
  * Sil et al. [2023] Sil, A., Sen, J., Iyer, B., Franz, M., Fadnis, K., Bornea, M., Rosenthal, S., McCarley, S., Zhang, R., Kumar, V., et al.: Primeqa: The prime repository for state-of-the-art multilingual question answering research and development. arXiv preprint arXiv:2301.09715 (2023)
  * Lan et al. [2022] Lan, Y., Wang, L., Zhang, Q., Lan, Y., Dai, B.T., Wang, Y., Zhang, D., Lim, E.-P.: Mwptoolkit: an open-source framework for deep learning-based math word problem solvers. In: AAAI, vol. 36 (2022)
  * Ushio et al. [2023] Ushio, A., Alva-Manchego, F., Camacho-Collados, J.: A practical toolkit for multilingual question and answer generation. arXiv preprint arXiv:2305.17416 (2023)
  * Guo et al. [2016] Guo, Q., Kulkarni, C., Kittur, A., Bigham, J.P., Brunskill, E.: Questimator: Generating knowledge assessments for arbitrary topics. In: IJCAI (2016)
  * Tsai et al. [2020] Tsai, C.-T., Chen, J.-J., Yang, C.-Y., Chang, J.S.: Lingglewrite: a coaching system for essay writing. In: ACL (2020)
  * Qorib et al. [2023] Qorib, M.R., Moon, G., Ng, H.T.: Allecs: A lightweight language error correction system. In: EACL (2023)
  * Zheng et al. [2023] Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Wang, Z., Shen, L., Wang, A., Li, Y., et al.: Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568 (2023)
  * Liang et al. [2023] Liang, Y., Wang, J., Zhu, H., Wang, L., Qian, W., Lan, Y.: Prompting large language models with chain-of-thought for few-shot knowledge base question generation. In: EMNLP (2023)
  * Wu et al. [2023] Wu, H., Wang, W., Wan, Y., Jiao, W., Lyu, M.: ChatGPT or Grammarly? Evaluating ChatGPT on grammatical error correction benchmark (2023)
  * Baladn et al. [2023] Baladn, A., Sastre, I., Chiruzzo, L., Ros, A.: Retuyt-inco at bea 2023 shared task: Tuning open-source llms for generating teacher responses. In: BEA (2023)
  * Dan et al. [2023] Dan, Y., Lei, Z., Gu, Y., Li, Y., Yin, J., Lin, J., Ye, L., Tie, Z., Zhou, Y., Wang, Y., et al.: Educhat: A large-scale language model-based chatbot system for intelligent education. arXiv preprint arXiv:2308.02773 (2023)