# A new way to accelerate the D-MORPH method to search for optimal quantum
control

Konstantin Zhdanov Department of applied mathematics and control processes,
Saint-Petersburg state university, Russia

###### Abstract

The paper introduces new corrections of different orders of smallness to the
D-MORPH method by using the full form of the derivative of the exponential
map, defined on a Lie algebra, to search for the optimal control of a quantum
system that implements a desired unitary evolution. The inclusion of such
corrections, which take into account information about different system
Hamiltonian’s commutators, results in faster optimal control’s finding, even
compared to the improved version of the method published earlier by the
author.

Keywords:  quantum systems, optimal control, evolution operator,
Lie algebras, quantum computations

## Introduction

The D-MORPH method [3, 4, 5, 7, 8] is one of the most frequently used methods
for constructing an optimal control of quantum systems that minimizes a given
objective function. The D-MORPH method is based on numerical integration of a
specially constructed system of ordinary differential equations with
independent variable $s$, which represents the optimization’s progress towards
the minimum of the objective function, using the variable-step Runge–Kutta’s
method of order four. That process usually takes a long time to reach a small
vicinity of the minimum. As was shown author’s previous paper [1], the D-MORPH
method can be speeded up with use of information about different quantum
system Hamiltonian’s commutators.

In this paper a new method for obtaining corrections of different orders of
smallness to the D-MORPH method is presented, such that they allow a speed-up
of quantum optimal control search by using the complete form of the derivative
of the exponential map defined on a Lie algebra. The corrections obtained
resemble those found by the author earlier in [1], hence these two forms of
corrections are compared with each other by numerically implementing a quantum
gate, and it’s shown that the use of the new form results in shorter times
spent to solve the underlying system.

## The previously obtained method

To implement a desired unitary operator $U_{D}$ at a moment $T$ in a $N$-level
quantum system with the Hamiltonian
$H(t)=H_{0}+\sum_{k=1}^{M}\epsilon_{k}(t)H_{k}$ subject to $M$ controls
$\\{\epsilon_{k}\\}_{k=1}^{M}$ the D-MORPH method requires solving of the
system

$\frac{\mathrm{d}\epsilon_{k}^{l}(s)}{\mathrm{d}s}=-\frac{\partial
J}{\partial\epsilon_{k}^{l}(s)},\quad l=1,\ldots,L,\quad k=1,\ldots,M,$ (1)

on the interval $[0,S]$ divided into $L$ equal-length subintervals
$[t_{l-1},t_{l}]$, where
$J=0.5-\operatorname*{Re}\operatorname*{Tr}\left(U^{*}_{D}U(T,0)\right)/(2N)$
— an objective function to be minimized, $\epsilon_{k}^{l}$ — constant
controls’ values on $[t_{l-1},t_{l}]$, $s$ — the parameter representing the
progress towards the minimum, $\operatorname*{Tr}(\ldots)$ — the trace
operation, $U^{*}_{D}$ — the Hermitian adjoint of $U_{D}$, $U(T,0)$ — the
system’s evolution operator on $[0,T]$.

In the previous paper [1] the following corrections of different orders to
D-MORPH were obtained by integrating system (1) with respect to time $t$:

$\frac{\mathrm{d}\epsilon_{k}^{l}}{\mathrm{d}s}=\frac{1}{2N}\operatorname*{Im}\operatorname*{Tr}\left(U_{D}^{*}U(T,t_{l-1})\left[\sum_{n=0}^{\infty}\frac{(\imath\Delta
t)^{n}}{(n+1)!}\operatorname{ad}^{n}_{H}\circ
H_{k}\right]U(t_{l-1},0)\right),$ (2)

where $H=H_{0}+\sum_{k=1}^{M}\epsilon_{l}^{k}H_{k}$ — the constant Hamiltonian
on $[t_{l-1},t_{l}]$, $\operatorname{ad}_{H}\circ
H_{k}=[H,H_{k}]=HH_{k}-H_{k}H$ — the commutator of matrices $H$ and $H_{k}$,
$\operatorname{ad}^{n}_{H}=\operatorname{ad}_{H}\operatorname{ad}^{n-1}_{H}$.
The original D-MORPH can be obtained by letting $\Delta t=0$, i. e. by taking
the first term only in the series (2). As it was shown in [1], method (2)
takes less time to reach the minimum of the objective function and requires a
smaller integration interval $[0,S]$ compared to D-MORPH.

## The new way to obtain corrections

There is one more way to obtain analogous corrections to D-MORPH based on more
accurate and theoretically correct calculation of the derivative of the
objective function $\nicefrac{{\partial J}}{{\partial\epsilon_{k}^{l}}}$.
Similar to the paper [2], the full form of the derivative of the exponential
matrix $\exp(-\imath\Delta tH)$ with respect to $\epsilon_{k}^{l}$ is used,
where the exponential matrix is the evolution operator of a quantum system
with the Hamiltonian $H$ on an interval of length $\Delta t$. This form of the
derivative has been known for a long time in the general theory of Lie groups
and algebras [9, p. 15] and in the case of a quantum system can be written as

$\frac{\partial}{\partial\epsilon_{k}^{l}}\exp(-\imath\Delta
tH)=\exp(-\imath\Delta tH)\frac{1-\exp(-\operatorname{ad}_{-\imath\Delta
tH})}{\operatorname{ad}_{-\imath\Delta tH}}\circ\left(-\imath\Delta
tH_{k}\right)=\\\ =-\imath\Delta t\exp(-\imath\Delta
tH)\sum_{n=0}^{\infty}\frac{(\imath\Delta
t)^{n}}{(n+1)!}\operatorname{ad}^{n}_{H}\circ H_{k}.$

In the author’s previous paper [1], only the term with $n=0$ was used, which
was similar to using the classic formula of the derivative of the scalar
exponential function. It can be easily seen that the error of such
approximation was $O(\Delta t)$ and it could have affected the accuracy of
numerical computations, especially in the case of large steps $\Delta t$.
Therefore, the full form of the derivative is employed in this paper to
potentially make the optimization process quicker and more accurate. After
substituting this form into system (1), a new system of differential equations
is obtained as

$\frac{\mathrm{d}\epsilon_{k}^{l}}{\mathrm{d}s}=\frac{\Delta
t}{2N}\operatorname*{Im}\operatorname*{Tr}\left(U_{D}^{*}U(T,t_{l-1})\left\\{\sum_{n=0}^{\infty}\frac{(\imath\Delta
t)^{n}}{(n+1)!}\operatorname{ad}^{n}_{H}\circ
H_{k}\right\\}U(t_{l-1},0)\right).$ (3)

This system looks similar to (2) obtained earlier in [1], except for the
presence of factor $\Delta t$ and for the fact that the new system more
accurately describes the derivative. It’s expected that method (3) will give
more precise results than method (2), which in turn gives more accurate
results than D-MORPH. Furthermore, it’s expected that the new method (3) will
find near-optimal controls faster than method (2).

## Numerical experiment

In order to confirm an increase in accuracy and speed of method (3) compared
to method (2), only the first two terms of the series were employed ($n=0$ and
$n=1$) for solving a problem in the area of quantum computations —
implementing the controlled NOT [6, p. 21] quantum gate (CNOT) in a quantum
system consisting of two spin-$\nicefrac{{1}}{{2}}$ particles ($N=4$), which
is described by the dimensionless Hamiltonian

$H=\sum_{i=1}^{2}S_{z}^{i}\omega_{i}+\sum_{k=1}^{2}\epsilon_{k}^{l}S_{x}^{k}+C_{x}^{(12)}S_{x}^{1}S_{x}^{2}+C_{y}^{(12)}S_{y}^{1}S_{y}^{2}+C_{z}^{(12)}S_{z}^{1}S_{z}^{2},$

where $\omega_{1}=20$, $\omega_{2}=30$, $C_{x}^{(12)}=110$,
$C_{y}^{(12)}=120$, $C_{z}^{(12)}=130$, $S_{i}^{2}=I\otimes S_{i}$,
$S_{i}^{1}=S_{i}\otimes I$,

$S_{x}=\begin{pmatrix}0&1\\\
1&0\end{pmatrix},S_{y}=\begin{pmatrix}0&-\imath\\\
\imath&0\end{pmatrix},S_{z}=\begin{pmatrix}1&0\\\
0&-1\end{pmatrix},U_{D}=\mathrm{e}^{\frac{\imath\pi}{4}}\begin{pmatrix}1&0&0&0\\\
0&1&0&0\\\ 0&0&0&1\\\ 0&0&1&0\end{pmatrix}.$

Therefore the following two models were compared:

$\frac{\mathrm{d}\epsilon_{k}^{l}}{\mathrm{d}s}=\frac{1}{2N}\operatorname*{Im}\operatorname*{Tr}\left(U_{D}^{*}U(T,t_{l-1})\left\\{H_{k}+\frac{\imath\Delta
t}{2}\left[H,H_{k}\right]\right\\}U(t_{l-1},0)\right),$ (4)
$\frac{\mathrm{d}\epsilon_{k}^{l}}{\mathrm{d}s}=\frac{\Delta
t}{2N}\operatorname*{Im}\operatorname*{Tr}\left(U_{D}^{*}U(T,t_{l-1})\left\\{H_{k}+\frac{\imath\Delta
t}{2}\left[H,H_{k}\right]\right\\}U(t_{l-1},0)\right).$ (5)

These systems were numerically solved with MATLAB’s ode45 method [10] on a
quad-core processor Intel Core i7 2.20 GHz with 12 Gb RAM. The initial control
fields were taken equal to zero everywhere on $[0,T]$ and the system’s
parameters were taken as follows: $T=5,10$; $L=150,300$; $S=5000$. The
numerical integration was stopped when either $S=5000$ was reached or the
objective function $J<10^{-7}$, i. e. when the error in the CNOT gate’s
implementation became less than $10^{-7}$. The comparison results are
summarized in the following table.

| $T=10$ | $T=10$ | $T=5$ | $T=5$
---|---|---|---|---
| $L=300$ | $L=150$ | $L=300$ | $L=150$
Method | (4) | (5) | (4) | (5) | (4) | (5) | (4) | (5)
Final S | 70 | 2089 | 73 | 1097 | 81.1 | 4866.5 | 117.8 | 3532.2
Time, sec. | 348 | 317 | 174.8 | 158.8 | 240.8 | 225.7 | 99 | 94.3
Max step | 0.19 | 5.6 | 0.14 | 2.1 | 0.3 | 19.4 | 0.4 | 13

It can be clearly seen from the table that in comparison with method (4),
method (5) proposed in this paper always reached the desired precision more
quickly but using a larger interval described in the table by the final S
value where the numerical integration was stopped. It’s worth mentioning that
methods (2) and (3) with only one term ($n=0$) were compared to each other as
well, and the new formula performed much worse than the original
D-MORPH method — it required larger computation times and larger intervals to
reach the desired precision than D-MORPH, and worst of all, sometimes this
method didn’t converge at all.

## Conclusion

A new way to obtain corrections of different orders of smallness to the
D-MORPH method for numerical construction of optimal quantum control was
presented in the paper. The corrections obtained turned out to be analogous to
the ones derived earlier by the author [1], except for the presence of a
factor proportional to the time step used to divide the interval of the
system’s evolution and for the fact that the new method more accurately
describes the derivative of the objective function. By solving the problem of
the CNOT gate’s implementation in a quantum system composed of two
spin-$\nicefrac{{1}}{{2}}$ particles, it was shown that the new method could
actually speed up the optimization process compared to both the original
D-MORPH and the previously obtained method [1] — the inclusion of only one
additional correction term allowed the new method to implement the desired
logical quantum operation more quickly. It’s noteworthy that in the case of
comparison between the original method and the new method without any
correction terms ($n=0$), the latter almost always performed much worse and
sometimes didn’t converge to the optimum at all — information about the system
Hamiltonian’s commutators turns out to be crucial for the method to work. If
no such information is available, then the best option is to use the original
D-MORPH method. Thus the use of the theoretically correct form of the
derivative of the exponential map to derive the corrections allows further
improvement to accuracy and performance of the method for finding optimal
quantum control, which should be beneficial when using computers with moderate
performance.

## References

  * [1] Zhdanov K. E. An improvement of D-MORPH method for finding quantum optimal control / K. E. Zhdanov // International research journal. 2016. Vol. 6(48). P. 94–99. DOI: 10.18454/IRJ.2016.48.057 [in Russian]
  * [2] de Fouquieresa P. Second order gradient ascent pulse engineering / P. de Fouquieresa, S. G. Schirmera, S. J. Glaserb, et. al. // Journal of Magnetic Resonance. 2011. Vol. 212(2). P. 412–417.
  * [3] Moore K. W. Search complexity and resource scaling for the quantum optimal control of unitary transformations / K. W. Moore, R. Chakrabarti, G. Riviello, et. al. // Phys. Rev. A. 2011. Vol. 83(1).
  * [4] Moore K. W. Exploring constrained quantum control landscapes / K. W. Moore, H. Rabitz // The Journal of Chemical Physics. 2012. Vol. 137(13).
  * [5] Moore Tibbetts K. Exploring the trade-off between fidelity and time optimal control of quantum unitary transformations / K. Moore Tibbetts, C. Brif, M. D. Grace, et. al. // Phys. Rev. A. 2012. Vol. 86(6).
  * [6] Nielsen M. A. Quantum Computation and Quantum Information: 10th Anniversary Edition / M. A. Nielsen, I. L. Chuang. New York: Cambridge University Press, 2010. 702 p.
  * [7] Riviello G. Searching for quantum optimal controls in the presence of singular critical points / G. Riviello, C. Brif, R. Long, et. al. // Phys. Rev. A. 2014. Vol. 90(1).
  * [8] Riviello G. Searching for quantum optimal controls under sever constraints / G. Riviello, K. Moore Tibbetts, C. Brif, et. al. // Phys. Rev. A. 2015. Vol. 91(4).
  * [9] Rossmann W. Lie Groups: An Introduction Through Linear Groups / W. Rossmann. New York: Oxford University Press, 2006. 265 p.
  * [10] Shampine L. F. The MATLAB ODE Suite / L. F. Shampine, M. W. Reichelt // SIAM Journal on Scientific Computing. 1997. Vol. 18(1).
P. 1–22.


utf8 utf8

# ChatGPT for Arabic Grammatical Error Correction

Sang Yun Kwon1 Gagan Bhatia 1 El Moatez Billah Nagoudi1
Muhammad Abdul-Mageed1,2
1Deep Learning & Natural Language Processing Group, The University of British
Columbia
2Department of Natural Language Processing & Department of Machine Learning,
MBZUAI
{skwon01@student.,gagan30@student.,moatez.nagoudi<EMAIL_ADDRESS>
##  Abstract

Recently, large language models (LLMs) fine-tuned to follow human instruction
have exhibited significant capabilities in various English NLP tasks. However,
their performance in grammatical error correction (GEC) tasks, particularly in
non-English languages, remains significantly unexplored. In this paper, we
delve into abilities of instruction fine-tuned LLMs in Arabic GEC, a task made
complex due to Arabic’s rich morphology. Our findings suggest that various
prompting methods, coupled with (in-context) few-shot learning, demonstrate
considerable effectiveness, with GPT-4 achieving up to $65.49$ F1 score under
expert prompting (approximately $5$ points higher than our established
baseline). This highlights the potential of LLMs in low-resource settings,
offering a viable approach for generating useful synthetic data for model
training. Despite these positive results, we find that instruction fine-tuned
models, regardless of their size, significantly underperform compared to fully
fine-tuned models of significantly smaller sizes. This disparity highlights a
substantial room for improvements for LLMs. Inspired by methods from low-
resource machine translation, we also develop a method exploiting synthetic
data that significantly outperforms previous models on two standard Arabic
benchmarks. Our work sets new SoTA for Arabic GEC, with $72.19\%$ and $73.26$
F1 on the 2014 and 2015 QALB datasets, respectively.

##  1 Introduction

As interest in second language learning continues to grow, ensuring the
accuracy and effectiveness of written language becomes increasingly
significant for pedagogical tools and language evaluation Rothe et al. (2021);
Tarnavskyi et al. (2022). A key component in this respect is grammatical error
correction (GEC), a sub-area of natural language generation (NLG), which
analyzes written text to automatically detect and correct diverse grammatical
errors. Figure 1 shows an instance of GEC from Mohit et al. (2014).

Despite the growing attention to GEC, it is predominantly studied within the
English language. One significant challenge in extending GEC systems to other
languages is the lack of high-quality parallel data and benchmark datasets. In
this work, our focus is on Arabic. Currently, the only available parallel data
and benchmark datasets for Arabic GEC(AGEC) is the Qatar Arabic Language Bank
(QALB) Mohit et al. (2014); Rozovskaya et al. (2015a), highlighting the
complexity of the task. Furthermore, Arabic, a language of complex grammar and
rich morphological features, presents significant challenges to GEC. This
further motivates our focus on Arabic in this work.

Figure 1: An example of an Arabic GEC system showcasing six types of errors:
character replacement, missing word, hamza error, missing punctuation,
additional character, and punctuation confusion.

Non-English settings aside, the field of GEC has witnessed significant
advancements specifically with the emergence of sequence-to-sequence
(seq-2-seq) models Chollampatt and Ng (2018); Gong et al. (2022) and sequence-
to-edit models (seq-2-edit) Awasthi et al. (2019); Omelianchuk et al. (2020)
achieving SoTA results in the CONLL-2014 shared tasks Ng et al. (2014).

Although these models have achieved prominent performance, their efficacy
relies heavily on large amounts of labeled data. Again, this presents
challenges in low-resource scenarios. Recently, scaled up language models, aka
large language models (LLMs) have demonstrated remarkable potential in various
NLP tasks. Their core strength lies in their capacity to generalize across a
wide range of languages and tasks, and in-context learning (ICL), enabling
them to take on various NLP tasks once fed with only few examples (i.e.,
few–shot learning). A key component of this learning process is instruction
fine-tuning, where these models are refined on a collection of tasks
formulated as instructions Chung et al. (2022). This process amplifies the
models’ ability to respond accurately to such directives, reducing the need
for few-shot examples Ouyang et al. (2022); Wei et al. (2022); Sanh et al.
(2021). With their unique features adeptly addressing the challenges of low-
resource NLP scenarios, LLMs have emerged as promising candidates for NLP
tasks in these scenarios. In our current study, we delve into the capabilities
of LLMs, taking ChatGPT as our focus. We examine the effectiveness of various
prompting strategies such as few-shot chain of thought (CoT) prompting Kojima
et al. (2022) and expert prompting Xu et al. (2023). Our research extends the
realm of GEC research by concentrating on the unique challenges posed by
Arabic, a complex and morphologically rich, low-resource language. Drawing
upon the work of Junczys-Dowmunt et al. (2018a), we frame these challenges
within the context of a low-resource MT task. We then carefully conduct a
thorough comparison of the different methodologies employed in addressing GEC
in Arabic. Our key contributions in this paper include:

  1. 1.

We conduct a comprehensive investigation of the potential of LLMs,
particularly focusing on ChatGPT, for tasks involving GEC in Arabic.

  2. 2.

We provide a detailed examination of different prompting methods such as few-
shot CoT and expert prompting, and an exploration into generating synthetic
data with ChatGPT to complement the performance of transformer-based language
models.

  3. 3.

We further carry out in-depth and meaningful comparisons between several
approaches (seq2seq, seq2-dit and instruction fine-tuning LLMs) in AGEC using
the QALB 2014 and 2015 L1 benchmark dataset, allowing us to offer novel
insights as to the utility of these approaches on Arabic.

The rest of this paper is organized as follows: In Section 2, we review the
related work on GEC, with a particular emphasis on Arabic. In Section 3, we
describe available benchmark datasets for Arabic GEC and related evaluation
metrics. In Section 4, we describe our experimental setup; Section 5 outlines
our experiments on LLMs. In Section 6, we introduce our seq-2-seq approach and
Section 7 discusses the sequence-2-edit approach. In 8, we conduct a
comprehensive analysis of error types using the ARETA Belkebir and Habash
(2021). Finally, in Section 9, we discuss our results, and in 10, we conclude
the paper summarizing our contributions and outlining future research
directions in the field of Arabic GEC.

##  2 Related Work

#### Progress in GEC.

Pre-trained Transformer-based models allowed for reframing GEC as a MT task Ng
et al. (2014); Felice et al. (2014); Junczys-Dowmunt et al. (2018b);
Grundkiewicz et al. (2019), leading to SoTA results. Meanwhile, sequence2edit
methods cast the task as text-editing of input into an output Malmi et al.
(2019); Awasthi et al. (2019); Omelianchuk et al. (2020). These methods have
simplified the complexity of model training while enhancing accuracy,
especially in data-scarce scenarios. Furthermore, instruction fine-tuning
Chung et al. (2022) and various prompting techniques, such as the CoT Kojima
et al. (2022), help optimize the performance of LLMs in the context of GEC.
Finally, there is recent work that treats application of LLMs such as ChatGPT
in GEC, demonstrating the effectiveness of these models. Further details
regarding each of these approaches can be found in Appendix A.

#### Arabic GEC.

For Arabic GEC (AGEC), challenges stem from the complexity and morphological
richness of Arabic. Arabic consists of a collection of diverse languages and
dialectal varieties. Modern Standard Arabic (MSA) is a current standard
variety of Arabic that is used in government and pan-arab media as well as
education, alongside numerous regional dialects defined at the country or
regional level Abdul-Mageed et al. (2020). The inherent ambiguity of Arabic at
the orthographic, morphological, syntactic, and semantic levels makes AGEC
particularly challenging. Optional use of diacritics further introduces
orthographic ambiguity Belkebir and Habash (2021), making AGEC even harder.

Despite these hurdles, progress has been made in AGEC. For example, the
QALB-2014 and 2015 shared task Mohit et al. (2014); Rozovskaya et al. (2015b),
released annotated datasets of comments and documents written by native (L1)
and Arabic learner (L2) speakers. More recently,the ZAEBUC corpus Habash and
Palfreyman (2022) a GEC corpus of essays written by first year university
students in Zayed University, UAE. In terms of model development, innovative
approaches have been introduced. Watson et al. (2018) develop the first
character-level seq2seq model that achieved SoTA results on AGEC L1 data.
Solyman et al. (2022, 2021) design a model that utilizes a dynamic linear
combination and EM routing algorithm with a seq2seq Transformer. Convolutional
neural network (CNN) have also been applied to AGEC, using unsupervised noise
injection techniques to generate synthetic parallel data Solyman et al. (2022,
2021, 2023). In spite of this progress, no work has considered the utility of
employing ChatGPT (or any LLM in general) for AGEC. Nor has been significant
work on exploring synthetic data generation, including from LLMs or adopting
more diverse machine learning approaches, been carried out. Our research fills
this existing gap.

##  3 Datasets & Evaluation

###  3.1 Datasets

In this study, we make use of the 2014 Mohit et al. (2014) and 2015 Rozovskaya
et al. (2015b) QALB Shared Task datasets to evaluate the performance of our
various models. Both datasets make use of the QALB corpus, a manually
corrected collection of Arabic texts. These texts include online commentaries
from Aljazeera articles in MSA by L1 speakers, as well as texts produced by L2
learners of Arabic. Both the QALB 2014 and 2015 dataset are split into
training (Train), development (Dev), and test (Test) sets based on their
annotated dates. QALB 2014 consists of $19,411$ sentences, $1,017$ sentences,
and $968$ sentences for the respective Train, Dev, and Test splits. QALB 2015
is an extension of the first 2014 shared task, including L1 commentaries and
L2 texts that cover different genres and error types. For the purposes of our
study, we exclusively utilize the L1 test set (2015), as we focus on sentence-
level AGEC, where L2 test sets are document-level. The parts of the 2015
dataset we use are comprised of $300$ sentences for Train, $154$ sentences for
Dev, and $920$ sentences for the L1 Test set. Statistics of these datasets are
in Table 1.

Dataset | Split | Lines | Words | Err. % | Level | Domain
---|---|---|---|---|---|---
QALB-2014 | Train | 19,411 | 1,021,165 | 30% | Native | Comments
Dev | 1,017 | 53,737 | 31% | Native | Comments
Test | 968 | 51,285 | 32% | Native | Comments
QALB-2015 | Train | 310 | 43,353 | 30% | L2 | Essays
Dev | 154 | 24,742 | 29% | L2 | Essays
Test-L2 | 158 | 22,808 | 27% | L2 | Essays
Test-L1 | 920 | 48,547 | 29% | Native | Comments

Table 1: Dataset descriptions and statistic for train, development (dev) and
test data.

###  3.2 Evaluation Metric

For evaluation, we utilize the overlap-based metric MaxMatch (M2) Dahlmeier
and Ng (2012), which aligns source and hypothesis sentences based on
Levenshtein distance , selecting maximal matching edits, scoring the precision
(P), recall (R), and F1 measure. Moreover, in alignment with recent works on
GEC, we also report the F0.5 score as another important evaluation metric, a
variation of the F1 score that place twice as much weight on precision than on
recall. This reflects a general consensus that precision holds greater
importance than comprehensive error correction in GEC systems.

Normalisation methods. Following the QALB shared task’s evaluation method, we
report system performance across three different categories, targeting
distinct types of mistakes. Namely, we assess the system on normalized text
with (1) Alif/Ya errors removed, (2) text without punctuation, and (3) text
devoid of both Alif/Ya errors and punctuation. Although we primarily focus on
the ’Exact Match’ results for analysis and discussion, scores for most
experiments are provided in our Appendixes. Examples of text under each
setting, along with the comprehensive results, can be found in the Appendix B.

##  4 Baseline and Experimental Setup

Our baseline settings include AraBart Eddine et al. (2022) and AraT5 Nagoudi
et al. (2021), text-to-text transformer-based models specifically tailored for
Arabic tasks. We also evaluate the performance of the mT0 Muennighoff et al.
(2022) and mT5 Xue et al. (2020) variants of the T5 model Raffel et al.
(2020), both of which are configured for multilingual tasks.

For our experiments, we fine-tune each models for 15 epochs. We employ a
learning rate of 5e-5 and a batch size of 32, then picking the best-performing
model on our Dev data before blind-testing on Test.

##  5 LLMs and Prompting Techniques

This section outlines our experiment designed to instruction fine-tune LLMs
and explore different prompting methods for ChatGPT in the context of GEC. We
begin by experimenting with various prompting strategies using ChatGPT,
comparing its performance against smaller LLMs and our listed baselines. We
evaluate the performance of ChatGPT-3.5 Turbo and ChatGPT-4 using the official
API, under two distinct prompting approaches: Few-shot CoT Fang et al. (2023)
and Expert Prompting Xu et al. (2023). We now describe our prompting
strategies.

###  5.1 ChatGPT Prompting

#### Preliminary experiment.

Initially, we experiment with a diverse set of prompt templates to assess
ChatGPT’s capabilities in zero-shot learning as well as two aspects of few-
shot learning: vanilla few-shot and few-shot CoT Fang et al. (2023). We also
experiment with prompts in both English and Arabic. However, we discover that
the responses from these prompt templates contain extraneous explanations and
are disorganized, necessitating substantial preprocessing for compatibility
with the M2 scorer. This problem was particularly notable in the zero-shot and
Arabic prompt setups, which failed to yield output to automatically evaluate.

Figure 2: An illustration of Few-Shot CoT and Expert Prompts applied to
CharGPT for Grammatical Error Correction.

#### Few-shot CoT.

Adopting the few-shot CoT prompt design strategy from Kojima et al. (2022) and
Fang et al. (2023), we implement a two-stage approach. Initially, we engage in
’reasoning extraction’, prompting the language model to formulate an elaborate
reasoning pathway. This is followed by an ’answer extraction’ phase, where the
reasoning text is combined with an answer-specific trigger sentence to form a
comprehensive prompt. These directives include tailored prompts that position
the model as an Arabic GEC tool. In the few-shot CoT setting, we include
labeled instances from the development set in our prompts to implement in-
context learning (ICL), facilitating learning from examples Brown et al.
(2020). This approach involves the use of erroneous sentences, indicated by
<input> SRC </input>, along with their corrected versions, coded by <output>
TGT </output>.

#### Expert prompting.

Xu et al. (2023) introduces a novel strategy, which leverages the expert-like
capabilities of LLMs. This method involves assigning expert personas to LLMs,
providing specific instructions to enhance the relevance and quality of the
generated responses. Following the framework proposed by Xu et al. (2023), we
ensure that our Arabic GEC correction tool exhibits three key characteristics:
being distinguished, informative, and automatic during the ’reasoning
extraction’ stage of our prompt. To achieve this, we curate a distinct and
informative collection of various error types rooted in the dataset,
referencing the taxonomy of the Arabic Learner Corpus Alfaifi and Atwell
(2012). Then we prompt to automate the system by instructing it to operate on
sentence labeled with <input> and <output> tags. Details of both prompts can
be found in Figure 2.

###  5.2 ChatGPT Results.

Table 2 presents the performance of ChatGPT under different prompting
strategies in comparison to the baseline settings. Noticeably, we observe
improvements particularly as we progress from the one-shot to five-shot
configurations under both the few-shot CoT and expert prompting (EP)
strategies. Under the CoT prompt, the F1.0 score for ChatGPT increased from
$53.59$ in the one-shot setting to $62.04$ in the five-shot setting. A
comparable upward trend was also evident for the EP strategy, with the F1.0
score improving from $55.56$ in the one-shot setup to $63.98$ in the five-shot
setup. Furthermore, among all the ChatGPT trials, the three-shot and five-shot
configurations of ChatGPT-4, under the CoT strategy, yield the highest scores.
These configurations achieve F1.0 scores of $63.88$ and $65.49$, respectively.

###  5.3 Instruction-Finetuning LLMs

InstructGPT Ouyang et al. (2022) and studies on ChatGPT 2, highlight the
potential of LLMs to excel in various downstream tasks just by leveraging a
few examples as instructions. Further advancements have been realized by fine-
tuning language models on a compilation of tasks presented as instructions,
enhancing models’ responsiveness, and minimizing the need for few-shot
exemplars Chung et al. (2022).

In this study, we extend the application of instruction-finetuning to AGEC
tasks across a broad range of models that range in size, including LLaMA-7B
Touvron et al. (2023), Vicuna-13B Chiang et al. (2023),
Bactrian-X${}_{\textit{bloom}}$-7B Li et al. (2023),
Bactrian-X${}_{\textit{llama}}$-7B Li et al. (2023).

#### LLM finetuning.

To instruct fine-tune relatively large language models, henceforth simply
LLMs, we first pre-train LLMs on the translated Alpaca dataset 111We translate
Alpaca datasets using NLLB MT model Costa-jussà et al. (2022) to help our
model gain deeper understanding of the Arabic language and its complexities.
Following this, we further fine-tune the models on our GEC dataset, targeting
specifically the task of GEC Taori et al. (2023). Then, we employ well-
structured task instructions and input prompts, enabling the models to take on
GEC tasks. Each model is assigned a task, given an instruction and an input
for output generation. A detailed illustration of the instructions utilized
for the models can be found in Appendix C.

Settings | Models | Exact Match
---|---|---
P | R | F1.0 | F0.5
Baselines | mT0 | $69.16$ | $52.63$ | $59.78$ | $65.07$
mT5 | $68.99$ | $52.22$ | $59.45$ | $64.83$
AraBART | $70.71$ | $60.46$ | $65.18$ | $68.39$
AraT5 | $\mathbf{73.04}$ | $\mathbf{63.09}$ | $\mathbf{67.70}$ | $\mathbf{70.81}$
\+ CoT | ChatGPT (1-shot) | $58.71$ | $49.29$ | $53.59$ | $56.55$
ChatGPT (3-shot) | $64.60$ | $60.37$ | $62.41$ | $63.71$
ChatGPT (5-shot) | $64.70$ | $59.59$ | $62.04$ | $63.61$
\+ EP | ChatGPT (1-shot) | $60.49$ | $51.37$ | $55.56$ | $58.42$
ChatGPT (3-shot) | $65.83$ | $61.41$ | $63.54$ | $64.90$
ChatGPT (5-shot) | $66.53$ | $61.62$ | $63.98$ | $65.49$
\+ CoT | GPT4 (1-shot) $*$ | $-$ | $-$ | $-$ | $-$
GPT4 (3-shot) | $69.31$ | $59.24$ | $63.88$ | $67.03$
GPT4 (5-shot) | $69.46$ | $61.96$ | $65.49$ | $67.82$

Table 2: Performance of ChatGPT under different prompting
strategies.$*$Results for GPT4 1-shot are not included due to the high cost of
producing these results, and a pattern has already been established showing
that scores increase with the number of N-shot examples

#### LLM Results.

As shown in Figure 3, larger models such as Vicuna-13B and models trained on
multilingual data like Bactrian-X${}_{\textit{llama}}$-7B, and
Bactrian-X${}_{\textit{bloom}}$-7B exhibit an overall trend of better
performance, achieving F1 scores of $58.30$, $50.1$, and $52.5$, respectively.
Despite these improvements, it is noteworthy that all these LLMs fall short of
ChatGPT’s performance in the AGEC tasks, reaffirming ChatGPT’s superior
ability in this context.

Figure 3: Performance of LLMs compared to ChatGPT.

##  6 Data Augmentation

Motivated by the significant improvements observed in low-resource GEC tasks
in languages such as German, Russian, and Czech through synthetic data
creation Flachs et al. (2021), and recognizing the recent efforts to develop
synthetic data for AGEC Solyman et al. (2021), we experiment with three
distinctive data augmentation methods, evaluating the efficacy of each method
in complementing performance of seq-2-seq models.

#### ChatGPT as corruptor.

With slight adaptation to our original prompt, we engage ChatGPT as an AI
model with the role of introducing grammatical errors into Arabic text to
generate artificial data. We randomly sample $10,000$ correct sentences from
the original training set and prompt ChatGPT to corrupt these, creating a
parallel dataset. In order to ensure a varied range of error types, we adopt
the taxonomy put forth by the Arabic Learner Corpus Alfaifi and Atwell (2012).

#### Token noising and error adaptation.

Adopting techniques known as token noising Xie et al. (2018) and error
adaptation Junczys-Dowmunt et al. (2018a), we generate artificial data by
introducing random alterations and matching the error rates of the original
benchmark dataset, in table 3. For token noising, random character-level and
word-level changes are introduced into clean texts, creating a parallel
dataset. These changes include random character manipulations, word
separations, space adjustments, Arabic text normalization, and inserting or
removing punctuation. To ensure domain compatibility with the original
benchmark dataset, we use commentaries from the same newspaper domain as our
clean inputs and adjust sentence lengths to align with the benchmark dataset.

| Edit | Add | Merge | Split | Delete | Move | Other
---|---|---|---|---|---|---|---
Train | 55.34% | 32.36% | 5.95% | 3.48% | 2.21% | 0.14% | 0.50%
Dev | 53.51% | 34.24% | 5.97% | 3.67% | 2.03% | 0.08% | 0.49%
Test | 51.94% | 34.73% | 5.89% | 3.48% | 3.32% | 0.15% | 0.49%

Table 3: Error type statistics for training, development, and test Sets.

#### Reverse noising.

We adopt a reverse noising approach Xie et al. (2018), training a reverse
model that converts clean sentences Y into noisy counterparts X. This involves
implementing a standard beam search to create noisy targets $\hat{Y}$ from
clean input sentences Y. Our approach incorporates two types of reverse
models: the first trains both QALB-2014 and 2015 datasets, and the second
makes use of a parallel dataset generated by ’ChatGPT as corruptor’.
Subsequently, we produce two parallel datasets by inputting clean ’in-domain’
and ’out-of-domain’ examples from each reverse model. In this context, the
’in-domain’ dataset refers to news article commentaries, the same as the
original training data, and ’out-of-domain’ refers to any Arabic sentences.

Figure 4: Scores trained on 100k to 1m sentences of training data Figure 5:
Scores of models fine-tuned on $10,000$ parallel sentences from different
sources: original training data, ’ChatGPT as Corruptor’, and reverse noising
on the ChatGPT dataset.

#### Data augmentation evaluation.

To evaluate the efficacy of ChatGPT in generating artificial data, we select
$10,000$ parallel sentences generated through ‘ChatGPT as corruptor’, $10,000$
examples from the parallel dataset from reverse noising on the ChatGPT dataset
as well as $10,000$ parallel sentences from the original training set. We then
further fine-tune each of the configurations on the original training dataset
and the ‘out-of-domain’ reverse noised dataset, aiming to assess whether these
artificially created datasets can replace the gold standard training set.
Figure 5 outlines the results. In our initial exploration, fine-tuning the
AraT5 model exclusively on $10,000$ samples, ChatGPT achieves an F1 of
$67.00$, scoring slightly below the original QALB 2014 training data
($68.45$). Subsequently, when further fine-tuned on the original training set,
our model (F1 score at $68.39$) is on par with the AraT5 model further fine-
tuned on the equivalent-sized gold dataset (F1 score at $68.05$) . This
confirms the utility of ChatGPT for generating synthetic data. Conversely,
when we further fine-tune the model on $10,000$ out-of-domain examples, its
performance drops significantly (F1 =$44.65$). This underscores the importance
of relevant and high-quality synthetic data over randomly generated samples.

We scale our data augmentation experiments comparing the ’token noising’ and
error adaptation’ and ’reverse-noising’. Results, outlined in Figure 4, show
consistent improvement over the baseline. The ’token noising and error
adaptation’ method helps improve the F1 scores, with a range of $68.09$ to
$68.85$, attaining optimal performance with the one million dataset size.
Similarly, the ’reverse noising’ method, yielding scores from $68.44$ to
$68.88$, also reaches its peak performance at the one million datasets. Both
methods exhibit similar performance trends when tested on the QALB-2015
dataset.

##  7 Sequence Tagging Approach

In this section, we detail our methods to adapt the GECToR model Omelianchuk
et al. (2020) for experimenting with the sequence-to-edit approach.

#### Token level transformations.

We first apply token-level transformation to recover the target text by
applying them to the source tokens. ‘Basic-transformations’ are applied to
perform the most common token-level edit operations, such as keeping the
current token unchanged ($KEEP$), deleting current token ($DELETE$), appending
new token t_$1$ next to the current token xi ($APPEND$_ t$1$) or replacing the
current token xi with another token t_$2$ ($REPLACE$_ t$2$). To apply tokens
with more task specific operations we employ ‘g-transformations’ such as the
($MERGE$) tag to merge the current token and the next token into a single one.
Edit space after applying token-level transformations results in KEEP ($725$K
op), DELETE ($13$K op), MERGE ($5.7$K op), APPEND_t$1$ ($75$K op), and
REPLACE_t2 ($201$K op) tags. As some corrections in a sentence depend on
others, applying edit sequences once may not be enough to fully correct the
sentence. To address this issue, GECToR employs an iterative correction
approach from Awasthi et al. (2019). However, in our experiments, we find that
the iterative correction approach does not result in any tangible improvement.
Therefore, we set our iterations to $3$.

#### Preprocessing and fine-tuning.

We start the preprocessing stage by aligning source tokens with target
subsequences, preparing them for token-level transformations. Subsequently, we
fine-tune ARBERT v2 Elmadany et al. (2022) and MARBERT v2Abdul-Mageed et al.
(2021) on the preprocessed data. Then we employ a three step training
procedure: an initial pre-training phase using artificially generated
sentences with errors; a fine-tuning phase that exclusively uses sentences
containing errors; and a final refinement phase that employs a combination of
sentences, both with and without errors.

#### Sequence tagging evaluation.

Outlined in Table 4, ARBERT v2 and MARBERT v2, exhibit high precision , with
ARBERT v2’s three-step training scoring the highest precision at $74.39$.
However, relatively lower recall scores indicate challenges in ability of the
two models to detect errors. The implementation of a three-stage training
approach yielded mixed results: while accuracy improves, recall scores
decrease, leading to a drop in the overall F1 score (by $0.36$ for ARBERT v2
and $1.10$ for MARBERT v2, respectively). Consequently, all models fall behind
the ’seq2seq’ models in performance. However, both ARBERT v2 and MARBERT v2
surpass ’mT0’ and ’mT5’ in terms of F0.5 scores highlighting their abilities
in correcting errors with precision.

Settings | Models | QALB-2014 | QALB-2015
---|---|---|---
P | R | F1.0 | F0.5 | P | R | F1.0 | F0.5
Encoder-Decoder | mT0 | $69.16$ | $52.63$ | $59.78$ | $65.07$ | $67.61$ | $58.41$ | $62.67$ | $65.55$
mT5 | $68.99$ | $52.22$ | $59.45$ | $64.83$ | $67.43$ | $57.30$ | $61.95$ | $65.13$
AraBART | $70.71$ | $60.46$ | $65.18$ | $68.39$ | $67.95$ | $65.62$ | $66.76$ | $67.47$
AraT5 | $73.04$ | $\mathbf{63.09}$ | $\mathbf{67.70}$ | $\mathbf{70.81}$ | $72.41$ | $\mathbf{74.12}$ | $\mathbf{73.26}$ | $\mathbf{72.75}$
Encoder-Only | ARBERTv2 | $73.89$ | $48.33$ | $58.43$ | $66.82$ | $73.10$ | $55.40$ | $63.03$ | $68.70$
ARBERTv2 (3-step) | $\mathbf{74.39}$ | $47.62$ | $58.07$ | $66.87$ | $\mathbf{74.20}$ | $53.80$ | $62.37$ | $68.96$
MARBERTv2 | $73.53$ | $48.21$ | $58.24$ | $66.54$ | $72.90$ | $54.90$ | $62.63$ | $68.41$
MARBERTv2 (3-step) | $74.21$ | $46.45$ | $57.14$ | $66.29$ | $74.00$ | $52.70$ | $61.56$ | $68.46$

Table 4: F1 and F0.5 scores of sequence tagging approach for QALB-2014 and
QALB-2015

##  8 Error Analysis

###  8.1 Error Types.

Using the Automatic Error Type Annotation (ARETA) tool Belkebir and Habash
(2021) we examine the performance of error types of our developed models. In
the following, we briefly describe the different error’s types included in
QALB, as well as the normalization methods used to evaluate the model
performances.

Figure 6: F1 scores for each fine-grained error type on the QALB-2014 test
set. The percentages in parentheses indicate the proportion of each error
type.

Type of errors. We concentrate on seven error types using ARETA: Orthographic,
Morphological, Syntactic, Semantic, Punctuation, and Merge and Split errors.
Table 5 presents examples of each error alongside their translations. Top-
performing systems from each approach, including ARBERT v2 (3-step), GPT-4
(5-shot) + CoT, and AraT5 fully trained on the AGEC dataset Solyman et al.
(2021), are analyzed in correcting these errors.

Error Type |  | Incorrect Sentence | Correct Sentence
---|---|---|---
Orthographic |  |  الفرس . > يرب > الرجل > |  الفرس . > يركب > الرجل >
The man rears the horse. | The man rides the horse.
Punctuations |  |  يركب الفرس . > ، > الرجل > |  الرجل يركب الفرس .>
The man, rides the horse. | The man rides the horse.
Syntax |  |  فرس .> وجد رجلا يركب> |  فرسا .> وجد رجلا يركب>
He found a man riding a hors. | He found a man riding a horse.
Merge |  |  الفرس . > سيركب > غداالرجل > |  الفرس . > سيركب > غدا الرجل >
Tomorrowtheman will ride the horse. |  Tomorrow the man will ride the horse.
Splits |  |  الفرس . > ير كب > غدا الرجل > |  الفرس . > يركب > غدا الرجل >
The man ri des the horse. | The man rides the horse.
Semantic |  |  ظهر الفرس . > في > الرجل يجلس> |  ظهر الفرس . > على > الرجل يجلس>
| The man is sitting in the horse’s back. | The man is sitting on the horse’s back.
Morphological |  |  الفرس . > ركب > غدا الرجل > |  الفرس . > سيركب > غدا الرجل >
| Tomorrow the man rode the horse. | Tomorrow the man will ride the horse.

Table 5: Examples of Merge, Morphological, Orthographic, Punctuation,
Semantic, Split, and Syntactic errors, along with their corresponding
corrections and English translations.

Figure 6 illustrates the performance of each model under various error type
categories. AraT5, fully trained on the AGEC dataset, surpasses all other
models across all error categories. In particular, it excels in handling
Orthographic (ORTH) errors, Morphological (MORPH) errors, and Punctuation
(PUNCT) errors, consistently achieving over $65$ F1 score. However, it is
worth observing that all models encounter challenges with Semantic (SEM) and
Syntactic (SYN) errors. These disparate outcomes underscore the significance
of selecting the appropriate model based on the error types prevalent in a
specific dataset.

###  8.2 Normalization Methods.

In addition to the ‘Exact Match’ score, we also analyze system performance
under different normalization methods. Looking at Table 6, setting under ‘No
punctuation’ leads to an increase in scores across all models, underscoring
the models’ limitations in handling punctuation errors. Another noteworthy
observation is the drop in F1 scores when Alif/Ya errors are removed,
illustrating the models’ dependency on Alif/Ya features in making correction.

Test Set |  Models | Exact Match | No Alif / Ya Errors | No Punctuation | No Punctuation and Alif / Ya Errors
---|---|---|---|---|---
P | R | F1.0 | F0.5 | P | R | F1.0 | F0.5 | P | R | F1.0 | F0.5 | P | R | F1.0 | F0.5
QALB-2014 | Solyman et al. (2021) | 79.06 | 65.79 | 71.82 | 75.99 | - | - | - | - | - | - | - | - | - | - | - | -
AraT5 (11m) | 77.12 | 67.85 | 72.19 | 75.07 | 62.04 | 52.69 | 56.99 | 59.91 | 86.57 | 82.70 | 84.59 | 85.77 | 79.32 | 67.19 | 72.75 | 76.56
GPT4 (5-shot) | 69.46 | 61.96 | 65.49 | 67.82 | 58.44 | 51.47 | 54.73 | 56.90 | 74.59 | 78.15 | 76.33 | 75.28 | 60.06 | 65.75 | 62.78 | 61.12
ARBERT V2 (3-step) | 74.39 | 47.62 | 58.07 | 66.87 | 65.25 | 41.58 | 50.79 | 58.58 | 77.00 | 46.00 | 57.60 | 67.85 | 56.99 | 28.90 | 38.35 | 47.71
Mohit et al. (2014) | 73.34 | 63.23 | 67.91 | 71.07 | 64.05 | 50.86 | 56.7 | 60.89 | 76.99 | 49.91 | 60.56 | 69.45 | 76.99 | 49.91 | 60.56 | 69.45
QALB-2015 | Solyman et al. (2021) | 80.23 | 63.59 | 70.91 | 76.24 | - | - | - | - | - | - | - | - | - | - | - | -
AraT5 (11m) | 72.41 | 74.12 | 73.26 | 72.75 | 55.95 | 43.53 | 48.96 | 52.93 | 85.46 | 72.56 | 78.48 | 82.53 | 75.23 | 52.56 | 61.88 | 69.26
ChatGPT (3-shot) + EP | 52.33 | 47.57 | 49.83 | 54.10 | 37.93 | 39.97 | 38.92 | 32.95 | 53.38 | 56.63 | 54.96 | 54.00 | 33.33 | 46.77 | 38.92 | 35.36
ARBERT V2 (3-step) | 74.20 | 53.80 | 62.37 | 68.97 | 57.30 | 38.50 | 46.06 | 52.20 | 66.70 | 61.50 | 63.99 | 65.59 | 71.24 | 38.50 | 49.99 | 60.88
Rozovskaya et al. (2015a) | 88.85 | 61.76 | 72.87 | 81.68 | 84.25 | 43.29 | 57.19 | 70.84 | 85.8 | 77.98 | 81.7 | 84.11 | 80.12 | 58.24 | 67.45 | 74.52

Table 6: Results on the Test sets of QALB-2014, QALB-2015 under Normalization
Methods

##  9 Discussion

LLMs and ChatGPT. ChatGPT demonstrates a remarkable ability to outperform
other fully trained models by learning from only a few examples, particularly
five-shot under both few-shot CoT and EP prompting strategies. Nevertheless,
ChatGPT’s performance lags behind AraT5 and AraBART, suggesting potential
areas for improvements in prompting strategies to fully exploit ChatGPT
models. Larger models, such as Vicuna-13B, as well as those trained on
multilingual datasets like Bactrian-X${}_{\textit{llama}}$-7B and
Bactrian-X${}_{\textit{bloom}}$-7B, tend to perform better, with F1 scores of
$58.30$, $50.10$, and $52.50$ respectively. However, these models fail to
match ChatGPT’s performance in AGEC tasks. This reinforces ChatGPT’s
superiority in this domain.

Data augmentation techniques. Data augmentation results underscore the
potential of synthetic data, generated by ChatGPT, in enhancing model
performance. Moreover, our findings reveal that not just the quantity, but the
quality of synthetic data, is crucial for achieving optimal performance. The
relative underperformance of models further trained with ’out of domain’ data
examples emphasizes this conclusion. Furthermore, our results on scaled
datasets indicate a trade-off between precision and recall. As the size of the
dataset increases, precision improves, while recall drops. This trend is
apparent across all dataset sizes.

Sequence tagging approach. These models exhibit high precision scores and
relatively low recall scores, suggesting their strengths in making corrections
rather than detecting errors. This trend can be explained by the absence of
G-transformations. For instance, in the case of English GECToR models,
g-transformations enable a variety of changes, such as case alterations and
grammatical transformations. However, crafting effective G-transformations for
Arabic, a language with rich morphological features, poses significant
challenges, limiting the model’s ability to effectively detect errors.

##  10 Conclusion

This paper provided a detailed exploration of the potential of LLMs, with a
particular emphasis on ChatGPT for AGEC. Our study highlights ChatGPT’s
promising capabilities, in low-resource scenarios, as evidenced by its
competitive performance on few-shot setttings. However, AraT5 and AraBART
still exhibit superior results across various settings and error types. Our
findings also emphasize the role of high-quality synthetic data, reinforcing
that both quantity and quality matter in achieving optimal performance.
Moreover, our work unveils trade-offs between precision and recall in relation
to dataset size and throughout all the other experimental settings. These
insight, again, could inform future strategies for improving GEC systems.
Although our exploration of ChatGPT’s performance on Arabic GEC tasks
showcases encouraging results, it also uncovers areas ripe for further study.
Notably, there remains significant room for improvement in GEC systems,
particularly within the context of low-resource languages. Future research may
include refining prompting strategies, enhancing synthetic data generation
techniques, and addressing the complexities and rich morphological features
inherent in the Arabic language.

## References

  * Abdul-Mageed et al. (2021) Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2021.  ARBERT & MARBERT: Deep bidirectional transformers for Arabic.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 7088–7105, Online. Association for Computational Linguistics.
  * Abdul-Mageed et al. (2020) Muhammad Abdul-Mageed, Chiyu Zhang, Houda Bouamor, and Nizar Habash. 2020.  NADI 2020: The first nuanced Arabic dialect identification shared task.  In _Proceedings of the Fifth Arabic Natural Language Processing Workshop_ , pages 97–110, Barcelona, Spain (Online). Association for Computational Linguistics.
  * Alfaifi and Atwell (2012) Abdullah Alfaifi and Eric Atwell. 2012.  Arabic learner corpora (alc): A taxonomy of coding errors.  In _The 8th International Computing Conference in Arabic_.
  * Awasthi et al. (2019) Abhijeet Awasthi, Sunita Sarawagi, Rasna Goyal, Sabyasachi Ghosh, and Vihari Piratla. 2019.  Parallel iterative edit models for local sequence transduction.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4260–4270, Hong Kong, China. Association for Computational Linguistics.
  * Belkebir and Habash (2021) Riadh Belkebir and Nizar Habash. 2021.  Automatic error type annotation for arabic.  _arXiv preprint arXiv:2109.08068_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Bryant et al. (2022) Christopher Bryant, Zheng Yuan, Muhammad Reza Qorib, Hannan Cao, Hwee Tou Ng, and Ted Briscoe. 2022.  Grammatical error correction: A survey of the state of the art.  _arXiv preprint arXiv:2211.05166_.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  * Chollampatt and Ng (2018) Shamil Chollampatt and Hwee Tou Ng. 2018.  A multilayer convolutional encoder-decoder neural network for grammatical error correction.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 32(1).
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Costa-jussà et al. (2022) Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022.  No language left behind: Scaling human-centered machine translation.  _arXiv preprint arXiv:2207.04672_.
  * Dahlmeier and Ng (2012) Daniel Dahlmeier and Hwee Tou Ng. 2012.  Better evaluation for grammatical error correction.  In _Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 568–572, Montréal, Canada. Association for Computational Linguistics.
  * Eddine et al. (2022) Moussa Kamal Eddine, Nadi Tomeh, Nizar Habash, Joseph Le Roux, and Michalis Vazirgiannis. 2022.  Arabart: a pretrained arabic sequence-to-sequence model for abstractive summarization.  _arXiv preprint arXiv:2203.10945_.
  * Elmadany et al. (2022) AbdelRahim Elmadany, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2022.  Orca: A challenging benchmark for arabic language understanding.  _arXiv preprint arXiv:2212.10758_.
  * Fang et al. (2023) Tao Fang, Shu Yang, Kaixin Lan, Derek F Wong, Jinpeng Hu, Lidia S Chao, and Yue Zhang. 2023.  Is chatgpt a highly fluent grammatical error correction system? a comprehensive evaluation.  _arXiv preprint arXiv:2304.01746_.
  * Felice et al. (2014) Mariano Felice, Zheng Yuan, Øistein E. Andersen, Helen Yannakoudakis, and Ekaterina Kochmar. 2014.  Grammatical error correction using hybrid systems and type filtering.  In _Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task_ , pages 15–24, Baltimore, Maryland. Association for Computational Linguistics.
  * Flachs et al. (2021) Simon Flachs, Felix Stahlberg, and Shankar Kumar. 2021.  Data strategies for low-resource grammatical error correction.  In _Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 117–122, Online. Association for Computational Linguistics.
  * Gong et al. (2022) Peiyuan Gong, Xuebo Liu, Heyan Huang, and Min Zhang. 2022.  Revisiting grammatical error correction evaluation and beyond.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 6891–6902, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Grundkiewicz et al. (2019) Roman Grundkiewicz, Marcin Junczys-Dowmunt, and Kenneth Heafield. 2019.  Neural grammatical error correction systems with unsupervised pre-training on synthetic data.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 252–263, Florence, Italy. Association for Computational Linguistics.
  * Habash and Palfreyman (2022) Nizar Habash and David Palfreyman. 2022.  ZAEBUC: An annotated Arabic-English bilingual writer corpus.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 79–88, Marseille, France. European Language Resources Association.
  * Junczys-Dowmunt et al. (2018a) Marcin Junczys-Dowmunt, Roman Grundkiewicz, Shubha Guha, and Kenneth Heafield. 2018a.  Approaching neural grammatical error correction as a low-resource machine translation task.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 595–606, New Orleans, Louisiana. Association for Computational Linguistics.
  * Junczys-Dowmunt et al. (2018b) Marcin Junczys-Dowmunt, Roman Grundkiewicz, Shubha Guha, and Kenneth Heafield. 2018b.  Approaching neural grammatical error correction as a low-resource machine translation task.  _arXiv preprint arXiv:1804.05940_.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  _arXiv preprint arXiv:2205.11916_.
  * Li et al. (2023) Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. 2023.  Bactrian-x: A multilingual replicable instruction-following model.  https://github.com/MBZUAI-nlp/Bactrian-X.
  * Malmi et al. (2019) Eric Malmi, Sebastian Krause, Sascha Rothe, Daniil Mirylenka, and Aliaksei Severyn. 2019.  Encode, tag, realize: High-precision text editing.  _arXiv preprint arXiv:1909.01187_.
  * Mohit et al. (2014) Behrang Mohit, Alla Rozovskaya, Nizar Habash, Wajdi Zaghouani, and Ossama Obeid. 2014.  The first qalb shared task on automatic text correction for arabic.  In _Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP)_ , pages 39–47.
  * Muennighoff et al. (2022) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2022\.  Crosslingual generalization through multitask finetuning.
  * Nagoudi et al. (2021) El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. 2021.  Arat5: Text-to-text transformers for arabic language generation.  _arXiv preprint arXiv:2109.12068_.
  * Ng et al. (2014) Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014.  The CoNLL-2014 shared task on grammatical error correction.  In _Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task_ , pages 1–14, Baltimore, Maryland. Association for Computational Linguistics.
  * Ng et al. (2013) Hwee Tou Ng, Siew Mei Wu, Yuanbin Wu, Christian Hadiwinoto, and Joel Tetreault. 2013\.  The CoNLL-2013 shared task on grammatical error correction.  In _Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task_ , pages 1–12, Sofia, Bulgaria. Association for Computational Linguistics.
  * Omelianchuk et al. (2020) Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020.  Gector–grammatical error correction: tag, not rewrite.  _arXiv preprint arXiv:2005.12592_.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551.
  * Rothe et al. (2021) Sascha Rothe, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. 2021.  A simple recipe for multilingual grammatical error correction.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 702–707, Online. Association for Computational Linguistics.
  * Rozovskaya et al. (2015a) Alla Rozovskaya, Houda Bouamor, Nizar Habash, Wajdi Zaghouani, Ossama Obeid, and Behrang Mohit. 2015a.  The second qalb shared task on automatic text correction for arabic.  In _Proceedings of the Second workshop on Arabic natural language processing_ , pages 26–35.
  * Rozovskaya et al. (2015b) Alla Rozovskaya, Houda Bouamor, Nizar Habash, Wajdi Zaghouani, Ossama Obeid, and Behrang Mohit. 2015b.  The second QALB shared task on automatic text correction for Arabic.  In _Proceedings of the Second Workshop on Arabic Natural Language Processing_ , pages 26–35, Beijing, China. Association for Computational Linguistics.
  * Sanh et al. (2021) Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021.  Multitask prompted training enables zero-shot task generalization.  _arXiv preprint arXiv:2110.08207_.
  * Solyman et al. (2022) Aiman Solyman, Zhenyu Wang, Qian Tao, Arafat Abdulgader Mohammed Elhag, Rui Zhang, and Zeinab Mahmoud. 2022.  Automatic arabic grammatical error correction based on expectation-maximization routing and target-bidirectional agreement.  _Knowledge-Based Systems_ , 241:108180.
  * Solyman et al. (2023) Aiman Solyman, Marco Zappatore, Wang Zhenyu, Zeinab Mahmoud, Ali Alfatemi, Ashraf Osman Ibrahim, and Lubna Abdelkareim Gabralla. 2023.  Optimizing the impact of data augmentation for low-resource grammatical error correction.  _Journal of King Saud University - Computer and Information Sciences_ , 35(6):101572.
  * Solyman et al. (2021) Aiman Solyman, Wang Zhenyu, Tao Qian, Arafat Abdulgader Mohammed Elhag, Muhammad Toseef, and Zeinab Aleibeid. 2021.  Synthetic data with neural machine translation for automatic correction in arabic grammar.  _Egyptian Informatics Journal_ , 22(3):303–315.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca.
  * Tarnavskyi et al. (2022) Maksym Tarnavskyi, Artem Chernodub, and Kostiantyn Omelianchuk. 2022.  Ensembling and knowledge distilling of large sequence taggers for grammatical error correction.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3842–3852, Dublin, Ireland. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Watson et al. (2018) Daniel Watson, Nasser Zalmout, and Nizar Habash. 2018.  Utilizing character and word embeddings for text normalization with sequence-to-sequence models.  _arXiv preprint arXiv:1809.01534_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_.
  * Wu et al. (2023) Haoran Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, and Michael Lyu. 2023.  Chatgpt or grammarly? evaluating chatgpt on grammatical error correction benchmark.  _arXiv preprint arXiv:2303.13648_.
  * Xie et al. (2018) Ziang Xie, Guillaume Genthial, Stanley Xie, Andrew Ng, and Dan Jurafsky. 2018.  Noising and denoising natural language: Diverse backtranslation for grammar correction.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 619–628, New Orleans, Louisiana. Association for Computational Linguistics.
  * Xu et al. (2023) Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, Yongdong Zhang, and Zhendong Mao. 2023.  Expertprompting: Instructing large language models to be distinguished experts.  _arXiv preprint arXiv:2305.14688_.
  * Xue et al. (2020) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020.  mt5: A massively multilingual pre-trained text-to-text transformer.  _arXiv preprint arXiv:2010.11934_.

##  Appendix A Related Works

#### Sequence to Sequence Approach.

Transformer-based Language Models (LMs) have been integral to advancements in
GEC. These models have substantially transformed the perception of GEC,
reframing it as a MT task. In this framework, erroneous sentences are
considered as the source language, and the corrected versions as the target
language. This perspective, which has led to SOTA results in the CONLL 2013
and 2014 shared tasks Bryant et al. (2022); Ng et al. (2013, 2014),
reinterprets GEC as a low-resource or mid-resource MT task. Building on this
paradigm, Junczys-Dowmunt et al. (2018a) successfully adopted techniques from
low-resource NMT and Statistical Machine Translation (SMT)-based GEC methods,
leading to considerable improvements on both the CONLL and JFLEG datasets.

#### Sequence Tagging Approach.

Sequence tagging methods, another successful route to GEC, are showcased by
models like GECToR Omelianchuk et al. (2020), LaserTagger Malmi et al. (2019),
and the Parallel Iterative Edit (PIE) model Awasthi et al. (2019). By viewing
GEC as a text editing task, these models make edits predictions instead of
tokens, label sequences rather than generating them, and iteratively refine
predictions to tackle dependencies. Employing a limited set of output tags,
these models apply edit operations on the input sequence, reconstructing the
output. This technique not only capably mirrors a significant chunk of the
target training data, but it also diminishes the vocabulary size and
establishes the output length as the source text’s word count. Consequently,
it curtails the number of training examples necessary for model accuracy,
which is particularly beneficial in settings with sparse human-labeled data
Awasthi et al. (2019).

#### Instruction Finetuning.

LLMs have revolutionized NLP, their vast data-learning capability enabling
diverse task generalizations. Key to their enhancement has been instructional
finetuning, which fortifies the model’s directive response and mitigates the
need for few-shot examples Ouyang et al. (2022); Wei et al. (2022); Sanh et
al. (2021). A novel approach, Chain of Thought (CoT), directs LLMs through a
series of natural language reasoning, generating superior outputs. Proven
beneficial in ’Let’s think step by step’ prompts Wei et al. (2022), CoT has
harnessed LLMs for multi-task cognitive tasks Kojima et al. (2022) and
achieved SOTA results in complex system-2 tasks like arithmetic and symbolic
reasoning.

#### ChatGPT.

In the specific realm of GEC, LLMs have demonstrated its potential. Fang et
al. (2023) applied zero-shot and few-shot CoT settings using in-context
learning for ChatGPT Brown et al. (2020) and evaluated its performance on
three document-level English GEC test sets. Similarly, Wu et al. (2023)
carried out an empirical study to assess the effectiveness of ChatGPT in GEC,
in the CoNLL2014 benchmark dataset.

##  Appendix B Normalisation Table

Normalisation Method | Example
---|---
Normal |  نحن معشر العرب نعرف إلا الشماتة ، ولكن يجب أن ندرس هذه الحالة ونحن المخرج منها من الاقتصاد الإسلامي.
No Alif/Ya |  نحن معشر العرب نعرف الا الشماتة ، ولكن يجب ان ندرس هذه الحالة ونحن المخرج منها من الاقتصاد الاسلامي.
No Punct |  نحن معشر العرب نعرف إلا الشماتة ولكن يجب أن ندرس هذه الحالة ونحن المخرج منها من الاقتصاد الإسلامي
No Alif/Ya & Punct |  نحن معشر العرب نعرف الا الشماتة ولكن يجب ان ندرس هذه الحالة ونحن المخرج منها من الاقتصاد الاسلامي

Table 7: Examples of normalized text: with Alif/Ya errors removed, punctuation
removed, and both Alif/Ya errors and punctuation removed.

##  Appendix C Instructions for LLaMa

Translated in English | Instructions Samples
---|---
Correct all written errors in the following text except for a thousand, ya and punctuation: |  قم بتصحيح كل الأخطاء الكتابية في النص التالي ماعدا المتعلقة بالألف والياء وعلامات الترقيم:
Please verify spelling, grammatical scrutiny, and correct all errors in the following sentence, except for punctuation: |  الرجاء التدقيق الإملائي والتدقيق النحوي و تصحيح كل الأخطاء في الجملة التالية إلا الخاصة بعلامات الترقيم:
Explore the grammatical errors and repair them except for punctuation marks such as a comma, or a question marks, etc: |  قم بإستكشاف أخطاء التدقيق الإملائي وإصلاحها ماعدا المتعلقة بعلامات الترقيم كالفاصلة أو علامة إستفهام ، إلخ:
Can you correct all errors in the following text except those related to punctuation such as commas, periods, etc: |  هل يمكنك كل الأخطاء الموجودة في النص التالي ماعدا المتعلقة بعلامات الترقيم كالفاصلة ، النقطة ، إلخ :
Can you fix all spelling and grammatical errors, except for the mistakes of the "Alif" and "Ya": |  هل يمكنك إصلاح كل الأخطاء الإملائية والنحوية ماعدا الأخطاء الخاصة بالألف والياء:
Please explore the grammatical spelling errors and repair them all, except for the mistakes related to the "Alif" and "Ya" |  الرجاء إستكشاف أخطاء التدقيق الإملائي النحوي وإصلاحها كلها ماعدا الأخطاء المتعلقة بالألف والياء:
Correct all the written errors in the following text except for the "Alif" and "Ya": |  قم بتصحيح كل الأخطاء الكتابية في النص التالي ماعدا المتعلقة بالألف والياء:
Please correct all errors in the following sentence: |  الرجاء تصحيح كل الأخطاء الموجودة في الجملة التالية:

Table 8: Different instructions used for instruction fine-tuning.

Fine-tune Instruction Example
---
فيما يلي أمر توجيه يصف مهمة مرتبطة بمدخل لتزويد
النص بسياق اضافي. يرجى صياغة ردود مناسبة لتحقق
الطلب بطريقة مناسبة و دقيقة.
### : الأمر/ التوجيه
قم بتصحيح كل الأخطاء الكتابية في النص التالي:
### : المدخل
الفرس . > يرب > الرجل >
### : الرد
الفرس . > يركب > الرجل >

Table 9: Modified data format for the LLaMA instruction fine-tuning step.

##  Appendix D ALC Error Type Taxonomy

Class | Sub-class | Description
---|---|---
Orthographic | OH | Hamza error
OT | Confusion in Ha and Ta Mutadarrifatin
OA | Confusuion in Alif and Ya Mutadarrifatin
OW | Confusion in Alif Fariqa
ON | Confusion Between Nun and Tanwin
OS | Shortening the long vowels
OG | Lengthening the short vowels
OC | Wrong order of word characters
OR | Replacement in word character(s)
OD | Additional character(s)
OM | Missing character(s)
OO | Other orthographic errors
Morphological | MI | Word inflection
MT | Verb tense
MO | Other morphological errors
XF | Definiteness
XG | Gender
XN | Number
XT | Unnecessary word
XM | Missing word
XO | Other syntactic errors
Semantic | SW | Word selection error
SF | Fasl wa wasl (confusion in conjunction use/non-use)
SO | Other semantic errors
Punctuation | PC | Punctuation confusion
PT | Unnecessary punctuation
PM | Missing punctuation
PO | Other errors in punctuation
Merge | MG | Words are merged
Split | SP | Words are split

Table 10:  The ALC error type taxonomy extended with merge and split classes

##  Appendix E ARETA Results

CLASS | GECToR_ARBERT | five-shot_2014_expertprompt | five-shot_2014-chatgpt4 | AraT5 (11M) | COUNT
---|---|---|---|---|---
OH | 73.73 | 89.80 | 92.91 | 87.34 | 4902
OT | 76.59 | 94.12 | 95.58 | 90.84 | 708
OA | 78.63 | 84.66 | 88.93 | 87.35 | 275
OW | 38.57 | 80.79 | 86.96 | 83.70 | 107
ON | 0.00 | 0.00 | 0.00 | 0.00 | 0
OG | 48.00 | 55.74 | 63.64 | 90.32 | 34
OC | 21.43 | 28.57 | 53.66 | 87.18 | 22
OR | 38.24 | 53.02 | 65.96 | 77.10 | 528
OD | 33.76 | 51.89 | 59.60 | 73.07 | 321
OM | 41.80 | 44.53 | 57.35 | 86.44 | 393
OO | 0.00 | 0.00 | 0.00 | 0.00 | 0
MI | 11.02 | 13.25 | 20.53 | 75.00 | 83
MT | 0.00 | 7.84 | 11.43 | 62.50 | 7
XC | 32.95 | 46.10 | 50.78 | 88.35 | 526
XF | 6.06 | 17.98 | 23.81 | 76.92 | 29
XG | 37.10 | 19.57 | 31.35 | 89.47 | 79
XN | 25.19 | 25.79 | 31.25 | 88.12 | 108
XT | 3.95 | 3.78 | 5.48 | 2.48 | 66
XM | 2.04 | 4.14 | 6.38 | 1.07 | 26
XO | 0.00 | 0.00 | 0.00 | 0.00 | 0
SW | 50.51 | 21.25 | 33.38 | 8.29 | 219
SF | 0.00 | 6.67 | 3.45 | 57.14 | 3
PC | 60.89 | 56.25 | 47.59 | 74.98 | 713
PT | 29.62 | 29.58 | 21.40 | 57.42 | 480
PM | 55.24 | 54.21 | 52.09 | 67.08 | 5599
MG | 25.05 | 75.96 | 79.70 | 64.80 | 434
SP | 42.27 | 90.93 | 91.61 | 86.70 | 805
micro avg | 55.67 | 60.05 | 64.51 | 57.28 | 16467
macro avg | 30.84 | 39.13 | 43.51 | 61.62 | 16467
weighted avg | 56.98 | 66.96 | 68.24 | 76.35 | 16467

Table 11:  Analysis of Error Type performances on the QALB-2014 Test set.

##  Appendix F Dev Results

Settings | Models | Exact Match | No Alif / Ya Errors | No Punctuation | No Puncation and Alif / Ya Errors
---|---|---|---|---|---
P | R | F1.0 | F0.5 | P | R | F1.0 | F0.5 | P | R | F1.0 | F0.5 | P | R | F1.0 | F0.5
Encoder-Only | ARBERTv2 | 73.30 | 47.85 | 57.90 | 66.25 | 65.60 | 44.20 | 52.81 | 59.81 | 72.38 | 48.75 | 58.26 | 65.98 | 57.40 | 33.90 | 42.63 | 50.41
ARBERTv2 3 Stage | 74.65 | 46.70 | 57.46 | 66.67 | 65.00 | 41.20 | 50.43 | 58.27 | 75.50 | 44.50 | 56.00 | 66.27 | 55.70 | 27.50 | 36.82 | 46.22
MARBERTv2 | 72.95 | 47.65 | 57.65 | 65.95 | 64.60 | 43.20 | 51.78 | 58.78 | 73.72 | 44.16 | 55.23 | 65.02 | 56.80 | 34.20 | 42.69 | 50.17
MARBERTv2 3 Stage | 74.55 | 45.75 | 56.70 | 66.21 | 65.10 | 41.30 | 50.54 | 58.37 | 75.41 | 45.52 | 56.77 | 66.66 | 56.00 | 29.20 | 38.38 | 47.31
Decoder-Only | LLama 7B Original | 58.20 | 32.50 | 41.71 | 50.25 | 35.50 | 16.70 | 22.71 | 28.98 | 19.60 | 54.30 | 28.80 | 22.47 | 65.10 | 32.00 | 42.91 | 53.94
Alpaca 7B | 42.20 | 31.20 | 35.88 | 39.42 | 42.20 | 33.40 | 37.29 | 40.09 | 82.20 | 62.20 | 70.81 | 77.23 | 62.20 | 39.50 | 48.32 | 55.79
Vicuna 13B | 63.90 | 51.00 | 56.73 | 60.82 | 51.40 | 39.30 | 44.54 | 48.42 | 83.90 | 73.90 | 78.58 | 81.69 | 68.50 | 49.00 | 57.13 | 63.45
bactrian-x-bloom-7b1-lora | 60.80 | 43.80 | 50.92 | 56.42 | 53.70 | 41.00 | 46.50 | 50.57 | 79.40 | 63.00 | 70.26 | 75.47 | 62.00 | 51.00 | 55.96 | 59.44
bactrian-x-llama-7b-lora | 58.60 | 41.40 | 48.52 | 54.10 | 51.00 | 38.10 | 43.62 | 47.77 | 77.00 | 59.20 | 66.94 | 72.63 | 58.60 | 48.10 | 52.83 | 56.15
Encoder Decoder Models | mT0 | 69.35 | 54.29 | 60.90 | 65.70 | 57.45 | 42.50 | 48.86 | 53.67 | 82.35 | 75.34 | 78.69 | 80.85 | 70.20 | 50.30 | 58.61 | 65.05
mT5 | 69.00 | 53.20 | 60.08 | 65.13 | 56.70 | 39.50 | 46.56 | 52.16 | 81.00 | 70.00 | 75.10 | 78.53 | 68.00 | 48.00 | 56.28 | 62.77
AraBART | 72.00 | 61.50 | 66.34 | 69.62 | 60.00 | 49.70 | 54.37 | 57.61 | 85.00 | 78.50 | 81.62 | 83.62 | 74.00 | 60.50 | 66.57 | 70.84
AraT5 | 74.50 | 64.50 | 69.14 | 72.26 | 63.50 | 52.70 | 57.60 | 61.00 | 88.00 | 84.50 | 86.21 | 87.28 | 81.50 | 69.50 | 75.02 | 78.78

Table 12: Dev Set results on the QALB-2014 benchmark dataset.