# Evaluation of really good grammatical error correction

Robert Östling1, Katarina Gillholm2, Murathan Kurfalı3, Marie Mattson,
Mats Wirén5
1Department of Linguistics / Stockholm University /<EMAIL_ADDRESS>
2Department of Linguistics / Stockholm University /<EMAIL_ADDRESS>
3Psychology Department / Stockholm University /<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
5Department of Linguistics / Stockholm University /<EMAIL_ADDRESS>

###### Abstract

Although rarely stated, in practice, Grammatical Error Correction (GEC)
encompasses various models with distinct objectives, ranging from grammatical
error detection to improving fluency. Traditional evaluation methods fail to
fully capture the full range of system capabilities and objectives. Reference-
based evaluations suffer from limitations in capturing the wide variety of
possible correction and the biases introduced during reference creation and is
prone to favor fixing local errors over overall text improvement. The
emergence of large language models (LLMs) has further highlighted the
shortcomings of these evaluation strategies, emphasizing the need for a
paradigm shift in evaluation methodology. In the current study, we perform a
comprehensive evaluation of various GEC systems using a recently published
dataset of Swedish learner texts. The evaluation is performed using
established evaluation metrics as well as human judges. We find that GPT-3 in
a few-shot setting by far outperforms previous grammatical error correction
systems for Swedish, a language comprising only 0.11% of its training data. We
also found that current evaluation methods contain undesirable biases that a
human evaluation is able to reveal. We suggest using human post-editing of GEC
system outputs to analyze the amount of change required to reach native-level
human performance on the task, and provide a dataset annotated with human
post-edits and assessments of grammaticality, fluency and meaning preservation
of GEC system outputs.

##  1 Introduction

Grammatical Error Correction (GEC) is typically used in an extended sense of
correcting language at multiple levels, including spelling errors, grammatical
errors, word choice and idiom usage.

In the literature on evaluating GEC systems, one is rarely explicit about the
purpose of the system. Following Sakaguchi et al. (2016), we see two somewhat
different objectives:

  1. 1.

Error detection and correction, where grammaticality has priority over
fluency. The goal is to point out individual language errors, which could
ideally be fixed one by one, resulting in an acceptable text that is as close
as possible to the original.

  2. 2.

General text improvement, where fluency is on equal footing with
grammaticality. The goal is to produce a text which is as close as possible to
what a highly proficient writer would have produced, assuming a perfect
understanding of the intended message of the original text.

The distinction between the two objectives is less clear for writers at high
proficiency levels, where changing an occasional spelling or grammar mistake
typically results in a high-quality text. For a less proficient writer, a text
may contain so many overlapping problems that it is difficult to identify
local changes that together result in a high-quality text. If a GEC system is
allowed to work directly at the level of general text improvement, its task
may become significantly simpler.

The choice of objective has practical implications for how to evaluate the
result. Traditional methods for GEC evaluation are reference-based, where
either the GEC system output is compared to a human-created reference (e.g.
Napoles et al., 2015), or the sets of edit operations produced by the GEC
system is compared to those needed to transform the original text to the human
reference (Bryant et al., 2017).

One important problem with reference-based evaluations is that there is
typically a large and varied set of possible ways to express the same
information. It is generally infeasible to approximate the full set of
possibilities, although providing multiple references is a common approach in
the machine translation community to alleviate this problem (e.g. Qin and
Specia, 2015). Results are also highly dependent on the way the references
were created. Freitag et al. (2020) show that biases due to “translationese”
effects in the creation of references negatively affect the accuracy of the
resulting evaluations, where interference from the source language may affect
the translation to become less idiomatic in the target language. They obtained
higher agreement between the automatic reference-based evaluations and human
judgments by first asking human annotators to maximally paraphrase the
reference sentences, to encourage diversity among the multiple references.

The references used for GEC evaluations suffer from the same bias, and often
annotators are explicitly instructed to stay as close to the original text as
possible (Volodina et al., 2019, Section 6.1). We are aware of no GEC
evaluation data which, in the style of Freitag et al. (2020), aims for a high
amount of diversity in the references. This has the effect of biasing existing
automatic evaluations against systems that perform paraphrasing rather than
conservatively fixing individual errors.

In the related field of text summarization, Goyal et al. (2022) found that
automatic evaluation metrics severely underestimate the performance of large
language models, further strengthening our suspicion that such powerful models
necessitate a paradigm shift in evaluation methodology. In this work, we
perform a comprehensive manual analysis of the output of multiple GEC systems,
and point towards analysis of human post-edits as the most promising way of
evaluating really good GEC systems.

##  2 Related work

###  2.1 Reference-free evaluation metrics

Yoshimura et al. (2020), building on earlier work by Asano et al. (2017),
propose a reference-free metric named SOME that learns a weighting of
grammatically, fluency, and semantic similarity scores obtained from three
separate models. Interestingly, they find that tuning these weights on a
dataset of human judgements results in 98% of the weight being put on the
fluency score computed as the difference between language model cross-entropy
for the system output and original text. This result aligns well with the
argument of Sakaguchi et al. (2016) that fluency is what GEC system ought to
aim for.

Islam and Magnani (2021) go one step further and dispose of the grammaticality
and semantic similarity models, and simple use language model scores combined
with a filter based on string similarity measures to reject “corrections” that
look too different from the original.111The paper somewhat unusually refers to
the string similarity measures as “syntactic similarity”. System-level Pearson
correlations with human scores are very high for all these systems: from about
$0.88$ (Yoshimura et al., 2020) to $0.98$ (Asano et al., 2017). However,
metrics that rely heavily on language model scores do not handle semantic
changes well. As Islam and Magnani (2021) point out, the SOME metric assigns a
very positive score when “He is going school.” is corrected into “He He He He
He He.”

###  2.2 Human evaluation

Grundkiewicz et al. (2015) performed a comprehensive ranking-based human
evaluation of then-current GEC systems, and compared the human rankings to a
number of automatic metrics. In their study, the edit distance-based MaxMatch
(M2) score (Dahlmeier and Ng, 2012) with a bias towards precision
($\beta=0.18$) achieved the highest correlation with the human rankings. Pure
machine translation metrics (BLEU, METEOR) were found to have _negative_
correlations with human rankings. Napoles et al. (2015) also performed a
ranking-based human evaluation, and obtained very similar results to
Grundkiewicz et al. (2015). However, in this case their proposed GLEU metric
achieved a higher correlation with the human rankings than did the M2 score.
In both studies, correlations were relatively modest, with Spearman $\rho$ and
Pearson $r$ in the order of 0.7–0.75 (Grundkiewicz et al., 2015) and 0.55
(Napoles et al., 2015) for the highest-correlated metrics. Náplava et al.
(2022) performed a similar human evaluation of more recent Czech GEC systems,
and found the agreement between human and GEC rankings to be very high, with
Pearson’s $r$ in the range 0.95–0.98 for GLEU, M2, I-measure and their Czech
adaptation of ERRANT (Felice et al., 2016; Bryant et al., 2017).

Rather than ranking system outputs akin to machine translation evaluation,
Yoshimura et al. (2020) applied a range of different GEC systems to the same
set of sentences and obtained human absolute scores in the dimensions of
grammaticality, fluency, and meaning preservation. This allows GEC systems to
be compared using each dimension individually or in combination.

###  2.3 Co-evolution of systems and metrics

As Yoshimura et al. (2020) demonstrated, it is possible to obtain a metric
with an extremely strong correlation (up to $\rho=0.98$) to human ratings, by
optimizing the parameters of the metric with respect to human ratings.
However, their best result was obtained by relying almost exclusively on the
fluency score. This indicates that their data does not contain enough examples
of GEC-“corrected” sentences that are scored high by a language model, but
suffer from problems like a low degree of meaning preservation. Since their
data is based on the outputs of existing GEC systems, it inherits their biases
towards certain types of mistakes. Most of these systems have been developed
against metrics such as GLEU and ERRANT, who tend to reward a conservative
approach where precision is prioritized over recall and more substantial
rewriting is discouraged. A GEC system optimized for high scores according to
GLEU, ERRANT, or similar metrics, is thus less likely to suggest major changes
that (when incorrect) may significantly alter the meaning of a text. The main
dimension along which the performance of such systems vary is to what extent
they can find and correct simple spelling, grammar or word choice errors – the
presence of which correlates strongly with a poor language model score.

We hypothesize that the co-evolution of GEC systems and their evaluation
metrics has resulted in reinforcing the bias towards certain types of
properties, namely a conservative approach which avoids paraphrasing.
Traditionally this has not been much of an issue, since we did not have
particularly good models for paraphrasing non-standard text. With the advent
of large language models that excel at this task, we argue that it is time to
break this circle of GEC system development and metric development.

###  2.4 Swedish GEC

We now briefly review published GEC systems for Swedish, focusing on general-
coverage methods that perform automated correction. We do not cover methods
specializing in specific error types, like spelling or collocations, or those
that are not able to automatically suggest corrections.

Granska (Domeij et al., 2000) is a mostly rule-based system for grammatical
error detection and correction, which has later been combined with a
probabilistic model (Bigert and Knutsson, 2002) that uses a language model to
score variants of the input sentence. Another contemporary rule-based system
based on constraint grammar was developed by Birn (2000). More recently,
Nyberg (2022) implemented Swedish versions of the following two methods.
First, the model of Bryant and Briscoe (2018), which is based on generating
variants of the input sentence and using a language model to choose the
highest-scoring one. Second, using a neural machine translation model trained
on artificially corupted data, generated in a fashion similar to that of
Grundkiewicz et al. (2019).

##  3 Purpose and aims

Most previous work on GEC evaluation has been performed using relatively
limited and conservative systems, and we see a need to extend this line of
work to systems based on large language models (LLMs) that are able to perform
more substantial corrections than previous methods. Given the problems pointed
out above with both reference-based and reference-free automated evaluation
metrics, we think it is important to consider manual evaluation methods in
addition to automated ones. Finally, since LLM-based systems are approaching
human-level performance, we also want to include text versions created by
humans in the evaluation, on equal footing with the automatic GEC systems in
order to ensure a fair comparison.

Our main contributions in this work are the following:

  * •

We evaluate a set of GEC systems, belonging to diverse paradigms ranging from
rule-based systems to large language models. In addition to automatic GEC
systems, we also include in the evaluation paraphrases from humans following
two different guidelines.

  * •

We investigate how different evaluation methods, automatic and manual, compare
across this range of different GEC paradigms, and point out relative strengths
and weaknesses of each paradigm.

  * •

We use human post-edits of GEC system outputs and human paraphrases, both to
evaluate the systems and to quantify the differences between the final
versions of each text after correction/paraphrasing and post-editing.

  * •

We make several new resources publicly available: Human evaluations and post-
edits of GEC system outputs, a novel annotation tool that was used to produce
the above, and a baseline GEC system for Swedish.222Our code and data is
available at https://github.com/robertostling/gec-evaluation

##  4 GEC systems

In this work we compare a total of five Swedish GEC systems:

  1. 1.

Granska: the web API version of the rule-based system Granska (Domeij et al.,
2000). We always accept its top suggestion for changes, but multiple
suggestions that change the same span are rejected.

  2. 2.

Nyberg MT: the MT-based method of Nyberg (2022, Section 3.3), training a
neural machine translation system to translate artificially corrupted text
back into its original form.

  3. 3.

Nyberg LM: the LM-based method of Nyberg (2022, Section 3.4), using a language
model to iteratively score local edits suggested by a heuristic procedure,
until no further changes sufficiently improve the score.

  4. 4.

MT: Similar to Nyberg MT, but with more data, a modified method for
introducing synthetic errors and a different architecture. Details can be
found in Kurfalı and Östling (2023).

  5. 5.

GPT-3: OpenAI’s (text-davinci-002) model (Brown et al., 2020) through their
public API. We use a two-shot prompt, with authentic student sentences taken
from the CrossCheck
corpus333https://www.csc.kth.se/tcs/projects/xcheck/korpus.html and manually
corrected by us. The prompt is entirely in Swedish, and is identical for all
processed sentences.

We also consider the dummy baseline Uncorrected, which simply leaves the text
unchanged, and the following three human-corrected versions:

  1. 1.

Human minimal: human-normalized sentences from the SweLL project (Volodina et
al., 2019). Annotators were asked to perform minimal edits in order to produce
a grammatically correct sentence while trying to preserve the meaning; for
normalization guidelines, see Rudebeck et al. (2021). The annotators had
access to the full context for each sentence.

  2. 2.

Human fluent: human-normalized sentences produced by having a native Swedish
annotator, different from the other annotators in the project, edit the Human
minimal sentences to achieve native-like fluency while staying as close as
possible to the original.

  3. 3.

Human free: human-normalized sentences produced by having a native Swedish
annotator, different from the other annotators in the project, edit the Human
minimal sentences to achieve native-like fluency while being encouraged to
change the sentence as much as needed to achieve the most idiomatic way of
expressing the given meaning.

##  5 Data and annotation

We use Swedish data from the SweLL project (Volodina et al., 2019), which
consists of 502 student texts collected from different levels of L2 Swedish
education. The texts are annotated with an approximate CEFR level, and have
been manually normalized by minimally editing them into a grammatically
correct version. We use the sentence segmentations and the division into a
test and a development set from Nyberg (2022). For the systems Nyberg MT and
Nyberg LM, we report evaluation results from Nyberg (2022).

Two independent annotators, both co-authors of this paper and native Swedish
speakers, were tasked with performing the following procedure for each
corrected sentence in two pilot datasets, containing ten sentences each:

  1. 1.

In a text box, read the system’s output and if necessary modify it to reach
the level of a native writer, considering both fluency and grammaticality. The
annotators are instructed to perform the minimum amount of editing to reach
this goal.

  2. 2.

When the (possibly) edited system output is submitted, the existing human-
normalized reference from the SweLL data is shown (Human minimal), and the
annotator is asked to confirm whether the meaning of the edited sentence
matches the reference. If the annotator thinks otherwise, the human reference
is hidden again and the tool returns to step 1.

  3. 3.

When the edited system output is accepted, the annotator is shown the student
sentence, the non-edited system output, and the SweLL reference. Then the
annotator chooses a score on a 4-level Likert scale (or “other”) for the three
dimensions of grammaticality, fluency and meaning preservation. We follow
Yoshimura et al. (2020) for the definition of the scales for each of these
three dimensions. Our annotation tool and full guidelines are publictly
available.444https://github.com/robertostling/gec-
evaluation/tree/main/annotator

| Gramm. | Fluency | Meaning
---|---|---|---
Round 1 | 0.45 | 0.69 | 0.51
Round 2 | 0.84 | 0.81 | 0.71
Table 1: Quadratically weighted kappa (QWK) between the two annotators during
the two-round pilot phase, for grammaticality, fluency and meaning
preservation.

Sentences were randomized from a pool containing the outputs of all systems
under evaluation, and a custom-made annotation tool was used for the task.
After discussions between the annotators and after the two pilot datasets, the
level of agreement was high enough (QWK in the range 0.71–0.84) to allow one
annotator to continue annotating the full data. Details are presented in
tab:iaa. The full dataset was subsampled from the development set of Nyberg
(2022) and contains 64 sentences each from CEFR proficiency levels A, B and C,
for a total of 192 sentences. Each sentence has been processed by three GEC
systems and two human paraphrasers, for a total of $192\times 5=960$ output
sentences with post-edits and scores.

As the final result, we have for each sentence produced by a GEC system: (a)
scores for grammaticality, fluency, and meaning preservation; (b) a post-
edited version of the output with the minimal edits required to obtain maximum
scores on grammaticality, fluency and meaning preservation.

In order to achieve comparability with previous work on Swedish GEC, we
adapted the sentence-level train/test split of Nyberg (2022), and continued
performing all analysis on the sentence level. The only instance where a wider
context is used, is for the minimal normalization from the SweLL project data.
These normalized versions are used as references, but since GEC systems have
access to less context than the reference was based on, models that are able
to take longer contexts into account are put at a disadvantage during
evaluation. In several cases, it is even impossible to correctly and
unambiguously interpret the sentence without further context. Only the human
corrections, which are directly or indirectly based on the full context, do
not suffer from this problem. We have manually inspected all cases of
“moderate” or “substantial” differences in meaning for the annotations of the
best-performing GEC system’s (GPT-3) output, and found that 8 out of 28 (29%)
such sentences require further context. While this is a methodological problem
to be addressed in future work, we see that the impact of this problem is
limited.

##  6 Results

###  6.1 Automatic evaluation metrics

| CEFR level
---|---
System | All | A | B | C
Uncorrected | 0.44 | 0.29 | 0.17 | 0.53
Granska | 0.47 | 0.35 | 0.24 | 0.55
Nyberg MT | 0.51 | 0.42 | 0.30 | 0.58
Nyberg LM | 0.52 | 0.42 | 0.32 | 0.58
MT | 0.57 | 0.48 | 0.38 | 0.63
GPT-3 | 0.63 | 0.60 | 0.52 | 0.65
Human minimal | 1.0 | 1.0 | 1.0 | 1.0
Table 2: Reference based evaluation: GLEU scores on the test set of Nyberg (2022). | CEFR level
---|---
System | All | A | B | C
Uncorrected | 0 | 0 | 0 | 0
Granska | 0.03 | 0.08 | 0.11 | -0.01
MT | 0.51 | 0.57 | 0.68 | 0.43
GPT-3 | 0.69 | 0.70 | 0.83 | 0.65
Human minimal | 0.68 | 0.67 | 0.77 | 0.65
Table 3: Reference-free evaluation: normalized scribendi scores on the test
set of Nyberg (2022).

tab:nyberg_gleu shows the performance of each system using the reference-based
GLEU metric, while tab:nyberg_scribendi contains the corresponding evaluation
using the reference-free Scribendi score (Islam and Magnani, 2021).555Since
figures for Nyberg MT and Nyberg LM are taken from Nyberg (2022), which only
reports GLEU, they are missing from tab:nyberg_scribendi. Both metrics yield
the same ranking of the systems: GPT-3 scores best, followed by the NMT
systems, followed in turn by the rule-based system. However, the relative
differences between the systems differ considerably between the metrics. In
particular, for the Scribendi score (tab:nyberg_scribendi) we see a very sharp
divide between the neural and the non-neural systems. For all different
levels, GPT-3 in fact scores higher than the human reference, even though its
output contains a substantial amount of errors (as shown in the human
evaluation, see tab:human_evaluations). This is not very surprising, since the
Scribendi score mainly represents the number of sentences where the Swedish
GPT-SW3 model (Ekgren et al., 2022) assigns a higher score to the system
output than to the original sentence, and the GPT-3 output was obtained from
the same family of language models.

###  6.2 Human evaluation

| CEFR level
---|---
System | All | A | B | C
Grammiticality
Granska | 3.0 | 3.1 | 2.5 | 3.3
MT | 3.3 | 3.4 | 3.0 | 3.5
GPT-3 | 3.7 | 3.8 | 3.6 | 3.8
Human fluent | 3.9 | 3.9 | 3.8 | 3.9
Human free | 3.9 | 3.9 | 3.9 | 3.8
Fluency
Granska | 2.8 | 3.0 | 2.3 | 3.2
MT | 3.1 | 3.2 | 2.7 | 3.3
GPT-3 | 3.6 | 3.7 | 3.4 | 3.7
Human fluent | 3.8 | 3.8 | 3.7 | 3.8
Human free | 3.8 | 3.8 | 3.9 | 3.8
Meaning preservation
Granska | 3.5 | 3.5 | 3.2 | 3.7
MT | 3.4 | 3.5 | 3.1 | 3.6
GPT-3 | 3.4 | 3.6 | 3.1 | 3.6
Human fluent | 3.9 | 4.0 | 3.9 | 3.8
Human free | 3.8 | 3.8 | 3.8 | 3.8
Normalized Levenshtein distance (NLD)
Granska | 0.126 | 0.119 | 0.180 | 0.079
MT | 0.113 | 0.095 | 0.158 | 0.087
GPT-3 | 0.076 | 0.068 | 0.112 | 0.050
Human fluent | 0.034 | 0.034 | 0.045 | 0.022
Human free | 0.029 | 0.030 | 0.034 | 0.025
Table 4: Human evaluation: mean score per system and assessment dimension. Higher is better for all assessments (range: 1–4), while lower is better for NLD.  System | Identical | Minor | Moderate | Substantial | Other
---|---|---|---|---|---
Granska | 125 | 34 | 11 | 13 | 9
MT | 122 | 35 | 19 | 13 | 3
GPT-3 | 126 | 36 | 14 | 14 | 2
Human fluent | 176 | 11 | 3 | 1 | 1
Human free | 160 | 26 | 5 | 0 | 1
Table 5: Human evaluation: actual distribution of meaning preservation scores.
The mean over each row in this table is summarized in the _All_ column under
_Meaning preservation_ in tab:human_evaluations.

The result of the human evaluation is summarized in tab:human_evaluations. As
for grammaticality and fluency, the ranking of the GEC systems is identical to
that of the automatic metrics. In both cases, GPT-3 performs at or near human
levels. The MT-based system follows, at a considerable distance, and the rule-
based system scores last. The trend is identical across CEFR proficiency
levels. For meaning preservation, the situation is different. Here, there are
no major differences between systems, all of them consistently perform below
human levels. The gap to the human paraphrases is highest for the B level
sentences, which contain the most errors and are generally the most difficult
to correct. This is presented in more detail in tab:meaning_preservation,
where the frequency of each individual score is given. While the human
corrections nearly always are classified as having no or minor differences,
all automatic systems have significant numbers of moderate and substantial
differences. Even given the fact that about 30% of these divergences are due
to insufficient context (see sec:data), the difference is large enough to
indicate that all GEC systems have problems producing corrections with
adequate semantics.

###  6.3 A tree of corrections

Figure 1: Multidimensional scaling view of all text versions, using normalized
Levenshtein distance. The resulting graph is a tree, with its root at the
“Original” node that represents the original text of the students. Edges
represent transformations made by either computers (solid black lines) or
humans (all other line types). Unlabeled leaf nodes represent human post-edits
to their parent nodes in order to achieve full grammaticality, fluency and
meaning preservation. Each human annotator is represented by a different
collor.

In this project, we have produced a total of ten different versions of each
sentence (three GEC systems and two humans, each with a post-edit), in
addition to the original and its minimal correction from the SweLL project
data. We visualize this by computing the normalized character-level
Levenshtein distance between each pair among the twelve versions of each
sentence, then using multidimensional scaling (Kruskal, 1964) as implemented
by Pedregosa et al. (2011).

In fig:mds, we show the result as a tree starting at the original student text
(“original”), through its immediate transformations by each of the GEC systems
(solid lines) and the minimal human correction (“grammatical”), followed by
the human rewrites for fluency (“fluent” and “free”) to the post-edited
versions (dashed lines). This figure complements the bottom part of
tab:human_evaluations, which gives the distances of the post-editing edges.

It is clear that GPT-3 by far performs the most extensive changes to the text
among the GEC systems. Still, the amount of post-editing required is much
higher than the human-corrected versions. To some extent this can be explained
by the fact that the “fluent” and “free” human corrections indirectly have
access to a wider context through the SweLL “grammatical” correction, but as
was shown in sec:data the size of this effect is limited.

We can also see in fig:mds that although the post-edits (leaf nodes) converge
somewhat, they still produce rather different versions. According to the human
annotator, all of the leaf nodes represent perfect corrections and none of
them should be penalized in an evaluation. From the figure, we get an
impression of the space of acceptable versions of the text in relation to the
space of unacceptable versions.

##  7 Discussion

We have compared a diverse set of GEC systems (rule-based, MT-based, LLM-
based, human), using a diverse set of metrics (reference-based, reference-
free, human scoring, human post-editing). This allows us to make some general
observations on the problem of GEC evaluation.

###  7.1 Metrics

The reference-free metric (tab:nyberg_scribendi) shows that the neural methods
(MT, GPT-3) achieve very high scores, with GPT-3 scoring above or on par with
the minimal human correction. The rule-based Granska system, however, achieves
scores comparable to the baseline of leaving the text uncorrected. By
comparing with the human evaluation (tab:human_evaluations), we see
considerable differences. To begin with, the human evaluation demonstrates
that all GEC systems are clearly below human-level preformance. We also see
that while MT is clearly better than Granska with respect to grammaticality,
fluency and post-edit distance, the gap is by no means as large as suggested
by the Scribendi score.

As seen in tab:nyberg_gleu, the reference-based GLEU suffers from ceiling
effects, in particular for data from the most proficient (level C) student
group. The GLEU scores are nearly identical (0.63 and 0.65) for the MT and
GPT-3 systems, but in terms of grammaticality, fluency and post-edit distance
there is a clear difference.

We argue that post-edit distance, when affordable, is perhaps the fairest
single-dimensional GEC evaluation metric. In this work we quantify the edit
distance using normalized character-level Levenshtein distance, but other edit
distances that better model moved text segments may be more appropriate.

###  7.2 Methodological issues

An important lesson for future work concerns the importance of producing test
sets with sufficiently long context, preferably whole documents. This would
allow models with long context windows to demonstrate their full potential,
and give a fair comparison to humans and to other computational models.

We also note that our choice of relying on the minimal corrections from the
SweLL data as gold standard is sometimes problematic, since multiple
corrections with different semantics can be plausible. In our work, we used
these minimal corrections as a basis for the other (“fluent” and “free”) human
corrections, which has the effect of reducing the diversity among the human
corrections. If multiple human corrections from the original text were to be
performed, we would also recommend annotating which cases are truly ambiguous
even to a human. In addition, it would be helpful to include annotations of
the grammaticality and fluency of the original sentence, for reference.

###  7.3 Summary and future work

In conclusion, we show that with the advent of large language models, Swedish
GEC has made enormous progress compared to early work. One of these models,
GPT-3, produces corrections with human-like grammaticality and fluency.
However, in the critical aspect of semantic accuracy, we see little improvment
compared to other types of models.

For evaluating GEC systems, we demonstrate that different types of automatic
evaluation metrics display different biases with respect to different types of
GEC systems. Reference-free metrics favor neural systems, even over human
corrections, while reference-based metrics struggle to differentiate systems
at high proficiency levels.

In this work, we used simple Normalized Levenshtein distance to quantify the
differences between post-edited corrections. For future work, we believe that
a more thorough analysis of these differences would provide valuable insights
into the weaknesses remaining even in very strong GEC systems. This could be
done manually, or in some cases automatically similar to Felice et al. (2016).

Given the strong and rapidly improving ability of LLMs to handle extended
contexts, we also see a need to perform future evaluations on longer segments
of texts, including entire documents. However, working at the document level
requires considerable adaptations of most existing evaluation methods, which
would be another interesting direction of future work.

## References

  * Asano et al. (2017) Hiroki Asano, Tomoya Mizumoto, and Kentaro Inui. 2017.  Reference-based metrics can be replaced with reference-less metrics in evaluating grammatical error correction systems.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 343–348, Taipei, Taiwan. Asian Federation of Natural Language Processing.
  * Bigert and Knutsson (2002) Johnny Bigert and Ola Knutsson. 2002.  Robust error detection: A hybrid approach combining unsupervised error detection and linguistic knowledge.  In _Proc. 2nd Workshop Robust Methods in Analysis of Natural language Data (ROMAND’02)_ , pages 10–19, Frascati, Italy.
  * Birn (2000) Juhani Birn. 2000.  Detecting grammar errors with lingsoft’s Swedish grammar checker.  In _Proceedings of the 12th Nordic Conference of Computational Linguistics (NODALIDA 1999)_ , pages 28–40, Trondheim, Norway. Department of Linguistics, Norwegian University of Science and Technology, Norway.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Bryant and Briscoe (2018) Christopher Bryant and Ted Briscoe. 2018.  Language model based grammatical error correction without annotated training data.  In _Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 247–253, New Orleans, Louisiana. Association for Computational Linguistics.
  * Bryant et al. (2017) Christopher Bryant, Mariano Felice, and Ted Briscoe. 2017.  Automatic annotation and evaluation of error types for grammatical error correction.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 793–805, Vancouver, Canada. Association for Computational Linguistics.
  * Dahlmeier and Ng (2012) Daniel Dahlmeier and Hwee Tou Ng. 2012.  Better evaluation for grammatical error correction.  In _Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 568–572, Montréal, Canada. Association for Computational Linguistics.
  * Domeij et al. (2000) Rickard Domeij, Ola Knutsson, Johan Carlberger, and Viggo Kann. 2000.  Granska–an efficient hybrid system for Swedish grammar checking.  In _Proceedings of the 12th Nordic Conference of Computational Linguistics (NODALIDA 1999)_ , pages 49–56, Trondheim, Norway. Department of Linguistics, Norwegian University of Science and Technology, Norway.
  * Ekgren et al. (2022) Ariel Ekgren, Amaru Cuba Gyllensten, Evangelia Gogoulou, Alice Heiman, Severine Verlinden, Joey Öhman, Fredrik Carlsson, and Magnus Sahlgren. 2022.  Lessons learned from GPT-SW3: Building the first large-scale generative language model for Swedish.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 3509–3518, Marseille, France. European Language Resources Association.
  * Felice et al. (2016) Mariano Felice, Christopher Bryant, and Ted Briscoe. 2016.  Automatic extraction of learner errors in ESL sentences using linguistically enhanced alignments.  In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_ , pages 825–835, Osaka, Japan. The COLING 2016 Organizing Committee.
  * Freitag et al. (2020) Markus Freitag, David Grangier, and Isaac Caswell. 2020.  BLEU might be guilty but references are not innocent.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 61–71, Online. Association for Computational Linguistics.
  * Goyal et al. (2022) Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022.  News summarization and evaluation in the era of gpt-3.
  * Grundkiewicz et al. (2015) Roman Grundkiewicz, Marcin Junczys-Dowmunt, and Edward Gillian. 2015.  Human evaluation of grammatical error correction systems.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 461–470, Lisbon, Portugal. Association for Computational Linguistics.
  * Grundkiewicz et al. (2019) Roman Grundkiewicz, Marcin Junczys-Dowmunt, and Kenneth Heafield. 2019.  Neural grammatical error correction systems with unsupervised pre-training on synthetic data.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 252–263, Florence, Italy. Association for Computational Linguistics.
  * Islam and Magnani (2021) Md Asadul Islam and Enrico Magnani. 2021.  Is this the end of the gold standard? a straightforward reference-less grammatical error correction metric.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3009–3015, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kruskal (1964) Joseph Bernard Kruskal. 1964.  Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis.  _Psychometrika_ , 29(1):1–27.
  * Kurfalı and Östling (2023) Murathan Kurfalı and Robert Östling. 2023.  A distantly supervised grammatical error detection/correction system for Swedish.  In _Proceedings of the 12th Workshop on NLP for Computer Assisted Language Learning_ , pages 35–39, Tórshavn, Faroe Islands. LiU Electronic Press.
  * Napoles et al. (2015) Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015.  Ground truth for grammatical error correction metrics.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 588–593, Beijing, China. Association for Computational Linguistics.
  * Nyberg (2022) Martina Nyberg. 2022.  Grammatical error correction for learners of swedish as a second language.  Master’s thesis, Uppsala University, Department of Linguistics and Philology.
  * Náplava et al. (2022) Jakub Náplava, Milan Straka, Jana Straková, and Alexandr Rosen. 2022.  Czech Grammar Error Correction with a Large and Diverse Corpus.  _Transactions of the Association for Computational Linguistics_ , 10:452–467.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830.
  * Qin and Specia (2015) Ying Qin and Lucia Specia. 2015.  Truly exploring multiple references for machine translation evaluation.  In _Proceedings of the 18th Annual Conference of the European Association for Machine Translation_ , Antalya, Turkey. European Association for Machine Translation.
  * Rudebeck et al. (2021) Lisa Rudebeck, Gunlög Sundberg, and Mats Wirén. 2021.  Swell normalization guidelines.  Technical Report GU-ISS-2021-03, Department of Swedish, University of Gothenburg, Gothenburg.
  * Sakaguchi et al. (2016) Keisuke Sakaguchi, Courtney Napoles, Matt Post, and Joel Tetreault. 2016.  Reassessing the goals of grammatical error correction: Fluency instead of grammaticality.  _Transactions of the Association for Computational Linguistics_ , 4:169–182.
  * Volodina et al. (2019) Elena Volodina, Lena Granstedt, Arild Matsson, Beáta Megyesi, Ildikó Pilán, Julia Prentice, Dan Rosén, Lisa Rudebeck, Carl-Johan Schenström, Gunlög Sundberg, and Mats Wirén. 2019.  The swell language learner corpus: From design to annotation.  _Northern European Journal of Language Technology (NEJLT)_ , 6:67–104.
  * Yoshimura et al. (2020) Ryoma Yoshimura, Masahiro Kaneko, Tomoyuki Kajiwara, and Mamoru Komachi. 2020.  SOME: Reference-less sub-metrics optimized for manual evaluations of grammatical error correction.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6516–6522, Barcelona, Spain (Online). International Committee on Computational Linguistics.


# Interlayer ferromagnetism and insulator-metal transition in element-doped
CrI3 thin films

Shiyang Sun‡ College of Physics and Hebei Advanced Thin Film Laboratory, Hebei
Normal University, Shijiazhuang, Hebei 050024, China  Xuyan Chen‡ School of
Gifted Young, University of Science and Technology of China, Hefei, Anhui
230026, China  Xuqi Li‡ College of Physics and Hebei Advanced Thin Film
Laboratory, Hebei Normal University, Shijiazhuang, Hebei 050024, China  Huihui
Zhang  College of Physics and Hebei Advanced Thin Film Laboratory, Hebei
Normal University, Shijiazhuang, Hebei 050024, China  Haidan Sang  College of
Physics and Hebei Advanced Thin Film Laboratory, Hebei Normal University,
Shijiazhuang, Hebei 050024, China  Shifei Qi<EMAIL_ADDRESS>College of
Physics and Hebei Advanced Thin Film Laboratory, Hebei Normal University,
Shijiazhuang, Hebei 050024, China  International Center for Quantum Design of
Functional Materials, CAS Key Laboratory of Strongly-Coupled Quantum Matter
Physics, and Department of Physics, University of Science and Technology of
China, Hefei, Anhui 230026, China  Zhenhua Qiao<EMAIL_ADDRESS>International Center for Quantum Design of Functional Materials, CAS Key
Laboratory of Strongly-Coupled Quantum Matter Physics, and Department of
Physics, University of Science and Technology of China, Hefei, Anhui 230026,
China

###### Abstract

The exploration of magnetism in two-dimensional layered materials has
attracted extensive research interest. For the monoclinic phase CrI3 with
interlayer antiferromagnetism, finding a static and robust way of realizing
the intrinsic interlayer ferromagnetic coupling is desirable. In this Letter,
we study the electronic structure and magnetic properties of the nonmagnetic
element (e.g., O, S, Se, N, P, As and C) doped bi- and triple-layer CrI3
systems via first-principles calculations. Our results demonstrate that O, P,
S, As, and Se doped CrI3 bilayer can realize interlayer ferromagnetism.
Further analysis shows that the interlayer ferromagnetic coupling in the doped
few-layer CrI3 is closely related to the formation of localized spin-polarized
state. This finding indicates that insulated interlayer ferromagnetism can be
realized at high doping concentration (larger than 8.33$\%$). When the doping
concentration is less than 8.33$\%$, but larger than 2.08$\%$, an insulator-
metal phase transition can occur since the localized spin-polarized states
percolate to form contiguous grids in few-layer CrI3.

Introduction—. Two-dimensional(2D) magnetic semiconductors have attracted
extensive attention due to the enormous potential for novel magneto-optic
Huang1 ; Gong ; Deng ; Gibertini ; Zhong ; Seyler , magnetoelectronic Zollner
; Song1 ; Klein ; Cardoso ; Wang1 ; Ghazaryan ; Wang2 ; Wang3 , and spintronic
devices Cummings ; Kim1 ; Karpiak . As a representative 2D layered material,
CrI3 possesses its own unique physical properties. The bulk CrI3 has two
different structures, i.e., high-temperature monoclinic phase and low-
temperature rhombohedral phase. The bilayer CrI3 with rhombohedral stacking
exhibits interlayer ferromagnetic coupling, while that with monoclinic
stacking exhibits interlayer antiferromagnetic coupling Sivadas ; Jiang1 ;
Jang ; Soriano1 ; Thiel ; Ubrig ; Kim2 , and their phase transition
temperature is 220 K McGuire . Thus, the magnetic order of CrI3 is susceptible
to the variation of layer thickness and stacking order. Previous first-
principles calculations have predicted that interlayer magnetic coupling can
be effectively modulated by stacking order in bilayer Sivadas ; Jiang1 ; Jang
. Later experiments Thiel ; Ubrig  approved that different stacking orders can
affect the observed magnetic states of CrI3 in both bulk and few-layer CrI3
systems.

In van de Waals layered systems, the relatively weak interlayer coupling
indicates that the interlayer magnetic order can be easily tuned via external
means. Indeed, it was experimentally reported that monoclinic bilayer CrI3 can
be transformed from interlayer antiferromagnetic to ferromagnetic coupling by
applying electric gating Jiang2 ; Jiang3 ; Huang2 ; Xu . A possible physical
mechanism describing this magnetic transition is the formation of magnetic
polaron, which was theoretically confirmed Soriano2 . Beside above external
electric gating, a natural question arises: whether it is possible to find a
static and robust way of realizing the intrinsic interlayer ferromagnetic
coupling in few-layer CrI3? In addition, for semiconductor materials, the
carrier doping concentration may destroy the physical properties. Therefore,
it is desirable to realize the interlayer ferromagnetically-coupled few-layer
CrI3 while maintaining its semiconducting characteristics without introducing
additional carriers.

In this Letter, we perform a systematic study on the magnetic and electronic
properties of nonmagnetic-element doped few-layer CrI3 by using first-
principles calculation methods. We first show that the interlayer
ferromagnetic coupling can be established in bilayer CrI3 doped with C, N, O,
P, S, As, or Se. We then find that the interlayer ferromagnetic coupling is
intimately related to the formation of magnetic polaron. Especially for the
As-doped bi- or tri-layer CrI3, it can achieve higher Curie temperature and
does not introduce extra carriers in the presence of increasing doping
concentration within certain scale, therefore maintaining the system’s
semiconducting properties. In addition, an insulator-metal phase transition
occurs with the help of percolated spin-polarized states at low doping
concentration.

Figure 1: (a) Side and top views of crystal structures of high-temperature
monoclinic bilayer CrI3 phase and the substitution sites are labeled as I1 and
I2. Formation energies of (b) O, S, Se, N and (c) P, As or C element-doped
bilayer CrI3 as a function of the host element chemical potentials.

Calculation Methods—. Our first-principle calculations were performed by using
the projected augmented-wave method Blochl  as implemented in the Vienna ab
initio simulation package (VASP) Kresse1 ; Kresse2 . The generalized gradient
approximation (GGA) of Perdew-Burke-Ernzerhof (PBE) type was used to treat the
exchange-correlation interaction Perdew . In our calculations, the lattice
constant of the high-temperature phase of CrI3 was chosen to be $a_{0}$=6.92 Å
Jiang1 . A vacuum buffer space of 15 Åwas used to prevent the coupling between
adjacent slabs. The kinetic energy cutoff was set to be 340 eV. With fixed
supercells, all structures were fully relaxed. The van der Waals (vdW) force
was taken into account by employing the Grimme’s method (DFT-D2) Grimme . The
Brillouin-zone integration was carried out by using $5\times 5\times 1$
Monkhorst-Pack grids. Unless mentioned otherwise, GGA+U Anisimov ; Dudarev
method was used with the on-site repulsion parameter $U=3.9~{}eV$ and the
exchange parameter $J=1.1~{}eV$ Jiang1 , where $U$ is for the more localized
$3d$ orbitals of Cr atoms. The Curie temperature $T{{}_{C}}$ was estimated
within the mean-field approximation by using $k{{}_{B}}T{{}_{C}}=2/3Jx$
Bergqvist , where $k{{}_{B}}$ is the Boltzmann constant, $x$ is the dopant
concentration, and $J$ is the exchange parameter obtained from the total
energy difference between ferromagnetic and antiferromagnetic configurations.

Figure 2: (a) Energy difference between interlayer ferromagnetic (FM) and
antiferromagnetic (AFM) states. (b) Difference of interlayer distance between
doping configuration and pristine CrI3. (c) Charge difference between Cr atoms
near doping site in the doped and pristine bilayer CrI3. (d) Schematic
illustration of spin-polarized state-mediated interlayer ferromagneitic
coupling in doped bilayer CrI3.

Experimental Possibility of Element Doping—. We first study the possibility of
element doping in bilayer CrI3, i.e., substituting I by nonmagnetic dopants.
Some typical candidates of nonmagnetic dopants including O, S, Se, N, P, As
and C are considered. As displayed in Fig. 1(a), there are two types of
I-doping sites labelled as I1 (at the surface) and I2 (inside the interlayer).
The formation energy was obtained by using the expression Zhang ; Qi ; Han
$\Delta H{{}_{F}}=E{{}_{tot}^{D}}-E{{}_{tot}}-\Sigma n{{}_{i}}\mu_{i}$, where
$E{{}_{tot}^{D}}$ is the total energy of the system including one nonmagnetic
impurity, $E{{}_{tot}}$ is the total energy of the system, ${\mu_{i}}$ is the
chemical potential for the species $i$ (host atoms or dopants), and ni is the
corresponding number that was added/removed from the system.

As displayed in Fig. 1(b), for O, S, Se, N substitutions at two I sites in the
same CrI3 layer, the formation energy is within the range of $-0.4\sim
1.5~{}eV$. It indicates that the I1 substitutional site is preferred due to
smaller formation energy than that at I2 substitution. For example, N
substitution leads to smaller formation energy (about $-0.6\sim 0.2eV$) than
those from O (about $-0.2\sim 0.2eV$), S (about $0.4\sim 0.9eV$) and Se (about
$1.1\sim 1.4eV$) substitution in the whole range of the accessible host
element chemical potentials. However, for P, As, and C substitutions [see Fig.
1(c)], they have larger formation energies than those with O, S, Se or N
dopants. In addition, we find that the I1 substitutional site is preferred by
P and C, while I2 substitutional site is preferred by As. The formation energy
shows that all candidate elements (except As) are more stable at I1 position.
The formation energy of As substituted I2 site is positive (about $2.3\sim
2.7eV$). It is noteworthy that C-doped ZnO has been experimentally fabricated
even the estimated formation energy is about 5.3 eV Pan , which is much larger
than all element-doped CrI3. Therefore, it is reasonable to believe that O, S,
Se, N, P and As doped CrI3 bilayer could be experimentally fabricated.

Magnetic Properties—. We now move to investigate the interlayer magnetic
coupling of the doped CrI3. Figure 2(a) displays the energy difference
$\Delta$${E}$=${E_{FM}}$-${E_{AFM}}$ between interlayer ferromagnetic and
antiferromagnetic states for different element-doped bilayer CrI3. As
reported, the pristine bilayer CrI3 exhibits interlayer antiferromagnetic
coupling Sivadas ; Jiang1 ; Jang . The introduction of dopants except C and N
leads to $\Delta E<0$, indicating the formation of interlayer ferromagnetism.
For the O, P, S, As, and Se substitution at I1(I2) site [see Fig. 2(a)],
$\Delta E$ are respectively -7.7(-2.2), -3.6(-6.3), -6.7(-4.0), -49.7(-113.9),
and -5.4(-12.3) meV, indicating the interlayer ferromagnetic coupling. As a
contrast, it maintains the interlayer antiferromagnetism in C or N doped case.
Hereinbelow, we choose the I2-site As-doped bilayer CrI3 as an example to
analyze the origin of the interlayer ferromagnetism.

Figure 3: Differential charge density of (a) pristine and (b) As-doped bilayer
CrI3. Spin density of (c) pristine and (d) As-doped bilayer CrI3. Local
density of states of (e) pristine and (f) As-doped bilayer CrI3. Yellow and
blue isosurfaces represent respectively charge accumulation and reduction. Red
and green isosurfaces represent respectively spin up and spin down. Cr-d, I-p
and As-p orbitals in each layer of CrI3 are displayed.

For vdW magnetic materials, many studies have shown that the interlayer
distance plays a crucial role in determining the interlayer magnetic coupling
Li1 ; Song2 ; Xia ; Zhu1 . Thus, we first investigate the relationship between
the interlayer distance and the energy difference $\Delta E$. Figure 2(b)
displays the difference of interlayer distances between doped and pristine
CrI3. One can find that the interlayer distances in nearly all doped systems
[except P-doped configuration at I1 substitution)] decrease with respect to
the pristine case. Particularly, the interlayer distances in C, N, and O doped
systems at I1-site substitution shrink respectively about 0.14, 0.17, and 0.18
Å, which are much larger than that in the As-doped system. These together show
that there is no obvious correlation between the strength of ferromagnetic
coupling and the interlayer distance.

It was reported that the interlayer magnetism of vdW materials is closely
related to different 3d electron occupation between different layers Xiao ;
Li2 ; Zhu2 . In Fig. 2(c), we display the charge difference of Cr atoms near
the dopants between the doped and pristine bilayer CrI3. It shows that the
charge of Cr atoms near dopants increases for all doped systems. However, in
As-doped bilayer, the change of 3d electron occupation due to doping is much
less than those of the C and N doped bilayers. Therefore, the difference of 3d
electrons occupation between two layers cannot explain the formation of the
interlayer ferromagnetism.

Figure 4: (a-c-e-g) Schematic plots of two spin-polarized states at different
distances (dP-P) in As-doped bilayer CrI3. (b-d-f-h) The density of states of
ferromagnetic state and the energy difference between interlayer ferromagnetic
(FM) and antiferromagnetic (AFM) states at different dP-P in As-doped bilayer.
The shadow part indicates the formation of spin-polarized state.

We now move to calculate the differential charge densities of pristine and As-
doped systems [see Figs. 3(a) and 3(b)] Jiang1 ; Liu . One can see that As-
doping indeed leads to obvious change of charge distribution inside the
interlayer space. Figures 3(c) and 3(d) display the spin densities of pristine
and doped cases, respectively. For pristine case, it exhibits intralayer
ferromagnetism and interlayer anti-ferromagnetism. After As-doping, the
interlayer anti-ferromagnetism transits to ferromagnetism, accompanying with a
strongly localized spin-polarized state near the doping site. To confirm this
finding, the local density of states of pristine and As doped bilayers are
displayed in Figs. 3(e) and 3(f). In Fig. 3(f), one can find that an extremely
sharp local density of state appears near the Fermi level, arising from the
hybridization among As p-orbital, bonded Cr d-orbital in the same layer, and I
p-orbital in the other layer. All together indicate that As-doping results in
a bound spin-polarized state, potentially forming the interlayer
ferromagnetism.

To further confirm this physical origin, in Fig. S1, we display the density of
states of the C, N, O, P, S, and Se doped bilayers, where C and N doped
systems exhibit interlayer antiferromagnetism, but O, P, S, and Se doped
systems exhibit interlayer ferromagnetism SM . Figures S1(a) and S1(b) display
that no spin-polarized bound states appear near the Fermi level for C and N
doped systems. While for other doped systems, spin-polarized bound states
arise near the Fermi level [see Figs. S1(c)-S1(f)]. These suggest a direct
evidence that the interlayer ferromagnetic coupling in doped bilayer CrI3 is a
consequence of the formation of spin-polarized bound state.

Another striking transport phenomenon is the insulating nature after doping.
It is known that doping or gating can result in the ferromagnetism of
semiconductors, but may also break the semiconducting property due to the
carrier injection. Surprisingly, for O, P, S, As, and Se doped bilayer CrI3,
our results show that they exhibit both ferromagnetic and insulating features.
In association with the experimental finding that different gate doping levels
do not lead to n- or p-type conduction of bilayer CrI3 dominantly with
affected magnetic properties Huang2 , it is believed that only insulated
interlayer ferromagnetism in few-layer CrI3 can be observed due to the
formation of spin-polarized bound state at certain doping concentrations.

When localized spin-polarized states percolate to form contiguous grids, an
insulator-metal transition can occur at certain doping concentration. As shown
in Figs. 4(a-c-e-g), we study the electronic structure and magnetic property
of two-As-atom doped bilayer CrI3 at different As-As distances (i.e., various
doping concentrations). For the nearest As-As distance dP-P=6.92 Å, in
comparison with density of states of only one-As-atom doped bilayer CrI3 (see
Fig. 3(f)), the spin-polarized states in Fig. 4(b) is more delocalized and
only one spin-polarized states is formed [A more larger spin-polarized states
is schematically plotted in Fig. 4(a)]. At this configuration (the estimated
doping concentration is about 8.33$\%$), it is a ferromagnetic insulator and
the interlayer ferromagnetic coupling ($\Delta E$= -177.0 meV) becomes
stronger than that($\Delta E$= -113.9 meV) with only one-As dopant. When the
As doping concentration decreases, from Figs. 4(c) and 4(e), we find that two
spin-polarized states can be formed and percolated. In such percolated
systems, they are p-type semiconductor with interlayer ferromagnetism (see
Figs. 4(d) and 4(f)), but the ferromagnetic coupling becomes weaker than that
of the insulating case. For longer As-As distance, two independent spin-
polarized states are formed and the interlayer magnetic coupling almost
disappears. For example, when the As-As distance dP-P is 13.17 Å, one spin-
polarized state is about 6.6 Å. If the doping concentration is further
decreased to 2.08$\%$, our results indicate that an insulated and
ferromagnetic bilayer CrI3 can be formed again due to the independent spin-
polarized states, similar to that in Fig. 3(f).

So far, we have shown that element doping in bilayer CrI3 can induce
interlayer ferromagnetism. The trilayer CrI3 has weak interlayer ferromagnetic
coupling Huang1 . Whether the element doping can enhance the interlayer
ferromagnetism? By taking As-doping as an example in Fig. S2(a), three I
substituted sites including I1, I2, and I3 are selected. The interlayer
magnetic couplings and their relative stabilities are respectively displayed
in Fig. S2(b) and Table S1. One can see that the state with interlayer
ferromagnetic coupling is more stable than the other three states with
interlayer antiferromagnetic coupling, which agrees well with the experimental
observation Huang1 . One can find that the total energy of As substitution at
I3 site is the lowest, indicating the most stable structure. Therefore, we
show that the strong interlayer ferromagnetic coupling can also be established
in trilayer CrI3 via As doping.

Conclusions—. In summary, we demonstrate that the interlayer ferromagnetic
coupling can be realized in both bilayer and trilayer CrI3 by doping
nonmagnetic elements. Our finding provides a new evidence that the interlayer
ferromagnetic coupling in CrI3 thin films may be related to the formation of
spin-polarized states, and also provides an alternative scheme for the
realization of CrI3 based ferromagnetic insulator.

Acknowledgements—. This work was financially supported by the NNSFC (11974098,
11974327, and 12004369), Natural Science Foundation of Hebei
Province(A2019205037), and Science Foundation of Hebei Normal University
(2019B08), Fundamental Research Funds for the Central Universities
(WK2030020032 and WK2340000082), Anhui Initiative in Quantum Information
Technologies. The supercomputing services of AM-HPC and USTC are gratefully
acknowledged.

‡ These authors contribute equally to this work.

## References

  * (1) B. Huang, G. Clark, E. Navarro-Moratalla, D. R. Klein, R. Cheng, K. L. Seyler, D. Zhong, E. Schmidgall, M. A. McGuire, D. H. Cobden, W. Yao, D. Xiao, P . Jarillo-Herrero, and X. Xu,Nature (London) 546, 270 (2017).
  * (2) C. Gong, L. Li, Z. Li, H. Ji, A. Stern, Y. Xia, T. Cao, W. Bao, C. Wang, Y . Wang, Z. Q. Qiu, R. J. Cava, S. G. Louie, J. Xia,and X. Zhang,Nature (London) 546, 265 (2017).
  * (3) Y. Deng, Y. Yu, Y. Song, J. Zhang, N. Z. Wang, Z. Sun, Y. Yi,Y. Z. Wu, S. Wu, J. Zhu, J. Wang, X. H. Chen, and Y. Zhang,Nature (London) 563, 94 (2018).
  * (4) M. Gibertini, M. Koperski, A. F. Morpurgo, and K. S. Novoselov,Nat. Nanotechnol. 14, 408 (2019).
  * (5) D. Zhong, K. L. Seyler, X. Linpeng, R. Cheng, N. Sivadas, B. Huang, E. Schmidgall, T. Taniguchi, K. Watanabe, M. A. McGuire, W. Yao, D. Xiao, K.-M. C. Fu, and X. Xu,Sci. Adv. 3, e1603113 (2017).
  * (6) K. L. Seyler, D. Zhong, B. Huang, X. Linpeng, N. P . Wilson, T. Taniguchi, K. Watanabe, W. Yao, D. Xiao, M. A. McGuire, K.-M. C. Fu, and X. Xu,Nano Lett. 18, 3823 (2018).
  * (7) K. Zollner, M. Gmitra, and J. Fabian,New J. Phys. 20, 073007(2018).
  * (8) T. Song, X. Cai, M. W.-Y. Tu, X. Zhang, B. Huang, N. P. Wilson, K. L. Seyler, L. Zhu, T. Taniguchi, K. Watanabe, M. A. McGuire, D. H. Cobden, D. Xiao, W. Yao, and X. Xu, Science 360, 1214 (2018).
  * (9) D. R. Klein, D. MacNeill, J. L. Lado, D. Soriano, E. Navarro-Moratalla, K. Watanabe, T. Taniguchi, S. Manni, P. Canfield, J. Fern$\acute{a}$ndez-Rossier, and P. Jarillo-Herrero, Science 360, 1218(2018).
  * (10) C. Cardoso, D. Soriano, N. A. Garc$\acute{i}$a-Mart$\acute{i}$nez, and J. Fern$\acute{a}$ndez-Rossier, Phys. Rev. Lett. 121, 067701 (2018).
  * (11) Z. Wang, I. Guti$\acute{e}$rrez-Lezama, N. Ubrig, M. Kroner, M. Gibertini, T. Taniguchi, K. Watanabe, A. Imamo$\breve{g}$lu, E. Giannini, and A. F. Morpurgo, Nat. Commun. 9, 2516(2018)
  * (12) D. Ghazaryan, M. T. Greenaway, Z. Wang, V. H. GuarochicoMoreira, I. J. Vera-Marun, J. Yin, Y. Liao, S. V. Morozov, O. Kristanovski, A. I. Lichtenstein, M. I. Katsnelson, F. Withers, A. Mishchenko, L. Eaves, A. K. Geim, K. S. Novoselov, and A. Misra, Nat. Electron. 1, 344 (2018).
  * (13) Z. Wang, T. Zhang, M. Ding, B. Dong, Y. Li, M. Chen, X. Li, J. Huang, H. Wang, X. Zhao, Y. Li, D. Li, C. Jia, L. Sun, H. Guo, Y. Ye, D. Sun, Y. Chen, T. Yang, J. Zhang, S. Ono, Z. Han and Z. Zhang, Nat. Nanotechnol. 13, 544 (2018).
  * (14) Z. Wang, M. Gibertini, D. Dumcenco, T. Taniguchi, K. Watanabe, E. Giannini, and A. F. Morpurgo, Nat. Nanotechnol. 14, 1116 (2019).
  * (15) A. W. Cummings, J. Phys. Mater. 2, 045007 (2019).
  * (16) M. Kim, P. Kumaravadivel, J. Birkbeck, W. Kuang, S. G. Xu, D. G. Hopkinson, J. Knolle, P. A. McClarty, A. I. Berdyugin, M. Ben Shalom, R. V. Gorbachev, S. J. Haigh, S. Liu, J. H. Edgar, K. S. Novoselov, I. V. Grigorieva, and A. K. Geim, Nat. Electron. 2, 457 (2019).
  * (17) B. Karpiak, A. W. Cummings, K. Zollner, M. Vila, D. Khokhriakov, A. M. Hoque, A. Dankert, P. Svedlindh, J. Fabian, S. Roche, and S. P. Dash, 2D Mater. 7, 015026 (2019).
  * (18) N. Sivadas, S. Okamoto, X. Xu, C. J. Fennie, and D. Xiao,Nano Lett. 18, 7658 (2018).
  * (19) P. Jiang, C. Wang, D. Chen, Z. Zhong, Z. Y uan, Z.-Y. Lu, and W. Ji,Phys. Rev. B 99, 144401 (2019).
  * (20) S. W. Jang, M. Y. Jeong, H. Yoon, S. Ryee, and M. J. Han, Phys. Rev. Mater. 3, 031001 (2019).
  * (21) D. Soriano, C. Cardoso, and J. Fernndez-Rossier,Solid State Commun. 299, 113662 (2019).
  * (22) L. Thiel, Z. Wang, M. A. Tschudin, D. Rohner, I. Guti rrez-Lezama, N. Ubrig, M. Gibertini, E. Giannini, A. F. Morpurgo, and P. Maletinsky, Science 364, 973 (2019).
  * (23) N. Ubrig, Z. Wang, J. Teyssier, T. Taniguchi, K. Watanabe, E. Giannini, A. F. Morpurgo, and M. Gibertini, 2D Mater. 7, 015007 (2019).
  * (24) H. H. Kim, B. Yang, T. Patel, F. Sfigakis, C. Li, S. Tian, H. L. Lei, and A. W. Tsen,Nano Lett. 18, 4885 (2018)
  * (25) M. A. McGuire, H. Dixit, V. R. Cooper, and B. C. Sales,Chem.Mater. 27, 612 (2015).
  * (26) S. Jiang, L. Li, Z. Wang, K. F. Mak, and J. Shan, Nat. Nanotechnol. 13, 549 (2018).
  * (27) S. Jiang, J. Shan, and K. F. Mak, Nat. Mater. 17, 406 (2018).
  * (28) B. Huang, G. Clark, D. R. Klein, D. MacNeill, E. Navarro-Moratalla, K. L. Seyler, N. Wilson, M. A. McGuire, D. H. Cobden, D. Xiao, W. Yao, P. Jarillo-Herrero, and X. Xu, Nat. Nanotechnol. 13, 544 (2018).
  * (29) R. Xu and X. Zou, J. Phys. Chem. Lett. 11, 3152(2020)
  * (30) D. Soriano and M. I. Katsnelson,Phys. Rev. B 101, 041402(R) (2020)
  * (31) P. E. Bl$\ddot{o}$chl, Phys. Rev. B 50, 17953 (1994).
  * (32) G. Kresse and J. Furthmuller, Phys. Rev. B 54, 11169 (1996).
  * (33) G. Kresse and D. Joubert,Phys. Rev. B 59, 1758 (1999).
  * (34) J. P. Perdew, K. Burke, and M. Ernzerhof, Phys. Rev. Lett. 77, 3865 (1996).
  * (35) S. Grimme, J. Comput. Chem. 27, 1787 (2006).
  * (36) V. I. Anisimov, J. Zaanen, and O. K. Anderson, Phys. Rev. B 44, 943 (1991).
  * (37) S. L. Dudarev, G. A. Botton, S. Y. Savrasov, C. J. Humphreys, and A. P. Sutton, Phys. Rev. B 57, 1505 (1998).
  * (38) L. Bergqvist, O. Eriksson, J. Kudrnovsky, V. Drchal, A. Bergman, L. Nordstrom, and I. Turek, Phys. Rev. B 72, 195210 (2005).
  * (39) J. M. Zhang, W. G. Zhu, Y. Zhang, D. Xiao, and Y. G. Yao, Phys. Rev. Lett. 109, 266405 (2012).
  * (40) S. Qi, R. Gao, M. Chang, T. Hou, Y. Han, and Z. Qiao,Phys. Rev. B 102, 085419 (2020).
  * (41) Y. Han, S. Sun, S. Qi, X. Xu,and Z. Qiao, Phys. Rev. B 103, 245403 (2021)
  * (42) H. Pan, J. B. Yi, L. Shen, R. Q. Wu, J. H. Yang, J. Y. Lin, Y. P. Feng, J. Ding, L. H. Van, and J. H. Yin, Phys. Rev. Lett. 99, 127201 (2007).
  * (43) T. Li, S. Jiang, N. Sivadas, Z. Wang, Y. Xu, D. Weber, J. E. Goldberger, K. Watanabe, T. Taniguchi, C. J. Fennie, K. F. Mak and J. Shan, Nat. Mater. 18, 1303(2019)
  * (44) T. Song, Z. Fei, M. Yankowitz, Z. Lin, Q. Jiang, K. Hwangbo, Q. Zhang, B. Sun, T. Taniguchi, K. Watanabe, M. A. McGuire, D. Graf, T. Cao, J-H Chu, D. H. Cobden, C. R. Dean, D. Xiao and X. Xu, Nat. Mater. 18, 1298(2019)
  * (45) J. Xia, J. Yan, Z. Wang, Y. He, Y. Gong, W. Chen, T. C. Sum, Z. Liu, P. M. Ajayan and Z. Shen, Nat. Phys.17,92 (2021).
  * (46) W. Zhu, C. Song, Y. Zhou, Q. Wang, H. Bai, and F. Pan, Phys. Rev. B 103, 224404 (2021).
  * (47) J. W. Xiao and B. H. Yan, 2D Mater. 7, 045010 (2020).
  * (48) Z. Li, J. Li, K. He, X. Wan, W. Duan, and Y. Xu, Phys. Rev. B 102, 081107(R) (2020).
  * (49) W. Zhu, C. Song, L. Liao, Z. Zhou, H. Bai, Y. Zhou, and F. Pan, Phys. Rev. B 102, 085111 (2020).
  * (50) N. Liu, S. Zhou and J. Zhao, Phys. Rev. Mater. 4, 094003 (2020).
  * (51) See Supplemental Material at [URL will be inserted by publisher] for [the density of states of the C, N, O, P, S, and Se doped bilayer CrI3, four magnetic states related interlayer magnetic coupling and their relative stability.].


* Lee [2010b] B. Lee,  Improved multiple-coarsening methods for sn discretizations of the boltzmann equation,  SIAM Journal on Scientific Computing 32 (2010b) 2497–2522.
  * Lee [2012] B. Lee,  Space-angle-energy multigrid methods for sn discretizations of the multi-energetic boltzmann equation,  Numerical Linear Algebra with Applications 19 (2012) 773–795.
  * Gao et al. [2012] H. Gao, L. Phan, Y. Lin,  Parallel multigrid solver of radiative transfer equation for photon transport via graphics processing unit,  Journal of Biomedical Optics 17 (2012).
  * Turcksin et al. [2012] B. Turcksin, J. C. Ragusa, J. E. Morel,  Angular multigrid preconditioner for krylov-based solution techniques applied to the sn equations with highly forward-peaked scattering,  Transport Theory and Statistical Physics 41 (2012) 1–22.
  * Buchan et al. [2012] A. Buchan, C. Pain, A. Umpleby, R. Smedley-Stevenson,  A sub-grid scale finite element agglomeration multigrid method with application to the boltzmann transport equation,  International Journal for Numerical Methods in Engineering 92 (2012) 318–342.
  * Slaybaugh et al. [2013] R. N. Slaybaugh, T. M. Evans, G. G. Davidson, P. P. H. Wilson,  Multigrid in energy preconditioner for Krylov solvers,  Journal of Computational Physics 242 (2013) 405–419.
  * Slaybaug et al. [2015] R. Slaybaug, T. M. Evans, G. Davidson, P. P. H. Wilson,  Rayleigh Quotient Iteration with a Multigrid in Energy Preconditioner for Massively Parallel Neutron Transport,  in: Proceedings of Joint International Conference on Mathematics and Computation, Supercomputing in Nuclear Applications, and the Monte Carlo Metho, Nashville, TN.
  * Drumm and Fan [2017] C. Drumm, W. Fan,  Multilevel acceleration of scattering-source iterations with application to electron transport,  Nuclear Engineering and Technology 49 (2017) 1114–1124.
  * Lathouwers and Perkó [2019] D. Lathouwers, Z. Perkó,  An angular multigrid preconditioner for the radiation transport equation with Fokker–Planck scattering,  Journal of Computational and Applied Mathematics 350 (2019) 165–177.
  * Manteuffel et al. [2019] T. A. Manteuffel, S. Münzenmaier, J. Ruge, B. Southworth,  Nonsymmetric Reduction-Based Algebraic Multigrid,  SIAM Journal on Scientific Computing 41 (2019) S242–S268. Publisher: Society for Industrial and Applied Mathematics.
  * Manteuffel and Southworth [2019] T. Manteuffel, B. S. Southworth,  Convergence in Norm of Nonsymmetric Algebraic Multigrid,  SIAM Journal on Scientific Computing 41 (2019) S269–S296. Publisher: Society for Industrial and Applied Mathematics.
  * Southworth et al. [2017] B. Southworth, T. A. Manteuffel, J. Ruge,  Nonsymmetric Algebraic Multigrid Based on Local Approximate Ideal Restriction (LAIR),  arXiv:1708.06065 [math] (2017).
  * Hanophy et al. [2020] J. Hanophy, B. S. Southworth, R. Li, T. Manteuffel, J. Morel,  Parallel Approximate Ideal Restriction Multigrid for Solving the S N Transport Equations,  Nuclear Science and Engineering 0 (2020) 1–20. Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/00295639.2020.1747263.
  * Hughes et al. [1998] T. J. R. Hughes, G. R. Feijóo, L. Mazzei, J.-B. Quincy,  The variational multiscale method—a paradigm for computational mechanics,  Computer Methods in Applied Mechanics and Engineering 166 (1998) 3–24.
  * Hughes et al. [2006] T. J. R. Hughes, G. Scovazzi, P. B. Bochev, A. Buffa,  A multiscale discontinuous galerkin method with the computational structure of a continuous galerkin method,  Computer Methods in Applied Mechanics and Engineering 195 (2006) 2761–2787.
  * Candy [2008] A. S. Candy, Subgrid scale modelling of transport processes., Thesis or dissertation, Imperial College London, 2008.
  * Buchan et al. [2010] A. G. Buchan, A. S. Candy, S. R. Merton, C. C. Pain, J. I. Hadi, M. D. Eaton, A. J. H. Goddard, R. P. Smedley-Stevenson, G. J. Pearce,  The inner-element subgrid scale finite element method for the boltzmann transport equation,  Nuclear science and engineering 164 (2010) 105–121.
  * Goffin et al. [2014] M. A. Goffin, A. G. Buchan, A. C. Belme, C. C. Pain, M. D. Eaton, P. N. Smith, R. P. Smedley-Stevenson,  Goal-based angular adaptivity applied to the spherical harmonics discretisation of the neutral particle transport equation,  Ann. Nucl. Energy 71 (2014) 60–80.
  * Dargaville et al. [2015] S. Dargaville, M. A. Goffin, A. G. Buchan, C. C. Pain, R. P. Smedley-Stevenson, P. N. Smith, G. Gorman,  Solving the boltzmann transport equation with multigrid and adaptive space/angle discretisations,  Annals of Nuclear Energy 86 (2015) 99–107.
  * Goffin [2015] M. Goffin, Goal-based adaptive methods applied to the spatial and angular dimensions of the transport equation, Ph.D. thesis, Imperial College London, 2015.
  * Buchan and Pain [2016] A. G. Buchan, C. C. Pain,  An efficient space-angle subgrid scale discretisation of the neutron transport equation,  Annals of Nuclear Energy 94 (2016) 440–450.
  * Adigun et al. [2018] B. J. Adigun, A. G. Buchan, A. Adam, S. Dargaville, M. A. Goffin, C. C. Pain,  A Haar wavelet method for angularly discretising the Boltzmann transport equation,  Progress in Nuclear Energy 108 (2018) 295–309.
  * Dargaville et al. [2019] S. Dargaville, A. G. Buchan, R. P. Smedley-Stevenson, P. N. Smith, C. C. Pain,  Angular adaptivity with spherical harmonics for Boltzmann transport,  Journal of Computational Physics 397 (2019).
  * Lygidakis and Nikolos [2014a] G. N. Lygidakis, I. K. Nikolos,  Using a Parallel Spatial/Angular Agglomeration Multigrid Scheme to Accelerate the FVM Radiative Heat Transfer Computation—Part I: Methodology,  Numerical Heat Transfer, Part B: Fundamentals 66 (2014a) 471–497. Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/10407790.2014.949561.
  * Lygidakis and Nikolos [2014b] G. N. Lygidakis, I. K. Nikolos,  Using a Parallel Spatial/Angular Agglomeration Multigrid Scheme to Accelerate the FVM Radiative Heat Transfer Computation—Part II: Numerical Results,  Numerical Heat Transfer, Part B: Fundamentals 66 (2014b) 498–525.
  * Southworth et al. [2021] B. S. Southworth, M. Holec, T. S. Haut,  Diffusion Synthetic Acceleration for Heterogeneous Domains, Compatible with Voids,  Nuclear Science and Engineering 195 (2021) 119–136. Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/00295639.2020.1799603.
  * Notay [2005] Y. Notay,  Algebraic multigrid and algebraic multilevel methods: a theoretical comparison,  Numerical Linear Algebra with Applications 12 (2005) 419–451. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/nla.435.
  * MacLachlan et al. [2006] S. MacLachlan, T. Manteuffel, S. McCormick,  Adaptive reduction-based AMG,  Numerical Linear Algebra with Applications 13 (2006) 599–620.
  * Brannick et al. [2010] J. Brannick, A. Frommer, K. Kahl, S. MacLachlan, L. Zikatanov,  Adaptive reduction-based multigrid for nearly singular and highly disordered physical systems,  Electronic transactions on numerical analysis 37 (2010) 276–295. Publisher: Institute of Computational Mathematics.
  * Zaman et al. [2022] T. Zaman, N. Nytko, A. Taghibakhshi, S. MacLachlan, L. Olson, M. West, Generalizing Reduction-Based Algebraic Multigrid, 2022\. ArXiv:2212.08371 [cs, math].
  * Brezina et al. [2004] M. Brezina, R. Falgout, S. MacLachlan, T. Manteuffel, S. McCormick, J. Ruge,  Adaptive Smoothed Aggregation(alphaSA),  SIAM Journal on Scientific Computing 25 (2004) 1896–1920.
  * Brezina et al. [2005] M. Brezina, R. Falgout, S. MacLachlan, T. Manteuffel, S. McCormick, J. Ruge,  Adaptive Smoothed Aggregation (alphaSA) Multigrid,  SIAM Review 47 (2005) 317–346.
  * Kolotilina and Yeremin [1993] L. Y. Kolotilina, A. Y. Yeremin,  Factorized Sparse Approximate Inverse Preconditionings I. Theory,  SIAM Journal on Matrix Analysis and Applications 14 (1993) 45–58. Publisher: Society for Industrial and Applied Mathematics.
  * Grote and Huckle [1997] M. J. Grote, T. Huckle,  Parallel Preconditioning with Sparse Approximate Inverses,  SIAM Journal on Scientific Computing 18 (1997) 838–853. Publisher: Society for Industrial and Applied Mathematics.
  * Vassilevski [2008] P. S. Vassilevski, Multilevel Block Factorization Preconditioners, Springer, 2008\.
  * Saad [2005] Y. Saad,  Multilevel ILU With Reorderings for Diagonal Dominance,  SIAM Journal on Scientific Computing 27 (2005) 1032–1057. Publisher: Society for Industrial and Applied Mathematics.
  * Van\vek et al. [1996] P. Van\vek, J. Mandel, M. Brezina,  Algebraic multigrid by smoothed aggregation for second and fourth order elliptic problems,  Computing 56 (1996) 179–196.
  * Schroder [2010] J. B. Schroder, Generalizing smoothed aggregation-based algebraic multigrid, Ph.D. thesis, University of Illinois at Urbana-Champaign, 2010.
  * Wiesner [2015] T. Wiesner, Flexible Aggregation-based Algebraic Multigrid Methods for Contact and Flow Problems, Ph.D. thesis, 2015.
  * Manteuffel et al. [2017] T. A. Manteuffel, L. N. Olson, J. B. Schroder, B. S. Southworth,  A Root-Node Based Algebraic Multigrid Method,  SIAM Journal on Scientific Computing 39 (2017) S723–S756. ArXiv: 1610.03154.
  * Xu and Zikatanov [2017] J. Xu, L. Zikatanov,  Algebraic multigrid methods*,  Acta Numerica 26 (2017) 591–721. Publisher: Cambridge University Press.
  * Olson et al. [2011] L. N. Olson, J. B. Schroder, R. S. Tuminaro,  A General Interpolation Strategy for Algebraic Multigrid Using Energy Minimization,  SIAM Journal on Scientific Computing 33 (2011) 966–991. Publisher: Society for Industrial and Applied Mathematics.
  * Saad and Schultz [1986] Y. Saad, M. H. Schultz,  GMRES: A generalized minimal residual algorithm for solving nonsymmetric linear systems,  SIAM Journal on scientific and statistical computing 7 (1986) 856–869.
  * Nachtigal et al. [1992] N. M. Nachtigal, L. Reichel, L. N. Trefethen,  A hybrid GMRES algorithm for nonsymmetric linear systems,  SIAM Journal on Matrix Analysis and Applications 13 (1992) 796–825.
  * Liu et al. [2015] Q. Liu, R. B. Morgan, W. Wilcox,  Polynomial Preconditioned GMRES and GMRES-DR,  SIAM Journal on Scientific Computing 37 (2015) S407–S428. Publisher: SIAM.
  * Loe and Morgan [2021] J. A. Loe, R. B. Morgan,  Toward Efficient Polynomial Preconditioning for GMRES,  Numerical Linear Algebra with Applications (2021). ArXiv: 1911.07065.
  * Meurant and Tebbens [2020] G. Meurant, J. D. Tebbens, Krylov Methods for Nonsymmetric Linear Systems: From Theory to Computations, Springer Series in Computational Mathematics, Springer International Publishing, 2020\.
  * Axelsson and Vassilevski [1989] O. Axelsson, P. S. Vassilevski,  Algebraic multilevel preconditioning methods. I,  Numerische Mathematik 56 (1989) 157–177.
  * Axelsson and Vassilevski [1990] O. Axelsson, P. S. Vassilevski,  Algebraic Multilevel Preconditioning Methods, II,  SIAM Journal on Numerical Analysis 27 (1990) 1569–1590. Publisher: Society for Industrial and Applied Mathematics.
  * Greenbaum and Trefethen [1994] A. Greenbaum, L. N. Trefethen,  GMRES/CR and Arnoldi/Lanczos as Matrix Approximation Problems,  SIAM Journal on Scientific Computing 15 (1994) 359–368. Publisher: Society for Industrial and Applied Mathematics.
  * Loe [2019] J. A. Loe, Polynomial preconditioning with the minimum residual polynomial., Thesis, 2019\. Accepted: 2020-09-09T13:49:00Z.
  * Hoemmen [2010] M. Hoemmen, Communication-avoiding Krylov subspace methods, Ph.D., University of California, Berkeley, United States – California, 2010\.
  * Chow and Patel [2015] E. Chow, A. Patel,  Fine-Grained Parallel Incomplete LU Factorization,  SIAM Journal on Scientific Computing 37 (2015) C169–C193. Publisher: Society for Industrial and Applied Mathematics.
  * Anzt et al. [2018] H. Anzt, E. Chow, J. Dongarra,  ParILUT—A New Parallel Threshold ILU Factorization,  SIAM Journal on Scientific Computing 40 (2018) C503–C519. Publisher: Society for Industrial and Applied Mathematics.
  * Mavriplis [1999] D. J. Mavriplis,  Directional agglomeration multigrid techniques for high-Reynolds-number viscous flows,  AIAA journal 37 (1999) 1222–1230.
  * Henson and Yang [2002] V. E. Henson, U. M. Yang,  BoomerAMG: A parallel algebraic multigrid solver and preconditioner,  Applied Numerical Mathematics 41 (2002) 155–177.
  * Brandt [2000] A. Brandt,  General highly accurate algebraic coarsening.,  ETNA. Electronic Transactions on Numerical Analysis [electronic only] 10 (2000) 1–20. Publisher: Kent State University, Department of Mathematics and Computer Science.
  * Brannick and Zikatanov [2007] J. Brannick, L. Zikatanov,  Algebraic Multigrid Methods Based on Compatible Relaxation and Energy Minimization,  in: O. B. Widlund, D. E. Keyes (Eds.), Domain Decomposition Methods in Science and Engineering XVI, Lecture Notes in Computational Science and Engineering, Springer, Berlin, Heidelberg, 2007, pp. 15–26.
  * Brannick and Falgout [2010] J. J. Brannick, R. D. Falgout,  Compatible Relaxation and Coarsening in Algebraic Multigrid,  SIAM Journal on Scientific Computing 32 (2010) 1393–1416. Publisher: Society for Industrial and Applied Mathematics.
  * Saad [1996] Y. Saad,  ILUM: A Multi-Elimination ILU Preconditioner for General Sparse Matrices,  SIAM Journal on Scientific Computing 17 (1996) 830–847. Publisher: Society for Industrial and Applied Mathematics.
  * Saad and Zhang [1999] Y. Saad, J. Zhang,  BILUTM: A Domain-Based Multilevel Block ILUT Preconditioner for General Sparse Matrices,  SIAM Journal on Matrix Analysis and Applications 21 (1999) 279–299.
  * Saad and Suchomel [2002] Y. Saad, B. Suchomel,  ARMS: an algebraic recursive multilevel solver for general sparse linear systems,  Numerical Linear Algebra with Applications 9 (2002) 359–378. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/nla.279.
  * MacLachlan and Saad [2007] S. MacLachlan, Y. Saad,  A Greedy Strategy for Coarse-Grid Selection,  SIAM Journal on Scientific Computing 29 (2007) 1825–1853.
  * Beckermann et al. [2005] B. Beckermann, S. A. Goreinov, E. E. Tyrtyshnikov,  Some Remarks on the Elman Estimate for GMRES,  SIAM Journal on Matrix Analysis and Applications 27 (2005) 772–778. Publisher: Society for Industrial and Applied Mathematics.
  * Liesen and Tichý [2020] J. Liesen, P. Tichý, The field of values bound on ideal GMRES, 2020. ArXiv:1211.5969 [cs, math].