# Small Language Models Improve Giants by Rewriting Their Outputs

Giorgos Vernikos1,2 Arthur Bražinskas3 Jakub Adamek3
Jonathan Mallinson3 Aliaksei Severyn3 Eric Malmi3
1EPFL, 2HEIG-VD, 3Google Research
<EMAIL_ADDRESS>
{abrazinskas, enkait, jonmall, severyn<EMAIL_ADDRESS>
Research conducted during a Google internship.

###### Abstract

Large language models (LLMs) have demonstrated impressive few-shot learning
capabilities, but they often underperform compared to fine-tuned models on
challenging tasks. Furthermore, their large size and restricted access only
through APIs make task-specific fine-tuning impractical. Moreover, LLMs are
sensitive to different aspects of prompts (e.g., the selection and order of
demonstrations) and can thus require time-consuming prompt engineering. In
this light, we propose a method to correct LLM outputs without relying on
their weights. First, we generate a pool of candidates by few-shot prompting a
LLM. Second, we refine the LLM-generated outputs using a smaller model, the
LM-corrector (LMCor), which is trained to rank, combine and rewrite the
candidates to produce the final target output. Our experiments demonstrate
that even a small LMCor model (250M) substantially improves the few-shot
performance of LLMs (62B) across diverse tasks. Moreover, we illustrate that
the LMCor exhibits robustness against different prompts, thereby minimizing
the need for extensive prompt engineering. Finally, we showcase that the LMCor
can be seamlessly integrated with different LLMs at inference time, serving as
a plug-and-play module to improve their performance.

Small Language Models Improve Giants by Rewriting Their Outputs

Giorgos Vernikos1,2††thanks:  Research conducted during a Google internship.
Arthur Bražinskas3 Jakub Adamek3 Jonathan Mallinson3 Aliaksei Severyn3 Eric
Malmi3 1EPFL, 2HEIG-VD, 3Google Research<EMAIL_ADDRESS>{abrazinskas, enkait, jonmall, severyn<EMAIL_ADDRESS>

Figure 1: An illustration of our approach. We first prompt an LLM to generate
multiple outputs via an API (dotted lines). Then we feed the generated
candidates to the LM-corrector, a small model that is trained to rewrite them
in order to generate the target sentence (solid lines).

##  1 Introduction

Large language models have recently demonstrated near state-of-the-art
performance on various tasks via in-context learning, which enables them to
generate outputs based on instructions and a handful of examples, without
task-specific training (Brown et al., 2020b, a; Chowdhery et al., 2022).
However, the effectiveness of this paradigm can vary significantly depending
on the task instruction (Shin et al., 2020; Jiang et al., 2021; Schick and
Schütze, 2021), the quantity, relevance and even the order of the in-context
examples (Brown et al., 2020a; Gao et al., 2021; Liu et al., 2022a; Zhang et
al., 2023a; Lu et al., 2022). As a result, in-context learning often requires
labour-intensive prompt engineering which does not always guarantee improved
performance (Jiang et al., 2021).

Fine-tuning, on the other hand, has been proven effective in scenarios where
task-specific datasets are available, demonstrating that even smaller fine-
tuned models can outperform few-shot prompted LLMs on challenging tasks
(Chowdhery et al., 2022; Lester et al., 2021). While LLMs can be fine-tuned to
enhance their performance in specific tasks there are several limitations.
Firstly, the fine-tuning process may have a detrimental effect on the few-shot
performance of LLMs on other tasks, leading to a tradeoff between versatility
and performance Fu et al. (2023). Secondly, the increasing scale of LLMs makes
fine-tuning on standard hardware computationally infeasible. In order to
address these issues parameter-efficient fine-tuning methods have been
proposed (Karimi Mahabadi et al., 2021; Lester et al., 2021; Li and Liang,
2021; Houlsby et al., 2019; Zhang et al., 2023b). While these methods are more
computationally efficient, they require access to the model weights and
substantial computational resources for loading and updating the model.
Furthermore, due to the commercialization of LLMs, they are often available
only through restricted inference APIs. In light of these challenges, we
propose a method – LMCor – which leverages only the outputs of LLMs to enhance
their performance in specific tasks.

Our approach is motivated by the observation that the outputs of the LLM can
be optimally edited and combined to generate the target sentence (Figure 2).
To achieve this, we propose LM-Corrector (LMCor), a smaller model that is used
to correct the predictions generated by the LLM. Unlike traditional fine-
tuning methods, our approach does not require access to the weights of the
LLM. Instead LMCor operates directly on the outputs of the LLM. The corrector
module receives a diverse set of candidates for a given source sentence,
representing alternative outputs sampled from the LLM. By considering multiple
candidates simultaneously, LMCor learns to optimally rank, combine, and even
edit them to produce more accurate and higher-quality outputs. Figure 1
illustrates our approach, where LMCor rewrites the first output of the LLM
while incorporating correct edits from the second (American) and the third
outputs (the Nobel) to produce the final, corrected output.

Our contributions can be summarized as follows. (1) We demonstrate that a
small LMCor model with only 250M parameters can improve the performance of a
large-scale LM with 62B parameters on a variety of natural language generation
tasks. Remarkably, LMCor achieves this without access to the model weights as
it operates directly on the predictions through API calls. (2) We showcase
that the corrector is robust to different prompts, alleviating the need for
extensive prompt engineering. (3) We demonstrate the versatility of our
approach showing that a single corrector can be effortlessly applied to
different LMs as a plug-and-play module during inference. (4) Our analysis
reveals that scaling the corrector can provide additional gains over standard
in-context learning.

Figure 2: Potential of ranking (oracle-rank) and combining (oracle-combine)
sampled candidates (k=10) from PaLM models of different scales for GEC.

##  2 Correcting the outputs of LLMs

In this section we present our approach that utilizes a small model, LMCor to
adapt the predictions of an LLM for a specific task. Our pipeline is designed
to be computationally efficient requiring only modest computational resources.
Furthermore, our approach does not require access to the weights of the LLM.
Instead, we interact with the LLM only through an API, as is the case for some
state-of-the-art commercial LLMs.

Our approach is based on the insight that the outputs of the LLM can be
improved with optimal editing and combinations. To illustrate this we conduct
experiments on the task of grammatical error correction (GEC) Ng et al. (2014)
using PaLM models Chowdhery et al. (2022) of varying size, depicted in Figure
2. We compare the outputs generated by the LLM, when prompted with the
description and demonstrations of the task, with those of a state-of-the-art
GEC model (11B parameters) that is pretrained on synthetic tasks and fine-
tuned on multiple GEC datasets Rothe et al. (2021). Surprisingly, even the
largest PaLM model (540B parameters) undeperforms compared to the smaller
task-specific model (sota). However, by sampling multiple times (10) from the
model, we generate a pool of candidates that includes high-quality outputs to
which the model does not assign high likelihood due to the absence of task-
specific training. To demonstrate this we employ an oracle that selects the
candidate closest to the target sentence (oracle-rank), resulting in a
significant boost in performance across all model scales. Furthermore, by
optimally combining the correct spans from the sampled candidates using
another oracle (oracle-combine111 For the oracle-combine we compute the
differing spans between the candidates and for each span we choose the one
that has the smallest edit distance with the target.) we achieve further
improvements. Specifically, the smallest model (PaLM-8B) performs on par with
the state-of-the-art while the larger models (PaLM-62/540B) surpass its
performance. This finding highlights the potential of leveraging multiple
generations through ranking or combinations to enhance the performance of the
LLM for a specific task.

Motivated by the above finding we train a smaller model to optimally edit and
combine the candidates generated by the LLM. As depicted in Figure 1 the
corrector module is trained to predict the target sentence given the original
input sequence and multiple candidates provided by the LLM . In this way,
LMCor learns to select the most promising among the generated outputs, combine
the different candidates and even edit them in order to compose the target
sentence.

#### Generating the candidates

Our pipeline starts with generating predictions from a large LM via in-context
learning. Specifically, we prompt the model with a verbal description of the
task and a handful of demonstrations to generate predictions for each sample
of the dataset. We generate k outputs from the model with temperature sampling
to obtain diverse candidates:

$c_{i}=p(y|x,d,e)\;\;\;\forall i=1,2,..,k\ $ (1)

where x is the source sentence, $d$ is the task description and $e$ are the
task examples.

#### Correcting the candidates

After the generation of the candidates, we combine them, via concatenation, to
create the input data for the corrector. The corrector receives the source
sentence and generated candidates as input to produce a refined output:

$\hat{y}=p(y|x,C)\;,\;\;C=c_{1},c_{2},...,c_{k}\ $ (2)

We use a small, pretrained LM for the corrector, that we fine-tune on the
augmented dataset for a few steps. During inference, we sample again from the
LLM and subsequently feed the generated outputs to the corrector to produce
the final, corrected output. We show that even a small corrector can improve
the performance of standard fine-tuning and in context learning, while
demonstrating robustness against different prompts.

##  3 Experiments & Results

###  3.1 Datasets and Models

We evaluate the LM-corrector on three text generation tasks: Grammatical Error
Correction (GEC) on CoNLL-14 Ng et al. (2014), summarization on XSum Narayan
et al. (2018), and data-to-text on E2E NLG Novikova et al. (2017); Dušek et
al. (2019).

In all our experiments, we use the 62B version of PaLM Chowdhery et al. (2022)
as our large LM except for Section 4.2 where we vary the size of the LLM up to
540B parameters. We prompt PaLM with a task description and a number of
demonstrations randomly selected from the respective training corpus. We
sample k = 4 times from PaLM with a temperature of 0.7. We use
T5.1.1222https://github.com/google-research/text-to-text-transfer-
transformer/blob/main/released_checkpoints.md base Raffel et al. (2020) (250M
parameters) as our model both for the LMCor and the standard fine-tuning
baseline. We train the LMCor on the task-specific dataset with candidates
sampled from the LLM. We also provide the greedy-decoded output as a candidate
since preliminary results showed that it improves performance. We choose the
model based on the performance on the validation set. The outputs of the
corrector and baselines are generated via beam search with beam size 5.

We compare our approach, LMCor, with the following baselines: 1) in-context
learning using PaLM-62B (ICL), prompted with the same number of
demonstrations, and 2) standard fine-tuning with a T5-base and PaLM. We also
provide scores for 3) the reranking approach of Suzgun et al. (2022a) where
they sample multiple times from the LLM and use an alignment function combined
with Minimum Bayes Risk Decoding (MBRD) to select one among the sampled
candidates. We use the same sampled candidates that are fed to the corrector
and use Sim-LCS, a lexical similarity function based on longest common
subsequence which achieved the best results across tasks among the alignment
functions. We additionally provide the scores of an oracle reranker that
selects the candidate with the smallest edit distance compared to the target
as an upper-bound of reranking methods. Finally, we provide the results for a
version of our approach that feeds the corrector with only the greedy-decoded
candidate (single).

###  3.2 Grammatical Error Correction

Grammatical Error Correction (GEC) is a text-to-text task that requires
correcting the grammatical errors while applying minimal changes to the
original input sentence. Despite being trained on vast amounts of text, LLMs
have been demonstrated to underperform task-specific models in this task
Yasunaga et al. (2021); Suzgun et al. (2022b).

We use the CoNLL-14 Ng et al. (2014) dataset as our testset. Following
previous work Rothe et al. (2021), we use the combination of the FCE
Yannakoudakis et al. (2011) and W&I Bryant et al. (2019) datasets (60k
examples) for training and validation. We report $F_{0.5}$ scores obtained
with the MaxMatch scorer Dahlmeier and Ng
(2012)333https://www.comp.nus.edu.sg/ nlp/conll14st.html. We use 5
demonstrations in the LLM prompt in order to generate the candidates during
training and inference.

| CoNLL-14
---|---
Model | F0.5
T5-base | 59.38
PaLM-62B (ICL) | 59.92
\+ MBRD-Sim-LCS | 58.87
\+ Oracle Re-ranker | 63.88
\+ LMCor (single) | 62.47
\+ LMCor (mult.) | 62.48
Table 1: Results of our approach in GEC (CoNLL-14). The first group indicates
fine-tuned models, the second group in-context learning with reranking and the
final group provides the scores for LMCor. The best scores are in bold and the
second best ones are underlined.

The results presented in Table 1 show that standard fine-tuning and in-context
learning exhibit comparable performance in GEC while our approach
significantly outperforms both, by 3 and 2.5 $F_{0.5}$ points respectively. It
is worth noting that MBRD does not yield any improvement over standard few-
shot prompting. However, as expected the use of an oracle to rank the produced
hypotheses results in a considerable performance boost. Although LMCor does
not surpass the performance of the oracle it manages to significantly close
the gap, demonstrating the ability of the corrector to identify high-quality
candidates from the model.

Additionally, we compare the performance of standard fine-tuning and LMCor as
a function of the number of training instances. As we can observe in Figure 3
our approach consistently outperforms the baseline across all dataset sizes.
The gap in performance is particularly pronounced when the training dataset is
limited, consisting of only 1k examples, resulting in a substantial difference
of 15 points in $F0.5$. We attribute the sample efficiency of LMCor to the
ability of the corrector module to leverage external information, in the form
of LLM-generated candidates, to produce more accurate outputs. We note that in
this low-resource scenario the scores for both models are below those achieved
by few-shot learning. This outcome is expected given the LLM’s pretraining
that is suitable for off-the-shelf grammatical error correction. As the
dataset increases to 10k examples we observe that LMCor performs on par with
the LLM while the baseline continues to underperform. Beyond this threshold,
LMCor surpasses both in-context learning and fine-tuning by utilizing both the
training data and the candidates.

Figure 3: The effect of dataset size for standard-finetunig and LMCor. Results
are reported on GEC.

###  3.3 Data-to-text

The next task we evaluate on is E2E NLG Novikova et al. (2017), a data-to-text
task where the input is a number of key-value pairs about a restaurant and the
output is a short description of the restaurant in natural language. We use
the cleaned version of the dataset, E2E NLG (cleaned) Dušek et al. (2019) and
the default splits for training (35k examples), validation and testing. We use
5 demonstrations to produce the candidates for the corrector both during
training and infrence. We report ROUGE-2 and ROUGE-L Lin (2004) scores.

Table 2 presents a comparison between standard fine-tuning with a T5-base
model, in-context learning, and fine-tuning using the much larger PaLM models.
Notably, standard fine-tuning with a T5-base significantly outperforms in-
context learning and achieves results comparable to fine-tuning with the much
larger PaLM models444Chowdhery et al. (2022) attribute the mediocre
performance of the fine-tuned PaLM models to the small dataset size and the
‘significant mismatch with the pre-training corpus’.. While reranking
techniques improve the performance of few-shot prompting, even the oracle
approach falls short of the performance achieved by the fine-tuned T5-base
model. This highlights the primary limitation of reranking approaches,
particularly for challenging tasks for LLMs, where their upper bound solely
depends on the quality of candidates.

In contrast, the performance of LMCor does not exclusively rely on the quality
of the candidates. The corrector module has the ability to edit the LLM-
generated candidates, leading to more accurate outputs. As a result, LMCor
demonstrates the best overall performance for the E2E task, surpassing even
the fine-tuned PaLM-540B model by 1 point in ROUGE-L. An important
characteristic of our approach is the ability of the corrector to observe
multiple candidates for a single input. This enables LMCor to combine
candidates in order to compose a more refined answer. This is supported by the
performance discrepancy between LMCor (single) and LMCor (mult.) highlighting
the effectiveness of leveraging multiple candidates simultaneously.

| E2E NLG
---|---
Model | R-2 | R-L
T5-base | 45.3 | 52.8
PaLM-62B* (FT) | 45.2 | –
PaLM-540B* (FT) | 45.3 | 52.3
PaLM-62B (ICL) | 35.1 | 45.6
\+ MBRD-Sim-LCS | 35.7 | 46.2
\+ Oracle Re-ranker | 37.1 | 50.4
\+ LMCor (single) | 44.8 | 52.8
\+ LMCor (mult.) | 45.6 | 53.4
Table 2: Results of our approach in E2E NLG (cleaned). Results with * are
reported from the original paper Chowdhery et al. (2022). The first group
indicates fine-tuned models, the second group in-context learning with
reranking and the final group provides the scores for LMCor. The best scores
are in bold and the second best ones are underlined.

###  3.4 Summarization

| XSum
---|---
Model | R-1 | R-2 | R-L
T5-base | 38.64 | 16.98 | 31.41
PaLM-62B* (FT) | – | 18.5 | –
PaLM-540B* (FT) | – | 21.2 | 36.5
PaLM-62B (ICL) | 28.18 | 10.50 | 22.38
PaLM-540B (ICL) | 29.88 | 11.75 | 23.83
\+ LMCor (single) | 36.98 | 16.41 | 30.20
\+ LMCor (mult.) | 37.62 | 16.50 | 30.67
Table 3: Results of our approach on XSum. Results with * are reported from the
original paper Chowdhery et al. (2022). The first group indicates fine-tuned
models, the second group in-context learning and the final group provides the
scores for LMCor. The best scores are in bold and the second best ones are
underlined.

The final task that we consider is abstractive summarization. Specifically, we
use XSum Narayan et al. (2018) as our dataset with the default train (204k
examples), validation and test splits. Due to the length of the articles we
truncate the inputs and only use 1 demonstration to prompt the LLM. To handle
the increased sequence length in the input of the corrector we again truncate
the articles and use a maximum sequence length of 2048 tokens. We report
ROUGE-1, ROUGE-2 and ROUGE-L scores.

The results of Table 3 reveal that standard fine-tuning outperforms in-context
learning for the XSum dataset. Specifically, the fine-tuned T5 and PaLM-62B
model outperform in-context learning by 15 and 18 points in ROUGE-2
respectively. The difficulty of the task, which involves summarizing an
article into a single sentence, poses challenges for in-context learning,
while the substantial dataset size of 204k examples favors fine-tuning.
However, the use of a corrector module leads to singificant improvements over
in-context learning with the 62B PaLM model, resulting in performance gains of
6 points in ROUGE-2 and 8 points in ROUGE-L. Notably, LMCor outperforms few-
shot learning with the largest 540B PaLM model. Again, the use of multiple
candidates by the corrector further enhances the performance of LMCor by
increasing the likelihood of discovering a more accurate candidate from the
LLM generations.

On the other hand, the LMCor performs slightly worse than standard fine-
tuning. This result is somewhat unexpected since the input for the corrector
is a strict superset of the model input in the fine-tuning setting. We
attribute the slight drop in performance to the poor quality of the provided
candidates, which introduces undesirable noise to the corrector’s input.
Additionally, the size of the dataset might also contribute to this
discrepancy, as observed in Section 3.2 where we showed that the gap between
standard fine-tuning and LMCor could potentially decrease for very large
datasets.

##  4 Robustness Analysis

###  4.1 Different prompts

We have demonstrated that LMCor enhances the performance of LLMs by refining
their generated predictions. However, the few-shot paradigm remains appealing
since it does not require training or access to a dataset, except for a
handful of examples used in the prompt. While prompt construction is not
computationally intensive, there is no consensus on the optimal selection,
number and even order of examples in the prompt which can lead to variations
in predictions and significantly impact LLM performance Lu et al. (2022). To
investigate whether LMCor is similarly affected by this variability we use
three different sets of 5 demonstrations to prompt the LLM for the task of
GEC. We then feed the generated predictions as input to the corrector. It is
important to note that we trained LMCor only once using the candidates
generated with the original set of demonstrations (set 1) and simply swapped
candidates during inference.

Model | set 1 | set 2 | set 3 | mean | std
---|---|---|---|---|---
PaLM-62B | 59.9 | 58.9 | 56.2 | 58.6 | 1.9
\+ LMCor | 62.5 | 62.3 | 62.9 | 62.6 | 0.3
Table 4: Mean $F_{0.5}$ score and standard deviation (std) using different
sets of demonstrations for in-context learning vs. our approach for GEC.

Table 4 highlights the significant variance in LLM performance depending on
the selection of demonstrations with a difference of $3.7$ between the highest
and the lowest $F_{0.5}$ score. In contrast, LMCor remains unaffected and
achieves competitive performance even when the quality of candidates
significantly deteriorates (set 3). This demonstrates the capability of LMCor
to compensate for candidates of poor quality by performing edits on them. The
robustness of the corrector against prompts of varying quality (with a
variance of $0.3$ compared to $1.9$ for the LLM) suggests that it can mitigate
the need for extensive prompt engineering.

###  4.2 Different LLMs

In the previous experiment we demonstrated the robustness of the corrector
against candidates of varying quality. In this set of experiments we further
examine the robustness of our approach by testing whether a corrector can be
used interchangeably with different LMs without retraining. It is important to
note that we only trained the corrector once and performed inference by
swapping the LLM responsible for generating the candidates.

Initially, we focus on LMs from the same family of models, namely PaLM
Chowdhery et al. (2022), which share similar architectures and training data
but vary in the number of parameters. Table 5 presents the results of applying
the corrector to different PaLM models than the one it was originally trained
on (62B). Across all scales, the LMCor consistently outperforms standard fine-
tuning and in-context learning, with the exception of the 540B model where the
performance is comparable. The corrector achieves significant gains when
applied to the 8B PaLM model, with an improvement of +13 points in $F_{0.5}$.
This further highlights the ability of LMCor to compensate for low-quality
candidates by ranking and rewriting in order to achieve competitive
performance. Furtherhmore, we observe that using a single candidate for LMCor
leads to inferior performance in all cases, except for the 62B PaLM model.
This finding suggests that the existence of diverse candidates prevents the
model from overfitting to the outputs of a specific LLM, thereby enhancing its
generalization capabilities.

T5-base | 59.38
---|---
PaLM (ICL) | 8B | 62B | 540B
48.62 | 59.92 | 65.37
\+ LMCor (single) | 61.40 | 62.48 | 63.55
\+ LMCor (mult.) | 61.89 | 62.47 | 65.16
Table 5: Results ($F_{0.5}$) of applying the corrector to LMs of different
scale during inference for GEC.

As a next step we exploret the application of LMCor to an LLM from a distinct
family of models, specifically Codex555We use the outputs of code-davinci-002
provided by Suzgun et al. (2022a). which is a GPT3-like model trained on code
Chen et al. (2021). To compare the effectiveness of LMCor with MRBD reranking
we utilize BLEURT Sellam et al. (2020) as the alignment function, as it has
been reported to achieve the highest scores for the E2E NLG task Suzgun et al.
(2022b). It is important to note that while the reranking approach samples 16
outputs from the LLM, we only use 5 for the corrector. The results in Table 6
demonstrate the superior performance of LMCor over reranking. In particular we
observe a performance boost of 10 points in ROUGE-2 when compared to in-
context learning, whereas MBRD achieves a mere 2-point improvement.

| E2E NLG (clean)
---|---
Model | R-2 | R-L
GPT3-Codex (ICL)* | 34.2 | 44.4
\+ MBRD-BLEURT* | 36.4 | 46.5
\+ LMCor (mult.) | 44.8 | 53.0
Table 6: Applying the corrector to different family of LMs during inference
for E2E NLG. *: the results as reported in the original paper.

The previous findings highlight the remarkable out-of-domain robustness of
LMCor and its ability to seamlessly integrate with various LMs, presenting a
versatile solution for enhancing their performance.

##  5 Analysis

###  5.1 Importance of the source

The input of the corrector consists of the source sentence and a number of
candidates generated by the LLM (Equation 2). In the previous section, we
demonstrated the effect of using multiple candidates in improved performance
and out-of-domain robustness. In this experiment, we focus on the importance
of the source sentence to LMCor. To examine this, we train a corrector that
has receives only the candidates as input, without access to the source. The
results for E2E NLG, presented in Table 7 reveal a noticeable decline in
performance when the source sentence is removed. This decrease can be
attributed to the inability of the corrector to select or successfully edit
candidates that are faithful to the input in the absence of the source
sentence.

| E2E NLG (clean)
---|---
Model | R-2 | R-L
PaLM-62B (ICL) | 35.1 | 45.6
\+ LMCor | 45.6 | 53.4
\- source sentence | 44.5 | 53.1
Table 7: The importance of the source sentence for the corrector.

###  5.2 Scaling the Corrector

We showed that a corrector model with 250M parameters, can effectively refine
the predictions of an LLM for a specific task. This raises the question: is
training a corrector still valuable if we have the computational resources to
train a very large model? To investigate this we train the largest version of
T5, T5-xxl with 11B parameters, both through standard fine-tuning and as a
LMCor for GEC. We note that, in this scenario, the sizes of the fine-tuned
model and the LLM are comparable (11B vs. 62B parameters).

Figure 4: The effect of scaling for LMCor and fine-tuning. Results are
reported on GEC.

As shown in Figure 4 both LMCor and the fine-tuned T5 benefit from scaling,
exhibiting higher $F0.5$ scores as their parameter count increases from 250
million to 11 billion. At the 11 billion scale, both models significantly
outperfrom in-context learning with the 62B PaLM model. The corrector
continues to outperform the baseline as model scale increases, although the
gap in performance narrows from 3 to 0.5 points in $F0.5$. We attribute this
reduction to the enhanced competence of the larger T5 model. At the scale of
11 billion parameters, the scores obtained by the T5 model alone surpass those
obtained by the LLM, indicating that the LLM-generated outputs are of lower
quality compared to the ones that the model would produce independently.
Therefore, while the additional input to the model remains beneficial, its
impact diminished to same extent, considering the performance disparity
between T5 and PaLM.

##  6 Related Work

Large language models have been the focal point of NLP research over the past
few years Brown et al. (2020a); Thoppilan et al. (2022); Black et al. (2022);
Zhang et al. (2022); Chowdhery et al. (2022). Their ability to perform in-
context learning when prompted, combined with their sheer scale has shifted
the paradigm away from fine-tuning.

Due to this paradigm shift, previous works focused on improving the few-shot
performance of LLMs. One line of work suggests prompting the model to generate
rationales aka chain-of-thoughts Nye et al. (2021); Wei et al. (2022); Kojima
et al. (2022) or break down the problem into simpler ones Press et al. (2022);
Zhou et al. (2023); Pilault et al. (2023), imitating a reasoning process
before generating the answer. Such prompting techniques are complementary to
our approach and can be used to provide better candidates to the corrector.

Another way to improve the performance of LLMs without training is reranking,
i.e. selecting the most promising from a pool of candidates obtained by
sampling from the model. Reranking approaches include training different
models as the ranker Cobbe et al. (2021), using task-specific ranking
functions Suzgun et al. (2022b), majority voting Wang et al. (2023b) or
minimum Bayes risk decoding (MBRD) Suzgun et al. (2022a). Although these
approaches can improve the few-shot performance of the LLM, they are upper-
bounded by the quality of the generated candidates.

Despite the massive compute requirements, there are cases in the literature
where LLMs have been fine-tuned to further boost performance Chowdhery et al.
(2022); Cobbe et al. (2021); Chung et al. (2022). However, due to the amount
of compute required to update models of such scale, parameter-efficient fine-
tuning methods (PEFT) have been proposed. These approaches introduce a small
number of additional parameters (relative to the full model) to be trained
while the rest of the model is frozen. The newly-added parameters can come in
the form of embeddings that are appended to the encoded sequence Li and Liang
(2021); Lester et al. (2021), MLPs that are added in-between layers, namely
adapters Houlsby et al. (2019) or rank-decomposition matrices that are added
in parallel to the existing layers Hu et al. (2022); Zhang et al. (2023b).
Although these works decrease the number of FLOPs needed for fine-tuning they
still require loading and backpropagating through the model, which can be
prohibitive for LLMs. Our work shares the same motivation with PEFT methods,
where in our case the introduced parameters are essentially another, smaller
model. However, our method does not have the memory requirements of PEFT since
the corrector operates directly on the model’s outputs and does not require
access to the model’s weights.

A line of work that proposes to adapt an LLM without updating the model’s
weights is derivative-free prompt learning Black et al. (2022); Chai et al.
(2022). Similar to prefix tuning Li and Liang (2021), this approach prepends a
number of soft prompt embeddings to the frozen model along with the standard
prompt. However, instead of performing back-propagation they use an objective
function to determine the fitness of the continuous prompt and use
evolutionary algorithms to iteratively optimize the embeddings. Although these
approaches are advertised as black-box, in practice, none of the existing APIs
provide the functionality to feed continuous embeddings to the LLM.

The interaction between LLMs and smaller task-specific models has been also
explored in the literature. The ability of LLMs to generate reasonable outputs
without any training can be used for data augmentation. The synthetic data can
be in the form of pseudo-labeled data Dai et al. (2023), external knowledge Yu
et al. (2023) or rationales Liu et al. (2022b); Wang et al. (2023a). Similar
to our findings, these works demonstrate that smaller models greatly benefit
from the external knowledge and that scaling brings further gains.

Most relevant to our approach is the concurrent work by Welleck et al. (2023)
where they train a corrector to improve sequences generated by language models
based on a predetermined score function. In contrast to our method, they rely
on unlabeled data and sample numerous times from the LLM to obtain a large
pool of candidates. They assume the availability of a value function that
assigns scores to each candidate and create input-output pairs by sorting
candidates based on their scores. Unlike their approach, we demonstrate that a
compact corrector can perform effectively across various tasks. One key
difference is that in our setting, the corrector has the ability to process
multiple candidates simultaneously, eliminating the need for multiple passes.
This characteristic of our approach significantly improves performance and
enhances the out-of-domain robustness of the corrector.

##  7 Conclusion

In this work, we introduced LMCor, a novel approach that leverages a small
corrector module to enhance the performance of LLMs for specific tasks. Unlike
parameter-efficient fine-tuning methods our approach does not require access
to the model or substantial computational resources. Our experiments
demonstrate that even a relatively small, 250M-parameter corrector can improve
the performance of a much larger LM with 62 billion parameters while
exhibiting robustness against different prompts. Furthermore, we showcase that
the corrector can be successfully applied to models of different scale or
architecture without any retraining. These findings offer a promising solution
for improving LLM performance in a practical and resource-efficient manner and
open up new possibilities for the utilization and deployment of LLMs in real-
world applications.

## References

  * Black et al. (2022) Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022.  GPT-NeoX-20B: An open-source autoregressive language model.  In _Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models_, pages 95–136.
  * Brown et al. (2020a) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020a.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901.
  * Brown et al. (2020b) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020b.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Bryant et al. (2019) Christopher Bryant, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. 2019\.  The BEA-2019 shared task on grammatical error correction.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 52–75.
  * Chai et al. (2022) Yekun Chai, Shuohuan Wang, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2022.  Clip-tuning: Towards derivative-free prompt learning with a mixture of rewards.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 108–117.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William H. Guss, Alex Nichol, Igor Babuschkin, S. Arun Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021.  Evaluating large language models trained on code.  _ArXiv_ , abs/2107.03374.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022\.  Palm: Scaling language modeling with pathways.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022.  Scaling instruction-finetuned language models.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021.  Training verifiers to solve math word problems.  _CoRR_ , abs/2110.14168.
  * Dahlmeier and Ng (2012) Daniel Dahlmeier and Hwee Tou Ng. 2012.  Better evaluation for grammatical error correction.  In _Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 568–572.
  * Dai et al. (2023) Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith Hall, and Ming-Wei Chang. 2023.  Promptagator: Few-shot dense retrieval from 8 examples.  In _International Conference on Learning Representations_.
  * Dušek et al. (2019) Ondřej Dušek, David M. Howcroft, and Verena Rieser. 2019.  Semantic noise matters for neural natural language generation.  In _Proceedings of the 12th International Conference on Natural Language Generation_ , pages 421–426.
  * Fu et al. (2023) Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023.  Specializing smaller language models towards multi-step reasoning.
  * Gao et al. (2021) Tianyu Gao, Adam Fisch, and Danqi Chen. 2021.  Making pre-trained language models better few-shot learners.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3816–3830.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for NLP.  In _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 2790–2799.
  * Hu et al. (2022) Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_.
  * Jiang et al. (2021) Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021.  How can we know when language models know? on the calibration of language models for question answering.  _Transactions of the Association for Computational Linguistics_ , 9:962–977.
  * Karimi Mahabadi et al. (2021) Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021.  Compacter: Efficient low-rank hypercomplex adapter layers.  _Advances in Neural Information Processing Systems_ , 34:1022–1035.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  In _ICML 2022 Workshop on Knowledge Retrieval and Language Models_.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81.
  * Liu et al. (2022a) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022a.  What makes good in-context examples for GPT-3? In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_ , pages 100–114.
  * Liu et al. (2022b) Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, and Hannaneh Hajishirzi. 2022b.  Generated knowledge prompting for commonsense reasoning.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3154–3169.
  * Lu et al. (2022) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022\.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8086–8098.
  * Narayan et al. (2018) Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  _arXiv preprint arXiv:1808.08745_.
  * Ng et al. (2014) Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014.  The CoNLL-2014 shared task on grammatical error correction.  In _Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task_ , pages 1–14.
  * Novikova et al. (2017) Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017.  The E2E dataset: New challenges for end-to-end generation.  In _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 201–206.
  * Nye et al. (2021) Maxwell I. Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021.  Show your work: Scratchpads for intermediate computation with language models.  _CoRR_ , abs/2112.00114.
  * Pilault et al. (2023) Jonathan Pilault, Xavier Garcia, Arthur Bražinskas, and Orhan Firat. 2023\.  Interactive-chain-prompting: Ambiguity resolution for crosslingual conditional generation with interaction.  _arXiv preprint arXiv:2301.10309_.
  * Press et al. (2022) Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2022.  Measuring and narrowing the compositionality gap in language models.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(1).
  * Rothe et al. (2021) Sascha Rothe, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. 2021.  A simple recipe for multilingual grammatical error correction.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 702–707.
  * Schick and Schütze (2021) Timo Schick and Hinrich Schütze. 2021.  It’s not just size that matters: Small language models are also few-shot learners.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2339–2352.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892.
  * Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020.  AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4222–4235.
  * Suzgun et al. (2022a) Mirac Suzgun, Luke Melas-Kyriazi, and Dan Jurafsky. 2022a.  Follow the wisdom of the crowd: Effective text generation via minimum bayes risk decoding.
  * Suzgun et al. (2022b) Mirac Suzgun, Luke Melas-Kyriazi, and Dan Jurafsky. 2022b.  Prompt-and-rerank: A method for zero-shot and few-shot arbitrary textual style transfer with small language models.  In _arXiv_.
  * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Kathleen S. Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed H. Chi, and Quoc Le. 2022.  Lamda: Language models for dialog applications.  _CoRR_ , abs/2201.08239.
  * Wang et al. (2023a) PeiFeng Wang, Aaron Chan, Filip Ilievski, Muhao Chen, and Xiang Ren. 2023a.  PINTO: Faithful language reasoning using prompted-generated rationales.  In _International Conference on Learning Representations_.
  * Wang et al. (2023b) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023b.  Self-consistency improves chain of thought reasoning in language models.  In _International Conference on Learning Representations_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  In _Advances in Neural Information Processing Systems_.
  * Welleck et al. (2023) Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2023.  Generating sequences by learning to self-correct.  In _The Eleventh International Conference on Learning Representations_.
  * Yannakoudakis et al. (2011) Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011.  A new dataset and method for automatically grading ESOL texts.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 180–189.
  * Yasunaga et al. (2021) Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2021.  LM-critic: Language models for unsupervised grammatical error correction.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7752–7763.
  * Yu et al. (2023) Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023.  Generate rather than retrieve: Large language models are strong context generators.  In _International Conference on Learning Representations_.
  * Zhang et al. (2023a) Biao Zhang, Barry Haddow, and Alexandra Birch. 2023a.  Prompting large language model for machine translation: A case study.  _arXiv preprint arXiv:2301.07069_.
  * Zhang et al. (2023b) Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Jiao Qiao. 2023b.  Llama-adapter: Efficient fine-tuning of language models with zero-init attention.  _ArXiv_ , abs/2303.16199.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022.  Opt: Open pre-trained transformer language models.
  * Zhou et al. (2023) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. 2023.  Least-to-most prompting enables complex reasoning in large language models.  In _The Eleventh International Conference on Learning Representations_.


# Real-Time Implementation of Dynamic State Estimation
for Microgrid Load Bus Protection

Sarbajit Basu 1*  —, Arthur K. Barnes 2*  —, Adam Mate 2*  —, and Olga Lavrova
1*  — Manuscript submitted: Mar. 3, 2023. Current version: Apr. 21, 2023. U.S.
Government work not protected by U.S. copyright. 1 The authors are with the
Klipsch School of Electrical and Computer Engineering at New Mexico State
University, Las Cruces, NM 88003 USA. Email<EMAIL_ADDRESS><EMAIL_ADDRESS>The authors are with the Analytics, Intelligence, and
Technology Division at Los Alamos National Laboratory, Los Alamos, NM 87545
USA.
Email<EMAIL_ADDRESS><EMAIL_ADDRESS>Approved for public
release; distribution is unlimited.

###### Abstract

Inverter-interfaced microgrids, owing to the lack of fault current, cannot be
protected using traditional over-current protections, while admittance or
differential relaying protection schemes are not practical to be implemented.
Dynamic state estimation can track and predict power system transients and has
been extensively investigated for setting-less protection. A novel real-time
application of dynamic state estimation for protection is proposed in this
paper, wherein parameter estimation and parallel processing is used to
identify the state of the system. The implementation scheme has low process
complexity and employs a data acquisition device and estimator that run on a
general-purpose computer. This proposed implementation extends the state-of-
the-art, under short-circuit conditions, to a real-time implementation with a
lumped-load radial microgrid and a grid-forming inverter with current-limiting
behavior.

###### Index Terms:

power system operation, microgrid, distribution network, protection, dynamic
state estimation.

##  I Introduction

Owing to their smaller carbon footprint, renewable resource-based generation –
often inverter-interfaced – is becoming increasingly prevalent. However, due
to their rapid current-limiting behavior, these resources requires novel
protection schemes. Traditional protection schemes are unable to detect faults
on microgrids, require complex coordination schemes among relays, and are
susceptible to misoperation owing to erroneous values provided by faulty
measurement devices [1, 2]. As inverter-dominated microgrids are unable to
supply high currents during faulted conditions, the inverters shut down during
faults. Additionally, bi-directional flow of current during switching over
from grid-connected to islanded mode can make fault detection difficult [3,
4].

Dynamic state estimation (DSE), coupled with high accuracy GPS-synchronized
measurements, can accurately track system dynamics and provide details about
system operating characteristics. The robustness, combined with the relatively
low time complexity of DSE, has paved the way to explore its real-time
applications for control and protection: DSE has been explored for real-time
parameter identification to develop dynamic load models and load model
behaviors [1, 5, 6, 7]; furthermore, DSE has been implemented as setting-less
protection scheme, using real-time measurement data, for the protection of
microgrids [8, 9].

DSE has also been explored for other setting-less protection applications on
both transmission and distribution networks [10, 11, 12, 13]; it promises to
eliminate or mitigate the protection issues described earlier. Owing to these
advantages, DSE performs better in protection of transmission lines,
transformers, and capacitor banks compared to distance relaying or other
traditional protection methods [14, 15]. Finally, DSE-based protection
schemes, with access to only the voltage and current measurements on terminals
of protected devices, has been investigated for the protection of microgrids
while identifying hidden failures in substations [16, 17].

The complexities of real-time state estimation, however, are several,
including but not limited to the availability of secure and reliable channels
[18], the volume of data to be considered for estimation [19], and the pre-
processing, handling, and storage of data received from the measurement
devices [20].

This paper introduces a real-time implementation of a state-of-the-art DSE
algorithm for adaptive load bus protection in an inverter-interfaced
microgrid. Earlier work focused on ideal voltage sources [21], whereas this
uses an inverter model with current-limiting behavior [22] that feeds a
balanced passive load. Most work on DSE considers one load model and
distinguishes the normal operational condition from the observed conditions
based on the chi-squared confidence; in this work, multiple load models –
corresponding to different operational conditions – are designed, which
clearly designate true operational state, making this implementation more
robust. A developed orchestrator module identifies the relevant operational
conditions from estimated system parameters, which in turn are applied to trip
the appropriate breakers. All of these were implemented on a general-purpose
computer, and experimental results verify that the proposed method accurately
identifies the correct model with low time delay and computational complexity.

##  II Problem Formulation for DSE

###  II-A Grounded-Wye Load with Three-Phase Fault

The problem formulation for the simplified grounded-wye-connected RL load with
a three-phase fault, displayed on Fig. 1, is described below. For other load
configurations, refer to [21].

The terminal equations for the case:

$\displaystyle v_{a}(t)=v_{f}(t)\quad v_{b}(t)=v_{f}(t)\quad
v_{c}(t)=v_{f}(t)\quad$ (1a) $\displaystyle i_{a}(t)=G_{f}v_{ra}(t)\quad
i_{b}(t)=G_{f}v_{rb}(t)\quad i_{c}(t)=G_{f}v_{rc}(t)\quad$ (1b)

The output and state of the system, respectively:

$\displaystyle y$
$\displaystyle=\begin{bmatrix}v_{a}(t)&v_{b}(t)&v_{c}(t)&i_{a}(t)&i_{b}(t)&i_{c}(t)\end{bmatrix}^{T}$
(2a) $\displaystyle x$
$\displaystyle=\begin{bmatrix}G_{f}&v_{ra}(t)&v_{rb}(t)&v_{rc}(t)\end{bmatrix}^{T}$
(2b)

Assuming that the signals are sampled at timesteps $n\in\\{1,2,\ldots,N\\}$
with sample time $\Delta t$, the discrete-time state-output mapping function,
$h(x)$, is therefore:

$\forall n\in\\{1,\ldots,N\\}$

$\displaystyle\begin{split}h_{n}(x)&=v_{a}(n)=v_{ra}(n)\\\
h_{n+N}(x)&=v_{b}(n)=v_{rb}(n)\\\ h_{n+2N}(x)&=v_{c}(n)=v_{rc}(n)\\\
h_{n+3N}(x)&=i_{a}(n)=G_{f}v_{ra}(n)\\\
h_{n+4N}(x)&=i_{b}(n)=G_{f}v_{rb}(n)\\\
h_{n+5N}(x)&=i_{c}(n)=G_{f}v_{rc}(n)\\\ \end{split}$ (3)

The Jacobian of the system, $\mathbf{H}$, is therefore:

$\forall n\in\\{1,\ldots,N\\}$

$\displaystyle\begin{split}\frac{\partial v_{a}(n)}{\partial
v_{ra}(n)}&=1\qquad\frac{\partial v_{b}(n)}{\partial
v_{rb}(n)}=1\qquad\frac{\partial v_{c}(n)}{\partial v_{rc}(n)}=1\\\
\frac{\partial i_{a}(n)}{\partial G_{f}}&=v_{ra}(n)\qquad\frac{\partial
i_{b}(n)}{\partial v_{rb}(n)}=G_{f}\qquad\frac{\partial i_{c}(n)}{\partial
G_{f}}=v_{rc}(n)\\\ \frac{\partial i_{a}(n)}{\partial
v_{ra}(n)}&=G_{f}\qquad\frac{\partial i_{b}(n)}{\partial
G_{f}}=v_{rb}(n)\qquad\frac{\partial i_{c}(n)}{\partial
v_{rc}(n)}=G_{f}\end{split}$ (4)

$\displaystyle\frac{\partial v_{a}(n)}{\partial v_{ra}(n)}$
$\displaystyle=1\qquad\frac{\partial v_{b}(n)}{\partial
v_{rb}(n)}=1\qquad\frac{\partial v_{c}(n)}{\partial v_{rc}(n)}=1$
$\displaystyle\frac{\partial i_{a}(n)}{\partial v_{ra}(n)}$
$\displaystyle=G_{f}\qquad\frac{\partial i_{b}(n)}{\partial
v_{rb}(n)}=G_{f}\qquad\frac{\partial i_{c}(n)}{\partial v_{rc}(n)}=G_{f}$
$\displaystyle\frac{\partial i_{a}(n)}{\partial G_{f}}$
$\displaystyle=v_{ra}(n)\qquad\frac{\partial i_{b}(n)}{\partial
G_{f}}=v_{rb}(n)\qquad\frac{\partial i_{c}(n)}{\partial G_{f}}=v_{rc}(n)$

Note that for simplicity of notation, the indexing of individual elements of
$\mathbf{H}$ is not presented.

Figure 1: Grounded-wye-connected RL load with a three-phase fault.

###  II-B DSE Solver Formulation

This process is repeated iteratively until either the maximum number of
iterations is reached or the algorithm has converged, indicated by the change
in the log of the squared error falling below a specified threshold:

$\Delta
J_{i}=\mathinner{\\!\left\lvert\log\mathinner{\\!\left\lvert\epsilon_{i}^{*}\cdot\epsilon_{i}\right\rvert}-\log\mathinner{\\!\left\lvert\epsilon_{i-1}^{*}\cdot\epsilon_{i-1}\right\rvert}\right\rvert}$
(5)

The measurement error test is performed as follows:

$p=F_{m-n}(J_{i})\geq 0.95$ (6)

where $F_{m-n}$ is the chi-squared cumulative distribution function for $m-n$
degrees of freedom, in which $m-n$ is the number of linearly independent
observables.

##  III Real-Time Implementation

The algorithm presented below was implemented as a collection of programs in
an interpreted language: separate programs to estimate each load model were
developed that run in parallel; these were executed on a general-purpose
computer running a POSIX operating system. The emphasis of the implementation
was on developing a system of communication that closely resembles industry
standards.

###  III-A Communication of Data

The load models were simulated offline and the measurements for different
operational conditions were stored in a comma-separated value (CSV) file. A
program was written in the Go language to emulate the behavior of an analog-
digital converter (ADC); this program reads data from the CSV files and
directs them to the STDIN channel of the orchestrator program, pausing after
each line for a time equal to the sample period. The measurements include
time-stamped voltage and current values, with data being sent line by line
from the CSV.

###  III-B Orchestrator

The orchestrator program receives and re-directs the measurements. Load
models, corresponding to different operational conditions, have been developed
in the C language. The orchestrator scans all the measurements from STDIN and
directs them to the different models, which then executed in parallel as
different processes. The communication is achieved by using two-way channels
via Unix pipes between the orchestrator and each model. Once each model was
run, the chi-squared confidence and estimated parameters are being returned to
the orchestrator, which then scans all values and selects the model with the
highest confidence; after this is completed, the corresponding protective
action is recommended and instructions are sent to the breakers to protect the
inverter and load.

Figure 2: Test system used for implementation.

1 Initialize BUFFER based on window size

2 while  _reading end of PIPE is non-empty_ do

3 Read measurements from PIPE

4 Write measurements to writing end of BUFFER

5 if  _BUFFER is full_ then

6 Run estimator

7 Calculate confidence

8 Write confidence to reading end of PIPE

9

10 Remove measurements for oldest timestamp from BUFFER

11

Algorithm 1 Model Implementation

1 Spawn subprocesses for each model

2 Create PIPEs for each model

3 while  _STDIN is non-empty_ do

4 Read measurements from STDIN

5 for  _each model[i]_ do

6 Write measurements to writing end of PIPE[i]

7 Read confidence for each model from reading end of PIPE[i]

8

9 Identify model with highest confidence

10 if  _new chosen model $=$ old chosen model_ then

11 $N\leftarrow N+1$

12

13 else

14 $N\leftarrow 0$

15

16 if  _N $>$ Hysteresis Samples_ then

17 Apply protection settings for model with highest confidence

18

19 else

20 Apply protection settings for previously chosen model

21

22

Algorithm 2 Orchestrator Implementation

##  IV Case Study and Experiments

To verify the implementation of the proposed method and demonstrate the use of
DSE in efficiently providing setting-less protection for microgrids, a system
with a balanced grounded-wye constant-impedance load was considered; Table I
summarizes the details of the system. The simulation was done in
MATLAB/Simulink®, using the Specialized Power Systems library on 64-bit MATLAB
R2019b® [23]. The load itself was connected to an inverter rated at 480
$V_{rms}$ line-line, through a 1000 ft of 1/0 AWG quadruplex overhead service
drop cable, as demonstrated in Fig. 2 (on Page 2).

TABLE I: Parameters for three-phase dynamic models Parameter | Values
---|---
Total Load Real Power | 10 kW
Total Load Reactive Power | 5kVAR
Load Resistance (per phase) | 18.432 $\Omega$
Load Inductance (per phase) | 24 mH
Fault Resistance | 1 m$\Omega$
Ground Resistance | 10 m$\Omega$

The modes of operation for the three-phase system investigated the following
scenarios: 1) Unfaulted; 2) Phase A line-ground fault; 3) Phase B line-ground
fault; 4) Phase C line-ground fault; 5) Phases A-B line-line fault; 6) Phases
B-C line-line fault; 7) Phases C-A line-line fault; and 8) Three-phase fault.
DSEs were devised corresponding to the modes of operation and run in parallel;
the current state of the system was indicated by the DSE with the lowest
error. The fault impedance for each line-ground fault case was 15 m$\Omega$
and the total fault impedance for each line-line fault case was 10 m$\Omega$.
For the Three-phase fault, the effective fault impedance was 15 m$\Omega$

Three different cases (Case I, Case II, and Case III) were devised, as
elaborated in Table II; their corresponding voltage and current variations are
demonstrated in Figs. 3–5. Each case started under the Unfaulted condition and
then went into a faulted condition. After that the orchestrator was tasked
with identifying the estimator model(s) that correspond to the different time
stamps associated with each case, thereby predicting the operational
conditions of the system.

TABLE II: Sequence of Events in Test Case | 0 – 0.25 sec | 0.25 – 0.5 sec
---|---|---
Case I. | Unfaulted | A-G fault
Case II. | Unfaulted | B-C fault
Case III. | Unfaulted | 3-$\phi$ fault
Figure 3: Plot of voltage and current for Case A. Figure 4: Plot of voltage
and current for Case B. Figure 5: Plot of voltage and current for Case C.

##  V Simulation Results

Results verify the orchestrator’s ability to be able to identify different
operating conditions; illustrated on Fig. 7, Fig. 9, and Fig. 11. The chi-
squared statistics – illustrated in Fig. 6, Fig. 8, and Fig. 10 – were always
considerably higher for the correct model than other models, thereby allowing
easy distinction; therefore, the orchestrator was able to identify the right
configuration and implement corresponding protection settings.

Figure 6: Chi-squared confidence for all fault models during Case I. Figure 7:
Fault models chosen by orchestrator during Case I. Figure 8: Chi-squared
confidence for all fault models during Case II. Figure 9: Fault models chosen
by orchestrator during Case II. Figure 10: Chi-squared confidence for all
fault models during Case III. Figure 11: Fault models chosen by orchestrator
during Case III.

For the DSE implementation, a sliding-window approach was used – as shown in
Fig. 12 – with 5 measurement samples considered for analysis to identify
system status; the samples were collected every 500 $\mu$s, comparable to that
of a PMU. A hysteresis of 5 samples were used to offset erroneous model
identifications during state transitions.

Figure 12: Sliding window implementation.

In Case I and Case II, the chi-squared confidences of relevant fault models
were significantly higher than the others after the orchestrators processed
measurements from post-fault conditions – i.e., 2.5 ms (5 samples) post-fault
– which is comparably faster than a conventional relay trip scheme. The
orchestrator, however, chose the correct fault model 9 samples later, owing to
the hysteresis implementation. There were no errors in the orchestrator’s
choice once all measurements corresponded to faulted conditions.

In Case III, however, the implementation became complicated, since during the
fault conditions several other models reported fairly high values of chi-
squared confidence. This took a toll on the orchestator implementation and the
model took significantly longer to identify the correct fault model; this was
a delay of about 20ms (38 samples). These spikes in chi-squared confidence for
incorrect models during the faulted conditions were of short duration, and
once the orchestrator identified the correct model, the hysteresis enabled
these false positives to not have an impact.

Lastly, when the transition between Unfaulted and faulted conditions passed
through the sliding window, there was a blackout period wherein the chi-
squared confidences were small for all the models, thereby causing the
orchestrator to not make a clear choice; the solution to this remains to be
addressed.

##  VI Conclusions

This paper investigated the implementation of a real-time application of DSE
for the protection of inverter-interfaced microgrids. The implementation,
based on parameter estimation to determine system state, examined parallel
processing of multiple models; DSE was able to identify the correct load
configurations for measurements corresponding to normal and different modes of
faulted operations.

The number of samples considered, along with the hysteresis, should also
impact the choice of model and should potentially solve the experienced
blackout issue. A delay in the availability of measurements, caused by channel
delays, would impact the overall implementation; the orchestrator needs to be
modified to address the same.

The impact of the ratio of the number of measurements to the number of states,
and the influence of window size on estimation efficiency is yet to be
addressed. To reduce model complexity, only balanced lumped radial loads were
considered; further work is to be performed to investigate the impact of load
imbalance on the quality of estimates.

The time and computational complexity of running several models in parallel
also require discussion, especially if measurements are received from a PMU
and the inverter feeds a critical load. Finally, the impact of noise on the
orchestrators performance has to be explored.

## References

  * [1] Y. Liu et al.  Dynamic State Estimation for Power System Control and Protection.  IEEE Trans. Power Syst., 36(6):5909–5921, May 2021.
  * [2] V. Telukunta et al.  Protection Challenges Under Bulk Penetration of Renewable Energy Resources in Power Systems.  CSEE Journal of Power and Energy Syst., 3:365–379, 2017.
  * [3] A. K. Barnes et al.  Optimization-Based Formulations for Short-Circuit Studies with Inverter-Interfaced Generation in PowerModelsProtection.jl.  MDPI Energies, 14(8):1–27, Apr. 2021.
  * [4] E. Dehghanpour et al.  A Protection System for Inverter Interfaced Microgrids.  IEEE Trans. Power Del., 37(3):2314–2325, Jun. 2022.
  * [5] S. Liu.  Dynamic-Data Driven Real-Time Identification for Electric Power Systems.  University of Illinois at Urbana-Champaign, 2009.
  * [6] C.-J. Lin et al.  Dynamic Load Models in Power Systems Using the Measurement Approach.  IEEE Trans. Power Syst., 8(1):309–315, Feb. 1993.
  * [7] A. Rouhani et al.  Real-Time Dynamic Parameter Estimation for an Exponential Dynamic Load Model.  IEEE Trans. on Smart Grid, 7(3):1530–1536, May 2016.
  * [8] S. Choi et al.  Effective Real-Time Operation and Protection Scheme of Microgrids Using Distributed Dynamic State Estimation.  IEEE Trans. Power Del., 32(1):504–514, Feb. 2017.
  * [9] S. Choi et al.  Autonomous State Estimation Based Diagnostic System in Smart Grid.  In Proc. of the 2013 IEEE PES Innovative Smart Grid Technologies Conf., pages 1–6, 2013.
  * [10] A. K. Barnes et al.  Dynamic State Estimation for Load Bus Protection on Inverter-Interfaced Microgrids.  In Proc. of the 2022 IEEE PES/IAS PowerAfrica Conf., 2022.
  * [11] B. Xie et al.  Dynamic State Estimation Based Unit Protection.  In Proc. of the 2019 IEEE PESGM, 2019.
  * [12] A. P. S. Meliopoulos et al.  Dynamic State Estimation-Based Protection: Status and Promise.  IEEE Trans. Power Del., 32(1):320–330, Feb. 2017.
  * [13] J. Zhao et al.  Power System Dynamic State Estimation: Motivations, Definitions, Methodologies, and Future Work.  IEEE Trans. Power Syst., 34(4):3188–3198, Jul. 2019.
  * [14] Y. Liu et al.  Dynamic State Estimation Based Protection of Microgrid Circuits.  In Proc. of the 2015 IEEE PESGM, 2015.
  * [15] J. Xie et al.  Reliable Dynamic State Estimation Based Protection during Geomagnetic Disturbances.  In Proc. of the 2020 IEEE PESGM, 2020.
  * [16] K. Liu et al.  Dynamic State Estimation-Based Protection of Distribution Systems with High Penetration of DERs.  In Proc. of the 2020 IEEE PESGM, 2020.
  * [17] H. F. Albinali et al.  Hidden Failure Detection Via Dynamic State Estimation in Substation Protection Systems.  In Proc. of the 2017 Saudi Arabia Smart Grid, pages 1–6, 2017\.
  * [18] M. N. Kurt et al.  Secure Distributed Dynamic State Estimation in Wide-Area Smart Grids.  IEEE Trans. Inf Forensics Security, 15(0):800–815, Jul. 2020\.
  * [19] J. Lin et al.  Design, Implementation and Application of Synchrophasor-Based Online State Estimation in a Large-Scale Real Power Grid.  CSEE Journal of Power and Energy Syst., pages 1–10, 2023.
  * [20] W. K. Chai et al.  An Information-Centric Communication Infrastructure for Real-Time State Estimation of Active Distribution Networks.  IEEE Trans. on Smart Grid, 6(4):2134–2146, Feb. 2015.
  * [21] A. K. Barnes et al.  Dynamic State Estimation for Radial Microgrid Protection.  In Proc. of the 2021 IEEE/IAS 57th Industrial and Commercial Power Syst. Tech. Conf., 2021.
  * [22] J. C. Vasquez et al.  Modeling, Analysis, and Design of Stationary-Reference-Frame Droop-Controlled Parallel Three-Phase Voltage Source Inverters.  IEEE Trans. Ind. Electron., 60(4):1271–1280, Apr. 2013.
  * [23] MATLAB.  9.7.0.1190202 (R2019b).  The MathWorks Inc., Natick, Massachusetts, 2019.