# Semiparametric Language Models Are
Scalable Continual Learners

Guangyue Peng1, Tao Ge2, Si-Qing Chen2, Furu Wei2, Houfeng Wang1
1 Peking University
2 Microsoft
Corresponding author: Tao Ge<EMAIL_ADDRESS>
###### Abstract

Semiparametric language models (LMs) have shown promise in continuously
learning from new text data by combining a parameterized neural LM with a
growable non-parametric memory for memorizing new content. However,
conventional semiparametric LMs will finally become prohibitive for computing
and storing if they are applied to continual learning over streaming data,
because the non-parametric memory grows linearly with the amount of data they
learn from over time. To address the issue of scalability, we present a simple
and intuitive approach called Selective Memorization (SeMem), which only
memorizes difficult samples that the model is likely to struggle with. We
demonstrate that SeMem improves the scalability of semiparametric LMs for
continual learning over streaming data in two ways: (1) data-wise scalability:
as the model becomes stronger through continual learning, it will encounter
fewer difficult cases that need to be memorized, causing the growth of the
non-parametric memory to slow down over time rather than growing at a linear
rate with the size of training data; (2) model-wise scalability: SeMem allows
a larger model to memorize fewer samples than its smaller counterpart because
it is rarer for a larger model to encounter incomprehensible cases, resulting
in a non-parametric memory that does not scale linearly with model size. We
conduct extensive experiments in language modeling and downstream tasks to
test SeMem’s results, showing SeMem enables a semiparametric LM to be a
scalable continual learner with little forgetting.

##  1 Introduction

Semiparametric language models (LMs) have been drawing attention in recent
years (Sukhbaatar et al., 2019; Guu et al., 2020; Yogatama et al., 2021; Alon
et al., 2022; Wu et al., 2022) for their photographic memorization
capabilities, which combine a parameterized neural LM with an extensible non-
parametric memory allowing them to incrementally learn from new data. However,
a conventional semiparametric LM has an obvious scalability issue for
continual learning (CL) over massive streaming data, as shown in Figure 1: (1)
with the increasing amount of streaming data it learns over time, its non-
parametric memory will proportionally grow and end up prohibitive for
computing and storing; (2) the size of memory also grows proportionally to the
parameterized LM’s size (more accurately, model dimension), which is
unfriendly and uneconomical for large LMs.

To address this scalability issue for CL over streaming data, we present a
simple and intuitive approach called Selective Memorization (SeMem). Unlike
conventional semiparametric LMs that fully memorize training data into non-
parametric memory, SeMem selectively memorizes only difficult samples that the
model is likely to struggle with, which is similar to the way humans learn
through memorization. SeMem can improve the scalability of semiparametric LMs
for CL over streaming data in two ways, as Figure 1 shows: (a) data-wise
scalability: as the model becomes more knowledgeable through continual
learning with more data, it will encounter fewer difficult cases that need to
be memorized; as a result, the growth of the non-parametric memory will slow
over time rather than growing at a linear rate with the size of training data;
(b) model-wise scalability: SeMem allows a larger model to memorize fewer
samples than a smaller model because it is rarer for a larger model to
encounter difficult cases than the smaller one, resulting in a non-parametric
memory that does not scale linearly with model size. We conduct extensive
experiments in language modeling and downstream tasks, showing that SeMem
allows a semiparametric LM to be a scalable continual learner with little
forgetting.

Figure 1: (a) The growth trend of memorized tokens of the conventional full
memorization (FullMem) and our proposed Selective Memorization (SeMem) policy.
Compared with FullMem, not only does SeMem have a significantly reduced memory
but also its memory growth will become slower as it continuously learns with
more training tokens than its initial growth (i.e., the green dashed line);
(b) The growth trend of memorized size (GB) with the increase of the model
size. The numbers in parentheses indicate the memory reduced by SeMem compared
to the FullMem, showing that SeMem’s effect is more significant in larger LMs.

Our contributions can be summarized as follows:

  * •

We propose Selective Memorization (SeMem) that allows the model to skip
memorizing easy cases and only focus on difficult cases. SeMem can largely
increase the non-parametric memory’s efficiency, allowing the model to
substantially reduce its external memory size without hurting performance.

  * •

Our extensive experiments in language modeling and downstream tasks show that
SeMem improves both data- and model-wise scalability of a semiparametric LM
for continual learning, which is one of the earliest explorations, to the best
of our knowledge, in scalable continual learning over streaming text data with
semiparametric LMs.

##  2 Preliminaries: Semiparametric LMs

A semiparametric LM, as its name implies, is a language model that consists of
a non-parametric memory besides a parametric LM. Formally, we use $\Theta$ to
denote a semiparametric LM:

$\Theta=(\theta,\mathcal{M})$ (1)

where $\theta$ denotes the parameterized LM and $\mathcal{M}$ denotes the non-
parametric memory.

In this paper, we use the $k$NN-LM (Khandelwal et al., 2020) as a
representative semiparametric LM for a case study. The $k$NN-LM enhances its
parametric LM’s prediction by leveraging information of the $k$-nearest
neighbors in its non-parametric memory during inference. Given a leftward
context $\boldsymbol{x}=(x_{1},\dots,x_{n})$, its next word $y$’s probability
is computed as follows:

$P(y|\boldsymbol{x};\Theta)=\underbrace{(1-\lambda)P(y|\boldsymbol{x};\theta)}_{{\footnotesize\raggedright\textrm{Parametric
LM}\@add@raggedright}}+\underbrace{\lambda
P(y|\widetilde{\boldsymbol{x}};\mathcal{M})}_{{\footnotesize\raggedright\textrm{Non-
parametric memory}\@add@raggedright}}$ (2)

where $\lambda$ is a hyperparameter to weigh the parametric LM and non-
parametric memory’s contribution to the final result.
$\widetilde{\boldsymbol{x}}$ denotes the contextualized representation111As
Khandelwal et al. (2020) suggests, we use the parametric LM’s last-layer
hidden representation before the feedforward network as the contextualized
representation. of $\boldsymbol{x}$ and $\mathcal{M}$ is the non-parametric
memory for $k$NN retrieval, which is a set of key-value pairs for looking up a
target word (value) of a leftward context (key):

$\mathcal{M}=\\{(\widetilde{\boldsymbol{x_{<t}}}\to
x_{t})|x_{t}\in\mathcal{D}\\}$ (3)

where $\mathcal{D}$ denotes a training set. As Eq 3 shows, conventional $k$NN-
LMs construct $\mathcal{M}$ with an entry for each training set token $x_{t}$
(as value), and the contextualized representation of its leftward context
$\widetilde{\boldsymbol{x_{<t}}}$ (as key). We refer to this method as full
memorization (FullMem) since it memorizes all tokens in a training set.

Figure 2: Intuition of SeMem where we consider cases with $\log P<-1.5$ as
hard: (a) Small-1 and Small-2 illustrate what training cases need to be
memorized/skipped; (b) Small-2 and Small-3 show that some hard cases will
become easy and thus be skipped if their similar cases are memorized before;
(c) Small-2 and Large-1 demonstrate that a hard case for a small model is
considered easy and thus skipped by a large model.

During inference, we first use $\boldsymbol{\tilde{x}}$ as a query to retrieve
$k$ nearest neighbors from the memory $\mathcal{M}$:

$\small\mathcal{N}(\boldsymbol{\tilde{x}})=\\{(\boldsymbol{\tilde{x}}^{(i)}\to
y^{(i)})|i=1,2,\dots,k\\}\subseteq\mathcal{M}$

Then, we obtain the non-parametric memory’s prediction by computing the
weighted sum of retrieved targets:

$\small
P(y|\boldsymbol{\tilde{x}};\mathcal{M})\propto\sum_{(\boldsymbol{\tilde{x}}^{(i)}\to
y^{(i)})\in\mathcal{N}(\boldsymbol{\tilde{x}})}\mathbbm{1}_{y^{(i)}=y}\exp(-d(\boldsymbol{\tilde{x}}^{(i)},\boldsymbol{\tilde{x}}))$

Where $\mathbbm{1}$ is the indicator function, and $d(.,.)$ denotes the
semantic similarity222As Khandelwal et al. (2020), we use $L^{2}$ distance in
this paper. between two vectors.

##  3 Selective Memorization

###  3.1 Memorization Policy

In contrast to the conventional full memorization policy that memorizes every
training case in semiparametric LMs, our proposed Selective Memorization
(SeMem) allows the model to selectively memorize training cases based on their
value to improve the model, as is inspired by the way in which humans learn
through memorization. As Figure 2 shows, SeMem assumes that an easy case
(e.g., Small-1) which is already well handled by the LM is less valuable to
improve the LM than a hard case (e.g., Small-2). This intuition allows the LM
to skip memorizing easy cases and only focus on memorizing hard cases that the
LM is likely to struggle with. SeMem can also be interpreted by analogy to
gradient-based learning where a simple case is less valuable than a difficult
case because the simple case only yields very weak learning signals (i.e.,
gradient) to update a model.

To formalize SeMem, we first define a memorization threshold $\delta$ for
determining if a training case is hard enough for the LM so that the LM needs
to memorize it:

$\begin{cases}\textrm{Memorize,~{}~{}}&\text{if}~{}\log
P(x_{t}|\boldsymbol{x_{<t}};\Theta)<\delta\\\
\textrm{Skip,}&\text{else}\end{cases}$ (4)

Then, we only memorize the cases whose (probability) confidence is below the
threshold $\delta$:

$\mathcal{M}\leftarrow\mathcal{M}~{}\cup~{}\\{(\boldsymbol{\widetilde{x_{<t}}}\to
x_{t})|\log P(x_{t}|\boldsymbol{x_{<t}};\Theta)<\delta\\}$ (5)

By looking into Eq 4-5 above, we can easily understand that the memorization
decision of SeMem dynamically changes over time even for the same case because
$\mathcal{M}$ keeps growing throughout continual learning. As shown in Figure
2(b), if a case (Small-2) was memorized previously, its similar cases (e.g.,
Small-3) as well as the case itself will be less likely to be memorized again
because the LM may not consider them hard as before. With the LM memorizing
more and more cases over time, it will become more knowledgeable and thus
rarer to confront incomprehensible cases that need to be memorized. As a
result, the non-parametric memory $\mathcal{M}$’s growth will not be
proportional to the size of training data but become slower and slower,
demonstrating the desirable data-wise scalability.

SeMem not only benefits scaling over data, but also it improves model-wise
scalability. As illustrated in Figure 2(c), a smaller and a larger
(parameterized) LM may have different memorization decisions even for the same
case: in general, a smaller LM tends to memorize more than a larger LM because
it is likely to confront more hard cases that need to be memorized than a
larger LM.

###  3.2 Inference Calibration

Given that SeMem allows the LM to skip memorizing easy cases during learning,
some easy cases that are well handled by the parameterized LM may never be
memorized in the non-parametric memory $\mathcal{M}$. Therefore, for these
easy cases, we should not consult the non-parametric memory but rely more on
the parameterized LM; while for hard cases that the parameterized LM cannot
address well, we should weigh the non-parametric memory more. This intuition
motivates us to calibrate the final output probability distributions (over the
output vocabulary) during inference instead of using a constant hyperparameter
($\lambda$ in Eq 2).

Inspired by previous studies (Zheng et al., 2021; Drozdov et al., 2022; He et
al., 2021) which propose to adaptively weigh the parametric LM and non-
parametric memory for either improving the prediction result or skipping
retrieval to reduce inference time, we train a neural network (NN) calibrator
to inform whether the case is too simple to consult the non-parametric memory
during inference with following features:

  * •

Distribution information of the parametric LM

    * –

$\boldsymbol{\tilde{x}}$: contextualized representation of $\boldsymbol{x}$ by
the parameterized LM

    * –

$conf(\boldsymbol{x})$: $\max_{y}P(y|\boldsymbol{x};\theta)$

    * –

$ent(\boldsymbol{x})$: entropy of $P(y|\boldsymbol{x};\theta)$

  * •

Lexical information of the training data

    * –

$\log freq(\boldsymbol{x_{-1}})$: log of frequency of the last token in the
context

    * –

$\log distinct(\boldsymbol{x_{-1}})$: log of the number of distinct values of
the last token in the context

  * •

Density information of the external memory

    * –

$d(\boldsymbol{\tilde{x}}^{(i)},\boldsymbol{\tilde{x}})$: $L^{2}$ distance
(semantic similarity) between the query and the top-$i$ retrieved neighbor,
$i=1,2,\dots,10$.

    * –

$\log distinct(\boldsymbol{\tilde{y}}^{(i)})$: log of the number of distinct
values of the top-$i$ retrieved values, $i=1,2,\dots,10$.

Following He et al. (2021), we use a 4-layer MLP network333Details are
included in Section A.2. as our NN calibrator (parameterized by $\theta_{c}$),
which can be learned to optimize perplexity of a small validation set:

$\displaystyle\theta_{c}^{*}=\arg\max_{\theta_{c}}(1-\lambda(\boldsymbol{x};\theta_{c}))P(y|\boldsymbol{x};\theta)+\lambda(\boldsymbol{x};\theta_{c})P(y|\widetilde{\boldsymbol{x}};\mathcal{M})$
(6)

##  4 Experiments

###  4.1 Experimental Setting

We use the news articles whose timestamps are during the first half (i.e., 6
months) of 2020 in the News Crawl corpus444https://data.statmt.org/news-crawl,
referred to as Newscrawl-20H1 as our streaming data for continual learning
(CL). We leave 1% of the streaming data per day as the validation set and
another 1% as the test set. Table 1 shows the statistics of Newscrawl-20H1. We
continually learn the streaming data in chronological order and update the
search index555Implementation details of search are included in Section A.1.
every day. In addition, we use the contemporaneous Wikipedia event
articles666An example event article is 2020 Caribbean earthquake. which
describe real-world events that happened during 20H1 mainly for testing our
approach in domains other than news articles. We refer to this dataset as
WikiEvent-20H1, which contains on average 10 Wikipedia articles per month with
around 100k tokens in total.

Newscrawl-20H1
---
| Daily | Monthly | Total
| Train | Valid | Test | Train | Valid | Test | Train | Valid | Test
Articles | 4,421 | 45 | 46 | 132,642 | 1,353 | 1,369 | 795,850 | 8,118 | 8,209
Tokens | 2.44M | 24.64K | 24.70K | 73.23M | 739.20K | 741.01K | 439.38M | 4.44M | 4.45M

Table 1: Statistics of Newscrawl-20H1.

We use the GPT-2777The release date of the GPT-2 is Feb 2019. Therefore, it
must be unaware of data after Feb 2019. (Radford et al., 2019) as our
parameterized LM to study CL over the streaming data (during 20H1). We
experiment with the GPT-2 small888The GPT-2 small is our default parametric LM
if we do not mention its size explicitly., medium and large size models whose
configurations are presented in Table 2.

Model | Layer | Dim | #Param
---|---|---|---
GPT-2 small | 12 | 768 | 123M
GPT-2 medium | 24 | 1024 | 355M
GPT-2 large | 36 | 1280 | 774M
Table 2: Model configurations of the GPT-2 models (as the parametric LMs) in
our experiments.

We evaluate SeMem’s result and scalability after CL over all the training data
of Newscrawl-20H1. We conduct extensive experiments in both language modeling
and downstream tasks, and compare SeMem with the following baselines:

  * •

Full memorization (FullMem): Conventional memorization policy in
semiparametric LMs that memorizes every token in the training set.

  * •

Random memorization (RandMem): Randomly memorizing $60\%$ of training data. We
conduct three runs with different random seeds and choose the best performing
one as the baseline.

We use the news articles during December 2019 in the News crawl as pilot
streaming data to estimate an appropriate memorization threshold $\delta$ for
SeMem. We find when $\delta=-1.5$, SeMem can achieve comparable performance to
FullMem with around 60% memory in the pilot data. Therefore, we set
$\delta=-1.5$ in our following experiments. The NN calibrator is trained on a
small subset of our Newscrawl-20H1 validation set, which contains only 1,800
articles.

###  4.2 Results of language modeling

Table 3 presents a result comparison between SeMem and FullMem as well as
RandMem for CL with Newscrawl-20H1, showing semiparametric LMs substantially
improve the bare GPT-2 small (24.1 PPL). Among them, SeMem achieves comparable
(even slightly better) language modeling performance to FullMem but with only
53% overall memory size, and largely outperforms RandMem (60% memory size),
demonstrating SeMem is a cost-effective memorization policy for CL.

Methods | PPL ($\downarrow$) | MemRate ($\downarrow$)
---|---|---
Bare GPT-2 small | 24.1 | 0%
+FullMem | 9.0 | 100%
+RandMem | 15.0 | 60%
+SeMem | 8.6 | 53%
Table 3: Perplexity (PPL) on the test set of Newscrawl-20H1 by different
memorization methods for CL over Newscrawl-20H1.

We look into the results by tracking SeMem’s memorization rate of each month
throughout CL. According to Figure 3, SeMem’s memorization rate each month
gradually decreases as it learns over time because the model becomes
increasingly knowledgeable and thus skips memorizing more training cases that
it already knows, which also accounts for the sublinear growth trend of
SeMem’s memory observed in Figure 1(a). We further confirm this by comparing
the memorization rate of continually learning the news data during July 2020
in the News crawl (referred to as Newscrawl-July) and WikiEvent-20H1 before
and after CL with Newscrawl-20H1 to demonstrate SeMem’s data-wise scalability,
as shown in Table 4.

Figure 3: SeMem’s memorization rate gradually decreases throughout CL over
months, demonstrating the data-wise scalability.

Model | Newcrawl-July (same domain) | WikiEvent-20H1 (different domain)
---|---|---
MemRate ($\downarrow$) | PPL ($\downarrow$) | MemRate ($\downarrow$) | PPL ($\downarrow$)
Bare GPT-2 small | 60% | 8.8 | 62% | 29.0
\+ CL with Newscrawl-2020H1 | 49% | 8.2 | 57% | 27.5

Table 4: Comparison of memorization rate and perplexity for continually
learning (CL) new data – Newscrawl-July and WikiEvent-20H1 before and after CL
with Newscrawl-2020H1: After CL with Newscrawl-2020H1, the model’s
memorization rate for new data (both Newscrawl-July and WikiEvent-20H1) is
significantly lower than the bare GPT-2 model but even with better results,
demonstrating that previous CL (with Newscrawl-2020H1) help the model skip
memorizing many cases that it learned before without hurting performance.

Table 5 shows the model size’s effect on SeMem. In the premise of comparable
results to FullMem, the SeMem’s effect becomes more significant as the
parametric model becomes larger: its overall memorization rate drops from 53%
(GPT-2 small) to 42% (GPT-2 large). This is because a larger LM tends to skip
more training cases than a smaller LM, as we assumed. The reduced memorization
rate well demonstrates the model-wise scalability of SeMem.

Model | Strategy | PPL ($\downarrow$) | MemRate ($\downarrow$)
---|---|---|---
GPT-2 small | FullMem | 9.0 | 100%
SeMem | 8.6 | 53%
GPT-2 medium | FullMem | 7.0 | 100%
SeMem | 7.2 | 48%
GPT-2 large | FullMem | 6.2 | 100%
SeMem | 6.3 | 42%
Table 5: SeMem’s model-wise scalability

We finally evaluate SeMem’s learning and forgetting performance by tracking
its result on 7 test sets throughout CL over the Newscrawl-20H1: 6 are each
month’s held-out data in Newscrawl-20H1, and the other is the test set of
Wiki-103 language modeling benchmark which has nothing to do with the
Newscrawl-20H1 training data. According to Figure 4, SeMem learns from the
streaming data well, reflected by a sharp decrease of PPL on a test set
(except Wiki-103) after learning the corresponding month’s training data. More
importantly, it does not suffer from the catastrophic forgetting issue
(French, 1999). PPL scores of all the 7 test sets do not significantly degrade
throughout CL, since SeMem will never erase previous memory or update the
parametric LM’s weights, showing desirable results for CL. The advantage can
be better understood by comparing with other CL baselines such as RecAdam
(Chen et al., 2020) and MixiReview (He et al., 2019)999Details of CL baselines
are included in Appendix B., as shown in Table 6.

Figure 4: Language modeling performance (i.e., perplexity) on 7 test sets
(Wiki-103 and 6 subsets of Newcrawl-20H1’s test set by month) throughout CL
via SeMem over Newscrawl-20H1.

Methods | Wiki-103 | 1 | 2 | 3 | 4 | 5 | 6
---|---|---|---|---|---|---|---
Bare GPT-2 | 29.1 | 24.3 | 24.0 | 24.1 | 24.0 | 24.7 | 23.8
\+ Fine-tune | 33.4 (+2.9) | 20.4 (+2.4) | 18.7 (+2.3) | 17.3 (+1.3) | 17.3 (+0.2) | 16.2 (+0.8) | 15.4 (+0.0)
(best) | 30.6 | 18.2 | 16.4 | 16.0 | 17.1 | 15.4 | 15.4
\+ RecAdam | 34.5 (+2.7) | 19.6 (+0.7) | 18.3 (+0.7) | 17.3 (+0.2) | 17.1 (+0.3) | 16.8 (+0.3) | 16.9 (+0.0)
(best) | 31.8 | 18.9 | 17.6 | 17.1 | 16.8 | 16.5 | 16.9
\+ MixReview | 33.6 (+3.0) | 19.9 (+1.8) | 18.5 (+2.1) | 17.3 (+1.4) | 17.2 (+0.1) | 15.8 (+0.4) | 15.6 (+0.0)
(best) | 30.6 | 18.1 | 16.4 | 15.9 | 17.1 | 15.4 | 15.6
\+ SeMem | 29.9 (+0.5) | 9.4 (+0.1) | 7.6 (+0.2) | 7.8 (+0.5) | 6.9 (+0.1) | 9.5 (+0.0) | 8.8 (+0.0)
(best) | 29.4 | 9.3 | 7.4 | 7.3 | 6.8 | 9.5 | 8.8

Table 6: Perplexity (the lower, the better) comparison evaluated on the 7 test
sets (Wiki-103 and Newscrawl-20H1) after CL over the streaming (training) data
of Newscrawl-20H1 between SeMem and representative CL approaches: RecAdam
(Chen et al., 2020) as the regularization-based CL baseline and Mix-Review (He
et al., 2019) as the replay-based CL baseline. The numbers in the second row
of each cell of CL methods denote the best result ever achieved during the
process of CL. SeMem not only achieves better results in learning from the new
data but also suffers less from catastrophic forgetting compared with other CL
approaches despite introducing additional memory.

After CL over the news streaming data which is not a significantly different
domain from the LM’s pretraining dataset (i.e., openwebtext), we test SeMem’s
CL performance over the ACL paper dataset (Lo et al., 2020), which contains
42K ACL papers. We hold off 80 papers ($\sim$200K tokens) to construct the
validation and test set, using the rest for training.101010We split training
data into 4 batches for CL and update the index after finishing each batch.
The result is shown in Table 7.

Model (Memrate) | Wiki-103 | Newscrawl-20H1 | ACL
---|---|---|---
Bare GPT-2 (0%) | 29.1 | 24.1 | 40.5
\+ SeMem (64%) | 29.9 $\to$ 30.4 (+0.5) | 8.6 $\to$ 9.2 (+0.6) | 22.7
Table 7: CL performance over ACL papers after learning the Newscrawl-20H1
dataset. SeMem consistently does well in learning new data (40.5 $\to$ 22.7 in
PPL) without significant forgetting (minor performance degradation compared
with the counterpart before CL over the ACL papers in the Wiki-103 and
Newscrawl-20H1 test set). Compared with FullMem, it only uses 64% memory.

###  4.3 Results of downstream tasks

####  4.3.1 Next-word prediction

For a language model, next-word prediction is its basic usage and most
straightforward end task which is especially important for the application
(e.g., input methods, Google’s smart compose and Microsoft’s text predictions)
of predicting texts in AI writing assistants.

To align this task with our CL setting (over Newscrawl-20H1), we test next-
word prediction on the WikiEvent-20H1 to verify if CL over Newscrawl-20H1 can
help write the contemporary event articles in Wikipedia. As in language
modeling, SeMem consistently shows comparable performance with increasingly
better scalability than FullMem as the model size gets larger (Table 8) and
achieves desirable CL results with little forgetting (Figure 5).

Figure 5: Next-word prediction accuracy on 6 subsets (by month) of WikiEvent-20H1 throughout CL over Newscrawl-20H1. Methods | Acc ($\uparrow$) |  | MemRate ($\downarrow$)
---|---|---|---
small | medium | large |  | small | medium | large
Bare GPT-2 | 39.5 | 41.2 | 44.6 |  | 0% | 0% | 0%
+FullMem | 43.0 | 45.5 | 48.9 |  | 100% | 100% | 100%
+RandMem | 40.3 | 41.3 | 45.5 |  | 60% | 60% | 60%
+SeMem (Ours) | 43.8 | 45.4 | 48.7 |  | 53% | 48% | 42%
Table 8: Next-word prediction accuracy (Acc) on the test set (WikiEvent-20H1)
and memorization rate (MemRate) by different memorization methods and models
after CL over Newscrawl-20H1.

####  4.3.2 Closed-book question answer

We use the RealTime QA (Kasai et al., 2022) benchmark which is a multiple-
choice question dataset about the current world, requiring a model to answer
questions about novel events or information, as our second testbed of
downstream tasks. To align with our streaming data, we use the subset of
question-answer pairs about the news during 20H1 in RealTime QA as our
downstream dataset and evaluate in the closed-book Multiple Choice setting. As
the original paper of RealTime QA suggests, we evaluate our largest111111The
GPT-2 small and medium are too weak to perform zero-shot learning. model
(i.e., the GPT-2 large) in a zero-shot learning setting.

Table 9 shows the results in the RealTime QA benchmark. Compared with the bare
GPT-2 large, CL through SeMem over Newscrawl-20H1 substantially improves
question-answering performance in the Realtime QA dataset because it learns
the world knowledge during 20H1 from the news stream that is necessary to
answer the questions. As in language modeling evaluation, SeMem performs as
well as FullMem with less memorization cost and outperforms RandMem with a
similar memory size. Moreover, little forgetting is consistently observed, as
shown in Table 10.

Methods (MemRate) | Acc ($\uparrow$)
---|---
Bare GPT-2 large (0%) | 29.8
+FullMem (100%) | 36.3
+RandMem (60%) | 30.7
+SeMem (42%) | 36.3
Table 9: Accuracy tested on RealTime QA of GPT-2 large model with CL over Newscrawl-20H1 in the zero-shot learning setting. CL over Test on | 1-2 | 3-4 | 5-6
---|---|---|---
1-2 | 36.0 | 29.3 | 29.5
3-4 | 37.2 | 35.8 | 30.6
5-6 | 37.0 | 36.5 | 35.5
Table 10: Accuracy of the GPT-2 large tested on every two months of RealTime
QA throughout CL over Newscrawl-20H1.

##  5 Analysis

###  5.1 Performance VS Memorization rate

We have confirmed that SeMem can achieve comparable performance to FullMem
with a substantially reduced memorization rate when $\delta=-1.5$.
Intuitively, if $\delta$ increases, more cases will be memorized and the
performance will be likely to further increase; on the contrary, if $\delta$
decreases, more cases will be skipped, resulting in less memory but weaker
performance. Table 11 confirms this intuition, demonstrating that we can
obtain a trade-off between scalability and performance through the
manipulation of $\delta$.

Methods | PPL ($\downarrow$) | MemRate ($\downarrow$)
---|---|---
Bare GPT-2 small | 24.1 | 0%
+FullMem | 9.0 | 100%
+SeMem ($\delta=-1.0$) | 8.2 | 63%
+SeMem ($\delta=-1.5$) | 8.6 | 53%
+SeMem ($\delta=-2.0$) | 9.9 | 46%
Table 11: SeMem with different memorization threshold $\delta$

###  5.2 NN calibrator

As introduced in Section 3.2, the NN calibrator plays an important role in
calibrating the semiparametric LM’s prediction by adapting $\lambda$ in Eq 2
at test time. Although such an adaptive method has been proven universally
effective in semiparametric LMs by previous work, we reveal in Table 12 that
it benefits SeMem most, significantly more than RandMem and conventional
FullMem which only introduces marginal improvement as in previous work,
demonstrating that calibration is more compatible with SeMem.

Methods | FullMem | RandMem | SeMem
---|---|---|---
Constant $\lambda$ | 9.0 | 15.0 | 14.3
NN calibrator | 8.3 (-0.7) | 12.5 (-2.5) | 8.6 (-5.7)
Table 12: Perplexity results on the Newscrawl-20H1 test data with and without
the NN calibrator for FullMem, RandMem and SeMem.

We ablate the features of the NN calibrator to study their effects on results.
According to Table 13, all our proposed features contribute positively to the
calibrator, among which the density information, especially the $L^{2}$
distance, is the most important one because it can directly reflect if the
non-parametric memory has much relevant information given a test case,
providing the most straightforward evidence to the calibrator.

Features | PPL ($\downarrow$)
---|---
All features | 8.6
-density features | 12.0
-distribution features | 9.9
-lexical features | 8.9
Table 13: The ablation study of features in the NN calibrator

###  5.3 In-context learning

We test if the in-context learning capability of a language model is affected
by CL through SeMem. We present the in-context learning result in the Realtime
QA benchmark that the memory can benefit in Table 14, showing that SeMem is
not in conflict with in-context learning and that the LM can still benefit
from more examples shown in the context.

Also, we evaluate SeMem’s effect on in-context learning for general tasks
collected by Shi et al. (2022a) that cannot benefit from its memory. According
to Table 15, despite no improvement observed, SeMem does not affect the
results in these tasks, demonstrating its robustness.

Methods (MemRate) | 0-shot | 1-shot | 2-shot
---|---|---|---
Bare GPT-2 large (0%) | 29.8 | 31.5 | 32.5
+FullMem (100%) | 36.3 | 36.9 | 37.2
+RandMem (60%) | 30.7 | 32.1 | 33.0
+SeMem (42%) | 36.3 | 36.7 | 37.7
Table 14: In-context learning accuracy of the GPT-2 large with CL over Newscrawl-20H1 on the RealTime QA in 0-, 1- and 2-shot learning. Methods (MemRate) | RTE | CB | RT | SST-2 | CR | MR | HYP
---|---|---|---|---|---|---|---
Bare GPT-2 large (0%) | 53.1 | 39.3 | 49.5 | 51.4 | 50.5 | 50.8 | 60.0
FullMem (100%) | 50.3 | 35.5 | 49.0 | 49.8 | 48.6 | 47.5 | 60.0
FullMem (100%) w/ NN calibrator | 52.8 | 41.1 | 49.5 | 51.8 | 50.8 | 50.9 | 60.0
SeMem (42%) | 53.1 | 41.1 | 49.5 | 51.5 | 50.8 | 50.0 | 60.0
Table 15: 0-shot learning accuracy of the GPT-2 large with CL over
Newscrawl-20H1 on general NLP tasks.

##  6 Related Work

Retrieval-augmented LMs There are many recent works that use retrieval
components for language tasks, in special language modeling. REALM (Guu et
al., 2020) puts a trainable document retriever onto the masked language model,
which is fine-tuned on an open-domain QA task. RETRO (Borgeaud et al., 2022)
enhances auto-regressive language models by conditioning on document chunks
retrieved from a large corpus by a frozen BERT retriever, a differentiable
encoder, and a chunked cross-attention mechanism. Unlike above approaches that
explicitly store text for retrieval, semiparametric LMs are proposed to store
text in memory which consists of key-value pairs. As a representative
semiparametric LM, the $k$NN-LM (Khandelwal et al., 2020) adopts non-
parametric external memory to store text examples seen before and uses it at
test time to enhance the parametric LM’s prediction, without the need for
training or retraining. Since $k$NN-LM came out as a powerful method for
making use of external data, a number of methods have been proposed to study
semiparametric LMs (Yogatama et al., 2021; Drozdov et al., 2022; Jin et al.,
2022b; Bhardwaj et al., 2022; Shi et al., 2022b; Trotta et al., 2022;
Khandelwal et al., 2021; Jiang et al., 2021; Zheng et al., 2021). Among them,
He et al. (2021) focuses on improving efficiency of semiparametric LMs,
proposing multiple methods (e.g., quantization, dimensionality reduction) to
optimize the inference speed. While it has similar motivation as our work, we
focus more on memory scalability in the setting of CL over streaming data with
orthogonal technical contributions.

Continual learning LMs Continual learning (CL) proposes to address the “new
knowledge - catastrophic forgetting” dilemma (French, 1999), which means
ingoing new information could always bring models to forget previously learned
knowledge severely. According to Ke et al. (2022), the main approach to
overcoming the difficulty of catastrophic forgetting could be categorized into
4 families: (1) regularization-based methods(Liu et al., 2019; Qin & Joty,
2022) add regularization to penalize excessive changes of parameters towards
new tasks; (2) replay-based methods(Sun et al., 2020; Scialom et al., 2022;
Qin et al., 2022) either memorize seen samples or generate pseudo samples to
enhance learning in new tasks; (3) parameter-isolation based methods(Ke et
al., 2022; Jin et al., 2021) use separate parameters for distinct tasks,
preventing them from interfering with each other; and (4) instruction-based
methods(Yin et al., 2022) relies on task-specific instructions (prompts) to
condition the LM. However, CL is still an under-explored field for LM. LAMOL
(Sun et al., 2020) simultaneously learns to solve the tasks and generate
training samples, so that it can replay pseudo-samples of previous tasks
without extra needs for memory or model capacity. ELLE (Qin et al., 2022)
performs continual pre-training by combining (2) and (3) methods, while CPT
(Ke et al., 2022) adopts continual post-training with help of its carefully-
designed adapter. The closest works to what we do are (Jin et al., 2022a) and
(Jang et al., 2022), which continually adapt language models to emerging
corpora across domains and timelines.

Our work is a pioneer exploration of using semiparametric LMs as a continual
learner over streaming data by addressing its painful scalability issue with
our proposed Selective Memorization method, which not only alleviates
catastrophic forgetting in continual learning but also reaches sublinear data-
wise and model-wise scalability in storage without the need for further
training or fine-tuning.

##  7 Conclusion

In this paper, we study improving the scalability of semiparametric LMs for
continual learning and propose Selective Memorization (SeMem) – a simple yet
intuitive approach. Extensive experiments in both language modeling and
downstream tasks show SeMem can largely reduce memory size without hurting
performance and that its desirable data-wise and model-wise scalability
enables a semiparametric LM to become a scalable continual learner with little
forgetting, shedding light on a new path towards evolving an LM without
changing its model parameters.

## References

  * Alon et al. (2022) Uri Alon, Frank F. Xu, Junxian He, Sudipta Sengupta, Dan Roth, and Graham Neubig.  Neuro-symbolic language modeling with automaton-augmented retrieval.  In _ICML_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 468–485. PMLR, 2022.
  * Bhardwaj et al. (2022) Rishabh Bhardwaj, George Polovets, and Monica Sunkara.  Adaptation Approaches for Nearest Neighbor Language Models, 2022\.  URL https://arxiv.org/abs/2211.07828.
  * Borgeaud et al. (2022) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre.  Improving language models by retrieving from trillions of tokens.  In _ICML_ , volume 162 of _Proceedings of Machine Learning Research_ , pp. 2206–2240. PMLR, 2022.
  * Chen et al. (2020) Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu.  Recall and learn: Fine-tuning deep pretrained language models with less forgetting.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 7870–7881, Online, 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.634.  URL https://aclanthology.org/2020.emnlp-main.634.
  * Drozdov et al. (2022) Andrew Drozdov, Shufan Wang, Razieh Rahimi, Andrew McCallum, Hamed Zamani, and Mohit Iyyer.  You can’t pick your neighbors, or can you? When and how to rely on retrieval in the $k$NN-LM, 2022.  URL https://arxiv.org/abs/2210.15859.
  * French (1999) Robert M. French.  Catastrophic forgetting in connectionist networks.  _Trends in Cognitive Sciences_ , 3(4):128–135, 1999.  ISSN 1364-6613.  doi: 10.1016/S1364-6613(99)01294-2.  URL https://www.sciencedirect.com/science/article/pii/S1364661399012942.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang.  REALM: Retrieval-Augmented Language Model Pre-Training, 2020\.  URL https://arxiv.org/abs/2002.08909.
  * He et al. (2021) Junxian He, Graham Neubig, and Taylor Berg-Kirkpatrick.  Efficient nearest neighbor language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 5703–5714, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.461.  URL https://aclanthology.org/2021.emnlp-main.461.
  * He et al. (2019) Tianxing He, Jun Liu, Kyunghyun Cho, Myle Ott, Bing Liu, James Glass, and Fuchun Peng.  Analyzing the forgetting problem in the pretrain-finetuning of dialogue response models.  _arXiv preprint arXiv:1910.07117_ , 2019.
  * Jang et al. (2022) Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo.  Towards continual knowledge learning of language models.  In _ICLR_. OpenReview.net, 2022.
  * Jiang et al. (2021) Qingnan Jiang, Mingxuan Wang, Jun Cao, Shanbo Cheng, Shujian Huang, and Lei Li.  Learning kernel-smoothed machine translation with retrieved examples.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pp. 7280–7290, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.579.  URL https://aclanthology.org/2021.emnlp-main.579.
  * Jin et al. (2021) Xisen Jin, Bill Yuchen Lin, Mohammad Rostami, and Xiang Ren.  Learn continually, generalize rapidly: Lifelong knowledge accumulation for few-shot learning.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pp. 714–729, Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.findings-emnlp.62.  URL https://aclanthology.org/2021.findings-emnlp.62.
  * Jin et al. (2022a) Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, and Xiang Ren.  Lifelong pretraining: Continually adapting language models to emerging corpora.  In _Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models_, pp. 1–16, virtual+Dublin, 2022a. Association for Computational Linguistics.  doi: 10.18653/v1/2022.bigscience-1.1.  URL https://aclanthology.org/2022.bigscience-1.1.
  * Jin et al. (2022b) Xuyang Jin, Tao Ge, and Furu Wei.  Plug and play knowledge distillation for knn-lm with external logits.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing_ , pp. 463–469, 2022b.
  * Johnson et al. (2019) Jeff Johnson, Matthijs Douze, and Hervé Jégou.  Billion-scale similarity search with GPUs.  _IEEE Transactions on Big Data_ , 7(3):535–547, 2019.
  * Kasai et al. (2022) Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui.  RealTime QA: What’s the Answer Right Now?, 2022.  URL https://arxiv.org/abs/2207.13332.
  * Ke et al. (2022) Zixuan Ke, Haowei Lin, Yijia Shao, Hu Xu, Lei Shu, and Bing Liu.  Continual Training of Language Models for Few-Shot Learning, 2022.  URL https://arxiv.org/abs/2210.05549.
  * Khandelwal et al. (2020) Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis.  Generalization through memorization: Nearest neighbor language models.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020.  URL https://openreview.net/forum?id=HklBjCEKvH.
  * Khandelwal et al. (2021) Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis.  Nearest neighbor machine translation.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021.  URL https://openreview.net/forum?id=7wCBOfJ8hJM.
  * Liu et al. (2019) Tianlin Liu, Lyle Ungar, and João Sedoc.  Continual learning for sentence representations using conceptors.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 3274–3279, Minneapolis, Minnesota, 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1331.  URL https://aclanthology.org/N19-1331.
  * Lo et al. (2020) Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld.  S2ORC: The semantic scholar open research corpus.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 4969–4983, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.447.  URL https://aclanthology.org/2020.acl-main.447.
  * Qin & Joty (2022) Chengwei Qin and Shafiq Joty.  LFPT5: A unified framework for lifelong few-shot language learning based on prompt tuning of T5.  In _ICLR_. OpenReview.net, 2022.
  * Qin et al. (2022) Yujia Qin, Jiajie Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou.  ELLE: Efficient lifelong pre-training for emerging data.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pp. 2789–2810, Dublin, Ireland, 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.findings-acl.220.  URL https://aclanthology.org/2022.findings-acl.220.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Scialom et al. (2022) Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan.  Continual-t0: Progressively instructing 50+ tasks to language models without forgetting.  _ArXiv preprint_ , abs/2205.12393, 2022.  URL https://arxiv.org/abs/2205.12393.
  * Shi et al. (2022a) Weijia Shi, Julian Michael, Suchin Gururangan, and Luke Zettlemoyer.  Nearest neighbor zero-shot inference.  _arXiv preprint arXiv:2205.13792_ , 2022a.
  * Shi et al. (2022b) Weijia Shi, Julian Michael, Suchin Gururangan, and Luke Zettlemoyer.  kNN-Prompt: Nearest Neighbor Zero-Shot Inference, 2022b.  URL https://arxiv.org/abs/2205.13792.
  * Sukhbaatar et al. (2019) Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, and Armand Joulin.  Augmenting Self-attention with Persistent Memory, 2019.  URL https://arxiv.org/abs/1907.01470.
  * Sun et al. (2020) Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee.  LAMOL: language modeling for lifelong language learning.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020.  URL https://openreview.net/forum?id=Skgxcn4YDS.
  * Trotta et al. (2022) Severino Trotta, Lucie Flek, and Charles Welch.  Nearest Neighbor Language Models for Stylistic Controllable Generation, 2022.  URL https://arxiv.org/abs/2210.15762.
  * Wu et al. (2022) Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy.  Memorizing transformers.  In _ICLR_. OpenReview.net, 2022.
  * Yin et al. (2022) Wenpeng Yin, Jia Li, and Caiming Xiong.  ConTinTin: Continual learning from task instructions.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 3062–3072, Dublin, Ireland, 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.218.  URL https://aclanthology.org/2022.acl-long.218.
  * Yogatama et al. (2021) Dani Yogatama, Cyprien de Masson d’Autume, and Lingpeng Kong.  Adaptive semiparametric language models.  _Transactions of the Association for Computational Linguistics_ , 9:362–373, 2021.  doi: 10.1162/tacl˙a˙00371.  URL https://aclanthology.org/2021.tacl-1.22.
  * Zheng et al. (2021) Xin Zheng, Zhirui Zhang, Junliang Guo, Shujian Huang, Boxing Chen, Weihua Luo, and Jiajun Chen.  Adaptive nearest neighbor machine translation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pp. 368–374, Online, 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-short.47.  URL https://aclanthology.org/2021.acl-short.47.

##  Appendix A Experiment details

###  A.1 Index building

We use the FAISS toolkit (Johnson et al., 2019) for index building and
searching. At each update, we sampled 1M keys randomly from memory to train 4K
cluster centroids, and then the whole keys in memory are added to the trained
index, all quantized to 64-bytes.

###  A.2 NN calibrator training

On each day during memorization, we extract from the validation set 10
articles and update them into the training set of the calibrator yesterday to
obtain the training set of the calibrator today. The validation set of the
calibrator is obtained as above, except for it only needs 5 articles each day.
Because the training set increases slowly every day, we reduce the number of
training epochs from 5 epochs to 1 epoch as time goes on, in case of
overfitting.

Each feature is fed into a 1-layer LeakyReLU network to be transformed into
hidden states of 128-dimension equally. Then all the hidden states are
concatenated to a long vector and fed into a 4-layer MLP network to predict
$\lambda$. We list the hyperparameters of the NN calibrator in Table 16.

Hyperparameters | Values
---|---
Layers | 4
Dimension of hidden state | 128
Learning rate | 3e-4
Optimizer | Adam
Activation function | ReLU
Dropout | 0.2
Table 16: Hyperparameters of the NN calibrator

###  A.3 Inference

During inference, we feed context into the parameterized LM, its
contextualized representation into the memory index, and its three types of
features into the calibrator. We search for top-1K nearest neighbors from 32
nearest cluster centroids using the memory index. The calibrator reweighs the
distributions of the parameterized LM and the memory, and we use this
calibrated distribution as the final output of our model.

##  Appendix B CL baselines

RecAdam (Chen et al., 2020) As a regularization-based method, RecAdam recalls
previously acquired knowledge by retaining the pretraining object through
frozen parameters, and it continually learns new information using a multi-
task learning object. As the learning process moves forwards, the
regularization is annealed to lessen the restriction.

Mix-Review (He et al., 2019) Assuming that the pretraining corpus is
obtainable, Mix-Review uses an empirical decreasing function to adjust the
quantity of the pretraining corpus mixed in the continued training data. As
the learning process moves forwards, the quantity of the pretraining corpus
tapers off to 0, resulting in the remaining training process being equivalent
to fine-tuning.