# TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving
Language Models

Joel Jang1, Seonghyeon Ye1,$\ast$ Changho Lee3 Sohee Yang1
Joongbo Shin2 Janghoon Han2 Gyeonghun Kim2 Minjoon Seo1
1KAIST 2LG AI Research 3Korea University
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>indicates equal contribution.

###### Abstract

Language Models (LMs) become outdated as the world changes; they often fail to
perform tasks requiring recent factual information which was absent or
different during training, a phenomenon called temporal misalignment. This is
especially a challenging problem because the research community still lacks a
coherent dataset for assessing the adaptability of LMs to frequently-updated
knowledge corpus such as Wikipedia. To this end, we introduce TemporalWiki, a
lifelong benchmark for ever-evolving LMs that utilizes the difference between
consecutive snapshots of English Wikipedia and English Wikidata for training
and evaluation, respectively. The benchmark hence allows researchers to
periodically track an LM’s ability to retain previous knowledge and acquire
updated/new knowledge at each point in time. We also find that training an LM
on the _diff_ data through continual learning methods achieves similar or
better perplexity than on the entire snapshot in our benchmark with 12 times
less computational cost, which verifies that factual knowledge in LMs can be
safely updated with minimal training data via continual learning. The dataset
and the code are available at this link.

##  1 Introduction

Large Language Models (LMs) pretrained on a vast amount of text corpus have
shown to be highly effective when finetuned or prompted to perform various
downstream tasks (Raffel et al., 2019; Brown et al., 2020; Sanh et al., 2022;
Wei et al., 2022). However, most of the datasets used to evaluate these LMs
are static benchmarks; the train and test data are both from similar points in
time. On the other hand, in the real world, factual knowledge is frequently
changed, added, or deprecated. For example, suppose a language model is asked
what the most dominant coronavirus variant is (Figure 1). The answer would
have been the Delta variant in the fall of 2021 but has changed to the Omicron
variant near the end of 2021. If LMs remain unchanged and are not periodically
trained to cope with the changing world, they will be outdated very quickly.
This means downstream tasks that directly depend on or are finetuned from the
LM will suffer from temporal misalignment (Luu et al., 2022; Lazaridou et al.,
2021), which refers to the misalignment in time between the train and test
data.

Temporal misalignment becomes a critical problem, especially when using
language models for knowledge-intensive tasks such as closed-book question
answering (Roberts et al., 2020; Petroni et al., 2021; Jang et al., 2022)
since they rely solely on the knowledge stored in their parameters.
Furthermore, LMs augmented with retrieval mechanism (Guu et al., 2020; Lewis
et al., 2020; Borgeaud et al., 2021) often suffer from hallucination even if
they successfully retrieve up-to-date information (Zhang and Choi, 2021; Chen
et al., 2021; Longpre et al., 2021). This means that the implicit knowledge
stored in the model parameters has to be updated as well because it may cause
conflicts with the explicit knowledge retrieved from external sources such as
up-to-date knowledge bases and ultimately cause the LM to hallucinate.

Recently, Lazaridou et al. (2021); Jang et al. (2022) have explored updating
the internal knowledge of LMs through continual pretraining on new and updated
data as a solution for mitigating temporal misalignment. However, these
datasets are still static in nature: as the world changes, they will
eventually get outdated as well. In order to comprehensively measure the
capability of ever-evolving LMs on addressing temporal misalignment, automated
periodic evaluation of the LMs is crucial.

Figure 1: An overview of using TemporalWiki, consisting of TWiki-Diffsets and
TWiki-Probes to train and evaluate ever-evolving LMs, respectively.
Differences between Wikipedia snapshots at different points in time are used
for temporal language modeling, and categorized factual instances in the
corresponding Wikidata snapshots are used for temporal evaluation.

In this paper, we introduce TemporalWiki, a lifelong benchmark for training
and evaluating ever-evolving LMs in a periodic and automated manner, shown in
Figure 1. The corpora used for updating LMs are constructed by comparing
articles from consecutive English Wikipedia snapshots and retrieving only
changed information, which we name as TWiki-Diffsets. The evaluation datasets
are constructed in a similar manner by comparing English Wikidata snapshots
that correspond to the Wikipedia snapshots in time and categorizing each
factual instance into Unchanged or Changed. Since Wikidata updates may not
exactly align with Wikipedia updates, we only retain factual instances that
can be grounded to articles in Wikipedia, ensuring the quality of the data and
name the resulting evaluation dataset as TWiki-Probes. The entire benchmark
creation process is done without any human annotation, thus allowing it to be
automated and lifelong as new English Wikipedia and English Wikidata snapshots
are released by Wikimedia111https://commons.wikimedia.org/ on a monthly basis.

Through TemporalWiki, we aim to tackle the following research questions: How
can we train ever-evolving LMs efficiently and automate the evaluation of each
update? How does updating LMs only on updated data from Wikipedia compare to
updating LMs on entire Wikipedia snapshots, especially in scenarios requiring
multiple updates? How problematic is catastrophic forgetting (McCloskey and
Cohen, 1989) when LMs are updated only on new data, and how can we effectively
mitigate catastrophic forgetting? Our main contributions are summarized as
follows:

  * •

We introduce TemporalWiki, a lifelong benchmark for ever-evolving LMs. Unlike
previous static benchmarks, TemporalWiki is responsive to the dynamic changes
in the world and can be utilized to automatically train and evaluate ever-
evolving LMs on each English Wikipedia and English Wikidata snapshot update.

  * •

We find that continually training LMs only on the updated portion of English
Wikipedia, which we call _temporal language modeling_ , is much more efficient
than updating LMs on entire English Wikipedia snapshots in terms of both
computation and stability-plasticity trade-off. It is still a challenging
task, especially when multiple updates are required due to catastrophic
forgetting.

  * •

As competitive baselines for temporal language modeling, we implement previous
continual learning approaches that mitigate forgetting while bolstering the
learning of new knowledge, thus providing an overall enhancement in terms of
both stability and plasticity. We hope that TemporalWiki will foster future
research towards training ever-evolving LMs.

##  2 Background

Recent works have introduced the need to tackle the issue of temporal
misalignment, which refers to neural networks showing poor performance due to
misalignment in time between the train and test data. Temporal misalignment
can be caused either by (1) the dynamic nature of language (Röttger and
Pierrehumbert, 2021; Hombaiah et al., 2021; Rosin et al., 2021; Loureiro et
al., 2022) or (2) the update of factual information (Chen et al., 2021;
Dhingra et al., 2021; Jang et al., 2022).

Luu et al. (2022) have emphasized the effect of temporal misalignment on eight
different NLP downstream tasks, asserting that misalignment between the train
and test sets of the downstream tasks causes severe performance degradation
that can be mitigate finetuning on the corpus from the target period. Agarwal
and Nenkova (2021) have argued this to be less of a concern when utilizing
representations from pretrained LMs and show that self-labeling on the
downstream task is more effective than continued pretraining on more recent
data for temporal adaptation. Note that these works have focused on
misalignment caused by the dynamic nature of language on tasks that are not
knowledge-intensive, such as text classification.

Others have tackled the problem of temporal misalignment caused by the update
of factual knowledge. Lazaridou et al. (2021) have shown that LMs deteriorate
significantly in performance when there is a misalignment in time between the
pretraining data and the downstream task and argued ever-evolving LMs are
necessary. Dhingra et al. (2021) have proposed explicitly including time
information during pretraining as a potential solution. Jang et al. (2022);
Jin et al. (2021) have implemented continual learning methods to mitigate
catastrophic forgetting that occurs during continued pretraining on new data.

Despite the recent surge of community interest in the need for ever-evolving
LMs, the community still lacks widely-available resources to train and
evaluate such LMs. Previous works have introduced benchmarks comprised of data
sources from Twitter feeds (Osborne et al., 2014; Yogatama et al., 2014;
Loureiro et al., 2022), recent news articles (Jang et al., 2022), and arXiv
papers (Lazaridou et al., 2021) where the temporal adaptability of LMs and the
effectiveness of different methodologies of updating LMs can be evaluated.
However, these data sources are domain-specific and inherently static.

On the other hand, Wikipedia and Wikidata are known to be great sources of
general world knowledge and thus have been widely used by the community (Dinan
et al., 2019; Thorne et al., 2018; Kwiatkowski et al., 2019; Piktus et al.,
2021). 120K volunteer editors make 120 updates to the English Wikipedia per
minute and add hundreds of new article entries every day (Logan IV et al.,
2021)222https://en.wikipedia.org/wiki/Wikipedia:Statistics. Even though every
Wikipedia and Wikidata update may not correspond to an actual change in the
real world, TemporalWiki leverages the dynamic nature of Wikipedia and
Wikidata to provide a _lifelong_ benchmark for developing and maintaining
ever-evolving LMs.

##  3 TemporalWiki

In this section, we delve into the process of creating TemporalWiki, which is
comprised of training corpora (TWiki-Diffsets) and evaluation datasets (TWiki-
Probes) constructed by comparing the consecutive snapshots of English
Wikipedia and English Wikidata, respectively (For brevity, we abbreviate them
by omitting ‘English’ throughout the paper). Moreover, we clarify that not all
Wikipedia/Wikidata updates equate to the actual updates of the world
knowledge. In Section 3.1, we first describe the process of constructing the
training corpora from Wikipedia snapshots. Then in Section 3.2, we describe
the process of generating the evaluation datasets from Wikidata snapshots. In
Section 3.3, we describe the quality control applied to the evaluation
datasets, including the alignment of instances from Wikidata with Wikipedia.
Lastly, in Section 3.4, we briefly discuss the current limitations of
TemporalWiki.

Algorithm 1 Generating TWiki-Diffsets

0: Wikipedia snapshots $WP_{prev}$ and $WP_{recent}$ where $WP_{recent}$ is
more recent.

$D$ := An empty array to store new and updated data.

*$article$ in $WP$ has attributes $id$ and $text$

for all $article$ $a_{r}\in WP_{recent}$ do

if $a_{r}.id=a_{p}.id$ for some $article$ $a_{p}\in WP_{prev}$ then

$D$.append($\textsc{GetDiff}(a_{p},a_{r}$))

else

$D$.append($a_{r}$)

end if

end for

function GetDiff($a_{p},a_{r}$)

$Diff$ := An empty string to append difference between $text$ in two
$article$s.

for all paragraph $p_{r}\in a_{r}.text$ do

if $p_{r}$ have no matching sentences with any paragraph $p_{p}\in a_{p}.text$
then

$Diff$ $\leftarrow Diff+p_{r}$

else if $p_{r}$ have some matching and some different sentences with any
paragraph $p_{p}\in a_{p}.text$ then

$Diff$ $\leftarrow Diff$ \+ sentences that differ between $p_{r}$ and $p_{p}$.

end if

end for

return $Diff$

###  3.1 Generating Training Corpora from Wikipedia

In terms of computational resources, it is highly inefficient to train an LM
on the entire Wikipedia snapshot every time the LM requires updates since most
part of Wikipedia is unchanged from the previous snapshot. Moreover, it is not
certain whether updating the LM on the entire Wikipedia snapshot is the best
approach for updating the factual knowledge stored in the LM. Therefore, we
compare the differences between consecutive Wikipedia snapshots in order to
use only updated and new text for training. We call these subsets TWiki-
Diffsets. Algorithm 1 shows the procedure for generating them.

As shown in Algorithm 1, a single TWiki-Diffset is generated by getting the
differences (similarly to git diff) between two consecutive Wikipedia
snapshots. If an article with a new unique id is included in the recent
snapshot, we append the entire article to TWiki-Diffset. For an article having
an existing id in the previous snapshot, we compare the two articles by
paragraphs and add new or updated sentences to TWiki-Diffsets. Examples of
TWiki-Diffset are shown in Figure 2, and detailed statistics are shown in
Section 4.

(a) Information Update

(b) New Information

Figure 2: Examples of TWiki-Diffsets constructed from comparing November 2021
and December 2021 Wikipedia Dumps. (a) shows an instance of information update
and (b) shows an instance of new information.

###  3.2 Generating Evaluation Datasets from Wikidata

In this work, the main objective for continually pretraining LMs is to add and
update the factual knowledge stored in the implicit parameters of LMs. The
success of an LM update can be evaluated by quantifying the stability-
plasticity dilemma (Mermillod et al., 2013): the dilemma of artificial and
biological neural systems having to sacrifice either stability, ability to
retain learned knowledge, or plasticity, ability to obtain new knowledge. In
order to evaluate whether each update is successful, we need evaluation
datasets that can quantify the amount of changed (updated or new) knowledge
successfully gained (plasticity) and the amount of knowledge that remains
unchanged as intended after the LM update (stability). Therefore, we
categorize factual instances from Wikidata snapshots that are temporally
aligned with Wikipedia snapshots and call the resulting datasets TWiki-Probes.

Wikidata snapshots are structured knowledge graphs that store factual
information in the form of (Subject, Relation, Object) such as (Barack Obama,
born-in, Hawaii). These factual instances can be used to probe the LM for
factual knowledge (Petroni et al., 2019). Through Algorithm 2, we distinguish
each factual instance into either Unchanged or Changed.

Algorithm 2 Generating TWiki-Probes

0: Wikidata snapshots $WD_{prev}$ and $WD_{recent}$ where $WD_{recent}$ is
more recent.

$Un$, $C$ := Arrays that store Unchanged and Changed factual instances,
respectively.

for all fact $(s_{r},r_{r},o_{r})\in WD_{recent}$ do

$\mathbb{P}\leftarrow\\{(s,r,o)\mid s=s_{r}\text{ where}\ (s,r,o)\in
WD_{prev}\\}$

if $\mathbb{P}=\emptyset$ then

$C$.append($s_{r},r_{r},o_{r}$)

else if $r_{r}\notin\mathbb{P}$ then

$C$.append($s_{r},r_{r},o_{r}$)

else if $r=r_{r}\text{ and }o=o_{r}\text{ for some}(s,r,o)\in\mathbb{P}$ then

$Un$.append($s_{r},r_{r},o_{r}$)

else

$C$.append($s_{r},r_{r},o_{r}$)

end if

end for

Table 1: Examples of successful alignment between Changed factual instances
from TWiki-Probes-0910 and articles from TWiki-Diffsets-0910. The alignment is
considered successful because for the given factual instance, the Subject
matches the title of the Wikipedia page and the Object exists in the article.

Subject | Relation | Object | Corresponding Sentence in Wikipedia
---|---|---|---
Carlo Alighiero | place of death | Rome | […] Carlo Alighiero died in Rome on 11 September 2021 at the age of 94.[…]
Shang-Chi and the | instance of | Film | […] Shang-Chi and the Legend of the Ten Rings is a 2021 American
Legend of the Ten Rings |  |  | superhero film based on Marvel Comics featuring the character Shang-Chi.[…]
Out of Shadows | language of work or name | Spanish | […] It was later translated into Portuguese, Turkish and Spanish.[…]
Mario Chalmers | member of sports team | Indios | […] On September 27, 2021, Chalmers signed with Indios de Mayagüez
|  | de Mayaguez | of the Baloncesto Superior Nacional.[…]

As shown in Algorithm 2, given two consecutive Wikidata snapshots, a single
TWiki-Probe is constructed, which is used to evaluate an LM updated with
TWiki-Diffset, constructed with the two consecutive Wikipedia snapshots with
the same timestamp. Algorithm 2 categorizes instances with new Relation or
instances with the same Relation, but a new Object into Changed, and unchanged
instances into Unchanged.

###  3.3 Quality Control for Evaluation Data

We apply several quality control steps to the categorized factual instances
from Section 3.2 (Algorithm 2) to best represent the actual change of
knowledge from the LM update.

#### Alignment with TWiki-Diffsets

We ensure correct alignment of Changed factual instances with articles in
TWiki-Diffsets and Unchanged factual instances with articles from the entire
Wikipedia since Wikidata updates do not necessarily entail Wikipedia updates
and vice versa. In order to do this, we take three steps. Step #1: We crawl
information from each Wikipedia article page to find the mapping to the
corresponding Wikidata entity id and store the information as a dictionary.
Step #2: Then, for each factual instance from Changed, we check if the Subject
id can be mapped to an article from TWiki-Diffsets using the dictionary of id
mappings. Likewise, for each instance from Unchanged, we check if the Subject
id can be mapped to an article from Wikipedia. Step #3: Lastly, for a
successfully mapped factual instance from Step 2 (whether it is Changed or
Unchanged), we finally keep the instances where Object exists in the text of
the article.

#### Heuristic Filtering

In addition to the alignment with TWiki-Diffsets, in order to further ensure
the quality of the evaluation datasets, we apply three heuristic filtering
rules to strengthen the quality of the data. Rule #1: We remove the instances
where either Subject or Object is a substring of the other. Rule #2: We remove
the instances where Object contains more than 5 words. Rule #3: We limit the
number of highly frequent entities and relations: $<1\%$ for each Subject
entity and $<5\%$ for each Object entity and Relation. Table 1 shows some
examples of TWiki-Probes after quality control.

###  3.4 Limitations of TemporalWiki

It is important to note that, as mentioned at the beginning of this Section,
each Wikipedia and Wikidata update does not necessarily align with an actual
change in the real world. For instance, an update in Wikidata or Wikipedia may
have actually happened in distant past, and some changes in the real world may
not be immediately reflected in their next snapshots.

Moreover, one aspect that is not covered in this work is knowledge deletion.
While maintaining Wikipedia and Wikidata, volunteer editors not only update or
add new information but also delete information that is incorrect or
misinformed. As removing the misinformation and bias stored in LMs is an
important issue and necessary for truly ever-evolving LMs, future work should
address this aspect utilizing deleted information from general knowledge
sources such as Wikipedia.

##  4 Dataset Statistics

In this paper, we construct TemporalWiki from 08.2021 to 12.2021333As new
Wikipedia and Wikidata dumps are available on a monthly basis, we provide the
source code for constructing new TWiki-Diffsets and TWiki-Probes at this link.
and its statistics are discussed below.

#### Training Corpora Statistics

Statistics of Wikipedia snapshots and TWiki-Diffsets are shown in Table 2. An
interesting aspect of TWiki-Diffsets is that the amount of information being
updated and added (i.e., number of tokens in each subset) is similar for each
month.

Table 2: Statistics of TWiki-Diffsets. The two digits indicate the month of
the year 2021 that the Wikipedia snapshot was obtained from. The four digits
for Wiki-Diffset indicate the months of the two snapshots being compared. For
instance, TWiki-Diffset-0809 indicates the difference between August (08) and
September (09).

| # of Articles | # of Tokens
---|---|---
Wikipedia-08 | 6.3M | 4.6B
TWiki-Diffset-0809 | 306.4K | 347.29M
Wikipedia-09 | 6.3M | 4.6B
TWiki-Diffset-0910 | 299.2K | 347.96M
Wikipedia-10 | 6.3M | 4.7B
TWiki-Diffset-1011 | 301.1K | 346.45M
Wikipedia-11 | 6.3M | 4.6B
TWiki-Diffset-1112 | 328.9K | 376.09M
Wikipedia-12 | 6.3M | 4.7B

Table 3: Detailed Statistics of TWiki-Probes during construction. Un and C
represents Unchanged and Changed factual instances, respectively.

| Initial Categorization | $\rightarrow$ | Alignment | $\rightarrow$ | Heuristic Filtering
---|---|---|---|---|---
Month | Un | C | Un | C | Un | C
0809 | 514,017 | 1,209,272 |  | 10,133 | 2,329 |  | 6,935 | 1,776
0910 | 544,708 | 1,196,806 |  | 10,625 | 2,621 |  | 7,340 | 1,982
1011 | 460,228 | 1,572,778 |  | 10,544 | 1,742 |  | 7,313 | 1,358
1112 | 463,623 | 1,653,709 |  | 10,580 | 3,472 |  | 7,293 | 1,951

#### Evaluation Dataset Statistics

The statistics of TWiki-Probes from the initial categorization from Algorithm
2 and quality control are shown in Table 3444A single Wikidata snapshot is
comprised of 93 million distinct entities, where there are around 30 facts for
each entity which amounts to roughly 2.8 billion factual instances. Since most
instances from Algorithm 2 are categorized into Unchanged, we randomly sample
0.1% of the factual instances after applying Algorithm 2..

For further analysis, we break down the entity types of Subject and Object,
and observe a similar proportion of each entity category for each month of
TWiki-Probes (Appendix A). We also show the distribution of the top 30 most
frequent Relation of Unchanged and Changed (Appendix B).

##  5 Experiments with TemporalWiki

In this section, we train and evaluate ever-evolving LMs with TemporalWiki,
which consists of TWiki-Diffsets and TWiki-Probes. Section 5.1 describes the
experimental settings. Section 5.2 describes the baseline methodologies for
updating LMs. Section 5.3 shows evaluation results on the training corpora.
Section 5.4 presents the experimental results on TWiki-Probes.

###  5.1 Experimental Settings

For our experiments, we continue pretraining GPT-2 Large (Radford et al.,
2019) (774M parameters), which serves as our baseline language model (LM). We
first compare the baseline performances between updating GPT-2 with TWiki-
Diffsets and updating it with entire Wikipedia snapshots and evaluate each
update using TWiki-Probes. We also implement continual learning methods from
literature known for mitigating catastrophic forgetting that occurs when
updating GPT-2 with only TWiki-Diffsets. Further detailed configuration of the
experimental settings is provided in Appendix C.

###  5.2 Baseline Models

Here we describe the baseline methods used for training and evaluation, namely
Initial, Full, Diff, RecAdam, Mix-review,K-Adapter, and LoRA as shown in Table
4 and 5.

#### Initial

As the starting model checkpoint for all of the experiments, we first bring
the initially pretrained GPT-2 from Radford et al. (2019) and continue
pretraining it on the 08.2021 Wikipedia snapshot for four epochs in total
(around 546K global steps) so that the initial GPT-2 used for all of the
experiments is updated with the last two years of world knowledge. We denote
this checkpoint as Initial, and it serves as the initial checkpoint for all of
the other methods.

#### Full

We start from Initial and continue pretraining it on the entire Wikipedia
snapshot of each month in a sequential manner. For example, after training on
the 09.2021 Wikipedia snapshot from Initial, we continue training it on the
10.2021 Wikipedia snapshot and move on to the next snapshot. We denote the
resulting model as Full. We iterate through the training data only once, which
corresponds to an average of 4.6 billion token updates (140K global steps) for
each month.

#### Diff

We start from Initial and continue pretraining it on TWiki-Diffsets in a
sequential manner. We denote the resulting model as Diff. Similarly to Full,
we iterate through the training data only once, which is an average of 347
million token updates (12K global steps) for each month.

#### RecAdam

We implement a regularization-based continual learning method for training
large LMs called RecAdam (Chen et al., 2020) which places a stronger
independent assumption among the model parameters, overcoming the limitations
of implementing traditional methods such as EWC (Kirkpatrick et al., 2017) for
training large language models. We set the hyperparameters of the optimizer
identical to the original implementation.

#### Mix-review

We implement a rehearsal-based continual learning method for training large
LMs called Mix-review (He et al., 2021) which mixes in random subsets of the
initial pretraining data (08.2021 Wikipedia data). We fix the mix-ratio as 2
in our experiments.

#### LoRA

We implement a parameter-expansion-based continual learning method called LoRA
(Hu et al., 2021) which freezes the original parameters while adding trainable
rank-decomposition matrices into each layer. We use hyperparameters identical
to the optimal setting of the original implementation.

#### K-Adapter

We implement another parameter-expansion-based continual learning method,
K-Adapter (Wang et al., 2021), which freezes the original parameters while
adding additional adapters (an increase of 103M parameters) to the LM. 555We
add the additional parameters once for the updates from 08.2021. Exploring the
optimal interval to add parameters for ever-evolving LMs is left for future
work.

###  5.3 Intrinsic Evaluation

(a) Non-TWiki-Diffsets

(b) TWiki-Diffsets

Figure 3: Relative proper noun perplexity of Full, Diff, and K-Adapter, LoRA,
RecAdam and Mix-review compared to Initial on TWiki-Diffsets and Non-TWiki-
Diffsets for each month. Lower ratio indicates better performance. The
performance of Diff (orange) and RecAdam (yellow) in (b) is is almost
identical. Table 4: Zero-shot perplexity of LMs measured on TWiki-Probes. Time
represents the average training time of a single update under the setting
described in Section 5.1. The description of each baseline model is explained
in Section 5.2. Best performance is marked as bold while the second best is
underlined.

|  | TWiki-Probes-0809 | TWiki-Probes-0910 | TWiki-Probes-1011 | TWiki-Probes-1112
---|---|---|---|---|---
| Time | Un | C | Avg | Un | C | Avg | Un | C | Avg | Un | C | Avg
Initial | 0 hours | 386.16 | 364.82 | 375.49 | 356.66 | 416.32 | 386.49 | 350.54 | 420.52 | 385.53 | 357.37 | 451.74 | 404.56
Full |  $\sim$24 hours | 379.43 | 360.46 | 369.95 | 388.85 | 437.15 | 413.00 | 337.34 | 383.06 | 360.20 | 381.11 | 435.47 | 408.29
Diff |  $\sim$2.5 hours | 409.31 | 284.34 | 346.83 | 409.86 | 336.55 | 373.21 | 465.20 | 367.72 | 416.46 | 391.77 | 365.07 | 378.42
RecAdam |  $\sim$4 hours | 358.10 | 253.07 | 305.59 | 376.12 | 306.64 | 341.38 | 439.14 | 338.17 | 388.66 | 400.56 | 356.60 | 378.58
Mix-review |  $\sim$6 hours | 337.59 | 274.91 | 306.25 | 394.20 | 381.21 | 387.71 | 375.85 | 369.50 | 372.68 | 313.94 | 323.49 | 318.72
LoRA |  $\sim$2 hours | 386.52 | 332.98 | 359.75 | 359.54 | 371.03 | 365.29 | 381.80 | 391.66 | 386.73 | 361.42 | 408.19 | 384.81
K-Adapter |  $\sim$2 hours | 340.47 | 297.39 | 318.93 | 326.53 | 338.16 | 332.35 | 325.11 | 332.61 | 328.86 | 333.53 | 374.67 | 354.10

We first perform intrinsic evaluation by measuring the perplexity of the
baseline models on their training corpora. For each month, we measure the
model’s perplexity on TWiki-Diffsets and Non-TWiki-Diffsets, where the latter
refers to the subset of the month’s entire Wikipedia snapshot that does not
include the data from TWiki-Diffsets. We sample 10,000 input instances from
each subset with a fixed length of 512 and measure the perplexity on proper
noun tokens determined by a Part-of-Speech (POS) tagger (Honnibal and Montani,
2017) as in (Lazaridou et al., 2021), which can be considered as a proxy for
tokens containing factual knowledge. Therefore, the result on Non-TWiki-
Diffsets is meant to indicate the performance on unchanged knowledge, while
the result on TWiki-Diffsets corresponds to updated and new knowledge. Figure
3 shows the relative perplexity of each baseline method compared to Initial
(i.e., dividing each model by Initial, and thus the lower, the better).

Results on Non-TWiki-Diffsets show that the relative perplexity of Diff
increases rapidly while that of Full remains constant as time goes on, which
implies that forgetting occurs when the LM is trained with TWiki-Diffsets. The
relative perplexities of continual learning methods increase less rapidly than
Diff, which means that applying continual learning mitigates catastrophic
forgetting. Mix-review, especially, shows the least amount of forgetting among
the continual learning methods, which indicates that training on the past
corpus is effective in retaining performance on the previous training corpora
in terms of perplexity.

On the other hand, the results on TWiki-Diffsets show the opposite trend: the
relative perplexity of Diff is much lower than Full. One thing to note is that
the perplexity of Full is very similar to that of Initial on TWiki-Diffsets,
which suggests that updating LMs on entire Wikipedia snapshots hinders the
effective learning of changed data compared to Diff, despite both having seen
the same instances of TWiki-Diffsets during training for the same number of
iterations. Among continual learning methods, K-Adapter and LoRA shows higher
overall perplexities than Diff while Mix-review and RecAdam shows similar
perplexity to Diff on TWiki-Diffsets.

###  5.4 Extrinsic Evaluation on TWiki-Probes

Figure 4: Average overall perplexity of TWiki-Probes. We average the
perplexities of Unchanged and Changed with equal importance placed on
stability and plasticity. The x-axis depicts the two-month intervals. A lower
score indicates better performance.

Performing only intrinsic evaluation on the training corpora is not sufficient
because the intrinsic evaluation itself only tests the capability of the LMs
for memorization (McCoy et al., 2021). Through extrinsic evaluation with
TWiki-Probes (Section 3.2), we specifically focus on evaluating factual
knowledge of the LMs from each update. Placing equal importance on stability
(Unchanged) and plasticity (Changed), we show the average of the perplexities
of Unchanged and Changed as well as individual perplexities in Table 4, and
show a bar graph of the average perplexities in Figure 4666The perplexity of
Unchanged and Changed were each calculated by measuring the average perplexity
of generating each factual instances..

As shown in Table 4, Diff and all continual learning methods show better
overall performance on Changed factual instances than Initial in all months,
bolstering the results from the intrinsic evaluation. For Unchanged, however,
Diff suffers from catastrophic forgetting, showing consistent performance
degradation as the number of updates increases. In contrast, continual
learning methods effectively mitigate much of the catastrophic forgetting
during temporal language modeling, resulting in lower perplexity on Unchanged,
except RecAdam which performs worse as the number of updates increases.
K-Adapter, especially, shows surprising results on Unchanged, outperforming
even Full throughout all of the months. Moreover, all continual learning
methods surpass or are on par with Diff on Changed factual instances, showing
that ability to learn new knowledge (plasticity) is not sacrificed to preserve
previous knowledge (stability).

Moreover, as shown in the average perplexity column of Table 4 and Figure 4,
K-Adapter shows the most robust performance throughout the time periods. It is
important to note that K-Adapter is around 12 times more computationally
efficient than Full in terms of total training time, under the same
computational constraint. Diff also outperforms Full in all months but 1011,
showing that temporal language modeling itself is an effective approach for
overall stability-plasticity trade-off.

We note that, as also shown in previous works (Lazaridou et al., 2021),
results in Table 4 present an overall high perplexity (>200) because the
sentences in TWiki-Probes are not natural sentences; they are factual phrases
synthetically generated from a naive concatenation of Subject, Relation, and
Object777Using the pre-defined templates of LAMA (Petroni et al., 2019) seems
to be an option, but we find that those templates do not fit well to our
experiments because there is a considerable distribution gap between LAMA and
TWiki-Probes; over half of the instances of TWiki-Probes are filtered out to
apply the templates, especially for Changed. . We discuss experiments with
light-tuning as an alternative in Appendix D.

(a) Full

(b) Diff

(c) K-Adapter

Figure 5: The zero-shot perplexity of the LMs updated and evaluated on various
time intervals of Changed of TWiki-Probes, showing the effect of temporal
misalignment. The better the results, the darker the performance is colored.
The color is compared within the same method and also the same evaluation set.

#### Effect of Temporal Misalignment

We quantify the effect of temporal misalignment on each method by training the
LMs and evaluating their zero-shot perplexity on Changed instances of TWiki-
Probes with various time intervals of training and evaluation. Among continual
learning methods, we select K-Adapter since it shows the most robust
performance for extrinsic evaluation across all time periods. As shown in
Figure 5, Full method is mostly influenced by the number of training updates
and not much by whether there is temporal alignment. Since Full is
continuously pretrained on the entire Wikipedia corpus in each month, it would
have likely seen the data containing Changed factual instances multiple times,
leading to lower perplexity as training steps increases.888Although directly
training Initial on the whole Wikipedia corpus of a specific month can be an
alternative, we exclude it here because it would only learn the knowledge of
the specific month and thus inappropriate for a truly ever-evolving setting.
For Diff and K-Adapter, there is a general trend of strong performance when
there is temporal alignment (diagonal entries), outperforming Full with much
fewer global training steps. It is important to note that K-Adapter shows
robustness against temporal misalignment, i.e., the perplexity does not
increase much even when the training and evaluation months do not match,
compared to Diff which suffers from a more severe perplexity spike.

##  6 Conclusion

In this paper, we provide answers to the four proposed questions in Section 1.
(1) How can we train ever-evolving LMs efficiently and automate the evaluation
of each update? We introduce TemporalWiki, a lifelong benchmark that can be
used for training and evaluating ever-evolving language models (LMs) in an
automated manner. It consists of TWiki-Diffsets as the training corpora for
temporal language modeling and TWiki-Probes as the evaluation datasets for
measuring the stability-plasticity trade-off at each LM update. (2) How does
updating LMs only on new and updated data from Wikipedia compare to updating
LMs on entire Wikipedia snapshots, especially in scenarios with multiple
updates? Through experiments on TemporalWiki, we show that updating LMs on
TWiki-Diffsets leads to better acquisition of new and updated knowledge than
updating on entire Wikipedia snapshots with much less computational cost (12
times less). (3) How serious is catastrophic forgetting when LMs are updated
only on new and updated data? We observe that temporal language modeling is a
challenging problem, especially as the number of LM updates increases.
However, results still show an overall enhancement in terms of stability-
plasticity compared to updating with entire Wikipedia snapshots, showing that
temporal language modeling is an effective alternative. (4) How can we
mitigate catastrophic forgetting? We find that continual learning methods
(regularization, rehearsal, and parameter-expansion) for large language model
training effectively mitigates forgetting and shows robust performance in
terms of enhancing the overall trade-off between stability and plasticity on
TWiki-Probes.

## References

  * Agarwal and Nenkova (2021) Oshin Agarwal and Ani Nenkova. 2021.  Temporal effects on pre-trained models for language processing tasks.  _arXiv preprint arXiv:2111.12790_.
  * Borgeaud et al. (2021) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2021.  Improving language models by retrieving from trillions of tokens.  _arXiv preprint arXiv:2112.04426_.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  In _NeurIPS_.
  * Chen et al. (2020) Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. 2020\.  Recall and learn: Fine-tuning deep pretrained language models with less forgetting.  In _EMNLP_.
  * Chen et al. (2021) Wenhu Chen, Xinyi Wang, and William Yang Wang. 2021.  A dataset for answering time-sensitive questions.  In _NeurIPS_.
  * Dhingra et al. (2021) Bhuwan Dhingra, Jeremy R Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W Cohen. 2021.  Time-aware language models as temporal knowledge bases.  _arXiv preprint arXiv:2106.15110_.
  * Dinan et al. (2019) Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019.  Wizard of wikipedia: Knowledge-powered conversational agents.  In _ICLR_.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.  Realm: Retrieval-augmented language model pre-training.  In _ICML_.
  * He et al. (2021) Tianxing He, Jun Liu, Kyunghyun Cho, Myle Ott, Bing Liu, James Glass, and Fuchun Peng. 2021.  Analyzing the forgetting problem in pretrain-finetuning of open-domain dialogue response models.  In _EACL_.
  * Hombaiah et al. (2021) Spurthi Amba Hombaiah, Tao Chen, Mingyang Zhang, Michael Bendersky, and Marc Najork. 2021.  Dynamic language models for continuously evolving content.  In _KDD_.
  * Honnibal and Montani (2017) Matthew Honnibal and Ines Montani. 2017.  spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing.  To appear.
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_.
  * Jang et al. (2022) Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. 2022.  Towards continual knowledge learning of language models.  In _ICLR_.
  * Jin et al. (2021) Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, and Xiang Ren. 2021.  Lifelong pretraining: Continually adapting language models to emerging corpora.  _arXiv preprint arXiv:2110.08534_.
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ , 114(13):3521–3526.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019.  Natural questions: a benchmark for question answering research.  _TACL_ , 7:453–466.
  * Lazaridou et al. (2021) Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Tomas Kocisky, Sebastian Ruder, et al. 2021.  Mind the gap: Assessing temporal generalization in neural language models.  In _NeurIPS_.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  In _NeurIPS_.
  * Lewis et al. (2021) Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021.  Question and answer test-train overlap in open-domain question answering datasets.  In _EACL_.
  * Logan IV et al. (2021) Robert L Logan IV, Alexandre Passos, Sameer Singh, and Ming-Wei Chang. 2021.  Fruit: Faithfully reflecting updated information in text.  _arXiv preprint arXiv:2112.08634_.
  * Longpre et al. (2021) Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021.  Entity-based knowledge conflicts in question answering.
  * Loureiro et al. (2022) Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, and Jose Camacho-Collados. 2022.  Timelms: Diachronic language models from twitter.  _arXiv preprint arXiv:2202.03829_.
  * Luu et al. (2022) Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Karishma Mandyam, and Noah A Smith. 2022.  Time waits for no one! analysis and challenges of temporal misalignment.  In _NAACL_.
  * McCloskey and Cohen (1989) Michael McCloskey and Neal J Cohen. 1989.  Catastrophic interference in connectionist networks: The sequential learning problem.  _Psychology of learning and motivation_.
  * McCoy et al. (2021) R Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. 2021\.  How much do language models copy from their training data? evaluating linguistic novelty in text generation using raven.  _arXiv preprint arXiv:2111.09509_.
  * Mermillod et al. (2013) Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. 2013.  The stability-plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects.  _Frontiers in Psychology_.
  * Osborne et al. (2014) Miles Osborne, Ashwin Lall, and Benjamin Van Durme. 2014.  Exponential reservoir sampling for streaming language models.  In _ACL_.
  * Petroni et al. (2021) Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al. 2021.  Kilt: a benchmark for knowledge intensive language tasks.  In _NAACL_.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019.  Language models as knowledge bases?  In _EMNLP_.
  * Piktus et al. (2021) Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Dmytro Okhonko, Samuel Broscheit, Gautier Izacard, Patrick Lewis, Barlas Oğuz, Edouard Grave, Wen-tau Yih, et al. 2021.  The web is your oyster–knowledge-intensive nlp against a very large web corpus.  _arXiv preprint arXiv:2112.09924_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _JMLR_.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model?  In _EMNLP_.
  * Rosin et al. (2021) Guy D Rosin, Ido Guy, and Kira Radinsky. 2021.  Time masking for temporal language models.  _arXiv preprint arXiv:2110.06366_.
  * Röttger and Pierrehumbert (2021) Paul Röttger and Janet B Pierrehumbert. 2021.  Temporal adaptation of bert and performance on downstream document classification: Insights from social media.  In _Findings of EMNLP_.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2022.  Multitask prompted training enables zero-shot task generalization.  In _ICLR_.
  * Smith (2018) Leslie N Smith. 2018.  A disciplined approach to neural network hyper-parameters: Part 1–learning rate, batch size, momentum, and weight decay.  In _CVPR_.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018\.  Fever: a large-scale dataset for fact extraction and verification.  In _NAACL_.
  * Wang et al. (2021) Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Cuihong Cao, Daxin Jiang, Ming Zhou, et al. 2021.  K-adapter: Infusing knowledge into pre-trained models with adapters.  In _Findings of ACL_.
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2022.  Finetuned language models are zero-shot learners.
  * Yogatama et al. (2014) Dani Yogatama, Chong Wang, Bryan R Routledge, Noah A Smith, and Eric P Xing. 2014\.  Dynamic language models for streaming text.  _TACL_.
  * Zhang and Choi (2021) Michael JQ Zhang and Eunsol Choi. 2021.  Situatedqa: Incorporating extra-linguistic contexts into qa.  In _EMNLP_.

##  Appendix A Details of Entity Types of Subject and Relation

Figure 6 shows the ratio of different entity types of Subject and Object of
Unchanged and Changed.

(a) Unchanged

(b) Changed

Figure 6: Entity types of Subject and Object in TWiki-Probes.

##  Appendix B Details of Relation Distribution

The distribution of Relation for Unchanged, Changed factual instances in
TWiki-Probes is shown in Figure 7.

(a) Unchanged

(b) Changed

Figure 7: TWiki-Probes distribution of the top 30 Relation.

##  Appendix C Continual Pretraining and Light Tuning Configuration

For each LM update, we use 8 32GB V100 GPUs with a global batch size of 64 and
a fixed input sequence length of 512. We use the max learning rate of 1e-4 and
one cycle learning rate scheduling policy (Smith, 2018). For light-tuning, the
training is done for only one epoch with a learning rate of 1e-5 and a batch
size of 32. Input and output sequence lengths are set to 25. For continual
learning-based methods, we unfreeze all of the parameters during light-tuning,
following Jang et al. (2022).

##  Appendix D Light-tuning results with TWiki-Probes

Table 5: Light-tuning perplexity of LMs measured on TWiki-Probes.

| TWiki-Probes-0809 | TWiki-Probes-0910 | TWiki-Probes-1011 | TWiki-Probes-1112
---|---|---|---|---
| Un | C | Un | C | Un | C | Un | C
Initial | 116.99 | 142.58 | 108.89 | 167.82 | 106.14 | 172.18 | 114.64 | 177.02
Full | 124.37 | 145.89 | 112.51 | 172.70 | 105.09 | 164.59 | 118.54 | 164.17
Diff | 120.52 | 116.44 | 125.80 | 142.82 | 132.83 | 156.60 | 144.61 | 164.34
RecAdam | 122.58 | 118.14 | 125.90 | 143.65 | 137.15 | 148.24 | 144.76 | 159.52
Mix-review | 116.53 | 121.57 | 119.39 | 154.72 | 119.16 | 157.59 | 118.64 | 145.29
LoRA | 123.62 | 130.41 | 115.54 | 156.07 | 115.26 | 165.51 | 122.11 | 169.59
K-Adapter | 115.93 | 134.46 | 116.27 | 154.11 | 110.17 | 158.21 | 117.22 | 167.44

Table 6: F1 score result of LMs on TWiki-Probes after light-tuning.

| TWiki-Probes-0809 | TWiki-Probes-0910 | TWiki-Probes-1011 | TWiki-Probes-1112
---|---|---|---|---
| Un | C | Un | C | Un | C | Un | C
Initial | 6.98 | 3.19 | 7.26 | 3.35 | 7.27 | 2.74 | 6.84 | 2.82
Full | 4.68 | 2.45 | 5.62 | 3.06 | 7.12 | 2.25 | 4.28 | 2.30
Diff | 7.51 | 4.38 | 6.91 | 4.46 | 5.24 | 2.65 | 5.45 | 4.38
RecAdam | 5.74 | 3.79 | 6.31 | 3.86 | 4.47 | 2.43 | 5.09 | 3.68
Mix-review | 7.12 | 3.31 | 6.16 | 3.56 | 6.63 | 2.08 | 6.84 | 3.67
LoRA | 7.36 | 4.48 | 7.23 | 3.89 | 7.19 | 3.87 | 6.82 | 3.81
K-Adapter | 7.54 | 3.99 | 7.34 | 3.73 | 7.38 | 3.91 | 6.87 | 3.30

To alleviate the distributional shift that causes high zero-shot perplexity,
we light-tune the LMs on 500 instances randomly sampled from WikiData that do
not overlap with instances from TWiki-Probes (details in Appendix E). Unlike
finetuning, light-tuning lets the LM only learn the input and output
distribution of the task, avoiding the problem of test-train overlap pointed
out by Lewis et al. (2021). Table 5 shows the results of light-tuning, which
demonstrate a similar trend as the zero-shot performance. Although light-
tuning avoids the problem of test-train overlap, results are largely affected
by the sampled instances for tuning, so a zero-shot evaluation setting is
preferred for reliability.

Many knowledge-intensive tasks such as closed-book question answering (Roberts
et al., 2020; Petroni et al., 2021; Jang et al., 2022) or slot filling Petroni
et al. (2021) use accuracy, EM, or F1 score to evaluate the task. We also show
the F1 score on TWiki-Probes in Table 6. Overall trend is consistent with
zero-shot perplexity metric; K-Adapter shows robust performance for both
Unchanged and Changed.

##  Appendix E Light-Tuning Data

Table 7: Statistics of the data used for Light-Tuning

| Size |  | # of
---
Relation
| Maximum
---
Repetition
of Relation
| # of
---
Subject
Unchanged | 500 | 102 | 58 | 499
Changed | 500 | 140 | 31 | 500

We sample 500 instances from WikiData for each time step that do not overlap
with instances from TWiki-Probes for each factual instance category. During
sampling, we keep the distribution of each Relation proportional to the
original distribution. Table 7 shows the size and distribution of Relation of
light-tuning datasets.


Firefighting is a complex, yet low automated task. To mitigate ergonomic and safety related risks on the human operators, robots could be deployed in a collaborative approach. To allow human-robot teams in firefighting, important basics are missing. Amongst other aspects, the robot must predict the human motion as occlusion is ever-present. In this work, we propose a novel motion prediction pipeline for firefighters' squads in indoor search and rescue. The squad paths are generated with an optimal graph-based planning approach representing firefighters' tactics. Paths are generated per room which allows to dynamically adapt the path locally without global re-planning. The motion of singular agents is simulated using a modification of the headed social force model. We evaluate the pipeline for feasibility with a novel data set generated from real footage and show the computational efficiency.

§ INTRODUCTION

Indoor search and rescue (SAR) is exceptionally stressful to the firefighting operators. We aim to deploy humans and mobile robots as a collaborative squad to reduce health risks of diverse nature. A collaborative squad consists of multiple firefighting operators (the squad) and a collaborative rescue robot. Similar to the human, the robot is challenged by obscured vision on the environment and that conditions may change drastically throughout each mission. Therefore, the robot must deploy multiple methodologies to perceive and predict the location of its assigned human squad. This is analogous to approaches established in service robotics, where the robot perceives the human locations through sensors but is also able to predict their future whereabouts through motion modeling. For indoor SAR missions, no motion and behavioral models are known. In this work, we propose an approach to model human behavior throughout such missions by combining a graph-based tactics-informed optimal planning approach and a modification of the headed social force model (HSFM) <cit.>. Optimal paths are planned according to common tactics considering restricted and unrestricted vision per room and per squad. The planning is based on a priori knowledge of the environment map and status, which are usually available at the start of a mission. Through the segmentation into room entities, paths are adapted to the current scene when predicted conditions change. Individual agents (i.e., the simulated firefighting operators), then, use the optimal paths as waypoints in their individual motion prediction. By this, inter- and intra-squad behavior is modeled. The models are parameterized according to real data and evaluated for usability in real-time systems.

Our main contributions in modeling indoor SAR are:

* New model and novel pipeline for motion prediction

* Composed novel data set to parameterize motion models

§ RELATED WORK

Modeling of human behavior is a common task outside of SAR. Of particular interest for this work are physics based models with dynamic environment and group cues. One of the earliest adoptions is the social force model (SFM) by Helbing et al. <cit.>, which was later adapted by many works: Moussaïd et al. <cit.> add group cues to the SFM. Pellegrini et al. <cit.> introduce collision prediction. Yamaguchi et al. <cit.> decide on the (hidden) group status by observing motion. Rudenko et al. <cit.> combine a velocity model based on a Markov decision process with SFM-based local interactions and a stochastic random walk policy.

Farina et al. <cit.> extend the SFM to the HSFM by applying a locomotion model which prefers forward motion in gazing direction of the agent. The locomotion model is defined by

\begin{gather}
\mathbf{R}(\theta_{i}) =
\begin{bmatrix}
\mathbf{e}_{x} & \mathbf{e}_{y}
\end{bmatrix} =
\begin{bmatrix}
\cos{\theta_{i}} & -\sin{\theta_{i}}\\
\sin{\theta_{i}} & \cos{\theta_{i}}
\end{bmatrix},\\
\dot{\mathbf{p}}_{i} = \mathbf{R}(\theta_{i})\mathbf{v}_{i}, \quad
\dot{\mathbf{v}}_{i} = \frac{1}{m_{i}}\mathbf{u}_i^{B}, \quad
\dot{\omega}_{i} = \frac{1}{I_{i}}u_{i}^{\theta}, %\addtocounter{equation}{1}\tag{\theequation,c,d}
\tag{\addtocounter{equation}{1}\theequation,\addtocounter{equation}{1}\theequation,\addtocounter{equation}{1}\theequation}
\end{gather}

where an agent $i$ is characterized by their pose , velocity , mass $m_{i}$, and inertia $I_{i}$, accordingly. The gazing direction $\theta_{i}$ is spanned between the global frame and a frame co-moving with the agent, hence, the distinction of velocities into $\dot{\mathbf{p}}_{i}$ (global) and $\mathbf{v}_{i}=\begin{bmatrix} v_{i,x} & v_{i,y} \end{bmatrix}^{T}$ (local). The lateral input forces $\mathbf{u}_{i}^{B}=\begin{bmatrix}u_{i}^{f} & u_{i}^{o}\end{bmatrix}^{T}$ and torque input $u_{i}^{\theta}$ are deconstructions of the social forces, defined by

\begin{align}
u_{i}^{f} &= \mathbf{f}_{i}\mathbf{e}_{x}(\theta_{i}),\\
u_{i}^{o} &= C^{o}(\mathbf{f}_{i}-\mathbf{f}_{i}^{acc})\mathbf{e}_{y}(\theta_{i})-C^{des}v_{i}^{des,o},\\
u_{i}^{\theta} &= -C^{\theta}(\theta_{i}-\phi_{i}^{acc})-C^{\omega}\omega_{i},
\end{align}

with the social forces resulting from external factors, i.e., agent-agent $\mathbf{f}_{i}^{j}$ and agent-border $\mathbf{f}_{i}^{B,k}$ interaction, and the agent's motivation to reach the goal $\mathbf{f}_{i}^{acc}$. The total social force $\mathbf{f}_{i}$ acting on the individual agent $i$ is the sum of all partial forces, defined by

\begin{equation}
\mathbf{f}_{i} = \mathbf{f}_{i}^{acc} + \sum_{j\ne i}^{\text{agents}}\mathbf{f}_{i}^{j} + \sum_{k}^{\text{borders}}\mathbf{f}_{i}^{B,k}.
\end{equation}

In this work, $\phi_{i}^{acc}$ is the phase of $\mathbf{f}_{i}^{acc}=\{d_{i}^{acc},\phi_{i}^{acc}\}$. $C^{o}=1$, $C^{des}=500$ are scaling parameters and $v_{i}^{des,o}$ is the desired velocity in orthogonal direction. The torque parameters $C^{\theta}$ and $C^{\omega}$ are configured according to <cit.>.

§ TACTICS AND RESTRICTIONS OF THE MODEL

Fire brigades worldwide apply different tactics, however, the overall approach is similar. A mission is usually split into multiple stages, e.g., the German THW employs five <cit.> and the Australian NDO six stages <cit.>. This work relates to firefighting in Germany, particularly, to the attack stage, which involves exploring indoor disaster environments, commonly, with respiratory protection <cit.>.

§.§ Motion Tactics in Indoor Attack

During indoor attack, motion tactics are applied per room based on its size and the visual conditions. In Germany, four different tactics are applied. In non-obscured vision, the squad explores the room until the entire space has been visually inspected without further instructions on exact motion (referred to as free traversal). In case of vision restriction, wall search (from German: “Wandtechnik”, Figure <ref>) is applied. Within, one firefighter constantly keeps their hand to the right- or left-hand wall, hence, the distinction in right-hand rule (RHR) and left-hand rule (LHR), respectively. They stretch out their arms and utilize squad mates and equipment to extend their reach. This technique is used in small to medium sized rooms, which are common in civil buildings, e.g., dorms or office buildings. In case of larger rooms, diving search (“Tauchertechnik”, Figure <ref>) and tree search (“Baumtechnik”, Figure <ref>) are applied. To establish a first baseline for firefighters' motion prediction, we focus on smaller rooms, hence, only free traversal and wall search (RHR and LRH) are modeled in this work.

§.§ Motion Data

For motion modeling, not only pure knowledge about the tactics, but also data is required to parameterize the models. However, real-world data is hard to come by as firefighters usually do not carry recording devices. Therefore, we base our data set on a thermal recording of the Institut der Feuerwehr NRW (IdF) generated in the project KOORDINATOR <cit.>, which depicts approaches under vision restriction with wall search, and two TV shows “Feuer & Flamme” <cit.> and “112: Feuerwehr im Einsatz” <cit.>. The TV shows display real footage of indoor SAR with and without vision restriction. Of particular interest are the velocities and distances the squad applies to borders and between squad members. The parameters we tune our model to are listed in Table <ref>. All parameters have high variance due to the low sample size. Particularly, velocity data is not trustworthy. However, as there is virtually no data for indoor SAR, these statistics implement vital reference values.

For motion modeling, not only pure knowledge about the tactics, but also data is required to parameterize the models. However, real-world data is hard to come by as firefighters usually do not carry recording devices. Therefore, we base our data set on a thermal recording of the Institut der Feuerwehr NRW (IdF) generated in the project KOORDINATOR <cit.>, which depicts approaches under vision restriction with wall search, and two TV shows <cit.>. The TV shows display real footage of indoor SAR with and without vision restriction. Of particular interest are the velocities and the distances the squad applies to borders and between squad members. The parameters we tune our model to are listed in Table <ref>. All parameters have high variance due to the low sample size. Particularly, velocity data is not trustworthy. However, as there is virtually no data for indoor SAR, these statistics implement vital reference values.

§ GRAPH-BASED TACTICS-INFORMED OPTIMAL PLANNING

The prediction framework is split into two stages. First, waypoints are generated by optimal path planning per room and squad. These are, then, used in the HSFM to generate agent-individual trajectories. The framework can generate motion trajectories of single agents as well as of individual agents in a single or in multiple squads. As input, a building plan is required, which is usually available for newer buildings and may be interpreted by other means (compare <cit.> for an overview). The map is segmented into individual rooms and their connecting doorways. These are, then, stored in a room graph in which nodes are rooms and vertices are doorways between them. For each room, a graph is generated which expands the room's node as a sub-graph embedded in the room graph. By this, global room sequences and local paths in each room may be planned isolated or together in a global optimization approach.

Parameters derived from <cit.>.

Parameter     Vision     Mean ($\mu$)     Variance ($s^2$)     Samples

Dist. in squad      Free       $0.634$     $0.301$     $105$

(in m)              Restr.     $0.275$     $0.073$     $325$

Dist. to border     Free       $0.392$     $0.106$     $76$

(in m)              Restr.     $0.297$     $0.134$     $188$

Velocity            Free       $1.500$     $0.212$     $22$

(in m/s)            Restr.     $0.326$     $0.056$     $29$

Wall search.

Diving search.

Tree search.

Search tactics under restricted vision in individual rooms. Wall and diving search use base tactic RHR to approach the room (SL: squad leader, SM: squad mate).

§.§ Graph Construction

For graph construction, we use three different types of graphs: Medial axis, visibility road map and pseudo-random sampling. The medial axis (Figure <ref>) is constructed from a Voronoi graph using the method proposed by Masehian and Amin-Naseri <cit.>. The seeds of the Voronoi regions are located in the occupied regions of the grid map. These are determined using distance transforms <cit.>. The medial axis is a graph that is constructed only from Voronoi border regions, i.e., where multiple Voronoi regions meet. Hence, it generates a graph with maximal distance to boundaries. The visibility road map <cit.> (Figure <ref>) generates a graph which guarantees that all space is visible. During construction, nodes are sampled pseudo-randomly using the Halton sequence <cit.>. Each sampled node is checked for visibility and added if it (a) is not visible from any other node (called guard) or (b) connects at least two guard nodes that were not connected beforehand (called connector). By checking the visibility road map it is determined whether space has been observed, which is particularly important when modeling free traversal behavior in SAR. Firefighters explore rooms until the entire space has been inspected to locate casualties. The remaining space is filled with nodes pseudo-randomly sampled with the Hammersley sequence <cit.>.

Medial axis.

Visibility road map.

Combined graph of sfig:graph_medial, sfig:graph_visibility, and pseudo-random sampling.

Greedy tree search on sfig:graph_visibility.

Graphs generated (blue) and path planned (red) in free traversal on complex artificial map.

Nodes are connected based on proximity, i.e., all neighbors in a specified radius are connected by vertices (Figure <ref>). The vertices are generated bidirectional, whereas each directed vertex carries information on the Euclidean length and in which motion tactics it is permissible for path planning. Free traversal allows all vertices. However, wall search requires locations close to walls, hence, vertices to far regions are blocked. Further, depending on the direction of motion, only clockwise (LHR) or counter-clockwise (RHR) vertices are allowed. On the combined graph, the shortest path within the given vertex restrictions is chosen. We observe good path quality with the A* algorithm in all three types of motion.

§.§ Special Case: Single Entry Rooms

In rooms with a single entry, the agent would be allowed to directly move back out, given the prior explanation. This is prevented by following considerations: Free traversal additionally requires all nodes in the visibility road map to be visited. To guarantee this, we build a tree from the entry node and follow the closest vertices in a greedy manner until a leaf is reached (Figure <ref>). Then new branches are generated from previously visited nodes until a new leaf is reached. In wall search, the first node in the path is moved by one margin clockwise (LHR) or counter-clockwise (RHR). As backtracking vertices are blocked, the agent is forced to move along the wall through the room instead of moving out of the doorway.

§ MODIFIED HEADED SOCIAL FORCE MODEL

After paths have been generated per squad, they are propagated to each agent in their respective squad. Motion trajectories are simulated using a modification of the HSFM.

§.§ Waypoint Management

An agent derives the full list of waypoints from its squad's path. Agents delete their waypoints individually if latter have been visited. A waypoint is marked visited if it is located (a) within a cone in the agent's gazing direction or (b) within a slim circle centered at the agent's position. The cone has a range of $50$m in free and $2$m in restricted vision, and an opening angle of $180$°. The circle is defined with a range of $0.2$m. Latter's main purpose is to delete waypoints if an agent is spawned exactly on top, in which case the cone alone is not sufficient. Waypoints are marked essential if they are located at key locations, e.g., doorways, or start/end points. An essential waypoint is only deleted using criterion (b), but with a relaxed threshold. Thus, agents maintain a goal over long distances which prevents them from sliding along walls.

§.§ Modified Contact Model

In the HSMF <cit.>, agents typically stay far from walls, e.g., in corridors they tend to walk in the middle. In firefighting, however, typical behavior (particularly in wall search) leads to agents staying much closer to walls. In addition, to safe computational resources our aim is to renounce the usage of high level extraction of semantic building entities besides rooms, e.g., the agglomeration of occupied pixels into walls. These two aspects lead to the challenge, that agents may get pushed through walls and once they are, there is no way back, as it is not possible to determine from which side an occupied area was penetrated. Note that no past data is stored.

Four quadrants and occupied space considered for border forces (green).

Partial forces $\mathbf{f}_{i}^{B,k}$ and resulting force $\mathbf{f}_{i}^{B}$ according to considered centers of mass.

Sketch of the agent-border interaction forces.

To mitigate this challenge, we define a new contact model. Occupied pixels are only considered for border forces, if they fall into a distance threshold and are closest in a quadrant relative to the agent's gazing direction (Figure <ref>). The quadrants are defined such that the first quadrant (I) is centered about the agent's gazing direction. In all four quadrants, the partial border forces $\mathbf{f}_{i}^{B,k}$ are computed and agglomerated into the total border force $\mathbf{f}_{i}^{B}$ (Figure <ref>). The partial border forces are computed as an exponential force superimposed by a linear force. We observe, that the pure exponential force used by Farina et al. <cit.> acts too slow to prevent agents from penetrating walls if configured according to Table <ref>. The linear force rises faster while the final force is typically lower. Hence, the agent is decelerated but may still approach the wall. In very close regions, the exponential force forces the agent away from the borders. As agents approach borders with lower velocity, they cannot skip the border within an update cycle, hence, penetration is prevented. We define the partial border force as

\begin{equation}
\label{eq:contact}
\begin{split}
\mathbf{f}_{i}^{B,k} &= \phi^{B}(r_{i}-d_{i}^{B,k})\mathbf{n}_{i}^{B,k}\\
\mathbf{0} & \text{, } d_{i}^{B,k}>r_{i}\\
\begin{split}
\end{split}
& \text{, otherwise.}
\end{cases}
\end{split}
\end{equation}

Hereby, $r_{i}$ is the radius of the agent and

\begin{equation}
d_{i}^{B,k}=||\mathbf{p}_{i} - \mathbf{p}_{k} -  \frac{\sqrt{2}}{2}w\mathbf{n}_{i}^{B,k}||_{2}
\end{equation}

is the distance from the agent to the center of the closest occupied pixel $\mathbf{p}_{k}$ considering half the pixel width $w$, simplified as circle with conservative radius $0.5\sqrt{2}w$. The contact direction is split into the normalized normal and normalized tangential direction vectors

\begin{gather}
\mathbf{n}_{i}^{B,k} =
\begin{bmatrix}
n_{i,x}^{B,k}\\ n_{i,y}^{B,k}
\end{bmatrix}
= \frac{\mathbf{p}_{i}-\mathbf{p}_{k}}{d_{i}^{B,k}}, \quad
\mathbf{t}_{i}^{B,k} =
\begin{bmatrix}
- n_{i,y}^{B,k}\\ n_{i,x}^{B,k}
\end{bmatrix},
\tag{\addtocounter{equation}{1}\theequation,\addtocounter{equation}{1}\theequation}
\end{gather}

\begin{align}
\mathbf{n}_{i}^{B,k} &=
\begin{bmatrix}
n_{i,x}^{B,k} & n_{i,y}^{B,k}
\end{bmatrix}^{T}
= \frac{\mathbf{p}_{i}-\mathbf{p}_{k}}{d_{i}^{B,k}},\\
\mathbf{t}_{i}^{B,k} &=
\begin{bmatrix}
- n_{i,y}^{B,k} & n_{i,x}^{B,k}
\end{bmatrix}^{T},
\end{align}

respectively. Note that the tangential part in Equation <ref> scales with the agent's orthogonal velocity $v_{i,y}$. The border potential is defined as

\begin{equation}
\label{eq:border_potential}
\phi^{B}(r_{i}-d_{i}^{B,k}) = \phi_{0}^{B}\cdot \exp{(\frac{r_{i}-d_{i}^{B,k}}{C^{B}})}.
\end{equation}

The soft potential is defined as

\begin{equation}
\phi^{s}(d_{i}^{B,k}) =
\begin{cases}
0 & \text{, } d_{i}^{B,k}> r_{i}\\
\frac{r_{i}-d_{i}^{B,k}}{r_{i}-d_{min}^{B}}\phi_{0}^{s} & \text{, } d_{i}^{B,k}\le d_{min}^{B}\\
\phi_{0}^{s} & \text{, otherwise,}
\end{cases}
\end{equation}

where $d_{min}^{B}<r_{i}$ and $\phi_{0}^{s}\gg0$. The parameter configuration for the contact model is listed in Table <ref>.

Parameter configuration of the contact model.

Parameter     Description     Range     Value

$C^{s}$            Directional weight          $[0,1]$          $0.5$

$\phi_{0}^{B}$     Base potential (border)     $(0,\infty)$     $11$

$C^{B}$            Scaling factor (border)     $(0,\infty)$     $0.2$

$\phi_{0}^{s}$     Base potential (soft)       $(0,\infty)$     $1200$

$d_{min}^{B}$      Inflation                   $[0,r_{i})$      $0$

§.§ Further Extensions of Standard HSFM

In addition to the already discussed alterations, we use certain features of state-of-the-art models. Firstly, we use the group cohesion proposed by Farina et al. <cit.> to better depict squad behavior. We use different configurations of the potential in agent-agent interaction, which are interchanged if the agents are part of the same or different squads. The repulsion of squad mates is lower than in inter-squad interaction. Secondly, we model agent-agent and agent-border forces with the Elliptical Specification (ES) II <cit.>. We experimentally compared ES I <cit.>, ES II, and Circular Specification <cit.>. We observed that the ES II works best with the new contact definition, resulting in less oscillation, i.e., better damping behavior, after first contact with borders.

§ VALIDATION

Test hardware consists of Intel i7-7700HQ and $8$Gb RAM. The modified HSFM is solved with a Runge-Kutta solver using the Dormand-Prince method of fifth order <cit.> (runge_kutta_dopri5 in boost) and a step size of $0.06$sec. Computational times for single squad traversal are listed in Table <ref> ($1000$ samples per test case). A squad consists of three agents, which is common for German fire brigades. With the low computational times, the robot employing the proposed methodology can generate squad trajectories in real or near-real time, depending on the length of the predicted path. Figure <ref> depicts the paths of a single squad traversing the building with varying motion tactics. Given the parameter configuration in Table <ref>, the motion prediction generates feasible results. Hence, we conclude that the method is well suited for deployment in collaborative rescue applications.

Characteristic measures of single squad traversal.

Tactic     2c|Path Length (in m)     2cSimulation Time (in ms)

Mean ($\mu_{d}$)     Variance ($s_{d}^2$)     Mean ($\mu_{t}$)     Variance ($s_{t}^2$)

Free     $8.835$      $0.156$     $21.154$     $2.973$

$48.365$     $0.155$     $105.937$     $18.528$

LHR     $11.568$      $0.008$     $78.044$     $30.453$

$108.445$     $0.818$     $842.665$     $634.517$

RHR     $14.787$      $0.021$     $93.298$     $27.704$

$65.981$      $0.163$     $482.274$     $201.914$

Simulation of a single squad. Displayed is the path of a single agent (yellow: free, blue: LHR, red: RHR).

§ CONCLUSIONS

In this work, we proposed a novel motion prediction pipeline for SAR. The method combines optimal path planning based on three specialized types of graphs with a modification of the HSFM <cit.>. The graphs model the tactics applied by firefighters in SAR, while the HSFM uses the paths to generate agent-individual motion trajectories. We modified the agent-border interaction by introducing a soft contact to allow traversal closer to borders. The models are configured with a novel data set. Finally, we showed that the proposed pipeline generates feasible trajectories in SAR and is computed fast depending on map size and agent count.

§ ACKNOWLEDGMENT

We like to thank the IdF NRW for providing data and Sebastian Döbler, Onur Akin, Lukas Clasen, and Till Waldermann for their support in data collection and implementation.