# Surface-Based Retrieval Reduces Perplexity of
Retrieval-Augmented Language Models

Ehsan Doostmohammadi1 Tobias Norlund2,4 Marco Kuhlmann1 Richard Johansson2,3
1 Linköping University 2 Chalmers University of Technology
3 University of Gothenburg 4 Recorded Future  Correspondence to
<EMAIL_ADDRESS>
###### Abstract

Augmenting language models with a retrieval mechanism has been shown to
significantly improve their performance while keeping the number of parameters
low. Retrieval-augmented models commonly rely on a semantic retrieval
mechanism based on the similarity between dense representations of the query
chunk and potential neighbors. In this paper, we study the state-of-the-art
Retro model and observe that its performance gain is better explained by
surface-level similarities, such as token overlap. Inspired by this, we
replace the semantic retrieval in Retro with a surface-level method based on
BM25, obtaining a significant reduction in perplexity. As full BM25 retrieval
can be computationally costly for large datasets, we also apply it in a re-
ranking scenario, gaining part of the perplexity reduction with minimal
computational overhead.

##  1 Introduction

The introduction of the Transformer architecture (Vaswani et al., 2017) has
led to a performance boost in language modeling (see, e.g., Brown et al.
2020), but also to a steep increase of computational cost, as the number of
parameters and data points is constantly growing. In reaction to this
development, there has recently been a surge in work on retrieval-augmented
language models Izacard and Grave (2021a); Li et al. (2022), which shows that
enabling models to retrieve from large corpora results in lower perplexity and
better accuracy in downstream tasks such as question answering, while at the
same time using considerably fewer parameters. In this paper, we specifically
focus on the Retrieval-Enhanced Transformer architecture (Retro; Borgeaud et
al., 2022).

By augmenting a language model with a retrieval mechanism, Retro, like similar
architectures, tries to decouple _memorization_ of the training data from the
additional _generalization_ that comes with increasing the number of
parameters. In Retro, when a chunk of text (a sequence of tokens) has been
generated, a dense representation of this chunk is used to retrieve the most
similar neighboring chunks from a large retrieval set, based on their L2
distance. Having the previously generated chunks and their nearest neighbors
in the retrieval set, the auto-regressive language model has now access to an
extended context when predicting the next chunk. The informativeness of this
context depends on the effectiveness of the retrieval method.

Borgeaud et al. (2022) note that part of Retro’s performance can be attributed
to the token overlap between the generated chunks and the retrieval set. Upon
further investigation, Norlund et al. (2023) show that the reduction in loss
observed in Retro “almost exclusively” stems from such overlap rather than
more sophisticated generalization. Our starting point in this paper is the
observation that the performance gain is actually _better_ explained by such
surface-level similarities than by the L2 distance between the dense
representations that Retro uses for retrieval. We then replace the semantic
retrieval method in Retro with one based on BM25 Robertson et al. (1995), a
surface-level measure, to exploit the findings in Norlund et al. (2023) and
here. Our results show that retrieving nearest neighbors using BM25 during
inference leads to a 13.6% lower perplexity, compared to dense retrieval based
on sentence transformers (ST) Reimers and Gurevych (2019), a model trained to
represent the semantic similarity between sentences.111The code and the data
for this study can be accessed at github.com/edoost/retro_bm25.

Finding the exact neighbors with BM25 is costly on large retrieval sets and
might not meet the speed requirements of all applications of retrieval-
augmented language models. We therefore explore a hybrid approach where we
first retrieve approximate neighbors using ST representations and then re-rank
them using BM25. We show that this approach yields 24.7% of the perplexity
reduction we get with BM25-based retrieval, with only minimal computational
overhead.

##  2 Method

We experiment with Retro (Borgeaud et al., 2022) as a state-of-the-art
retrieval-augmented language model.

###  2.1 Model

Retro is very similar to a standard auto-regressive language model such as T5
Raffel et al. (2020), the main differences being the introduction of the
retrieval mechanism and how the retrieved neighbors are used for language
modeling.

#### Nearest Neighbor Retrieval

In Retro, all textual data is stored and used in chunks of 64 tokens. When the
model has generated a chunk $C_{u}$, it retrieves the $k$ nearest neighbors
$N_{1:k}$ to that chunk, together with the chunks $F_{1:k}$ following these
neighbor chunks in the retrieval data. It then generates the next chunk
$C_{u+1}$ conditioned on the retrieved chunk pairs. Retrieval uses the squared
L2 distance on a dense representation ($\mathit{DR}$) of chunks:

$d(C_{u},N_{i})=\|\mathit{DR}(C_{u})-\mathit{DR}(N_{i})\|^{2}_{2}$

This leaves us with

$\textsc{Ret}(C_{u})=([N_{u}^{1};F_{u}^{1}],\dots,[N_{u}^{k};F_{u}^{k}])$

as the retrieved neighbors that the model receives as additional context when
generating the next chunk. The likelihood of the very first chunk does not
depend on any neighbors ($\textsc{Ret}(C_{1})=\emptyset$). During training and
perplexity evaluation, the retrieval process is filtered such that chunks
originating from the same source document as the training sequence are never
considered as neighbors.

#### Integration of the Neighbors

Retro improves auto-regressive language modeling by conditioning the next
token prediction on the retrieved chunks of text. This means that the
probability of generating the next token $x_{t+1}$ depends not only on the
previously generated tokens $x_{1:t}$ but also on the retrieved neighbors of
the previously generated chunks, as well as their following chunks:

$P\left(x_{t+1}\,|\,x_{1:t},\textsc{Ret}(C_{1}),\dots,\textsc{Ret}(C_{u});\theta\right)$

The neighbors as well as the current chunk $C_{u+1}$ are passed through a
transformer encoder. In the Decoder, cross-attention then attends to the
output of the Encoder and the intermediary embeddings of the last tokens in
the chunk $C_{u}$, a mechanism called _chunked cross-attention_. For more
details, see Borgeaud et al. (2022).

#### Implementation Details

As an official implementation of Retro is not publicly available, we draw upon
our implementation in Norlund et al. (2023), which is based on the description
in Borgeaud et al. (2022). Our implementation deviates only in that (1) we use
learnable relative positional biases as in T5 Raffel et al. (2020), with a
bucket for each unique relative position; (2) instead of BERT Devlin et al.
(2019), we use the pre-trained sentence transformers (ST) Reimers and Gurevych
(2019) model to embed the chunks for the offline retrieval. ST is preferable
over BERT, as it is trained for the task of similarity search, and produces
embeddings of lower dimensionality, which makes it more efficient. We use
PyTorch (Paszke et al., 2019) and PyTorch Lightning for distributed training.
For the tokenization, we use the pre-trained T5 tokenizer HuggingFace . For
retrieving approximate neighbors, we use faiss (Johnson et al., 2019), which
performs efficient similarity search between dense representations with GPU
support for faster indexing and retrieval.

###  2.2 Data

Borgeaud et al. (2022) use the _MassiveText_ dataset Rae et al. (2021) for
both training and retrieval. As this dataset is not publicly available, we set
out to replicate it using open sources. _MassiveText_ consists of multilingual
text data in five categories: Wikipedia articles, books, GitHub code, news,
and common crawl web data. We use _Pile_ (Gao et al., 2021) and _RealNews_
(Zellers et al., 2019) to build a large dataset resembling _MassiveText_ ’s
composition. The new dataset (see Norlund et al. (2023) for details) consists
of 36M documents containing 52B tokens. For _Pile_ , we keep the training and
validation splits, while for _RealNews_ , we use the full training set but
downsample the validation set to 16,400 news articles to match the proportions
of the categories in _Pile_. For details on the deduplication process refer to
Gao et al. (2021) and Zellers et al. (2019).

###  2.3 Training

We use our dataset to train a Retro model with approximately 630M parameters.
For more details refer to Norlund et al. (2023). During training, we retrieve
from the training set; during validation, we retrieve from the union of the
training and validation sets. We train the model on sequences truncated to
1,024 tokens. The chunk size is 64, as in Borgeaud et al. (2022), and the
number of retrieved neighbors is $k=2$ for training and validation. We train
the model for 140k training steps with a batch size of 16, taking seven days
on 16 A100 GPUs. This means that we use 6% of the training data during
training, not including the retrieved neighbors. As our optimizer, we use Adam
Kingma and Ba (2015) with a fixed learning rate of $110-4$.

##  3 A Study on Correlations

We experiment with two settings: Retro[on], the language model with retrieval
enabled, and Retro[off], where there are no chunk cross-attention layers and
therefore no retrieval, leaving us with a decoder-only language model. As
shown by Borgeaud et al. (2022), the Retro[on] model performs better when it
can exploit an overlap between the generated text and the retrieved neighbor.
This is more apparent in text categories with higher token overlap, such as
GitHub. The studies in the Retro paper also show that allowing more overlap
when deduplicating the data results in a lower bits-per-byte (Bpb222Bpb
$=(L_{T}/L_{B})\mathcal{L}/ln(2)$, where $L_{T}$ and $L_{B}$ are the lengths
of the validation set in T5 tokens and UTF-8 encoded bytes, respectively, and
$\mathcal{L}$ stands for log likelihood loss. $L_{T}/L_{B}$ is $0.258415$ for
our validation set.). Norlund et al. (2023) take this further to show even
minimal overlap results in significant loss reduction, demonstrating the large
extent Retro relies on surface-level similarities. These findings lead us to
hypothesize that having a retrieval method that can find the highest
overlapping neighbors will yield lower perplexity (Ppl). Because BERT, ST and
similar deep representations of sentences do not always capture surface-level
similarities, we set out to investigate where performance gains come from.

$X$ | $Y$ | $\rho$ | $r$
---|---|---|---
L22 (ST) | $\Delta\textsc{Ppl}$ | 0.349 | 0.160
token overlap | $\Delta\textsc{Ppl}$ | 0.528 | 0.488
L22 (ST) | token overlap | 0.464 | 0.515
Table 1: Spearman $\rho$ and Pearson $r$ between variables $X$ and $Y$ where
L22 (ST) is the (negative) squared L2 distance between the ST embeddings of
$\textsc{Ret}(C_{u-1})$ and $C_{u}$; token overlap is the unigram overlap
between these chunks; and
$\Delta\textsc{Ppl}=\textsc{Ppl}_{\textsc{Retro[off]}}-\textsc{Ppl}_{\textsc{Retro[on]}}$
for $C_{u}$.

To this end, we measure how Ppl difference between Retro[on] and Retro[off]
($\Delta\text{{Ppl}}=(\Delta\text{{Ppl}}_{C_{u}})_{u\geq 2}$) correlates with
(1) squared L2 distance between the ST embeddings of $\textsc{Ret}(C_{u-1})$
and $C_{u}$ (ST), and (2) unigram token overlap based on T5 tokenization,
between the generated chunk ($C_{u\geq 2}$) and the ST retrieved chunk
($\textsc{Ret}(C_{u-1})_{u\geq 2}$). The results, reported in Table 1, show a
considerably stronger correlation between $\Delta\textsc{Ppl}$ and unigram
token overlap than negative L22 distance between ST representations. The trend
is similar between Spearman and Pearson correlation coefficients.

##  4 Changing the Retrieval Method

As the results from the previous section show a stronger correlation between
performance gain and surface-level similarity than ST similarity, we
experiment with a retrieval method based on BM25.

###  4.1 BM25

Okapi BM25, introduced by Robertson et al. (1995), is a bag-of-words retrieval
method based on tf–idf scores and some free parameters. These parameters are
$k_{1}$, which normalizes the term frequency, and $b$, which controls how much
the length of a document would affect the term frequency values. We use
Pyserini Lin et al. (2021), a Python interface to Lucene’s BM25
implementation. We build the BM25 index on the training set and leave the free
parameters at their default values ($k_{1}=0.9$, $b=0.4$). These values were
also shown to perform the best by Karpukhin et al. (2020a). Using Lucene’s
analyzer333Analyzer is the policy for extracting index terms from text Lucene
, which includes tokenization and preprocessing. results in more than 50M
unique words for our corpus. We instead use the T5 tokenizer from Hugging Face
Transformers Wolf et al. (2020) and limit our vocabulary to 32k words for the
reranking experiments.

###  4.2 Retrieving with BM25

We use the model described in Section 2.3 and change the retrieval method only
at inference time to retrieve better neighbors. The results can be found in
Table 2. The perplexity is 21.44 for Retro[off] and 15.30 for Retro[on] with
ST retrieval (Retro[on]-ST), corresponding to a 28.6% reduction in Ppl.
Replacing the retrieval method with BM25 (Retro[on]-BM25) gives an additional
13.6% reduction, which is 47.5% of the initial drop. For comparability with
Borgeaud et al. (2022), we also report Bpb. The results show that using
neighbors with more surface-level similarity to the generated chunk is a solid
method for leveraging the retrieval mechanism to reduce the perplexity. If the
retrieval augmentation is meant to act as an external memory, or to offload
memorization from the model Borgeaud et al. (2022), then BM25 is a more
suitable method to achieve this goal.

Model | Ppl | Bpb
---|---|---
Retro[off] | 21.44 | 1.142
Retro[on]-ST | 15.30 | 1.017
Retro[on]-ST+BM25-R | 14.62 | 1.000
Retro[on]-BM25 | 12.55 | 0.943
Table 2: Bpb and Ppl on validation set. R stands for reranking. The Retro
model is the same for all rows.

###  4.3 Reranking

While the performance gain is significant, finding the _exact_ neighbors using
BM25 could be costly, depending on the size of the datasets. On the other
hand, faiss provides an efficient similarity search for dense vectors to find
the _approximate_ neighbors. Therefore, if enough of the BM25-retrieved
neighbors could be found among top-$k$ faiss-retrieved ones, with an efficient
reranking, we could expect at least part of the performance gain with minimal
computational overhead, as long as $k$ is not significantly large. To find an
optimal $k$, we first need to know how many of BM25 neighbors could be found
in top-$k$ faiss-retrieved chunks.

Looking at the faiss-retrieved neighbors, we see that of top-$4$
BM25-retrieved neighbors, $17.6\%$ appear in top-$100$ faiss-retrieved chunks,
while the overlap is $22.1\%$ for top-$1000$. We decide to continue our
experiment with top-$1000$ neighbors, but it is obvious that one could get an
even higher overlap with a higher $k$, with diminishing returns. The results
in Table 2 show that with the proposed reranking, Retro[on]-ST+BM25-R could
achieve $24.7\%$ of the Ppl reduction of Retro[on]-BM25 compared to
Retro[on]-ST. The reranking results are interesting not only due to their
practical implications but also as an analysis revealing the limited number of
high-quality neighbors that can be retrieved using semantic retrieval, even
when assuming a large $k$ is feasible.

##  5 Related Work

Augmenting language models with mechanisms that help them incorporate larger
contexts has been approached extensively in different forms, such as Guu et
al. (2018)’s retrieve-and-edit approach to reduce the Ppl in language
generation, and Asai et al. (2020) that make use of lexical overlap to improve
the performance in question answering. While retrieval-augmentation has been
used with different objectives in mind, such as language modeling Khandelwal
et al. (2020); Wu et al. (2022) and machine translation Khandelwal et al.
(2021), question answering has been the application to attract the most
interest Guu et al. (2020); Karpukhin et al. (2020b); Izacard and Grave
(2021b).

An extensive study was performed by Izacard et al. (2022), showing that while
we get performance gains using retrieval augmentation, training the retrieval
part of the model would yield even more benefits. Retro Borgeaud et al.
(2022), on the other hand, aims at scaling such language models and therefore
opts for keeping the retriever frozen, showing substantial Ppl reduction with
increasing either the number of language model parameters or the size of
retrieval set.

Among the more recent work, Xu et al. (2023) found that training using
approximate neighbors resulted in a 2.6% decrease in perplexity. This suggests
that non-exact neighbors may have a regularization effect, leading to improved
generalization ability. Additionally, Ram et al. (2023) report a drop in
perplexity using BM25 over BERT retrieval using in-context retrieval-augmented
language models.

##  6 Conclusions and Future Work

In this paper, we study the source of performance gains in Retro, which could
be generalized to similar retrieval-augmented language models. After observing
that the Ppl drop correlates more strongly with surface-level overlap between
the query and the retrieved text, we replace the retrieval method with BM25,
and observe a significant drop in Ppl, which confirms us in the findings of
the correlation study. This is also an interesting insight as to how these
models work, which could be leveraged for performance gain in tasks like
question answering where model relies on retrieving facts. In the end, we also
conduct an analysis to find out how much BM25 neighbors overlap with those
retrieved using ST. The results show that while faiss is able to find some of
the neighbors with high token overlap, the majority of them remain
unretrieved. This is however, enough to gain part of the loss reduction
achieved with a pure BM25 retrieval system.

The proposed methods could also be used during training. By retrieving more
overlapping neighbors during training, the process of guiding the model to use
retrieved neighbors for language modeling could be done more efficiently. This
is particularly true when augmenting an already trained language model with a
retrieval mechanism. As reported by Borgeaud et al. (2022), retrieval
augmentation results in a larger drop in Bpb as the number of model parameters
and the size of retrieval data grow. This calls for more efficient methods
based on surface level similarities, if we wish to exploit this potential.
Furthermore, the retrieval system in Retro is based on semantic retrieval, the
model seems to rely more on surface level similarities. This could affect the
generalizability capabilities of such models, which necessitates further
investigations. Lastly, we only evaluate our modified Retro model on language
modeling. It would be interesting to know the impacts of BM25 retrieval on
down-stream tasks where retrieval is of use.

## Limitations

We only experiment with one type of retrieval-augmented language models, i.e.,
Retro. However, the ways the other models retrieve neighbors and integrate
them are not so much different to affect the results in this paper. The
experiments in this paper are done with a small size Retro model and data
compared to the sizes considered by Borgeaud et al. (2022), due to
computational limitations. According to the same paper however, the gains
should be constant with the increase of the model and retrieval set size. The
larger models are mainly different in their behavior when there is no overlap.
However, this should not affect the _copying_ tendency of these models
tremendously, as it is still the easiest way to generate the next token. It is
also worth noting that Retro[off] is trained using retrieval and then the
retrieval is turned off. So it is not a complete retrieval-free model. The
results presented by Borgeaud et al. (2022) however, show that Retro[off] is
on a par with their retrieval-free baseline in terms of Bpb. Finally, we note
that our evaluations have only considered the perplexity under teacher
forcing, and we have not investigated the behavior of the model in free-form
generation or with any kind of fine-tuning.

## Acknowledgements

This work was partially supported by the Wallenberg AI, Autonomous Systems and
Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.
The computations were enabled by resources provided by the National Academic
Infrastructure for Supercomputing in Sweden (NAISS) at Alvis partially funded
by the Swedish Research Council through grant agreement no. 2022-06725, and by
the Berzelius resources provided by the Knut and Alice Wallenberg Foundation
at the National Supercomputer Center.

## References

  * Asai et al. (2020) Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. 2020.  Learning to retrieve reasoning paths over wikipedia graph for question answering.  In _International Conference on Learning Representations_.
  * Borgeaud et al. (2022) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. 2022\.  Improving language models by retrieving from trillions of tokens.  In _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 2206–2240. PMLR.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Gao et al. (2021) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2021.  The pile: An 800gb dataset of diverse text for language modeling.  _CoRR_ , abs/2101.00027.
  * Guu et al. (2018) Kelvin Guu, Tatsunori B. Hashimoto, Yonatan Oren, and Percy Liang. 2018.  Generating sentences by editing prototypes.  _Transactions of the Association for Computational Linguistics_ , 6:437–450.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020.  Realm: Retrieval-augmented language model pre-training.  In _International Conference on Machine Learning_ , pages 3929–3938. PMLR.
  * (8) HuggingFace.  Huggingface T5.
  * Izacard and Grave (2021a) Gautier Izacard and Edouard Grave. 2021a.  Leveraging passage retrieval with generative models for open domain question answering.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 874–880, Online. Association for Computational Linguistics.
  * Izacard and Grave (2021b) Gautier Izacard and Edouard Grave. 2021b.  Leveraging passage retrieval with generative models for open domain question answering.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 874–880, Online. Association for Computational Linguistics.
  * Izacard et al. (2022) Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022.  Few-shot learning with retrieval augmented language models.  _arXiv preprint arXiv:2208.03299_.
  * Johnson et al. (2019) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019.  Billion-scale similarity search with GPUs.  _IEEE Transactions on Big Data_ , 7(3):535–547.
  * Karpukhin et al. (2020a) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020a.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online. Association for Computational Linguistics.
  * Karpukhin et al. (2020b) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020b.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online. Association for Computational Linguistics.
  * Khandelwal et al. (2021) Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2021\.  Nearest neighbor machine translation.  In _International Conference on Learning Representations_.
  * Khandelwal et al. (2020) Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020\.  Generalization through memorization: Nearest neighbor language models.  In _International Conference on Learning Representations (ICLR)_.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _ICLR (Poster)_.
  * Li et al. (2022) Huayang Li, Yixuan Su, Deng Cai, Yan Wang, and Lemao Liu. 2022.  A survey on retrieval-augmented text generation.  arXiv preprint 2202.01110.
  * Lin et al. (2021) Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021.  Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’21, page 2356–2362, New York, NY, USA. Association for Computing Machinery.
  * (20) Apache Lucene.  Lucene analyzer.
  * Norlund et al. (2023) Tobias Norlund, Ehsan Doostmohammadi, Richard Johansson, and Marco Kuhlmann. 2023\.  On the generalization ability of retrieval-enhanced transformers.  In _Findings of the Association for Computational Linguistics: EACL 2023_ , pages 1485–1493, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc.
  * Rae et al. (2021) Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew J. Johnson, Blake A. Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. 2021.  Scaling language models: Methods, analysis & insights from training gopher.  _CoRR_ , abs/2112.11446.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Ram et al. (2023) Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023.  In-context retrieval-augmented language models.  _arXiv preprint arXiv:2302.00083_.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Robertson et al. (1995) Stephen Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, and M. Gatford. 1995\.  Okapi at TREC-3.  In _Overview of the Third Text REtrieval Conference (TREC-3)_ , pages 109–126. Gaithersburg, MD: NIST.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wu et al. (2022) Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy. 2022.  Memorizing transformers.  In _International Conference on Learning Representations_.
  * Xu et al. (2023) Frank F Xu, Uri Alon, and Graham Neubig. 2023.  Why do nearest neighbor language models work? _arXiv preprint arXiv:2301.02828_.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019.  Defending against neural fake news.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.