# Plug-and-Play Adaptation for Continuously-updated QA

Kyungjae Lee5 Wookje Han1 Seung-won Hwang1
Hwaran Lee2 Joonsuk Park2,4 Sang-Woo Lee2,3
1Seoul National University 2NAVER AI Lab 3NAVER CLOVA
4University of Richmond 5LG AI Research  correspond to<EMAIL_ADDRESS>
###### Abstract

Language models (LMs) have shown great potential as implicit knowledge bases
(KBs). And for their practical use, knowledge in LMs need to be updated
periodically. However, existing tasks to assess LMs’ efficacy as KBs do not
adequately consider multiple large-scale updates. To this end, we first
propose a novel task—Continuously-updated QA (CuQA)—in which multiple large-
scale updates are made to LMs, and the performance is measured with respect to
the success in adding and updating knowledge while retaining existing
knowledge. We then present LMs with plug-in modules that effectively handle
the updates. Experiments conducted on zsRE QA and NQ datasets show that our
method outperforms existing approaches. We find that our method is 4x more
effective in terms of updates/forgets ratio, compared to a fine-tuning
baseline.

##  1 Introduction

LM-as-KB is a new paradigm in which pre-trained language models (LMs) are used
as implicit knowledge bases (KBs) Petroni et al. (2019). This is made possible
by LMs’ impressive ability to memorize factual knowledge Heinzerling and Inui
(2021); Brown et al. (2020). Recently, two tasks have been used to assess such
ability: LAMA, a knowledge probing benchmark, challenges LMs to fill in masked
words over relational knowledge Petroni et al. (2019); and closed-book QA
(CBQA) examines whether LMs can correctly answer natural language questions
Roberts et al. (2020).

Figure 1: Examples of CuQA showcasing two scenarios.

For practical usage, LM-as-KB requires that LMs are updated periodically to
stay current with the ever-evolving world. Thus, LMs’ ability to update
knowledge should also be evaluated. To this end, we present Continuously-
updated QA (CuQA), which tests the ability to continuously inject knowledge to
update (or target knowledge), while retaining existing knowledge (or source
knowledge). Specifically, we consider multiple large-scale knowledge updates
(8k to 60k) covering two scenarios: injecting new knowledge (Scenario 1 in
Figure 1) and updating existing knowledge (Scenario 2 in Figure 1) .

Our goal is to organize the implicit storage of knowledge, to add target
knowledge (yellow box in Figure 1) and anchor to select target knowledge. A
simple approach is to train updated LMs from scratch; however, this is far too
expensive considering the parameter sizes of recent LMs, such as 175B for
GPT-3 Brown et al. (2020) and about 11B for T5 Raffel et al. (2020). There has
also been related work for the two scenarios. For Scenario 1, a method for
continual learning can be adopted, constraining the distance between
parameters before and after fine-tuning Chen et al. (2020). However, this
approach still suffers from so-called catastrophic forgetting, where the LMs
fail to retain large amounts of source knowledge. For Scenario 2, one may
consider knowledge editing methods, where we see reasonable performances for a
single knowledge edit while retaining the rest De Cao et al. (2021); Mitchell
et al. (2021). However, this line of work does not perform well when multiple
edits are accumulated, e.g., only 67% of 125 edits were updated, as reported
in Mitchell et al. (2021).

We propose to efficiently extend LMs with plug-and-play modules that store
target knowledge. More specifically, we adopt a parameter-expansion method in
which the LM storing existing knowledge is extended with plug-in feed-forward
modules storing updated knowledge. Depending on the input, the LM selectively
uses either the original LM or a plug-in module. We stress that, by keeping
the original LM intact, we retain (a) not only source knowledge, (b) but also
those outdated from updates (red arrow in Figure 1). (a) is important to avoid
catastrophic forgetting, while (b) is useful when updates need to be reverted
due to ethical concerns—for example, there can be malicious attempts to
override facts.

We evaluate our approach on zsRE Levy et al. (2017) and Natural Questions
Kwiatkowski et al. (2019) to showcase successful updates of new knowledge and
retention of existing knowledge. We measure the accuracies on both previous
and updated knowledge and find that ours show x4 higher updates/forgets ratio,
compared to fine-tuning. We will release our code and dataset.

Our key contributions are as follows:

  * •

We present CuQA, a novel task to assess LMs’ ability to continuously inject
knowledge to update.

  * •

We propose a new methodology, plug-and-play adaptation, to continually learn
new knowledge while better retaining existing knowledge.

##  2 Related Work

The relevant research can be categorized into three groups: Knowledge Editing,
Continual Learning, and Adaptation. In Table 1, we compare these with our
method.

#### Editing Implicit Knowledge

In Table 1(a), knowledge editing methods De Cao et al. (2021); Mitchell et al.
(2021); Dai et al. (2021) aim to efficiently edit model’s parameters on
examples that have conflicts with old facts, while preserving the outputs of
untargeted examples. Instead of directly updating gradients by fine-tuning,
these methods transform the gradients for new edit parameters. As
representative methods for knowledge editing, KnowledgeEditor (KE) De Cao et
al. (2021) using LSTM produces gate vectors, then the gated sum of gradients
is updated into the model, while MEND Mitchell et al. (2021) uses simple MLP
layers and residual connections for the same purpose. Although these methods
succeeded in updating the target examples less forgetting, their target
scenario is a single edit, such that the cumulative effect of multiple edits
does not reflect well, which disqualifies its use for our target task of
update large-scale data (8K$\sim$60K). As reported in Mitchell et al. (2021),
MEND successfully updates only 67% of edits when applying 125 edits, while our
finding was consistent when none of the 125 edits was applied in our
evaluation.111In the case of KE, we reimplement the released code for testing:
https://github.com/nicola-decao/KnowledgeEditor. In addition, for editing
previous knowledge, KE and MEND simulate knowledge updates, by generating
synthetic knowledge from LM. Such generations may not be realistic data and
also give unfair advantages to LM-based methods, while we use actual up-to-
date knowledge as new data, which were annotated on recent corpus Zhang and
Choi (2021).

---
Method |  Forgetting less? |  Scales to a large set? |  Conflict with old facts?
(a) Editing | ✓ | ✗ | ✓
(b) CL | ✓ | ✓ | ✗
(c) Adaptation | ✗ | ✓ | ✗
Our Method | ✓ | ✓ | ✓

Table 1: Conceptual comparison of existing approaches.

#### Continual Learning (CL) for NLP

For our task, we can adopt CL methods, learning a new task while preserving
the accuracy on previous tasks. Kirkpatrick et al. (2017) proposed Elastic
Weight Consolidation, alleviating catastrophic forgetting. This method
regularizes learning on a new task, by constraining the parameters trained on
the previous task. For NLP tasks, RecAdam Chen et al. (2020) uses the
regularization and annealing technique, which is a CL baseline in our
experiment. While CL approaches focusing on forgetting do not consider
conflicts between old and new knowledge, our work deals with such a realistic
scenario. Additionally, previous work Dhingra et al. (2021) proposed
benchmarks for probing temporal language models, asking “Fill-in-the-Blank
(FIB)" questions. Meanwhile, FIB questions are limited to evaluate masked
language models, such as BERT and RoBERTa. We extend to evaluate arbitrary
questions for a knowledge-intensive task; closed-book QA, which can evaluate
generative LMs with broader applicability, to include T5 and GPT.

#### Task-aware Adaptation for Transformers

Recent works Hu et al. (2021); Wang et al. (2020); Lin et al. (2020) study LM
adaptation to new labeled data in a new domain, which has a different data
distribution from that at pretraining. These works show performance
improvements on downstream tasks in the new domain, while fine-tuning a small
number of parameters. However, these adaptation methods do not consider
sequential training, and overwrite the new data into the parameters that store
previous knowledge. In our experiment, it is observed that the adaptation
methods are rapidly forgetting previously seen data, while performing well on
new knowledge.

##  3 A Continuously-updated QA Task

#### Task Description

In this section, we propose Continuously-updated QA (CuQA), a new continual
learning task for knowledge updates in LMs based on closed-book QA (CBQA)
Roberts et al. (2020). In CBQA, LMs answer factual questions with the implicit
knowledge stored in the model, without any external context (i.e., in contrast
to open-domain QA), so that LMs are required to adequately update their
parameters to the target knowledge. In our CuQA, LMs learn source (original)
knowledge first, then update them with target (new) knowledge without source
knowledge access. For the above setting, source knowledge (to be retained) and
target knowledge (to be added) in CuQA do not have any overlap of QA pairs (or
paraphrases) for any given fact.

Specifically, we denote a factual pair of question and answer as $(q,a)$,
source knowledge as $\mathcal{K}_{s}$, and target as $\mathcal{K}_{t}$. We
first build an initial model $\theta^{old}$ pre-trained on source knowledge
$\mathcal{K}_{s}$. Then, we inject target knowledge $\mathcal{K}_{t}$ into the
pre-trained model and obtain the infused model $\theta^{new}$. Our goal is to
memorize $\mathcal{K}_{t}$ on model $\theta^{new}$, with less forgetting
$\mathcal{K}_{s}$. If knowledge in $\mathcal{K}_{t}$ conflicts one in
$\mathcal{K}_{s}$, the model is required to adjust its parameters by
reflecting the target knowledge. Note that multiple target knowledge can be
sequentially updated to the model (see details in Section 4).

#### Research Questions

CuQA is designed to address the following research questions:

  * •

RQ1: Can the method learn target knowledge while retaining source knowledge?

  * •

RQ2: How does sequentially learning multiple target knowledge affect the
performance?

  * •

RQ3: How does the size of each target knowledge affect the performance?

#### Metric

For evaluation, we measure the success of updates, retaining of source
knowledge, and generality using exact match (EM) scores. Additionally, we
measure the ratio of forgets to updates.

  * •

Accuracy on $\mathcal{K}_{t}$ : we evaluate how much model $\theta^{new}$
successfully updates examples in $\mathcal{K}_{t}$.

  * •

Accuracy on $\mathcal{K}_{s}$: how much model $\theta^{new}$ forgets examples
in $\mathcal{K}_{s}$. This indicates performance degradation, when replacing
$\theta^{old}$ with $\theta^{new}$.

  * •

Accuracy on $\mathcal{P}_{s}$, $\mathcal{P}_{t}$: how well model
$\theta^{new}$ generalizes on semantically equivalent questions (or
paraphrases).

  * •

F/U Ratio ($\\#$ of forgets$/\\#$ of updates): how many examples in
$\mathcal{K}_{s}$ are forgotten per an update of one example in
$\mathcal{K}_{t}$. ($\\#$ of forgets) is equal to the difference of correct
prediction cases in $\mathcal{K}_{s}$, between $\theta^{old}$ and
$\theta^{new}$.

##  4 Method

In this section, we describe baseline approaches (Section 4.1), and introduce
our proposed method for plug-and-play adaptation (Section 4.2).

Figure 2: An overview of our proposed architecture.

###  4.1 Baseline Approaches

We establish three baseline for (a), (b), and (c), in Table 1. Since we found
that a knowledge editing approach is outperformed by fine-tuning, we exclude
it as baselines, and add fine-tuning instead.

#### Fine-tuning on target knowledge

As a naive baseline, we start with the previous work Roberts et al. (2020) for
CBQA, by fine-tuning T5 Raffel et al. (2020) with encoder-decoder structure.
This baseline is to fine-tune the pre-trained model $\theta^{old}$ on facts in
$\mathcal{K}_{t}$ to minimize the loss:

$\mathcal{L}_{FT}=\sum_{(q,a)\in\mathcal{K}_{t}}L((q,a);\theta)$ (1)

where $L$ refers to a seq2seq loss. This baseline is expected to optimize
accuracy on target knowledge $\mathcal{K}_{t}$, thus increases the distance
between the before- ($\theta^{old}$) and after-parameters ($\theta^{new}$)
resulting in the risk of forgetting. For other baselines and our method, we
adopt the same transformer: T5 as backbone network.

#### Regularized fine-tuning for CL

We adopt RecAdam Chen et al. (2020) aiming to reduce the forgetting risk by
adding a constraint to minimize the distance between $\theta^{old}$ and
$\theta^{new}$ as follow:

$\mathcal{R}=\|(\theta-\theta^{old})\|_{p}$ (2)

where $\|\cdot\|_{p}$ indicates $L_{p}$ norm. In addition, RecAdam uses an
annealing technique, controlling the ratio between $\mathcal{R}$ and the fine-
tuning loss (Eq. (1)) as follows:

$\displaystyle\mathcal{L}_{total}=$
$\displaystyle~{}\lambda(t)\mathcal{L}_{FT}+(1-\lambda(t))\mathcal{R},$ (3)
$\displaystyle\lambda(t)=$
$\displaystyle~{}\frac{1}{1+\textrm{exp}(-k\cdot(t-t_{0}))}$ (4)

where $k$ and $t_{0}$ are hyper-parameters.

#### Adapters for knowledge updates

For adaptation approaches, we implement two parameter-expansion methods:
K-adapter Wang et al. (2020) and LoRA Hu et al. (2021). The approaches freeze
the parameters $\theta^{old}$ in pre-trained LM and augment additional new
parameters $\tilde{\theta}$ in the LM to train target knowledge as following:

$\mathcal{L}_{adap}=\sum_{(q,a)\in\mathcal{K}_{t}}L((q,a);\theta^{old},\tilde{\theta}).$
(5)

For $\tilde{\theta}$, K-adapter Wang et al. (2020) uses augmented self-
attention layers, while LoRA Hu et al. (2021) utilizes extra low-rank
matrices.

###  4.2 Our Method

Motivated by the intuition of regularization to preserve source knowledge and
that of adapters to inject target knowledge into new parameters, we show their
strengths can be combined for our task. At the inference phase, our method
selectively uses the plug-in modules to keep source knowledge intact, while
tasks requiring target knowledge will be redirected to new plug-in modules.

Specifically, our distinction is augmenting function $f$ (in an original LM)
with function $g$, representing source and target knowledge respectively. The
function $f$ is a single layer in transformer trained on source knowledge
$\mathcal{K}_{s}$, and $g$ is an augmented function with new parameters for
$\mathcal{K}_{t}$. Existing work, such as LoRA, can be interpreted by adding
the two functions:

$h=f(x)+g(x)$ (6)

where $f$ is one-linear layer in self-attention or feed-forward layers. That
is, $f(x)=W_{0}x$, where $W_{0}\in\mathrm{R}^{d\times k}$ denotes the pre-
trained and fixed parameters. LoRA uses low-rank matrices as $g(x)$, i.e.,
$g(x)=BAx$, where $B\in\mathrm{R}^{d\times r}$, $A\in\mathrm{R}^{r\times k}$,
and $r<\\!\\!<min(d,k)$. The low-rank matrices $A$ and $B$ are trainable
parameters for updating target knowledge. The new layer with the additional
matrices is denoted as follows:

$h=W_{0}x+BAx=(W_{0}+BA)x$ (7)

However, the above add-aggregation has a limitation, as $g(x)$ can affect the
model’s outputs, and increase the distance between hidden states in
$\theta^{old}$ and $\theta^{new}$, which causes a forgetting problem.

Our key distinction is adding a selector, that is selectively activated for
$q$ requiring the use of plug-in module $g$, as follows:

$h=f(x)+\sigma(q)\cdot g(x)$ (8)

where $\sigma(q)$ is 1 or 0 depending on query $q$. While there can be various
ways to train the selector in a sophisticated way, supervised either directly,
or indirectly in an end-to-end manner, we show a simple unsupervised selector
is already sufficient to show gains. Specifically, our selector is a key-value
lookup where the key is $m_{i}$ and value is $g$. At inference time, when
given query $q$ is based on facts in $\mathcal{K}_{t}$, we activate the
augmented $g$ for generating its output. If $q$ is not from $\mathcal{K}_{t}$,
we use only the original model $\theta^{old}$ for generation. To classify
whether the input is from $\mathcal{K}_{t}$ or not, we build explicit memory
with embeddings of $\mathcal{K}_{t}$ and leverage the distance with nearest
neighbor (NN) in the memory.

Let $\mathcal{M}\in\mathrm{R}^{N\times d}$ be memory embeddings that stores
embeddings of input questions in $\mathcal{K}_{t}$, where $N$ is the total
number of examples in $\mathcal{K}_{t}$. As shown in Figure 2, question
embedding can be extracted from the encoder, by averaging the hidden states of
input sequence. In T5 model with encoder-decoder, this averaging method is
known to be effective on semantic textual similarity, as in Ni et al. (2021).
Given question $q$, cosine similarity with NN is calculated as follows:

$s_{q}=\textrm{max}_{i}(\textrm{sim}(m_{i},q)),~{}~{}m_{i}\in\mathcal{M}$ (9)

where sim indicates cosine similarity. Based on $s_{q}$, if the score is
greater than or equal to threshold $\delta$, we assume $q$ is from target
knowledge $\mathcal{K}_{t}$. We build a indicator function as follows:

$\sigma(q)=\begin{cases}~{}~{}1~{}~{}~{}~{}~{}\textrm{if}~{}~{}s_{q}\geq\delta,\\\
~{}~{}0~{}~{}~{}~{}~{}\textrm{if}~{}~{}s_{q}<\delta.\\\ \end{cases}$ (10)

In other words, $s_{q}\geq\delta$ indicates that input $q$ is semantically
similar with one fact in $\mathcal{K}_{t}$. At that time, our model is
augmented with $g$ that stores new and updated knowledge.

Meanwhile, as shown in Figure 2, we apply the selective use of parameters to
only a decoder in a transformer architecture, not a encoder. The switch
$\sigma$ depends on query embedding $q$, and the embedding $q$ is extracted
from T5 encoder. If we apply the switch $\sigma$ to hidden states in T5
encoder, this causes a recursion relation, or inefficient computations. By
augmenting $g$ for the decoder, embedding $q$ is not changing during updating
target knowledge, and depends on only pre-trained $\theta^{old}$.

#### General case of multiple knowledge updates

Our new perspective has another benefit of naturally generalizing to
sequential ($>$2) sources. Assume that there are multiple target knowledge to
be sequentially updated, i.e.,
$\mathcal{K}_{t}^{1},\mathcal{K}_{t}^{2},...,\mathcal{K}_{t}^{M}$. We build
multiple functions $g_{k}$ and memories $\mathcal{M}_{k}$ (where $k=1,...M$),
according to each target knowledge. The new function considering the multiple
knowledge is denoted as follows:

$h=f(x)+\sum_{k=1}^{M}\sigma_{k}(q)\cdot g_{k}(x)$ (11)

During training $j$-th target $\mathcal{K}_{t}^{j}$, the switch
$\sigma_{k}(q)$ is activated where $1\leq k\leq j$. At inference time, our
selector extracts top1-NN fact $m^{*}$, which is closest to a query $q$. If
$m^{*}$ is in $\mathcal{M}_{k}$, the switch $\sigma_{j}(q)$ is activated where
$1\leq j\leq k$, as follows:

$m^{*}=\operatorname*{argmax}_{m}(\textrm{sim}(m,q)),~{}~{}m\in\mathcal{M}_{1:M}\\\
$ (12)

If the NN fact $m^{*}$ is in $\mathcal{M}_{j}$, we estimate that its implicit
knowledge is stored in the accumulated function $\sum_{k=1}^{j}g_{k}(x)$. That
is, when $m^{*}$ is in $\mathcal{M}_{j}$, the activation is decided as
follows:

$\begin{split}\sigma_{k}(q)=&\begin{cases}~{}~{}1~{}~{}~{}~{}~{}\textrm{if}~{}~{}s_{q}\geq\delta~{}~{}\textrm{and}~{}~{}1\leq
k\leq j,\\\ ~{}~{}0~{}~{}~{}~{}~{}\textrm{if}~{}~{}s_{q}<\delta.\\\
\end{cases}\end{split}$ (13)

#### An alternative adapter

We can replace LoRA with K-adapter Wang et al. (2020). In K-adapter, $f$ is a
transformer layer (denoted as $\textrm{TRM}(x)$), and $g$ is multiple
transformer layers with two projection layers (denoted as $\textrm{KIA}(x)$)).
That is, $f(x)=\textrm{TRM}(x)$, consisting of one self-attention & two feed-
forward layers. In the original paper Wang et al. (2020), $g(x)$ consists of
multiple transformer layers and up&down projection layers. For K-adapter, we
set a simple version with only a single transformer layer, as follows:

$h=\textrm{TRM}(x)+\textrm{KIA}(x)$ (14)

where the parameters in TRM are fixed and that in KIA is trainable on target
knowledge.

##  5 Experiment

In this section, we demonstrate the effectiveness of our approach on CuQA.

#### Datasets

We evaluate our method on the following closed-book QA datasets:

(1) Zero-shot Relation Extraction (zsRE): Levy et al. (2017) build relation-
specific QA pairs, and De Cao et al. (2021) utilize this dataset for a closed-
book QA task. This set provides question paraphrases based on the same fact
and answer. We split this set into two groups ($\mathcal{K}_{s}$ and
$\mathcal{K}_{t}$) that do not share the same facts. To validate
generalization, we build held-out sets ($\mathcal{P}_{s}$ and
$\mathcal{P}_{t}$) that are not used in training process. For this, we sample
one QA pair among paraphrases based the same fact as $\mathcal{P}$.

(2) Natural Questions (NQ) \+ SituatedQA: Kwiatkowski et al. (2019) build NQ –
a large-scale QA dataset based on user queries. We consider NQ as source
knowledge $\mathcal{K}_{s}$ except outdated facts based on SituatedQA. Zhang
and Choi (2021) proposed SituatedQA identifying temporal- and geographical-
dependent questions on a subset of NQ. We use the temporal-dependent QA pairs
as $\mathcal{K}_{t}$, which are annotated based on 2021 dump of Wikipedia. For
$\mathcal{P}_{s}$ and $\mathcal{P}_{t}$, as both NQ and SituatedQA do not
provide paraphrases, we follow De Cao et al. (2021) using back-translation for
generating paraphrases.

#### Implementation

For T5 model, we use a large version with total 770M parameters. In our
experiment, we assume that the old model $\theta^{old}$ storing source
knowledge is available. For NQ, we used the open-source pre-trained
model222https://huggingface.co/google/t5-large-ssm-nq as the model
$\theta^{old}$. For zsRE, we load and train T5
model333https://huggingface.co/google/t5-large-ssm on source knowledge. For
training, we set batch size 64 on 4 RTX3090 GPUs, and used Adam Kingma and Ba
(2015) optimizer with learning rate 4e-4. For development set, we sample each
1K from $\mathcal{K}_{s}$, $\mathcal{K}_{t}$, and select the maximum harmonic
mean of their accuracies as a best model. As a hyper-parameter, we search
$\delta$ in a range of [0,1] with 0.05 step size, and found the best value
($\delta$=0.9) based on development set. As embedding memory $\mathcal{M}$, we
used additional parameters: 60M for zsRE and 8.5M for NQ. The size of the
memories can be reduced by several techniques, such as random projection Luan
et al. (2020) and binary encoding Yamada et al. (2021), which is left out of
our focus.

#### Comparison with baselines

We compare our method with baselines, as mentioned in Section 3.2; Fine-tuning
(B-I), RecAdam (B-II), LoRA (B-III), and K-adapter (B-IV). When re-
implementing K-adapter, we do not freeze the parameters of decoder, unlike in
the original paper Wang et al. (2020), because the performance is not changing
when freezing. We train each model until 80 epochs and select a best model by
the harmonic mean of source/target knowledge in development set.

---
| The total # of examples
| $\mathcal{K}_{s}$ | $\mathcal{P}_{s}$ | $\mathcal{K}_{t}$ | $\mathcal{P}_{t}$
zsRE (Large) | 60K | 24K | 60K | 24K
zsRE (Medium) | 60K | 24K | 30K | 12K
zsRE (Small) | 60K | 24K | 15K | 6K
NQ + SituatedQA | 59K | 32K | 8.3K | 1.6K

Table 2: Statistics of datasets.

---
|  | zsRE Question Answering | NQ (with SituatedQA)
Method |  # of Prams (train/total) | $\mathcal{K}_{s}$ | $\mathcal{P}_{s}$ | $\mathcal{K}_{t}$ | $\mathcal{P}_{t}$ |  F/U Ratio | $\mathcal{K}_{s}$ | $\mathcal{P}_{s}$ | $\mathcal{K}_{t}$ | $\mathcal{P}_{t}$ |  F/U Ratio
Model $\theta^{old}$ | - | 95.6 | 95.2 | 25.7 | 28.5 | - | 96.6 | 94.9 | 35.3 | 33.7 | -
B-I: Fine-tuning | 737M / 737M | 76.7 | 70.6 | 92.6 | 85.9 | 0.284 | 92.9 | 82.5 | 94.9 | 92.9 | 0.435
B-II: RecAdam | 737M / 737M | 80.5 | 74.7 | 91.6 | 83.5 | 0.230 | 93.1 | 82.1 | 93.8 | 92.1 | 0.419
B-III: K-adapter | 538M / 840M | 80.5 | 70.8 | 96.4 | 89.6 | 0.215 | 94.4 | 81.4 | 94.8 | 89.4 | 0.259
B-IV: LoRA |  62M / 799M | 71.1 | 62.9 | 92.9 | 84.8 | 0.366 | 89.8 | 74.0 | 94.0 | 90.5 | 0.800
Ours (+K-adapter) | 538M / 840M | 86.3 | 78.9 | 96.4 | 91.1 | 0.132 | 95.6 | 88.1 | 94.9 | 90.3 | 0.118
Ours (+LoRA) |  62M / 799M | 90.5 | 90.6 | 95.3 | 89.4 | 0.073 | 95.6 | 95.2 | 95.1 | 90.0 | 0.117

Table 3: The comparison of the continual learning results on zsRE (Large) and
NQ datasets. We measure the accuracies on the knowledge
$\mathcal{K}_{s},\mathcal{K}_{t}$, and the paraphrase knowledge
$\mathcal{P}_{s},\mathcal{P}_{t}$, with the F/U ratio.  Figure 3: Accuracies
of ours and baselines over training epochs.

###  5.1 R1: Comparing Ours with Baselines

Table 3 shows our main experimental results on two CBQA datasets. First, the
model $\theta^{old}$ memorizes the source knowledge $\mathcal{K}_{s}$ well and
generalizes on the paraphrase set $\mathcal{P}_{s}$ as well, showing high
accuracy on both datasets. After training on $\mathcal{K}_{t}$, all models
perform well on $\mathcal{K}_{t}$ and $\mathcal{P}_{t}$. These results
indicate that these models are at least appropriate for memorizing training
data in the current task.

Meanwhile, while acquiring $\mathcal{K}_{t}$, the models show variant results
on $\mathcal{K}_{s}$ and $\mathcal{P}_{s}$, which have the different ability
of retaining previous knowledge against forgetting. In Fine-tuning (B-I), its
performances on source knowledge $\mathcal{K}_{s}$ and $\mathcal{P}_{s}$
decrease as training epochs (see Figure 3). RecAdam (B-II) alleviates the
forgetting problem of fine-tuning, but the performance gains are marginal on
two datasets. K-adapter (B-III) shows the strong performance on
$\mathcal{K}_{s}$ with less forgetting, however, does not perform well on
$\mathcal{P}_{s}$ and $\mathcal{P}_{t}$ showing low generalization. Because
LoRA (B-IV) has the fewest trainable parameters, its forgetting is more
aggravated, showing the worst performance on $\mathcal{K}_{s}$ and
$\mathcal{P}_{s}$ in both zsRE and NQ. Ours with either K-adapter or LoRA
shows the best performance on $\mathcal{K}_{s}$ and $\mathcal{K}_{t}$. In
terms of the F$/$U ratio, our method also shows the lowest loss when updating
one new example. Figure 3 shows how the performance of each model changes over
training epochs, on the development set.

#### Ablation study

In an ablation study, we test which component has the higher impact on
memorizing implicit knowledge, on paraphrase set $\mathcal{P}_{s}$ and
$\mathcal{P}_{t}$. In our method with LoRA, the function $f$ in Eq. (8) can be
applied to any projection layer in transformers. While the original work Hu et
al. (2021) applies to query- and value-matrices ($W_{Q},W_{V}$) in self-
attention, we consider feed-forward layers ($W_{FF}$), as well as self-
attention. In addition, we observe how does the performance vary when the
number of parameters increases by controlling rank $r$. In Table 4, we
empirically found applying feed-forward layers is more effective than query
and value projection, especially on target knowledge $\mathcal{P}_{t}$. These
results indicate that memorizing factual knowledge is more relevant with a
feed-forward module, which is consistent with the views in Sukhbaatar et al.
(2019); Geva et al. (2020).

---
Type |  $W_{Q}$,$W_{V}$ | $W_{FF}$ | All
Rank $r$ | 16 | 64 | 256 | 16 | 64 | 256 | 256
$\mathcal{P}_{s}$ | 94.9 | 95.2 | 95.5 | 95.1 | 95.0 | 95.2 | 95.2
$\mathcal{P}_{t}$ | 59.6 | 65.1 | 65.5 | 87.1 | 89.2 | 90.0 | 89.3

Table 4: An ablation study Figure 4: The accuracies on multiple knowledge
sources (K=5) over training epochs for zsRE. Figure 5: Accuracies over varying
size of zsRE.

###  5.2 R2: Accumulating over Multiple $\mathcal{K}_{t}$

To evaluate the scalability of our method on multiple $\mathcal{K}_{t}$
($>$2), we assume multiple updates (five-phase) with smaller amount of
examples, by splitting target knowledge $\mathcal{K}_{t}$ in zsRE (Large,
60K), into four sets, from $\mathcal{K}_{t}^{1}$ to $\mathcal{K}_{t}^{4}$
(each 15K). In this experiment, we train models during 40 epochs/phase. To
generalize for LoRA baseline, we aggregate multiple $g_{k}$ by addition, by
activating all the switches at inference, i.e., $\sigma_{1:M}(x)=1$ in Eq.
(13). This setting assumes that this baseline cannot leverage our selector to
organize the storage of implicit knowledge. Figure 4 shows the performances of
Fine-tuning, LoRA, and Ours, over training epochs. In fine-tuning, the
accuracy on source knowledge keeps dropping during the whole training process.
In LoRA, multiple updating deteriorates memorizing target knowledge stored in
adapters, faster than source knowledge stored in the original parameters. This
indicates that the fewer parameters, the faster the forgetting. In contrast,
our method consistently outperforms the baselines, by retaining five
knowledge, with forgetting less. To summarize these results, sequential
updates aggravate forgetting of the fine-tuning method, which can be overcome
through the selective use of adapters.

###  5.3 R3: Over varying Size of $\mathcal{K}_{t}$

As the size of target knowledge increases, it makes LMs suffer from more
forgetting, increasing the distance between before- and after-parameters. In
this section, we observe how does the performance of each model vary as
different sizes of $\mathcal{K}_{t}$. Figure 5 shows the the accuracies of
zsRE datasets (Large-60K, Medium-30K, Small-15K), over training epochs. On
source knowledge $\mathcal{K}_{s}$, the performance of fine-tuning and LoRA
keeps dropping, and the accuracy drops are proportional to the size of target
knowledge. Meanwhile, our method with LoRA consistently maintains high
performance, which is not sensitive to training epochs. On target knowledge
$\mathcal{K}_{t}$, the performances of three models reach high accuracy.
However, our method on Large zsRE shows unstable performance at the end of
training, which may need to use early stopping.

| Ground-truth
---|---
| Source | Target
Selector Prediction | Source |  19527 (40.7%) |  854 (1.8%)
Target |  4473 (9.3%) |  23146 (48.2%)

Table 5: The confusion matrix of Selector.

|  | Ground-truth
---|---|---
|  | Source | Target
Selector Prediction | Source | 95.3 | 35.1
Target | 70.8 (0.0) | 91.7 (97.4)

Table 6: The accuracies of Ours/Retrieval in four cases.

###  5.4 Analysis of Selector

In Table 5, we show the distribution of selector’s predictions and the ground-
truths, in our experiment on zsRE (Large). Nearest Neighbor-based selector
successfully classifies 88.9% of examples, while 11.1% failed. In our method,
if the selector classifies an input as target knowledge, the plug-in $g$ is
activated. Instead of the use of $g$, we can retrieve answers aligned with
questions in $\mathcal{M}$, not generate them. We compare our generation with
the retrieval in each case of Table 5. Table 6 shows the accuracy of
predicting the answers, where the numbers in each cell indicate EM of our
generation (retrieval: in parentheses). If an example in source knowledge is
incorrectly classified as target, there is no relevant fact in $\mathcal{M}$,
thus the accuracy in this case is zero. In contrast to Retrieval, our
generative method is robust in this case, achieving 70.8% EM, because ours
with $g$ learned the source knowledge.

##  6 Conclusion

This paper studies how to accumulate new knowledge to LMs that stores existing
knowledge. We propose a simple yet effective method to update target knowledge
into new parameters, preventing from forgetting source knowledge. On two
datasets: zsRE and NQ, our empirical results show that our proposed method can
improve existing approaches for continual learning or task adaptation.

##  7 Acknowledgement

This research was supported by SNU-NAVER Hyperscale AI Center, and IITP grants
funded by the Korea government (MSIT) [2021-0-02068 SNU AIHub,
IITP-2022-2020-0-01789].

## References

  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_.
  * Chen et al. (2020) Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. 2020\.  Recall and learn: Fine-tuning deep pretrained language models with less forgetting.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7870–7881.
  * Dai et al. (2021) Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. 2021.  Knowledge neurons in pretrained transformers.  _arXiv preprint arXiv:2104.08696_.
  * De Cao et al. (2021) Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021.  Editing factual knowledge in language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Dhingra et al. (2021) Bhuwan Dhingra, Jeremy R Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W Cohen. 2021.  Time-aware language models as temporal knowledge bases.  _arXiv preprint arXiv:2106.15110_.
  * Geva et al. (2020) Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2020.  Transformer feed-forward layers are key-value memories.  _arXiv preprint arXiv:2012.14913_.
  * Heinzerling and Inui (2021) Benjamin Heinzerling and Kentaro Inui. 2021.  Language models as knowledge bases: On entity representations, storage capacity, and paraphrased queries.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1772–1791.
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the National Academy of Sciences of the United States of America_ , 114(13):3521–3526.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7\.
  * Levy et al. (2017) Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017.  Zero-shot relation extraction via reading comprehension.  In _Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)_ , pages 333–342.
  * Lin et al. (2020) Zhaojiang Lin, Andrea Madotto, and Pascale Fung. 2020.  Exploring versatile generative language model via parameter-efficient transfer learning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_ , pages 441–459.
  * Luan et al. (2020) Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2020.  Sparse, dense, and attentional representations for text retrieval.  _arXiv preprint arXiv:2005.00181_.
  * Mitchell et al. (2021) Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2021.  Fast model editing at scale.  _arXiv preprint arXiv:2110.11309_.
  * Ni et al. (2021) Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B Hall, Daniel Cer, and Yinfei Yang. 2021.  Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models.  _arXiv preprint arXiv:2108.08877_.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language models as knowledge bases?  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2463–2473.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21:1–67.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model?  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5418–5426.
  * Sukhbaatar et al. (2019) Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, and Armand Joulin. 2019.  Augmenting self-attention with persistent memory.  _arXiv preprint arXiv:1907.01470_.
  * Wang et al. (2020) Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Guihong Cao, Daxin Jiang, Ming Zhou, et al. 2020.  K-adapter: Infusing knowledge into pre-trained models with adapters.  _arXiv preprint arXiv:2002.01808_.
  * Yamada et al. (2021) Ikuya Yamada, Akari Asai, and Hannaneh Hajishirzi. 2021.  Efficient passage retrieval with hashing for open-domain question answering.  _arXiv preprint arXiv:2106.00882_.
  * Zhang and Choi (2021) Michael JQ Zhang and Eunsol Choi. 2021.  Situatedqa: Incorporating extra-linguistic contexts into qa.  _arXiv preprint arXiv:2109.06157_.


# The frequency of metal-enrichment of cool helium-atmosphere white dwarfs
using the DESI Early Data Release

Christopher J. Manser,1,2 Boris T. Gänsicke,2 Paula Izquierdo,2 Andrew Swan,2
Joan Najita,3 C. Rockosi,4,5,6 Andreia Carrillo,7,8 Bokyoung Kim,9 Siyi Xu,10
Arjun Dey,3 J. Aguilar,11 S. Ahlen,12 R. Blum,3 D. Brooks,13 T. Claybaugh,11
K. Dawson,14 A. de la Macorra,15 P. Doel,13 E. Gaztañaga,16,17,18 S. Gontcho A
Gontcho,11 K. Honscheid,19,20,21 R. Kehoe,22 A. Kremin,11 M. Landriau,11 L. Le
Guillou,23 Michael E. Levi,11 T. S. Li,24 A. Meisner,3 R. Miquel,25,26 J.
Nie,27 M. Rezaie,28 G. Rossi,29 E. Sanchez,30 M. Schubnell,31,32 G. Tarlé,32
B. A. Weaver,3 Z. Zhou,26 H. Zou26
1 Astrophysics Group, Department of Physics, Imperial College London, Prince
Consort Rd, London, SW7 2AZ, UK
2 Department of Physics, University of Warwick, Coventry CV4 7AL, UK
3 NSF’s NOIRLab, 950 N. Cherry Avenue, Tucson, AZ 85719, USA
4 Department of Astronomy and Astrophysics, University of California, Santa
Cruz, 1156 High Street, Santa Cruz, CA 95065, USA
5 Department of Astronomy and Astrophysics, UCO/Lick Observatory, University
of California, 1156 High Street, Santa Cruz, CA 95064, USA
6 University of California Observatories, 1156 High Street, Sana Cruz, CA
95065, USA
7 Institute for Computational Cosmology, Department of Physics, Durham
University, Durham DH1 3LE, U.K
8 Centre for Extragalactic Astronomy, Department of Physics, University of
Durham, South Road, Durham DH1 3LE, UK
9 Institute for Astronomy, University of Edinburgh, Royal Observatory,
Blackford Hill, Edinburgh EH9 3HJ, UK
10 Gemini Observatory/NSF’s NOIRLab, 670 N. A’ohoku Place, Hilo, Hawaii,
96720, USA
11 Lawrence Berkeley National Laboratory, 1 Cyclotron Road, Berkeley, CA
94720, USA
12 Physics Dept., Boston University, 590 Commonwealth Avenue, Boston, MA
02215, USA
13 Department of Physics & Astronomy, University College London, Gower Street,
London, WC1E 6BT, UK
14 Department of Physics and Astronomy, The University of Utah, 115 South 1400
East, Salt Lake City, UT 84112, USA
15 Instituto de Física, Universidad Nacional Autónoma de México, Cd. de México
C.P. 04510, México
16 Institut d’Estudis Espacials de Catalunya (IEEC), 08034 Barcelona, Spain
17 Institute of Cosmology & Gravitation, University of Portsmouth, Dennis
Sciama Building, Portsmouth, PO1 3FX, UK
18 Institute of Space Sciences, ICE-CSIC, Campus UAB, Carrer de Can Magrans
s/n, 08913 Bellaterra, Barcelona, Spain
19 Center for Cosmology and AstroParticle Physics, The Ohio State University,
191 West Woodruff Avenue, Columbus, OH 43210, USA
20 Department of Physics, The Ohio State University, 191 West Woodruff Avenue,
Columbus, OH 43210, USA
21 The Ohio State University, Columbus, 43210 OH, USA
22 Department of Physics, Southern Methodist University, 3215 Daniel Avenue,
Dallas, TX 75275, USA
23 Sorbonne Université, CNRS/IN2P3, Laboratoire de Physique Nucléaire et de
Hautes Energies (LPNHE), FR-75005 Paris, France
24 Department of Astronomy & Astrophysics, University of Toronto, Toronto, ON
M5S 3H4, Canada
25 Institució Catalana de Recerca i Estudis Avançats, Passeig de Lluís
Companys, 23, 08010 Barcelona, Spain
26 Institut de Física d’Altes Energies (IFAE), The Barcelona Institute of
Science and Technology, Campus UAB, 08193 Bellaterra Barcelona, Spain
27 National Astronomical Observatories, Chinese Academy of Sciences, A20 Datun
Rd., Chaoyang District, Beijing, 100012, P.R. China
28 Department of Physics, Kansas State University, 116 Cardwell Hall,
Manhattan, KS 66506, USA
29 Department of Physics and Astronomy, Sejong University, Seoul, 143-747,
Korea
30 CIEMAT, Avenida Complutense 40, E-28040 Madrid, Spain
31 Department of Physics, University of Michigan, Ann Arbor, MI 48109, USA
32 University of Michigan, Ann Arbor, MI 48109, USA
E-mail<EMAIL_ADDRESS>
(Accepted XXX. Received YYY; in original form ZZZ)

###### Abstract

There is overwhelming evidence that white dwarfs host planetary systems;
revealed by the presence, disruption, and accretion of planetary bodies. A
lower limit on the frequency of white dwarfs that host planetary material has
been estimated to be $\simeq$ 25 – 50 per cent; inferred from the ongoing or
recent accretion of metals onto both hydrogen-atmosphere and warm helium-
atmosphere white dwarfs. Now with the unbiased sample of white dwarfs observed
by the Dark Energy Spectroscopic Instrument (DESI) survey in their Early Data
Release (EDR), we have determined the frequency of metal-enrichment around
cool-helium atmosphere white dwarfs as $21\pm 3$ per cent using a sample of
234 systems. This value is in good agreement with values determined from
previous studies. With the current samples we cannot distinguish whether the
frequency of planetary accretion varies with system age or host-star mass, but
the DESI data release 1 will contain roughly an order of magnitude more white
dwarfs than DESI EDR and will allow these parameters to be investigated.

###### keywords:

white dwarfs – surveys – planetary systems – exoplanets

††pubyear: 2023††pagerange: The frequency of metal-enrichment of cool helium-
atmosphere white dwarfs using the DESI Early Data Release–The frequency of
metal-enrichment of cool helium-atmosphere white dwarfs using the DESI Early
Data Release

##  1 Introduction

Planetary systems around main-sequence stars are ubiquitous. Multiple
occurrence-rate studies of planetary-mass bodies show that on average there is
at least one planet for every star in the Milky Way (Cassan et al., 2012;
Poleski et al., 2021; Zhu & Dong, 2021). The fraction of stars that host
planetary systems is harder to constrain, with estimates that $\simeq$ 30 per
cent of Sun-like stars host planets with radii greater than one Earth radius,
and orbital periods less than 400 d (Zhu et al., 2018). As the host stars of
planetary systems evolve into the giant-phase, it is expected that many of
their planets orbiting at distances greater than several au will survive and
continue to orbit their stars after they exhaust their fuel and evolve off the
main sequence (Villaver & Livio, 2009; Mustill et al., 2014; Veras & Gänsicke,
2015; Lagos et al., 2021; Veras & Hinkley, 2021).

White dwarfs are the dense, degenerate stellar remnants of main-sequence stars
with initial masses $\leq 8$ $\mathrm{M}_{\odot}$ (Iben et al., 1997; Dobbie
et al., 2006). Their high surface gravities lead to the chemical
stratification of their atmospheres, resulting in the majority of white dwarfs
having either hydrogen- or helium-dominated atmospheres. Heavier elements
settle out of the observable atmosphere on the diffusion timescale, which
varies from a few days in hot ($\simeq$ 20 000 K) hydrogen-atmosphere white
dwarfs, to several million years in cool ($<$ 12 000 K) helium-atmosphere
white dwarfs (Koester, 2009). Thus the presence of photospheric metal-line
absorption features in the spectra of white dwarfs usually***Radiative
levitation can sustain the presence of heavier elements in the atmosphere of
hot ($\gtrsim$ 25 000 K) white dwarfs (Chayer et al., 1995), and convection
can “dredge-up” substantial amounts of carbon up from the core in cool
($\lesssim$ 10 000 K), helium-atmosphere white dwarfs (Pelletier et al.,
1986). reveals the recent or ongoing accretion of metals.

There is a large wealth of evidence that these metals arrive at the white
dwarf predominantly from a remnant planetary system that has survived the
evolution of its host star to the white dwarf phase. This is identified by (i)
direct observations of disintegrating and transiting planetary-sized bodies
(Vanderburg et al., 2015, 2020; Gänsicke et al., 2019; Vanderbosch et al.,
2020, 2021; Guidry et al., 2021; Farihi et al., 2022b), (ii) compact ($\simeq$
1 $\mathrm{R}_{\odot}$) debris discs made of dust (Zuckerman & Becklin, 1987;
Jura, 2003; Rocchetto et al., 2015; Wilson et al., 2019), and/or gas (Gänsicke
et al., 2006; Manser et al., 2020; Dennihy et al., 2020; Melis et al., 2020;
Gentile Fusillo et al., 2021) produced from the complete or partial disruption
of a planetary body like an asteroid or comet, (iii) the emission of x-rays
from the direct bombardment of the white dwarf surface from the accretion of
such a disc (Cunningham et al., 2022), and (iv) the resultant metal-enrichment
of white dwarf atmospheres with material consistent in composition with
primordial chondritic material (Trierweiler et al., 2023), bulk-Earth
(Zuckerman et al., 2007; Gänsicke et al., 2012; Hollands et al., 2017, 2018a),
and more diverse bodies showing volatiles (Farihi et al., 2013; Xu et al.,
2017; Johnson et al., 2022).

For white dwarfs, the frequency of metal-enrichment of their atmospheres can
be used as a proxy for a lower limit on the occurrence rate of planetary
systems that orbit them. This has been performed in several studies using
hydrogen-atmosphere white dwarfs (Zuckerman et al., 2003; Koester et al.,
2014; Wilson et al., 2019), as they are the most abundant type of white dwarf.
Zuckerman et al. (2003) looked at a sample of approximately 100 hydrogen-
atmosphere white dwarfs without close stellar companions, with the majority of
systems having relatively cool effective temperatures ($T_{\mathrm{eff}}$ $<$
10 000 K), and identified that roughly 25 per cent of them had absorption
features consistent with metal-enrichment from planetary material. This result
provided the first lower limit on the occurrence rate of planetary systems
around white dwarfs. Koester et al. (2014) utilised the Hubble Space Telescope
(HST) to look at a sample of 85 hydrogen-atmosphere white dwarfs with
effective temperatures in the range 27 000 K $>$ $T_{\mathrm{eff}}$ $>$ 17 000
K. This sample of warm hydrogen-dominated white dwarfs provided a frequency of
white dwarfs that have accreted planetary material ranging from $\simeq$ 25 –
50 per cent. A more recent study using HST by Wilson et al. (2019) using 143
hydrogen-atmosphere (DA) white dwarfs (which includes the sample of Koester et
al. 2014) corroborates these findings with an occurrence rate of the accretion
of planetary material of $45\pm 4$ per cent.

The frequency of planetary systems around warm helium-atmosphere white dwarfs
has been investigated by Zuckerman et al. (2010), who determined an occurrence
rate of white dwarfs accreting planetary material of $\simeq$ 30 per cent in
the temperature range 19 500 K $>$ $T_{\mathrm{eff}}$ $>$ 13 500 K. This is
again consistent with the occurrence rates determined from hydrogen-atmosphere
white dwarfs, and the frequency of planets around main-sequence stars. This
would suggest that the frequency of planetary systems around white dwarfs with
hydrogen- and helium-dominated atmospheres do not differ.

The Dark Energy Spectroscopic Instrument (DESI, DESI Collaboration et al.
2016a, b) observes white dwarfs as part of its 5-year survey (Allende Prieto
et al., 2020), and the Early Data Release (EDR, DESI Collaboration et al.
2023) contains 2706 spectroscopically confirmed white dwarfs (Manser et al.,
submitted to MNRAS). These white dwarfs were selected using Gaia photometry
and astrometry with the selections described in Cooper et al. (2023), and
based on those of Gentile Fusillo et al. (2019, 2021). The DESI EDR white
dwarf sample closely follows the $G<20$ magnitude limited sample of high-
confidence white dwarfs of Gentile Fusillo et al. (2021), making it far less
biased than the sample of white dwarfs obtained by the Sloan Digital Sky
Survey (SDSS, Gunn et al. 2006; Abdurro’uf et al. 2022) which were obtained
through multiple differing targeting strategies (Kleinman et al., 2004), and
therefore applicable for statistical analyses.

In this paper, we use a subset of cool ($11\,500$ K $>T_{\mathrm{eff}}>5\,000$
K) helium-atmosphere white dwarfs with and without signs of metal-enrichment
to determine a lower limit on the occurrence rate on planetary systems around
white dwarfs with cooling ages longer than $\simeq$ 400 Myr. We give details
on the white dwarfs we use for this calculation obtained from the DESI EDR
white dwarf sample in Section 2. We then calculate the frequency of white
dwarf planetary systems and test the robustness of our calculations in Section
3, and discuss our results and conclude in Section 4.

##  2 Sample selection

In this analysis we use the set of spectroscopically confirmed (i.e. excluding
uncertain classifications, noted by “:”) systems in the DESI EDR white dwarf
sample (Manser et al., submitted to MNRAS). DESI on the Mayall 4 m telescope
at Kitt Peak National Observatory (KPNO) is a multi-object spectroscopic
instrument capable of collecting fibre spectroscopy on up to $\simeq 5000$
targets per pointing (DESI Collaboration et al., 2022). The fibres are
positioned by robot actuators and are grouped into ten petals which feed ten
identical three-arm spectrographs, each spanning 3600 – 9824 Å at a FWHM
resolution of $\simeq$ 1.8 Å. A full description of the DESI reduction
pipeline is given by Guy et al. (2023).

To determine the frequency of metal-enriched white dwarfs, we need to identify
a subsample of white dwarfs with similar properties where the presence or
absence of extrinsic metal can be deduced. Helium-atmosphere white dwarfs that
display only the signatures of metals (excluding carbon), the DZ spectral
class, are relatively easy to identify, with calcium abundances being
detectable several orders of magnitude lower than their hotter counterparts
that show He i features (DBZs) and optical absorption features that
dramatically alter the emergent spectral energy distribution (Dufour et al.,
2007; Koester & Kepler, 2015; Hollands et al., 2017). Additionally, the
convection zones of DZs are deep, leading to long diffusion timescales ($>1$
Myr) for metals to sink out of their atmospheres and thus making metal-
enrichment detectable for significantly longer than their warmer counterparts
(Koester, 2009). These attributes make DZs good proxies for determining limits
on the occurrence rate of planetary systems around white dwarfs.

Figure 1: Top panel: The Gaia Hertzsprung-Russell Diagram (HRD) showing the
metal-enriched (orange and magenta triangles) and non-enriched (blue stars and
cyan squares) systems used in the calculation of the frequency of planetary
systems around cool, helium-atmosphere white dwarfs, $f$ (see Section 3).
Cooling tracks from Bédard et al. (2020) for pure helium-atmosphere white
dwarfs with masses from top to bottom of 0.4 $\mathrm{M}_{\odot}$, 0.6
$\mathrm{M}_{\odot}$, 0.8 $\mathrm{M}_{\odot}$ and 1.0 $\mathrm{M}_{\odot}$
are plotted as dashed lines on the Gaia HRD, with vertical tabs highlighting
$T_{\textrm{eff}}$ values in steps of 1000 K. These cooling tracks are only
strictly applicable to helium-atmosphere white dwarfs with no traces of metals
or carbon. The red shaded regions shows the ranges excluded from the
$0<(G_{\mathrm{BP}}-G_{\mathrm{RP}})<1$ selection used in our $f$
calculations. One DC, WD J095106.36+645400.46, sits far below the cooling
track with $G_{\mathrm{Abs}}>16$ and is thought to have a spectrum dominated
by collisionally-induced absorption (Kilic et al., 2020; Bergeron et al.,
2022). We still include this system in our analyses. Bottom panel: The
fraction of metal-enriched helium-dominated white dwarfs with no trace
hydrogen detectable as a function of $G_{\textrm{BP}}$ – $G_{\textrm{RP}}$ is
shown in six, equal sized bins of 39 white dwarfs. Vertical lines on the gray
points represent 1 $\sigma$ errors on the value of $f$ in the bin, while the
horizontal lines represent the span of the $(G_{\mathrm{BP}}-G_{\mathrm{RP}})$
values for each bin. The horizontal dashed line and shaded region is set to
the average value and uncertainty of $f=0.21\pm 0.03$ calculated using all 234
white dwarfs in the range $0<(G_{\mathrm{BP}}-G_{\mathrm{RP}})<1$.

A DZ white dwarf can arise from the accretion of planetary material onto a
featureless DC white dwarf, or a DQ showing carbon dredge-up features. The
presence of metals in a white dwarf atmosphere can eradicate the spectral
features of dredged-up carbon at optical wavelengths (Pelletier et al., 1986;
Blouin, 2022; Hollands et al., 2022), but at sufficiently low metal
abundances, carbon dredge-up features and metal-enrichment can be observed
simultaneously, such as in the DZQ†††The ordering of spectral identifications
is based on the strengths of the features present. For a DZQ, the non-carbon
metal features (Z) are stronger than the carbon Swan bands (Q) present., WD
J101453.59+411416.94, and the DQZ, WD J142018.91+324921.12, identified within
the DESI EDR sample. This observational bias is thought to be the reason
metal-enriched DQs appear to be so rare.

As the Balmer lines remain present in the optical spectrum of white dwarfs
down to cooler temperatures than He i lines (Bergeron et al., 1997), cool
helium-atmosphere white dwarfs with traces of hydrogen can have spectral types
DA, DAZ, and DZA (e.g. GD362; Zuckerman et al. 2007). These are excluded from
the following analysis, as without full modelling of their spectra it is
difficult to distinguish them from the more common hydrogen-atmosphere white
dwarfs that share those spectral types.

Based on the above arguments, we select the metal-enriched helium-atmosphere
DZs, DQZs, and DZQs, along with their non-enriched counterparts: DCs and DQs,
from the DESI EDR sample (Manser et al., submitted to MNRAS). This results in
a subsample of 298 white dwarfs containing 197 DCs, 50 DQs, 49 DZs, 1 DQZ, and
1 DZQ (Fig. 1). We further restrict our sample to exclude likely contaminants
using a Gaia colour cut $0<(G_{\mathrm{BP}}-G_{\mathrm{RP}})<1$. The majority
of DQs bluer than $(G_{\mathrm{BP}}-G_{\mathrm{RP}})=0$ sit significantly
below the white dwarf cooling track due to them having high masses. These
systems are thought to be white dwarfs produced in binary mergers (Dunlap &
Clemens, 2015; Kawka et al., 2023), and therefore do not have the same origin
as the more abundant cool DQs which are likely to be the non-enriched
equivalents to DZ, DZQ and DQZ white dwarfs. DCs redder than
$(G_{\mathrm{BP}}-G_{\mathrm{RP}})=1$ are thought to be dominated by hydrogen-
rich atmosphere white dwarfs as the Balmer absorption lines stop being present
in the emergent spectrum (Caron et al., 2023), and we therefore exclude
systems above this colour limit. For helium-dominated atmospheric models with
no carbon or metals, the $0<(G_{\mathrm{BP}}-G_{\mathrm{RP}})<1$ colour range
spans the broad temperature range of $\simeq$ $11\,500$ K
$>T_{\mathrm{eff}}>5000$ K (Bédard et al., 2020).

In the $0<(G_{\mathrm{BP}}-G_{\mathrm{RP}})<1$ range the number of white
dwarfs in our selection reduces to 141 DCs, 43 DQs, 48 DZs, 1 DQZ and 1 DZQ,
for a total of 234 systems, while removing 1 DZ, 56 DCs, and 7 DQs.

##  3 Results

We calculate the fraction, $f$, of cool metal-enriched helium-atmosphere white
dwarfs as the number of DZ, DQZ, and DZQ white dwarfs divided by both the
number of these metal-enriched systems and the number of the types they would
appear as without the accretion of planetary bodies - DCs and DQs,
$f=(n_{\mathrm{DZ}}+n_{\mathrm{DQZ}}+n_{\mathrm{DZQ}})/(n_{\mathrm{DC}}+n_{\mathrm{DQ}}+n_{\mathrm{DZ}}+n_{\mathrm{DQZ}}+n_{\mathrm{DZQ}})$
arriving at a value of $f$ = 0.21 $\pm$ 0.03. The uncertainty on $f$ has been
determined by sampling from a binomial distribution, and this method is used
throughout this study. The value of $f$ obtained here is in reasonable
agreement with values determined using cool hydrogen-atmosphere white dwarfs
(Zuckerman et al., 2003), but significantly lower than those determined for
warmer hydrogen- and helium-atmosphere white dwarfs (Zuckerman et al., 2010;
Koester et al., 2014; Wilson et al., 2019).

We investigate the dependence of $f$ on $(G_{\mathrm{BP}}-G_{\mathrm{RP}})$ in
Fig. 1, where we calculate $f$ in six equal-sized bins of 39 white dwarfs
across the $0<(G_{\mathrm{BP}}-G_{\mathrm{RP}})<1$. $f$ varies between
$\simeq$0.15 – 0.36 throughout the selected colour range of
$0<(G_{\mathrm{BP}}-G_{\mathrm{RP}})<1$, with five out of six bins in
excellent agreement with the constant value of $f=0.21\pm 0.03$, which would
suggest there is no detectable trend between $f$ and
$(G_{\mathrm{BP}}-G_{\mathrm{RP}})$.

Table 1: The fraction of white dwarfs that have accreted planetary material, $f$ from various studies. The sample size column gives both the total sample size, with the number of metal-enriched white dwarf in brackets. Cooling ages, $\tau$, are estimated based on the temperature range of the sample and white dwarf evolutionary models of Bédard et al. (2020).  Sample | WD type | Sample size | Fraction ($f$) | Cooling age ($\tau$) [Myr]
---|---|---|---|---
This work, $0<(G_{\mathrm{BP}}-G_{\mathrm{RP}})<1$ | helium-atmosphere (DC + DQ) | 234 (50) | $0.21\pm 0.03$ |  $\sim$ 300 - 10 000
This work, $0<(G_{\mathrm{BP}}-G_{\mathrm{RP}})<1$ | helium-atmosphere (DC) | 189 (48) | $0.25\pm 0.03$ |  $\sim$ 300 - 10 000
This work | DC + DQ a | 298 (51) | $0.17\pm 0.02$ |  $\sim$ 100 - 10 000
This work | DC a | 246 (49) | $0.20\pm 0.03$ |  $\sim$ 100 - 10 000
Hollands et al. (2018b) | DC + DQ a | 51 (9) | $0.18\pm 0.05$ |  $\sim$ 100 - 10 000
Hollands et al. (2018b) | DC a | 38 (9) | $0.24\pm 0.07$ |  $\sim$ 100 - 10 000
Zuckerman et al. (2010) | helium-atmosphere (DB + DBA) | 54 (16) | 0.30 |  $\sim$ 100 - 300
Zuckerman et al. (2003) | hydrogen-atmosphere (DA) |  $\simeq$ 80 ($\simeq$ 20) |  $\simeq$ 0.25 |  $\sim$ 600 - 10 000
Koester et al. (2014) b actively accreting | hydrogen-atmosphere (DA) | 85 (23) | 0.27 |  $\sim$ 10 - 300
Koester et al. (2014) b metals present | hydrogen-atmosphere (DA) | 85 (48) | 0.56 |  $\sim$ 10 - 300
Wilson et al. (2019) b | hydrogen-atmosphere (DA) | 143 (65) | $0.45\pm 0.04$ |  $\sim$ 10 - 300
a No colour cuts have been placed and it is likely that the sample includes
hydrogen-atmosphere DCs and high-mass binary-merger DQ white dwarfs.
b Sample observed exclusively with HST rather than ground-based observatories.

Our calculation also relies on the assumption that the colour cut
$0<(G_{\mathrm{BP}}-G_{\mathrm{RP}})<1$ we make cleanly selects a sample of
cool-helium dominated white dwarfs and rejects contaminant systems. Relaxing
this cut and using all 298 white dwarfs with spectral types of DC, DQ, DZ, DZQ
and DQZ results in a value of $f$ = 0.17 $\pm$ 0.03, and can be taken as a
conservative lower-limit on the fraction of metal-enriched cool helium-
atmosphere white dwarfs. This value is also consistent with our previous
estimate and a larger sample size is needed to identify a significant
difference between the two values of $f$ obtained so far.

We also compare our values of $f$ to those obtainable using the volume limited
20 pc sample of white dwarfs from Hollands et al. (2018b), which contains 29
DC and DCP white dwarfs, 13 DQ, DQPec and DQPecP white dwarfs, and 9 DZ white
dwarfs, where ‘P’ and ‘Pec’ identify the detection of polarisation due to
magnetism and peculiar/unknown features respectively. The resulting value of
$f$ = 0.18 $\pm$ 0.06 obtained from the entire 20 pc sample is in excellent
agreement with the value of $f$ we obtain from the DESI EDR sample. Finally,
there are some arguments that all DQ white dwarfs are the product of binary
merger events (Farihi et al., 2022a), and as such should not be considered to
be in the same population as the standard, isolated cool DC white dwarfs. If
this is the case, we also calculate $f$ values excluding DQs, DQZs and DZQs,
which are given in Table 1, and are still in good agreement with $f$ values
including carbon-line DQ white dwarfs.

These tests demonstrate the robustness of our assumptions with the available
data sets and that the frequency we determine here is representative for
planetary accretion at cool helium-dominated white dwarfs. We present all the
occurrence rates of planetary accretion at white dwarfs we discuss in Table 1.

##  4 Discussion and Conclusions

Helium-atmosphere white dwarfs in the temperature range $\simeq$ $11\,500$ K
$>T_{\mathrm{eff}}>5000$ K have cooling ages, $\tau$, ranging from 460 Myr
$<\tau<$ 6.4 Gyr for a 0.6 $\mathrm{M}_{\odot}$white dwarf (Bédard et al.,
2020). Our sample therefore probes a significantly older range of observed
white dwarfs compared with the relatively young samples of Koester et al.
(2014), Wilson et al. (2019), and Zuckerman et al. (2010) which span ages in
the range 9 Myr $<\tau<$ 300 Myr. The sample of hydrogen-dominated white
dwarfs investigated by Zuckerman et al. (2003) extends over a temperature and
age range consistent with our sample.

The occurrence rates of planetary systems around white dwarfs obtained by
Wilson et al. (2019) and Koester et al. (2014) are at least as high as
$0.45\pm 0.04$ and $0.56\pm 0.10$ respectively, significantly higher than the
value of $0.21\pm 0.03$ we derive here. Conversely the occurrence of planetary
systems in the samples of Zuckerman et al. (2003) and Zuckerman et al. (2010)
reduces to $\simeq$ 0.25 - 0.30, slightly higher than but still in agreement
with the value we obtain here. While this could be suggestive of the depletion
of planetary material around white dwarfs as a function of cooling age,
Koester et al. (2014) note that this discrepancy is likely due to the high
resolution, high signal-to-noise ratio, and ultraviolet wavelength-range
obtained by the HST Cosmic Origins Spectrograph compared with the ground-based
observations of Zuckerman et al. (2003, 2010) and the DESI EDR sample. Higher-
resolution and deeper follow-up of the full DESI EDR sample of cool helium-
dominated white dwarfs could reveal additional metal-enriched systems
undetected by DESI or identify contaminant systems (e.g. white dwarfs with
weak or magnetically-split hydrogen lines), both of which would increase the
lower limit of $f$ determined here.

Hollands et al. (2018a) found a depletion in the accretion rates on an
$e$-folding timescale of $\simeq$ 1 Gyr around cool DZ white dwarfs in the
cooling-age range 1 Gyr $<\tau<$ 8 Gyr. However, this result has been disputed
by (Blouin & Xu, 2022) who used updated white dwarf models along with Gaia
parallaxes and identified at best only a factor ten drop off in accretion
rates over the range of 1 Gyr $<\tau<$ 8 Gyr. While these studies suggest a
potential depletion or constant rate of planetary material accreted by white
dwarfs over several billion years, they cannot be used to directly probe
changes to the occurrence rate of planetary systems around white dwarfs over
the same time-frame. The detection sensitivity of planetary material in the
atmospheres of white dwarfs is dependent on both the amount of material that
is being delivered to and accreted by white dwarfs in addition to the
frequency of planetary systems around them. Forward-modelling of the samples
of white dwarfs in Table 1 would allow the evolution of the accretion and
frequency of planetary systems to be constrained based on the current
understanding of model atmospheres (Bauer & Bildsten, 2018; Cunningham et al.,
2019), and inform future observing campaigns.

While we ignore the presence of hydrogen in our analysis, there is a
significant link between the presence of hydrogen in a helium-atmosphere white
dwarf, and the presence of metals (Gentile Fusillo et al., 2017). This
correlation in the presence of hydrogen and metals has been explained as
either: (i) the accretion of planetary debris being linked to the accretion of
hydrogen in the form of water (Gentile Fusillo et al., 2017), or the accretion
of giant planet atmospheres (Schreiber et al., 2019), or (ii) that helium-
atmosphere white dwarfs born without detectable trace hydrogen have a
significantly reduced planetary system occurrence rate (Bédard et al., 2023).
Investigating the link between the presence of hydrogen and metals is beyond
the scope of this paper, but the frequencies we calculate here for cool
helium-atmosphere white dwarfs along with those in Table. 1 will allow the
degeneracy between the two hypotheses listed to be lifted.

The DESI survey is well into its five year observing program, and the first
data release (DR1, currently internal) contains spectroscopy of over 47 000
white dwarf candidates (Manser et al., 2023). Based on the DESI EDR sample of
spectroscopically confirmed white dwarfs, it is expected that the DR1 sample
with contain over 2 000 helium-dominated white dwarfs, and roughly 500 metal-
enriched white dwarfs in the colour range
$0<(G_{\mathrm{BP}}-G_{\mathrm{RP}})<1$. Such a sample would allow the
frequency of planetary systems around white dwarfs to be probed in much
greater detail, such as whether there is a drop-off in the occurrence of
planetary systems at late cooling ages. Additionally, this sample would allow
other studies to be performed, such as an independent test of observed
accretion rates with white dwarf cooling ages, or differences in accreted
material as a function of white dwarf mass and therefore host-star progenitor
mass, studies of which are currently limited (Johnson et al., 2010; Zhu &
Dong, 2021).

## Acknowledgements

The authors acknowledge financial support from Imperial College London through
an Imperial College Research Fellowship grant awarded to CJM. This project has
received funding from the European Research Council (ERC) under the European
Union’s Horizon 2020 research and innovation programme (Grant agreement No.
101020057). SX is supported by NOIRLab, which is managed by the Association of
Universities for Research in Astronomy (AURA) under a cooperative agreement
with the National Science Foundation. This material is based upon work
supported by the U.S. Department of Energy (DOE), Office of Science, Office of
High-Energy Physics, under Contract No. DE–AC02–05CH11231, and by the National
Energy Research Scientific Computing Center, a DOE Office of Science User
Facility under the same contract. Additional support for DESI was provided by
the U.S. National Science Foundation (NSF), Division of Astronomical Sciences
under Contract No. AST-0950945 to the NSF’s National Optical-Infrared
Astronomy Research Laboratory; the Science and Technology Facilities Council
of the United Kingdom; the Gordon and Betty Moore Foundation; the Heising-
Simons Foundation; the French Alternative Energies and Atomic Energy
Commission (CEA); the National Council of Science and Technology of Mexico
(CONACYT); the Ministry of Science and Innovation of Spain (MICINN), and by
the DESI Member Institutions: https://www.desi.lbl.gov/collaborating-
institutions. Any opinions, findings, and conclusions or recommendations
expressed in this material are those of the author(s) and do not necessarily
reflect the views of the U. S. National Science Foundation, the U. S.
Department of Energy, or any of the listed funding agencies.

The authors are honored to be permitted to conduct scientific research on
Iolkam Du’ag (Kitt Peak), a mountain with particular significance to the
Tohono O’odham Nation.

This work has made use of data from the European Space Agency (ESA) mission
Gaia (https://www.cosmos.esa.int/gaia), processed by the Gaia Data Processing
and Analysis Consortium (DPAC,
https://www.cosmos.esa.int/web/gaia/dpac/consortium). Funding for the DPAC has
been provided by national institutions, in particular the institutions
participating in the Gaia Multilateral Agreement.

## Data availability

The data presented here are all available from the public archives of DESI.
Data used to produce the figures presented in this paper are available under
DOI: 10.5281/zenodo.10226244 and can be found here
https://zenodo.org/records/10226244.

## References

  * Abdurro’uf et al. (2022) Abdurro’uf et al., 2022, ApJS, 259, 35
  * Allende Prieto et al. (2020) Allende Prieto C., et al., 2020, Research Notes of the American Astronomical Society, 4, 188
  * Bauer & Bildsten (2018) Bauer E. B., Bildsten L., 2018, ApJ, 859, L19
  * Bédard et al. (2020) Bédard A., Bergeron P., Brassard P., Fontaine G., 2020, ApJ, 901, 93
  * Bédard et al. (2023) Bédard A., Bergeron P., Brassard P., 2023, ApJ, 946, 24
  * Bergeron et al. (1997) Bergeron P., Ruiz M. T., Leggett S. K., 1997, ApJS, 108, 339
  * Bergeron et al. (2022) Bergeron P., Kilic M., Blouin S., Bédard A., Leggett S. K., Brown W. R., 2022, ApJ, 934, 36
  * Blouin (2022) Blouin S., 2022, A&A, 666, L7
  * Blouin & Xu (2022) Blouin S., Xu S., 2022, MNRAS, 510, 1059
  * Caron et al. (2023) Caron A., Bergeron P., Blouin S., Leggett S. K., 2023, MNRAS, 519, 4529
  * Cassan et al. (2012) Cassan A., et al., 2012, Nat, 481, 167
  * Chayer et al. (1995) Chayer P., Fontaine G., Wesemael F., 1995, ApJS, 99, 189
  * Cooper et al. (2023) Cooper A. P., et al., 2023, ApJ, 947, 37
  * Cunningham et al. (2019) Cunningham T., Tremblay P.-E., Freytag B., Ludwig H.-G., Koester D., 2019, MNRAS, 488, 2503
  * Cunningham et al. (2022) Cunningham T., Wheatley P. J., Tremblay P.-E., Gänsicke B. T., King G. W., Toloza O., Veras D., 2022, Nature, 602, 219
  * DESI Collaboration et al. (2016a) DESI Collaboration et al., 2016a, preprint,  (arXiv:1611.00036)
  * DESI Collaboration et al. (2016b) DESI Collaboration et al., 2016b, preprint,  (arXiv:1611.00037)
  * DESI Collaboration et al. (2022) DESI Collaboration et al., 2022, AJ, 164, 207
  * DESI Collaboration et al. (2023) DESI Collaboration et al., 2023, arXiv e-prints, p. arXiv:2306.06308
  * Dennihy et al. (2020) Dennihy E., et al., 2020, ApJ, 905, 5
  * Dobbie et al. (2006) Dobbie P. D., et al., 2006, MNRAS, 369, 383
  * Dufour et al. (2007) Dufour P., et al., 2007, ApJ, 663, 1291
  * Dunlap & Clemens (2015) Dunlap B. H., Clemens J. C., 2015, in Dufour P., Bergeron P., Fontaine G., eds, Astronomical Society of the Pacific Conference Series Vol. 493, 19th European Workshop on White Dwarfs. p. 547
  * Farihi et al. (2013) Farihi J., Gänsicke B. T., Koester D., 2013, Science, 342, 218
  * Farihi et al. (2022a) Farihi J., Dufour P., Wilson T. G., 2022a, arXiv e-prints, p. arXiv:2208.05990
  * Farihi et al. (2022b) Farihi J., et al., 2022b, MNRAS, 511, 1647
  * Gänsicke et al. (2006) Gänsicke B. T., Marsh T. R., Southworth J., Rebassa-Mansergas A., 2006, Science, 314, 1908
  * Gänsicke et al. (2012) Gänsicke B. T., Koester D., Farihi J., Girven J., Parsons S. G., Breedt E., 2012, MNRAS, 424, 333
  * Gänsicke et al. (2019) Gänsicke B. T., Schreiber M. R., Toloza O., Fusillo N. P. G., Koester D., Manser C. J., 2019, Nature, 576, 61
  * Gentile Fusillo et al. (2017) Gentile Fusillo N. P., Gänsicke B. T., Farihi J., Koester D., Schreiber M. R., Pala A. F., 2017, MNRAS, 468, 971
  * Gentile Fusillo et al. (2019) Gentile Fusillo N. P., et al., 2019, MNRAS, 482, 4570
  * Gentile Fusillo et al. (2021) Gentile Fusillo N. P., et al., 2021, MNRAS, 508, 3877
  * Guidry et al. (2021) Guidry J. A., et al., 2021, ApJ, 912, 125
  * Gunn et al. (2006) Gunn J. E., et al., 2006, AJ, 131, 2332
  * Guy et al. (2023) Guy J., et al., 2023, AJ, 165, 144
  * Hollands et al. (2017) Hollands M. A., Koester D., Alekseev V., Herbert E. L., Gänsicke B. T., 2017, MNRAS, 467, 4970
  * Hollands et al. (2018a) Hollands M. A., Gänsicke B. T., Koester D., 2018a, MNRAS, 477, 93
  * Hollands et al. (2018b) Hollands M. A., Tremblay P. E., Gänsicke B. T., Gentile-Fusillo N. P., Toonen S., 2018b, MNRAS, 480, 3942
  * Hollands et al. (2022) Hollands M. A., Tremblay P. E., Gänsicke B. T., Koester D., 2022, MNRAS, 511, 71
  * Iben et al. (1997) Iben I. J., Ritossa C., Garcia-Berro E., 1997, ApJ, 489, 772
  * Johnson et al. (2010) Johnson J. A., Aller K. M., Howard A. W., Crepp J. R., 2010, PASP, 122, 905
  * Johnson et al. (2022) Johnson T. M., Klein B. L., Koester D., Melis C., Zuckerman B., Jura M., 2022, ApJ, 941, 113
  * Jura (2003) Jura M., 2003, ApJ Lett., 584, L91
  * Kawka et al. (2023) Kawka A., Ferrario L., Vennes S., 2023, MNRAS, 520, 6299
  * Kilic et al. (2020) Kilic M., Bergeron P., Kosakowski A., Brown W. R., Agüeros M. A., Blouin S., 2020, ApJ, 898, 84
  * Kleinman et al. (2004) Kleinman S. J., et al., 2004, ApJ, 607, 426
  * Koester (2009) Koester D., 2009, A&A, 498, 517
  * Koester & Kepler (2015) Koester D., Kepler S. O., 2015, A&A, 583, A86
  * Koester et al. (2014) Koester D., Gänsicke B. T., Farihi J., 2014, A&A, 566, A34
  * Lagos et al. (2021) Lagos F., Schreiber M. R., Zorotovic M., Gänsicke B. T., Ronco M. P., Hamers A. S., 2021, MNRAS, 501, 676
  * Manser et al. (2020) Manser C. J., Gänsicke B. T., Gentile Fusillo N. P., Ashley R., Breedt E., Hollands M., Izquierdo P., Pelisoli I., 2020, MNRAS, 493, 2127
  * Manser et al. (2023) Manser C. J., et al., 2023, MNRAS, 521, 4976
  * Melis et al. (2020) Melis C., Klein B., Doyle A. E., Weinberger A., Zuckerman B., Dufour P., 2020, ApJ, 905, 56
  * Mustill et al. (2014) Mustill A. J., Veras D., Villaver E., 2014, MNRAS, 437, 1404
  * Pelletier et al. (1986) Pelletier C., Fontaine G., Wesemael F., Michaud G., Wegner G., 1986, ApJ, 307, 242
  * Poleski et al. (2021) Poleski R., et al., 2021, Acta Astron., 71, 1
  * Rocchetto et al. (2015) Rocchetto M., Farihi J., Gänsicke B. T., Bergfors C., 2015, MNRAS, 449, 574
  * Schreiber et al. (2019) Schreiber M. R., Gänsicke B. T., Toloza O., Hernandez M.-S., Lagos F., 2019, ApJ, 887, L4
  * Trierweiler et al. (2023) Trierweiler I. L., Doyle A. E., Young E. D., 2023, arXiv e-prints, p. arXiv:2306.03743
  * Vanderbosch et al. (2020) Vanderbosch Z., et al., 2020, ApJ, 897, 171
  * Vanderbosch et al. (2021) Vanderbosch Z. P., et al., 2021, arXiv e-prints, p. arXiv:2106.02659
  * Vanderburg et al. (2015) Vanderburg A., et al., 2015, Nat, 526, 546
  * Vanderburg et al. (2020) Vanderburg A., et al., 2020, Nature, 585, 363
  * Veras & Gänsicke (2015) Veras D., Gänsicke B. T., 2015, MNRAS, 447, 1049
  * Veras & Hinkley (2021) Veras D., Hinkley S., 2021, MNRAS, 505, 1557
  * Villaver & Livio (2009) Villaver E., Livio M., 2009, ApJ Lett., 705, L81
  * Wilson et al. (2019) Wilson T. G., Farihi J., Gänsicke B. T., Swan A., 2019, MNRAS, p. 1000
  * Xu et al. (2017) Xu S., Zuckerman B., Dufour P., Young E. D., Klein B., Jura M., 2017, ApJ Lett., 836, L7
  * Zhu & Dong (2021) Zhu W., Dong S., 2021, ARA&A, 59, 291
  * Zhu et al. (2018) Zhu W., Petrovich C., Wu Y., Dong S., Xie J., 2018, ApJ, 860, 101
  * Zuckerman & Becklin (1987) Zuckerman B., Becklin E. E., 1987, Nat, 330, 138
  * Zuckerman et al. (2003) Zuckerman B., Koester D., Reid I. N., Hünsch M., 2003, ApJ, 596, 477
  * Zuckerman et al. (2007) Zuckerman B., Koester D., Melis C., Hansen B. M., Jura M., 2007, ApJ, 671, 872
  * Zuckerman et al. (2010) Zuckerman B., Melis C., Klein B., Koester D., Jura M., 2010, ApJ, 722, 725