Sauradip Nags.nag@surrey.ac.uk1,2 Xiatian Zhu<EMAIL_ADDRESS>Tao
Xiangt.xiang@surrey.ac.uk1,2 Centre for Vision Speech and Signal Processing
(CVSSP)
University of Surrey, UK
iFlyTek-Surrey Joint Research
Centre on Artificial Intelligence Few-Shot TAL with Query Adaptive Transformer

# Few-Shot Temporal Action Localization
with Query Adaptive Transformer

###### Abstract

Existing temporal action localization (TAL) works rely on a large number of
training videos with exhaustive segment-level annotation, preventing them from
scaling to new classes. As a solution to this problem, few-shot TAL (FS-TAL)
aims to adapt a model to a new class represented by as few as a single video.
Exiting FS-TAL methods assume trimmed training videos for new classes.
However, this setting is not only unnatural – actions are typically captured
in untrimmed videos, but also ignores background video segments containing
vital contextual cues for foreground action segmentation. In this work, we
first propose a new FS-TAL setting by proposing to use untrimmed training
videos. Further, a novel FS-TAL model is proposed which maximizes the
knowledge transfer from training classes whilst enabling the model to be
dynamically adapted to both the new class and each video of that class
simultaneously. This is achieved by introducing a query adaptive Transformer
in the model. Extensive experiments on two action localization benchmarks
demonstrate that our method can outperform all the state-of-the-art
alternatives significantly in both single-domain and cross-domain scenarios.
The source code can be found in https://github.com/sauradip/fewshotQAT

##  1 Introduction

Temporal action localization (TAL) aims to identify the temporal duration
(_i.e_ , the start and end points) and class label of action instances in
naturally untrimmed videos [Jiang et al.(2014)Jiang, Liu, Zamir, Toderici,
Laptev, Shah, and Sukthankar, Caba Heilbron et al.(2015)Caba Heilbron,
Escorcia, Ghanem, and Carlos Niebles]. Existing TAL methods [Lin et
al.(2018)Lin, Zhao, Su, Wang, and Yang, Xu et al.(2020)Xu, Zhao, Rojas,
Thabet, and Ghanem, Buch et al.(2017)Buch, Escorcia, Shen, Ghanem, and Carlos
Niebles, Wang et al.(2017)Wang, Xiong, Lin, and Van Gool, Zhao et
al.(2017)Zhao, Xiong, Wang, Wu, Tang, and Lin] use training datasets composed
of a large number of videos (e.g., hundreds) per class with exhaustive
segment-level annotation. The annotation is tedious and costly. Further, for
some rare classes collecting sufficient video instances may not even be
feasible. These have severely limited the scalability and general usability of
existing TAL methods. Inspired by the success of few-shot image classification
[Sung et al.(2018)Sung, Yang, Zhang, Xiang, Torr, and Hospedales, Snell et
al.(2017)Snell, Swersky, and Zemel, Finn et al.(2017)Finn, Abbeel, and Levine,
Chen et al.(2019)Chen, Liu, Kira, Wang, and Huang, Rusu et al.(2018)Rusu, Rao,
Sygnowski, Vinyals, Pascanu, Osindero, and Hadsell], few-shot learning (FSL)
has been recently introduced to TAL [Yang et al.(2018)Yang, He, and Porikli,
Yang et al.(2020)Yang, Hu, Mettes, and Snoek, Zhang et al.(2020a)Zhang, Dai,
and Wang]. A few-shot learning model is designed to eliminate the annotation
of large training data. This is achieved by meta-learning which enables a
model to adapt to any new class with as few as a single video. One of the key
challenges in FS-TAL is how to capture the intra-class variation using only a
handful (_e.g_ , 1-5) training instances of a new class. One of the key
objectives of meta-learning is to thus transfer such intra-class variation
information from a large set of seen training classes to the new class to
compensate for lack of training data.

Nonetheless, existing few-shot TAL (FS-TAL) methods [Yang et al.(2018)Yang,
He, and Porikli, Yang et al.(2020)Yang, Hu, Mettes, and Snoek, Zhang et
al.(2020a)Zhang, Dai, and Wang] all adopt a setting under which trimmed videos
are used to represent the new classes for model adaptation. This setting seems
to problematic: (1) As mentioned earlier, the TAL problem exists because most
action instances are first captured in untrimmed videos sandwiched by
background segments. An analogy is that objects always co-exist with
background (_e.g_ , tree/road/wall) in an image. So to obtain the trimmed new
class video, one needs to first manually annotate (trim) the untrimmed videos.
This begs the question: why not use the untrimmed video together with the
annotation for model adaptation? (2) Each new action class occurs in its own
specific context (background), which carries important cues on how to segment
it. Using trimmed videos means that a FS-TAL model is unable to exploit the
contextual information for both knowledge transfer from seen classes and new
unseen class adaptation.

In this work, we first introduce a new and more practical few-shot TAL (FS-
TAL) problem setting. During both the training (meta-learning) and inference
(model adaptation) stages, each class is represented by a support set
comprising untrimmed videos with temporal annotation. A segmentation model is
then built using the support set and applied to a query set containing
untrimmed videos of the same class to locate the foreground action instances.
This change of setting means that instead of meta-learning a model to
temporally align the support set instances with the foreground segments of the
query video as in [Yang et al.(2018)Yang, He, and Porikli, Yang et
al.(2020)Yang, Hu, Mettes, and Snoek, Zhang et al.(2020a)Zhang, Dai, and
Wang], we aim to meta-learn a foreground/background classifier that can be
quickly adapted to new classes. To this end, we propose a novel FS-TAL model
which meta-learns a query adaptive Transformer (QAT) for fast adaption of
foreground/background classifier to a new class. In particular, this leverages
the attention mechanism across the query video and few-shot classifier in
order to better capture the intra-class invariance. As shown in Figure 1, our
model has two key components, a snippet classifier that labels each video
snippet into foreground or background, and a query adaptation module designed
for query video adaptation. The former is a simple binary classifier
constructed using the annotated untrimmed support set videos. The latter is
formulated as a Transformer that takes both the classifier weight vector and
query video features as input and outputs an updated classifier adapted to
each query video. This QAT module is meta-learned and fixed during inference;
therefore the whole model is inductive. Importantly, our model is flexible in
that it can work in both the new setting proposed in this paper and the
existing setting with trimmed support set. We make the following
contributions: (1) We introduce a new and more practical FS-TAL problem
setting. (2) We propose a novel FS-TAL model with a query adaptive Transformer
for model adaptation to both a given new class and each query video. (3)
Extensive experiments show that the proposed method yields new state-of-the-
art performance on two TAL datasets (ActivityNet-v1.3 and Thumos’14). Under a
more challenging and more realistic cross-domain setting, the advantage of our
method remains.

##  2 Related Works

Temporal Action Localization An intuitive approach to temporal action
localization (TAL) is based on sliding window – first generating multi-scale
segments and then classifying them [Dai et al.(2017)Dai, Singh, Zhang, Davis,
and Qiu Chen]. A key limitation with this pipeline is that a large number
(thousands) of possible segments are necessary for achieving reasonable
accuracy, which is computationally expensive. To overcome this issue,
foreground/background modeling is introduced to generate action proposals
[Shou et al.(2016)Shou, Wang, and Chang, Shou et al.(2016)Shou, Wang, and
Chang, Gao et al.(2017)Gao, Yang, Chen, Sun, and Nevatia, Zhao et
al.(2017)Zhao, Xiong, Wang, Wu, Tang, and Lin, Lin et al.(2018)Lin, Zhao, Su,
Wang, and Yang, Lin et al.(2019b)Lin, Liu, Li, Ding, and Wen]. When proposal
generation is poor, incorporating sliding windows could be helpful [Gao et
al.(2018)Gao, Chen, and Nevatia]. For improving local segment-level feature
representation, [Zeng et al.(2019)Zeng, Huang, Tan, Rong, Zhao, Huang, and
Gan, Xu et al.(2020)Xu, Zhao, Rojas, Thabet, and Ghanem] exploit graph
convolutional networks to capture long-range contextual information.
Nonetheless, assuming a pre-collected dataset of all action class during
training, all these methods have poor scalability to large number of classes,
due to the high annotation cost.

Few-shot Learning For fast adaptation of a model to any given new class with
few training samples, few-shot learning (FSL) provides a solution [Vinyals et
al.(2016)Vinyals, Blundell, Lillicrap, Kavukcuoglu, and Wierstra, Sung et
al.(2018)Sung, Yang, Zhang, Xiang, Torr, and Hospedales, Snell et
al.(2017)Snell, Swersky, and Zemel]. It is often realized by meta-learning
which simulates the behaviour of new tasks with novel classes represented by
only a handful of labeled samples. This eliminates the requirement of labeling
a large dataset for a new class. Representative approaches include
hallucination (data augmentation) [Hariharan and Girshick(2017), Wang et
al.(2018)Wang, Girshick, Hebert, and Hariharan], initialization optimization
[Finn et al.(2017)Finn, Abbeel, and Levine, Rusu et al.(2018)Rusu, Rao,
Sygnowski, Vinyals, Pascanu, Osindero, and Hadsell, Ravi and
Larochelle(2016)], metric learning [Koch et al.(2015)Koch, Zemel, and
Salakhutdinov, Sung et al.(2018)Sung, Yang, Zhang, Xiang, Torr, and
Hospedales]. Beyond image classification, FSL has also been introduced to
object detection [Kang et al.(2019)Kang, Liu, Wang, Yu, Feng, and Darrell,
Dong et al.(2018)Dong, Zheng, Ma, Yang, and Meng, Hu et al.(2019)Hu, Mettes,
Huang, and Snoek] and semantic segmentation [Shaban et al.(2017)Shaban,
Bansal, Liu, Essa, and Boots, Zhang et al.(2020b)Zhang, Wei, Yang, and Huang,
Zhang et al.(2019)Zhang, Lin, Liu, Yao, and Shen, Wang et al.(2020)Wang,
Zhang, Hu, Yang, Cao, and Zhen]. In contrast to these image analysis problems,
here we focus on the more challenging TAL problem. Note that the model in [Hu
et al.(2019)Hu, Mettes, Huang, and Snoek], though developed for object
detection in images, can also work in the FS-TAL setting with trimmed support
set. More specifically, unlike our query adaptive Transformer for classifier
adaptation at the sample level, it leverages self-attention to contrast the
regional features exhaustively across the query and support samples. We will
compare with [Hu et al.(2019)Hu, Mettes, Huang, and Snoek] in our experiments
(Table 1).

Few-shot Temporal Action Localization FSL has been introduced to temporal
action location recently [Yang et al.(2018)Yang, He, and Porikli, Zhang et
al.(2020a)Zhang, Dai, and Wang, Yang et al.(2020)Yang, Hu, Mettes, and Snoek].
Yang et al. [Yang et al.(2018)Yang, He, and Porikli] propose the first FS-TAL
setting with trimmed support set. It incorporates the sliding window idea in
the matching network [Vinyals et al.(2016)Vinyals, Blundell, Lillicrap,
Kavukcuoglu, and Wierstra] to localize action instances in untrimmed query
videos. Later on, Zhang et al. [Zhang et al.(2020a)Zhang, Dai, and Wang]
consider weak video-level annotation of untrimmed training videos. Similar to
our proposed setting, the latest work [Yang et al.(2018)Yang, He, and Porikli]
also focuses on a singe new class at one time. However, a common limitation
with these existing FS-TAL problem settings stems from the assumption of
trimmed support set. As explained earlier, trimmed videos do not exist
naturally and need to be obtained with the same amount of manual annotation as
our setting. Importantly, the ignorance of background content in the original
untrimmed video leads to the failure to exploit useful context information. We
will compare the two FS-TAL settings in our experiments (Table 1).

##  3 Proposed Methodology

Problem Formulation Given only a few videos from any unseen action class, we
aim to learn a TAL model for that class. For FS-TAL, we assume a base category
set $C_{base}$ for training, and a novel category set $C_{novel}$ for testing.
For testing cross-class generalization, we ensure that the two class sets are
disjoint: $C_{base}\bigcap C_{novel}=\varnothing$. Accordingly, the base and
novel datasets are denoted as
$D_{base}=\left\\{\left(V_{i},Y_{i}\right),Y_{i}\in C_{base}\right\\}$ and
$D_{novel}=\left\\{\left(V_{i},Y_{i}\right),Y_{i}\in C_{novel}\right\\}$
respectively. Under the proposed new setting, each training video $V_{i}$ is
associated with segment-level annotation
$Y_{i}=\left\\{(s_{t},e_{t},c),t\in\\{1,..,M\\},c\in\textit{C}\right\\}$
including $M$ segment labels each with the start and end time locations and
action class. In evaluation, for each task, we randomly sample a class
$\textit{L}\sim C_{novel}$ from which K and one labeled videos are randomly
sampled to construct the support set S and the query set Q respectively. The
labels of $S$ are accessible for model few-shot learning whilst that of $Q$
are used for performance evaluation.

###  3.1 Model Architecture

Our FS-TAL model is illustrated in Figure 1. It consists of a task-generic
video embedding module (Sec. 3.2), and a task-specific snippet classification
module (Sec. 3.3). We aim to achieve optimal video embedding and
classification for any new task with only a few (1 or 5) labeled support
videos. To that end, we share video embedding component across all tasks, and
exploit the classification component for tackling the task specificity. With
the output of task adapted classification on every snippet of a test video, we
apply a non-parametric localization process to obtain the segment predictions
(Sec. 3.4).

Figure 1: Overview of the proposed FS-TAL deep learning architecture. There
are two main modules: (1) Video embedding for feature representation: It is
pre-trained on the whole training set, and shared by all different tasks for
more effective knowledge transfer from training classes to test classes. (2)
Snippet classification for foreground prediction: It is learned specifically
for every individual task in two steps. Initialized with the average of
foreground snippet features, the first step learns the classifier on the
support videos in a supervised manner. The second step further adapts the
classifier weights to every query video with a query adaptive Transformer. The
Transformer is meta-trained. The final localization result is obtained by
thresholding snippet-level classification scores and temporal non-maximum
suppression.

###  3.2 Task-Generic Video Embedding

To capture action location information of a video, we construct a video
embedding module with two components including feature backbone and snippet
embedding.

Feature backbone  In general, any video action models can be used such as C3D
[Tran et al.(2015)Tran, Bourdev, Fergus, Torresani, and Paluri], I3D [Carreira
and Zisserman(2017)] and TSM [Lin et al.(2019a)Lin, Gan, and Han]. For fair
comparison with [Yang et al.(2020)Yang, Hu, Mettes, and Snoek], we adopt the
same backbone C3D as our default choice. It is characterized by conducting 3D
convolution and pooling operations in 2D spatial and 1D temporal dimensions
simultaneously, capturing both appearance and motion information. Given an
input video $V$, we extract RGB $X_{r}\in\mathbb{R}^{T\times d_{1}}$ and
optical flow $X_{o}\in\mathbb{R}^{T\times d_{1}}$ features at the snippet
level, where $T$ denotes the number of snippets and $d_{1}$ denotes the
feature dimension. Each snippet is a short sequence of (_e.g_ , 16 in our
case) consecutive frames. We denote the concatenated features as
$X=[X_{r};X_{o}]\in\mathbb{R}^{T\times 2d_{1}}$. As in most TAL methods [Xu et
al.(2020)Xu, Zhao, Rojas, Thabet, and Ghanem, Xu et al.(2017)Xu, Das, and
Saenko, Lin et al.(2019b)Lin, Liu, Li, Ding, and Wen], the feature backbone is
pre-trained on a large video classification dataset (_e.g_ , Kinetics [Kay et
al.(2017)Kay, Carreira, Simonyan, Zhang, Hillier, Vijayanarasimhan, Viola,
Green, Back, Natsev, et al.]) and then frozen to serve as a feature extractor.

Snippet embedding  Whilst C3D features have already encoded local motion
information due to using 3D convolution and optical flow, long-term structural
information is lacking but critical for action localization. To address this
issue, we adopt an off-the-shelf temporal proposal model called GTAD [Xu et
al.(2020)Xu, Zhao, Rojas, Thabet, and Ghanem]. Other proposal models can be
similarly integrated [Lin et al.(2018)Lin, Zhao, Su, Wang, and Yang, Lin et
al.(2019b)Lin, Liu, Li, Ding, and Wen, Xu et al.(2020)Xu, Zhao, Rojas, Thabet,
and Ghanem]. In particular, GTAD exploits a temporal graph and a semantic
graph for modeling long-term temporal and contextual information concurrently.
In our context, we utilize GTAD as a means for refining the C3D snippet
features in a way that they become more sensitive to foreground (action
content) and background. We use the output of some intermediate layer of GTAD
as the snippet embedding. The layer selection will be evaluated in our
experiments (Sec. 4).

Formally, taking C3D features $X\in\mathbb{R}^{T\times 2d_{1}}$ of a video as
input, GTAD can output the snippet embedding as
${X}_{se}\in\mathbb{R}^{T\times C}$ where $C$ is the embedding dimension.
Support and query videos share the same GTAD model. We denote $X^{s}_{se}$ and
$X^{q}_{se}$ as the embedding of the support and query videos. Consider that
snippet embedding would be largely shareable among different tasks, we train
the GTAD model on the base dataset in a standard supervised learning way. The
objective function includes a classification loss and a localization loss with
respect to the ground-truth foreground mask [Xu et al.(2020)Xu, Zhao, Rojas,
Thabet, and Ghanem]. The trained GTAD and C3D form the video embedding module
which provides generic video representations for the subsequent few-shot
learning stage.

###  3.3 Task-Specific Snippet Classification

In our architecture, few-shot learning is focused on the snippet
classification component for capturing each task’s specificity. We aim to
build a binary classifier $h_{\phi}$ (with $\phi$ the parameters) that can
distinguish foreground action from background content in a video. Formally,
the classifier model for predicting the foreground likelihood is a simple
linear classifier as

$\textit{p}(t)=h_{\phi}(X_{se}(t))=\sigma(\tau\cdot[cos(X_{se}(t),\phi)]),$
(1)

where $\sigma$ specifies the sigmoid function, $\tau$ is a temperature hyper-
parameter, and $cos$ is the cosine similarity. The snippet is indexed by
$t\in\\{1,\cdots,T\\}$.

To make a classifier discriminative for each specific task, we introduce a
two-step learning-and-adapting strategy. In the first step, we learn the
classifier weights on the support set in a supervised way. In the second step,
we further adapt the support-set trained classifier weight to every query
video with a query adaptive Transformer model. This aims to solve the intra-
class variation problem.

New class adaptation As the support set is composed of untrimmed videos with
segment-level annotation, we can adapt the classifier to a new class with
standard supervised learning. Given the ground-truth annotation, we label each
snippet with foreground or background. To train the classifier, we use the
cross entropy loss as the objective function:

$\displaystyle{L}_{ce}=-\frac{1}{2K}\sum_{k=1}^{K}[{L}_{fg}\left({X}^{s}_{k}\right)+{L}_{bg}\left({X}^{s}_{k}\right)],$
(2)
$\displaystyle{L}_{fg}\left({X}_{k}^{s}\right)=\frac{l_{fg}+l_{bg}}{\epsilon+l_{fg}}\sum_{t\in\\{1,\cdots,T\\}}\hat{y}^{s}_{k}(t)\log[p_{k}^{s}(t)],$
(3)
$\displaystyle{L}_{bg}\left({X}_{k}^{s}\right)=\frac{l_{fg}+l_{bg}}{\epsilon+l_{bg}}\sum_{t\in\\{1,\cdots,T\\}}(1-\hat{y}^{s}_{k}(t))\log[1-p_{k}^{s}(t)],$
(4)

where $p_{k}^{s}(t)$ is the prediction of the $t$-th snippet $X_{k}^{s}(t)$
from the $k$-th support video. $\epsilon$ is used to tackle extreme cases such
as zero background/foreground. To balance the effect of foreground and
background snippets in training, we introduce a balancing policy based on
their sizes ($l_{fg}$ and $l_{bg}$). The idea is intuitive – less is more
important.

The classifier can be trained by a small number of (_e.g_ , 50$\sim$100)
iterations. We denote $\phi^{*}$ as the support-set trained classifier’s
weights. Given only a handful of labeled samples, how to initialize the
classifier weights becomes more critical. Instead of random initialization, we
found that the mean of foreground snippet’s embedding serves as a stronger
choice.

#### Query video adaptation

Under the few-shot setting, a key challenge to overcome is the insufficient
training samples in the support set for capturing the intra-class invariance
of the new class. As a result, training the classifier only on the support
videos often fails to capture the discriminative informative generalizable to
individual query videos. To address this limitation, we propose a query
adaptive Transformer model (with the parameters $\psi$) which is based on
self-attention [Vaswani et al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit,
Jones, Gomez, Kaiser, and Polosukhin].

Taking an input in a triplet of $\\{(query,key,value)\\}$, our Transformer
outputs an undated query with attentive association with respect to the value.
As our objective is to associate the classifier weights $\phi^{*}$ with the
query video $X_{se}^{q}$, we set
${(query,key,value)}=(\phi^{*},X_{se}^{q},X_{se}^{q})$. The attentive learning
is then formulated as

$A_{i}(\phi^{*})=\phi^{*}+softmax(\frac{\phi^{*}W_{Q}({X}_{se}^{q}W_{K})^{T}}{\sqrt{d}})({X}_{se}^{q}W_{V}),$
(5)

where $W_{Q}/W_{K}/W_{V}$ are learnable parameters (each is realized by a
fully-connected layer) that projects the respective input to a $d$-dimension
latent space. In a multi-head attention (MA) design, we combine a set of
independent $A_{i}$ to form a richer learning process:

$\phi^{**}=\underset{MA}{\underbrace{[A_{1}(\phi^{*})..A_{m}(\phi^{*})]}}+MLP(\phi^{*})\
\in\ \mathbb{R}^{L\times 256}.$ (6)

The MLP block has one fully-connected layer with residual skip connection.
Layer norm is applied before both the MA and MLP block.

Learning objective  After the classifier has been learned on both support and
query videos, it can be applied to the query video. We classify each snippet
with Eq. (1) with the foreground probability as:

$p{{}^{\prime}}(t):=h_{\phi^{**}}(X_{se}^{q}(t)).$ (7)

For training our Transformer ($\psi$), this prediction is then used to compute
a cross-entropy loss (Eq. (2)) as objective. In meta-training, we conduct loss
gradient back-propagation only once for each episode. We denote $\psi^{*}$ as
the optimized Transformer’s parameters.

###  3.4 Model Inference

During testing, each time we are given a new task with one random unseen
action class sampled from the novel dataset $D_{novel}$. With the frozen video
embedding module, we need to obtain a task-specific snippet classifier in two
steps: supervised learning with $K$ shots of support videos (Eq. (2)) and
classifier weight adaptation on a query video by applying the meta-trained
Transformer (note that our Transformer itself is frozen here). The classifier
is then applied to predict the foreground probability of every snippet of a
query/test video.

Action instance generation  After we obtain the snippet-level classification
results, we threshold on their foreground probabilities and take those
consecutive snippets as action instance candidates. To indicate the prediction
confidence of each candidate, we use the highest snippet foreground
probability. We then adjust the confidence scores using temporal soft Non-
Maximal Suppression (NMS) [Bodla et al.(2017)Bodla, Singh, Chellappa, and
Davis, Lin et al.(2019b)Lin, Liu, Li, Ding, and Wen]. Finally, we select top
$N$ candidates as the localization result.

##  4 Experiments

Datasets We evaluate on two large-scale temporal action localization datasets.
ActivityNet-v1.3 [Caba Heilbron et al.(2015)Caba Heilbron, Escorcia, Ghanem,
and Carlos Niebles] is a popular TAL benchmark. It contains 19,994 temporally
annotated untrimmed videos in 200 action categories. THUMOS’14 [Jiang et
al.(2014)Jiang, Liu, Zamir, Toderici, Laptev, Shah, and Sukthankar] is another
widely used benchmark for action recognition and localization. There are 413
untrimmed videos from 20 different categories. The 20 classes are a subset of
the 101 classes in UCF101 [Soomro et al.(2012)Soomro, Zamir, and Shah].

Few-shot learning setting  To facilitate performance comparison, we use the
same class split as introduced in [Yang et al.(2020)Yang, Hu, Mettes, and
Snoek]. For both datasets, we split the videos into single instance and multi-
instance according to the number of action instances per video. For the single
instance case, we divide the videos with multiple action instances into
independent single-instance videos. Every newly generated video is no longer
than 768 frames. For each of the two cases, we divide all the classes into
three non-overlapping subsets for training (80%), validation (10%) and testing
(10%), respectively. The validation set is used for model parameter tuning and
best model selection. We consider 1-shot and 5-shot. In our setting we adopt
untrimmed support set, as opposed to [Yang et al.(2020)Yang, Hu, Mettes, and
Snoek] using trimmed videos. For each test, we use 5000 random tasks and
report their average result.

Implementation Details For each untrimmed video, we extract its RGB frames at
16 FPS and at the resolution of 256 × 256. We averagely divide each video into
100 (256 for THUMOS) non-overlapping snippets and sample 8 frames for each
snippet (_i.e_ , $T=100$). As [Yang et al.(2020)Yang, Hu, Mettes, and Snoek],
we filter out videos having less than 768 frames. We consider single-instance
and multi-instance test videos, separately. The dimension of C3D feature is
500 (_i.e_ , $d_{1}=500$). We take the penultimate layer’s output (Layer-5) of
GTAD’s localization module as video embedding (256-D). The latent feature
dimension $d$ (Eq. (5)) of our query adaptive Transformer is 256. Dropout is
used in our Transformer to alleviate model overfitting. We set the NMS
threshold of 0.7/0.6 for ActivityNet/THUMOS. As the final TAL result, we take
top 100/200 for ActivityNet/THUMOS. We adopt the Adam optimizer [Sutskever et
al.(2013)Sutskever, Martens, Dahl, and Hinton] with learning rate 0.004. We
train the model for $50$ epochs each with $200$ episodes.

Single instance videos | Multi-instance videos
---|---
| ActivityNet-v1.3 | THUMOS’14 | ActivityNet-v1.3 | THUMOS’14
map@ | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | mean | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | mean | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | mean | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | mean
1 Shot | 1 shot
Hu et al. [Hu et al.(2019)Hu, Mettes, Huang, and Snoek] | 41.0 | 33.0 | 27.1 | 15.9 | 6.8 | 24.8 | - | - | - | - | - | - | 29.6 | 23.2 | 12.7 | 7.4 | 3.1 | 15.2 | - | - | - | - | - | -
Feng et al. [Feng et al.(2018)Feng, Ma, Liu, Zhang, and Luo] | 43.5 | 35.1 | 27.3 | 16.2 | 6.5 | 25.7 | - | - | - | - | - | - | 31.4 | 25.5 | 16.1 | 8.9 | 3.2 | 17.0 | - | - | - | - | - | -
Yang et al. [Yang et al.(2020)Yang, Hu, Mettes, and Snoek] | 53.1 | 40.9 | 29.8 | 18.2 | 8.4 | 29.5 | 48.7 | - | - | - | - | - | 42.1 | 36.0 | 18.5 | 11.1 | 7.0 | 22.9 | - | - | - | - | - | -
Ours | 55.1 | 45.2 | 35.5 | 25.3 | 13.2 | 32.5 | 49.2 | 36.9 | 24.3 | 16.5 | 10.1 | 27.2 | 44.1 | 37.8 | 29.5 | 21.4 | 11.5 | 25.8 | 7.3 | 4.2 | 3.1 | 2.0 | 1.5 | 3.7
Ours† | 55.6 | 44.6 | 35.7 | 24.6 | 12.7 | 31.8 | 51.2 | 38.1 | 22.7 | 14.8 | 9.2 | 27.0 | 44.9 | 38.0 | 29.2 | 21.4 | 11.2 | 25.9 | 9.1 | 6.8 | 4.9 | 3.5 | 2.3 | 5.3
5 Shot | 5 shot
Buch et al. [Buch et al.(2017)Buch, Escorcia, Shen, Ghanem, and Carlos Niebles] | 39.7 | 33.6 | 27.0 | 14.0 | 4.6 | 23.3 | 35.7 | 29.4 | 20.8 | 11.7 | 3.4 | 20.2 | 30.4 | 25.1 | 19.6 | 12.9 | 6.6 | 18.9 | 2.7 | 1.9 | 1.4 | 0.9 | 0.4 | 1.5
Hu et al. [Hu et al.(2019)Hu, Mettes, Huang, and Snoek] | 45.4 | 35.0 | 29.9 | 17.6 | 5.2 | 27.0 | 42.2 | 32.6 | 20.3 | 13.7 | 5.2 | 22.8 | 38.9 | 27.2 | 18.3 | 12.7 | 7.3 | 20.9 | 6.8 | 3.1 | 2.2 | 1.8 | 1.3 | 3.1
Yang et al. [Yang et al.(2020)Yang, Hu, Mettes, and Snoek] | 56.5 | 47.0 | 37.4 | 21.5 | 11.9 | 34.9 | 51.9 | 42.7 | 24.4 | 17.7 | 10.1 | 29.3 | 43.9 | 37.4 | 20.2 | 13.4 | 7.7 | 24.5 | 8.6 | 5.6 | 3.8 | 2.5 | 1.7 | 4.4
Ours | 63.0 | 54.5 | 44.2 | 30.9 | 15.8 | 38.4 | 54.3 | 43.6 | 35.8 | 24.5 | 12.2 | 31.6 | 48.2 | 39.1 | 29.7 | 22.5 | 12.8 | 28.2 | 10.4 | 7.1 | 5.7 | 4.8 | 2.9 | 5.4
Ours† | 63.8 | 54.2 | 43.9 | 31.4 | 16.4 | 38.5 | 56.1 | 47.2 | 32.4 | 24.3 | 13.7 | 32.7 | 51.8 | 42.7 | 32.6 | 23.4 | 11.9 | 30.2 | 13.8 | 11.3 | 8.4 | 6.3 | 4.2 | 7.1

Table 1: FS-TAL results (%). †: Using untrimmed support set (i.e., the new
setting).

###  4.1 Comparison with state-of-the-art

Competitors  For comparative evaluation, we consider a few-shot object
detection model [Hu et al.(2019)Hu, Mettes, Huang, and Snoek], a one-shot
video re-localization model [Feng et al.(2018)Feng, Ma, Liu, Zhang, and Luo],
and the latest FS-TAL model [Yang et al.(2020)Yang, Hu, Mettes, and Snoek].
Because [Feng et al.(2018)Feng, Ma, Liu, Zhang, and Luo] cannot tackle
multiple support videos, we compare with a modified version of temporal action
proposal model SST [Buch et al.(2017)Buch, Escorcia, Shen, Ghanem, and Carlos
Niebles] for 5-shot case. As in [Yang et al.(2020)Yang, Hu, Mettes, and
Snoek], a fusion layer is added on top of SST’s GRU layer to incorporate the
support video features, and the proposal with the largest confidence score is
taken as the prediction. All the methods use the same C3D video feature
backbone. For all the competitors, we use trimmed support set to keep their
original designs. We evaluate the proposed model under both the previous
setting (trimmed support set) and our new setting (untrimmed support set).
This allows for absolute fair model comparison as well as setting comparison.
When feeding trimmed support videos into our model, the background loss term
$L_{bg}$ in Eq. (2) will become zero; without any other formulation change,
our model can be applied to the old setting. The difference is that now the
Transformer is used to adapt a foreground template/prototype to each query
video, instead of a foreground/background classifier. Note that none of the
existing methods can be easily extended to operate under our new setting.

Results  The results are compared in Table 1. It is evident that our method
achieves the best performance in all test settings when using the same trimmed
support set. This suggests the superiority of our model over all alternative
designs, verifying the proposed few-shot learning architecture. The margin is
even larger in more strict metrics. Importantly, we see that the margin
further increases in 5-shot case, indicating the superior capability of our
method in leveraging multiple training videos. This is mainly due to the
proposed query adaptive Transformer that can amplify the benefit of larger
support-set via attentive query video adaptation, which is lacking in all
existing methods. In the multi-instance setting on THUMOS’14, all the methods
do not work well due to longer videos and short action instances. However, it
is still encouraging that our model can double or triple the performance of
alternatives at<EMAIL_ADDRESS>in such challenging test.

We further examine the two FS-TAL problem settings with the proposed method.
We make the following observations. In the single-instance setting, the model
performance is marginally better in previous setting with trimmed videos in
most cases. Our observation suggests that this is potentially due to lack of
background diversity. However, when it comes to the more practical and
challenging multi-instance setting, the opposite is true especially in the
5-shot case. This indicates that background helps model learning with useful
context cues co-existing with action instances. Given these observations, we
consider that the proposed setting is not only more practical but also
provides more information for better modeling, as compared to the previous
settings.

###  4.2 Effect of Query Video Adaptation

In Section 3.3 we introduce a query adaptive Transformer for fast adapting the
support-set trained classifier’s weights to each query video. This aims to
solve intra-class variation with FS-TAL as there is no sufficient training
samples in support set to capture this variation. There may exist big
appearance difference between the support and query video action instances
(see Figure 2 in Supplementary). Query video adaptation is thus critical. From
Table 2 we can see that without the proposed query video the performance will
drop significantly ($3\sim 8$%) in 1/5-shot settings of both datasets. This
verifies the importance of learning the intra-class invariance problem and the
ability of our Transfer model in adapting the classifier’s weight to each
query video, _i.e_ , video-specific adaptation. In Figure 2 we visually show
that our query adaptive transformer is effective in adapting the classifier’s
weight to capture the specificity of the query video’s foreground content.

Dataset | ActivityNet | Thumos
---|---|---
mAP | 0.5 | mean | 0.5 | mean
Without QVA
Ours @ 1-shot | 37.3 | 21.7 | 3.6 | 2.3
Ours @ 5-shot | 43.8 | 25.3 | 7.9 | 4.0
With QVA
Ours @ 1-shot | 44.9 ($\uparrow\textbf{7.6}$) | 25.9 ($\uparrow\textbf{4.2}$) | 9.1 ($\uparrow\textbf{5.5}$) | 5.3 ($\uparrow\textbf{3.0}$)
Ours @ 5-shot | 51.8 ($\uparrow\textbf{8.0}$) | 30.2 ($\uparrow\textbf{4.9}$) | 13.8 ($\uparrow\textbf{5.9}$) | 7.1 ($\uparrow\textbf{3.1}$)

Table 2: Effect of query video adaptation (QVA) in the multi-instance setting.
Figure 2: The effect of Query Video Adaptation with t-SNE visualization. It is
shown that with the proposed query video adaptation, the classifier weight can
be effectively pushed to be aligned with the foreground content of the query
video sample. This improves learning the intra-class invariance of the new
class.  Figure 3: Intra-class variation example in the “Doing Fencing” class
on ActivityNet-v1.3. As can be seen, the two videos present clear difference
in viewpoint, scene setup, background, illumination, as well as instance
length (c).

###  4.3 Cross-Domain Localization

Following the above single domain (dataset) FS-TAL evaluation, we further
introduce a more challenging and more realistic cross-domain setting. As
THUMOS’14 and ActivityNet-v1.3 present large differences in action instance
length and background characteristics, they are suitable for cross-domain
evaluation. We consider the single-instance setting. We compare our method
with the state-of-the-art model [Yang et al.(2020)Yang, Hu, Mettes, and
Snoek].

$\textbf{THUMOS}\rightarrow\textbf{ActivityNet}$ In the first cross-domain
experiment, we train a model on the base classes of THUMOS’14 (source domain)
and test the model on the novel classes of ActivityNet-v1.3 (target domain).
The results are reported in Table 3. It is shown that the performance
advantage of our method remains compared to the single-domain setting. For
example, the<EMAIL_ADDRESS>margin of our model over [Yang et al.(2020)Yang, Hu,
Mettes, and Snoek] is 4.8%/6.2% in the 1/5-shot cases. Comparing the single-
domain results in Table 1, we can see that domain shift indeed negatively
affects the performance of both models.

$\textbf{ActivityNet}\rightarrow\textbf{THUMOS}$ The second experiment
considers the opposite transfer direction. At large we have similar
observations with our model again outperforming [Yang et al.(2020)Yang, Hu,
Mettes, and Snoek] in both 1/5-shot setting. This suggests that our model can
generalize to different transfer setups with consistent performance
advantages.

Cross Domain | $\textbf{Thumos}\rightarrow\textbf{ActivityNet}$ | $\textbf{ActivityNet}\rightarrow\textbf{Thumos}$
---|---|---
mAP | 0.5 | mean | 0.5 | mean
Yang et al. [Yang et al.(2020)Yang, Hu, Mettes, and Snoek] @ 1-shot | 41.1 | 25.2 | 36.2 | 21.4
Ours @ 1-shot | 45.9 | 26.6 | 38.1 | 22.5
Yang et al. [Yang et al.(2020)Yang, Hu, Mettes, and Snoek] @ 5-shot | 48.2 | 27.8 | 37.5 | 23.6
Ours @ 5-shot | 54.4 | 31.6 | 43.8 | 27.2

Table 3: Cross-domain FS-TAL.

###  4.4 Effect of Video Embedding Module

We evaluate the generality of our FS-TAL architecture in different video
embedding designs. In this test we select BMN [Lin et al.(2019b)Lin, Liu, Li,
Ding, and Wen]. Table 4 shows that BMN is slightly inferior to GTAD for video
embedding, which is consistent with the previous finding [Xu et al.(2020)Xu,
Zhao, Rojas, Thabet, and Ghanem].

###  4.5 Inference Efficiency

In inference, our model runs a small number of iterations for learning the
linear classifier’s weights on the support set, which increases slightly the
computational overhead. We conduct a quantitative cost analysis in 5-shot
multi-instance setting on ActivityNet-v1.3. We compared to the state-of-the-
art model [Yang et al.(2020)Yang, Hu, Mettes, and Snoek]. For both methods, we
track the speed of 100 FS-TAL tasks on a machine with one RTX2080Ti GPU. Table
5 shows that our method has very similar inference speed as [Yang et
al.(2020)Yang, Hu, Mettes, and Snoek], without efficiency disadvantage.

Dataset | ActivityNet-v1.3
---|---
mAP | 0.5 | mean
BMN [Lin et al.(2019b)Lin, Liu, Li, Ding, and Wen] | 61.6 | 37.5
GTAD [Xu et al.(2020)Xu, Zhao, Rojas, Thabet, and Ghanem] | 63.8 | 38.5

Table 4: Effect of video embedding in the 5-shot multi-instance setting.

Dataset | ActivityNet-v1.3
---|---
Metrics | Speed (seconds / task)
Yang et al. [Yang et al.(2020)Yang, Hu, Mettes, and Snoek] | 0.81
Ours | 0.83

Table 5: Inference efficiency test in the 5-shot multi-instance setting with a
RTX2080 GPU.

###  4.6 Qualitative Analysis

For visual analysis, we provide two qualitative examples in Figure 4. To
visualize the intra-class variation challenge which our method in particular
the proposed query adaptive Transformer aims to address, we show some common
examples in Figure 3.

Figure 4: Qualitative results of (a) “BreakDancing“ class on ActivityNet and
(b) “Throw Discuss“ class on THUMOS.

###  4.7 Choice of Video Embedding Layer

We examine which layer of GTAD [Xu et al.(2020)Xu, Zhao, Rojas, Thabet, and
Ghanem] is a good choice for video snippet embedding. In particular, we test
five GTAD layers. The result curve in Figure 5 shows that deeper layers are
usually better than shallow ones, suggesting that snippet-level contextual
information is useful for action localization. We select the layer-5 as our
embedding layer as it has best cost-effectiveness.

Figure 5: Ablation of GTAD video embedding layer in the single-instance
setting on ActivityNet-v1.3. The number in round bracket is the embedding
dimension.

##  5 Conclusion

We have presented a new and more practical few-shot temporal action
localization (FS-TAL) problem. Unlike all existing settings, in our setting a
new action class is represented by untrimmed support set with useful
background segments to provide contextual information. We introduce a novel
FS-TAL architecture that effectively transfers class-generic representation
knowledge from training classes to any unseen test classes whilst adapting the
model to any new class. To solve the large intra-class variation problem, we
introduce a query adaptive Transformer that further dynamically adapts the
support-set trained classifier’s weights to each query video. Experiments on
two popular TAL datasets verify the superiority of our method over existing
alternatives in both the newly proposed setting with untrimmed labeled support
set and previous settings with trimmed counterpart. Moreover, our method
remains to be advantageous under a more realistic and challenging cross-domain
setting.

## References

  * [Bodla et al.(2017)Bodla, Singh, Chellappa, and Davis] Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S Davis.  Soft-nms–improving object detection with one line of code.  In _ICCV_ , 2017.
  * [Buch et al.(2017)Buch, Escorcia, Shen, Ghanem, and Carlos Niebles] Shyamal Buch, Victor Escorcia, Chuanqi Shen, Bernard Ghanem, and Juan Carlos Niebles.  Sst: Single-stream temporal action proposals.  In _CVPR_ , 2017.
  * [Caba Heilbron et al.(2015)Caba Heilbron, Escorcia, Ghanem, and Carlos Niebles] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles.  Activitynet: A large-scale video benchmark for human activity understanding.  In _CVPR_ , 2015.
  * [Carreira and Zisserman(2017)] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In _CVPR_ , 2017.
  * [Chen et al.(2019)Chen, Liu, Kira, Wang, and Huang] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang.  A closer look at few-shot classification.  _arXiv preprint arXiv:1904.04232_ , 2019.
  * [Dai et al.(2017)Dai, Singh, Zhang, Davis, and Qiu Chen] Xiyang Dai, Bharat Singh, Guyue Zhang, Larry S Davis, and Yan Qiu Chen.  Temporal context network for activity localization in videos.  In _ICCV_ , 2017.
  * [Dong et al.(2018)Dong, Zheng, Ma, Yang, and Meng] Xuanyi Dong, Liang Zheng, Fan Ma, Yi Yang, and Deyu Meng.  Few-example object detection with model communication.  _TPAMI_ , 41(7), 2018.
  * [Feng et al.(2018)Feng, Ma, Liu, Zhang, and Luo] Yang Feng, Lin Ma, Wei Liu, Tong Zhang, and Jiebo Luo.  Video re-localization.  In _ECCV_ , 2018.
  * [Finn et al.(2017)Finn, Abbeel, and Levine] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _ICML_. PMLR, 2017.
  * [Gao et al.(2017)Gao, Yang, Chen, Sun, and Nevatia] Jiyang Gao, Zhenheng Yang, Kan Chen, Chen Sun, and Ram Nevatia.  Turn tap: Temporal unit regression network for temporal action proposals.  In _ICCV_ , 2017.
  * [Gao et al.(2018)Gao, Chen, and Nevatia] Jiyang Gao, Kan Chen, and Ram Nevatia.  Ctap: Complementary temporal action proposal generation.  In _ECCV_ , 2018.
  * [Hariharan and Girshick(2017)] Bharath Hariharan and Ross Girshick.  Low-shot visual recognition by shrinking and hallucinating features.  In _ICCV_ , 2017.
  * [Hu et al.(2019)Hu, Mettes, Huang, and Snoek] Tao Hu, Pascal Mettes, Jia-Hong Huang, and Cees GM Snoek.  Silco: Show a few images, localize the common object.  In _CVPR_ , 2019.
  * [Jiang et al.(2014)Jiang, Liu, Zamir, Toderici, Laptev, Shah, and Sukthankar] Yu-Gang Jiang, Jingen Liu, A Roshan Zamir, George Toderici, Ivan Laptev, Mubarak Shah, and Rahul Sukthankar.  Thumos challenge: Action recognition with a large number of classes, 2014\.
  * [Kang et al.(2019)Kang, Liu, Wang, Yu, Feng, and Darrell] Bingyi Kang, Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and Trevor Darrell.  Few-shot object detection via feature reweighting.  In _ICCV_ , 2019.
  * [Kay et al.(2017)Kay, Carreira, Simonyan, Zhang, Hillier, Vijayanarasimhan, Viola, Green, Back, Natsev, et al.] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al.  The kinetics human action video dataset.  _arXiv preprint arXiv:1705.06950_ , 2017.
  * [Koch et al.(2015)Koch, Zemel, and Salakhutdinov] Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov.  Siamese neural networks for one-shot image recognition.  In _ICML deep learning workshop_ , volume 2. Lille, 2015.
  * [Lin et al.(2019a)Lin, Gan, and Han] Ji Lin, Chuang Gan, and Song Han.  Tsm: Temporal shift module for efficient video understanding.  In _CVPR_ , 2019a.
  * [Lin et al.(2018)Lin, Zhao, Su, Wang, and Yang] Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang.  Bsn: Boundary sensitive network for temporal action proposal generation.  In _ECCV_ , 2018.
  * [Lin et al.(2019b)Lin, Liu, Li, Ding, and Wen] Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen.  Bmn: Boundary-matching network for temporal action proposal generation.  In _CVPR_ , 2019b.
  * [Ravi and Larochelle(2016)] Sachin Ravi and Hugo Larochelle.  Optimization as a model for few-shot learning.  2016\.
  * [Rusu et al.(2018)Rusu, Rao, Sygnowski, Vinyals, Pascanu, Osindero, and Hadsell] Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell.  Meta-learning with latent embedding optimization.  _arXiv preprint arXiv:1807.05960_ , 2018.
  * [Shaban et al.(2017)Shaban, Bansal, Liu, Essa, and Boots] Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots.  One-shot learning for semantic segmentation.  In _BMVC_ , 2017.
  * [Shou et al.(2016)Shou, Wang, and Chang] Zheng Shou, Dongang Wang, and Shih-Fu Chang.  Temporal action localization in untrimmed videos via multi-stage cnns.  In _CVPR_ , 2016.
  * [Snell et al.(2017)Snell, Swersky, and Zemel] Jake Snell, Kevin Swersky, and Richard S Zemel.  Prototypical networks for few-shot learning.  _arXiv preprint arXiv:1703.05175_ , 2017.
  * [Soomro et al.(2012)Soomro, Zamir, and Shah] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  _arXiv preprint arXiv:1212.0402_ , 2012.
  * [Sung et al.(2018)Sung, Yang, Zhang, Xiang, Torr, and Hospedales] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales.  Learning to compare: Relation network for few-shot learning.  In _CVPR_ , 2018.
  * [Sutskever et al.(2013)Sutskever, Martens, Dahl, and Hinton] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton.  On the importance of initialization and momentum in deep learning.  In _ICML_. PMLR, 2013.
  * [Tran et al.(2015)Tran, Bourdev, Fergus, Torresani, and Paluri] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri.  Learning spatiotemporal features with 3d convolutional networks.  In _ICCV_ , 2015.
  * [Vaswani et al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _arXiv preprint arXiv:1706.03762_ , 2017.
  * [Vinyals et al.(2016)Vinyals, Blundell, Lillicrap, Kavukcuoglu, and Wierstra] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra.  Matching networks for one shot learning.  _arXiv preprint arXiv:1606.04080_ , 2016.
  * [Wang et al.(2020)Wang, Zhang, Hu, Yang, Cao, and Zhen] Haochen Wang, Xudong Zhang, Yutao Hu, Yandan Yang, Xianbin Cao, and Xiantong Zhen.  Few-shot semantic segmentation with democratic attention networks.  In _ECCV_ , 2020.
  * [Wang et al.(2017)Wang, Xiong, Lin, and Van Gool] Limin Wang, Yuanjun Xiong, Dahua Lin, and Luc Van Gool.  Untrimmednets for weakly supervised action recognition and detection.  In _CVPR_ , 2017.
  * [Wang et al.(2018)Wang, Girshick, Hebert, and Hariharan] Yu-Xiong Wang, Ross Girshick, Martial Hebert, and Bharath Hariharan.  Low-shot learning from imaginary data.  In _CVPR_ , 2018.
  * [Xu et al.(2017)Xu, Das, and Saenko] Huijuan Xu, Abir Das, and Kate Saenko.  R-c3d: Region convolutional 3d network for temporal activity detection.  In _ICCV_ , 2017.
  * [Xu et al.(2020)Xu, Zhao, Rojas, Thabet, and Ghanem] Mengmeng Xu, Chen Zhao, David S. Rojas, Ali Thabet, and Bernard Ghanem.  G-tad: Sub-graph localization for temporal action detection.  In _CVPR_ , June 2020.
  * [Yang et al.(2018)Yang, He, and Porikli] Hongtao Yang, Xuming He, and Fatih Porikli.  One-shot action localization by learning sequence matching network.  In _CVPR_ , 2018.
  * [Yang et al.(2020)Yang, Hu, Mettes, and Snoek] Pengwan Yang, Vincent Tao Hu, Pascal Mettes, and Cees GM Snoek.  Localizing the common action among a few videos.  In _ECCV_. Springer, 2020.
  * [Zeng et al.(2019)Zeng, Huang, Tan, Rong, Zhao, Huang, and Gan] Runhao Zeng, Wenbing Huang, Mingkui Tan, Yu Rong, Peilin Zhao, Junzhou Huang, and Chuang Gan.  Graph convolutional networks for temporal action localization.  In _ICCV_ , 2019.
  * [Zhang et al.(2019)Zhang, Lin, Liu, Yao, and Shen] Chi Zhang, Guosheng Lin, Fayao Liu, Rui Yao, and Chunhua Shen.  Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning.  In _CVPR_ , 2019.
  * [Zhang et al.(2020a)Zhang, Dai, and Wang] Da Zhang, Xiyang Dai, and Yuan-Fang Wang.  Metal: Minimum effort temporal activity localization in untrimmed videos.  In _CVPR_ , 2020a.
  * [Zhang et al.(2020b)Zhang, Wei, Yang, and Huang] Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas S Huang.  Sg-one: Similarity guidance network for one-shot semantic segmentation.  _IEEE Transactions on Cybernetics_ , 2020b.
  * [Zhao et al.(2017)Zhao, Xiong, Wang, Wu, Tang, and Lin] Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin.  Temporal action detection with structured segment networks.  In _ICCV_ , 2017.


# Viability of novae as sources of Galactic lithium

Alex J. Kemp School of Physics & Astronomy, Monash University, Clayton 3800,
Victoria, Australia  Centre of Excellence for Astrophysics in Three Dimensions
(ASTRO-3D), Melbourne, Victoria, Australia  Amanda I. Karakas  School of
Physics & Astronomy, Monash University, Clayton 3800, Victoria, Australia
Centre of Excellence for Astrophysics in Three Dimensions (ASTRO-3D),
Melbourne, Victoria, Australia  Andrew R. Casey  School of Physics &
Astronomy, Monash University, Clayton 3800, Victoria, Australia  Centre of
Excellence for Astrophysics in Three Dimensions (ASTRO-3D), Melbourne,
Victoria, Australia  Benoit Côté  Department of Physics and Astronomy,
University of Victoria, Victoria, BC V8P 5C2, Canada  Konkoly Observatory,
Research Centre for Astronomy and Earth Sciences, Eötvös Loránd Research
Network (ELKH), Konkoly Thege Miklós út 15-17, H-1121 Budapest, Hungary
Robert G. Izzard  Astrophysics Research Group, University of Surrey,
Guildford, Surrey GU2 7XH, UK  Zara Osborn  School of Physics & Astronomy,
Monash University, Clayton 3800, Victoria, Australia  Centre of Excellence for
Astrophysics in Three Dimensions (ASTRO-3D), Melbourne, Victoria, Australia

(Received January 1, 2018; Revised January 7, 2018)

###### Abstract

Of all the light elements, the evolution of lithium (Li) in the Milky Way is
perhaps the most difficult to explain. Li is difficult to synthesize and is
easily destroyed, making most stellar sites unsuitable for producing Li in
sufficient quantities to account for the proto-solar abundance. For decades,
novae have been proposed as a potential explanation to this ‘Galactic Li
problem’, and the recent detection of ${}^{7}{\rm B}$e in the ejecta of
multiple nova eruptions has breathed new life into this theory. In this work,
we assess the viability of novae as dominant producers of Li in the Milky Way.
We present the most comprehensive treatment of novae in a galactic chemical
evolution code to date, testing theoretical- and observationally-derived nova
Li yields by integrating metallicity-dependent nova ejecta profiles computed
using the binary population synthesis code binary_c with the galactic chemical
evolution code OMEGA+. We find that our galactic chemical evolution models
which use observationally-derived Li yields account for the proto-solar Li
abundance very well, while models relying on theoretical nova yields cannot
reproduce the proto-solar observation. A brief exploration of physical
uncertainties including single-stellar yields, the metallicity resolution of
our nova treatment, common-envelope physics, and nova accretion efficiencies
indicates that this result is robust to physical assumptions. Scatter within
the observationally-derived Li yields in novae is identified as the primary
source of uncertainty, motivating further observations of ${}^{7}{\rm B}$e in
nova ejecta.

editorials, notices — miscellaneous — catalogs — surveys

††journal: ApJ

##  1 Introduction

Lithium (Li) is a notoriously troublesome element. It is extremely fragile,
destroyed in H-capture reactions at temperatures as low as $2\times 10^{6}$ K.
This is cool enough for stars to deplete their surface Li on, or before, the
main sequence by convecting surface material deeper into the star where the Li
is destroyed. This makes it almost impossible to accurately calculate the Li
abundance at birth for most stars. The exception to this is the Sun, for which
we have access to meteorites which preserve the proto-solar Li abundance
(A(Li)=$3.26\pm 0.05$, see Lodders et al.,, 2009; Asplund et al.,, 2009).
Galactic chemical evolution (GCE) models typically under-produce this Li
abundance by roughly an order of magnitude, a discrepancy known as the
‘Galactic Li problem’ (D’Antona and Matteucci,, 1991; Matteucci et al.,,
1995).

The fragility of Li is detrimental to its production. Its dominant isotope
${}^{7}{\rm L}$i is formed as the sole decay product of ${}^{7}{\rm B}$e,
which is unstable with a half-life of 53.3 days. ${}^{7}{\rm B}$e forms via
the proton-proton (pp) chains during H-burning, but quickly decays to
${}^{7}{\rm L}$i via electron capture (pp II). The ${}^{7}{\rm L}$i itself is
then also destroyed by proton capture, with the net result that very little
survives H-burning. The Cameron-Fowler mechanism (Cameron and Fowler,, 1971)
proposes that some of the ${}^{7}{\rm B}$e is transported to a cooler region,
where electron captures and the natural decay of ${}^{7}{\rm B}$e can form Li
in an environment where it can survive.

Although the yield is model dependent, relatively little Li can be produced in
this way by AGB stars (e.g., Karakas,, 2010; Pignatari et al.,, 2016), and
core-collapse supernova models produce orders of magnitude less (e.g.,
Kobayashi et al.,, 2006). Li can also be produced through spallation reactions
caused by cosmic rays at levels comparable to AGB stars (Prantzos,, 2012).

The list of viable Li sources at this point is short but not exhausted. It has
been proposed for many years now that classical novae could be viable sites
for Li production (Arnould and Norgaard,, 1975). Novae are transients caused
by explosive H-burning episodes on the surface of accreting white dwarfs
(WDs). Their high burning temperatures, rapid evolution, and high mass loss
rates during outburst make them promising Li producers due to their ability to
synthesize ${}^{7}{\rm B}$e in significant quantities and then transport that
material into cooler mass-losing regions during outburst, where it can decay
to ${}^{7}{\rm L}$i without that Li being destroyed. Early theoretical work on
explosive H burning and nova yields found significant overproduction of Li
relative to solar composition material (Arnould and Norgaard,, 1975;
Starrfield et al.,, 1978). More sophisticated modelling with comprehensive
reaction networks qualitatively confirmed these early results (José and
Hernanz,, 1998; Starrfield et al.,, 2000), solidifying the idea of novae as Li
factories.

Since the first detection of ${}^{7}{\rm B}$e in 2015 (Tajitsu et al.,, 2015),
${}^{7}{\rm B}$e has been been detected in the spectra of a handful of nova
outbursts (Tajitsu et al.,, 2015; Molaro et al.,, 2016; Tajitsu et al.,, 2016;
Selvelli et al.,, 2018; Molaro et al.,, 2020; Arai et al.,, 2021; Izzo et
al.,, 2022). These observations not only provided direct evidence of novae
producing Li, but also implied a far greater over-production factor than that
found in theoretical models, reigniting interest in novae as a solution to the
Galactic Li problem. Where older chemical evolution models that relied on
theoretical nova yields identified novae as promising candidates (Romano et
al.,, 2001), newer works making use of the observationally-derived yields
demonstrated that novae could account for the high proto-solar Li abundance
(Cescutti and Molaro,, 2019; Grisoni et al.,, 2019).

However, existing calculations on the viability of novae as dominant sources
of Li have relied upon simplified models for how novae behave. For example,
the recent work of Grisoni et al., (2019) assume each nova system to undergo
$10^{4}$ eruptions (Bath and Shaviv,, 1978), assigning each a constant ejecta
mass of Li bounded by observations of V1369 Cen (Izzo et al.,, 2015). The
birth rate of nova systems is set as a fraction of the WD formation rate, and
all nova eruptions generated from a period of star formation are assumed to
occur instantaneously, offset by a fixed amount time. Cescutti and Molaro,
(2019) features a more sophisticated treatment which approximates the
continuous release of nova ejecta, and instead treats the average Li ejecta-
mass per event as a free parameter, ultimately determined to be consistent
with observations of ${}^{7}{\rm B}$e in nova ejecta. These assumptions were
necessary as more sophisticated models for nova populations did not exist.

In this work, we instead rely on the results of previously computed
theoretical nova populations (Kemp et al.,, 2022), where each nova eruption is
treated individually based the white dwarf mass $M_{\rm WD}$ and accretion
rate $\dot{M}$. Our chemical evolution model and treatment of novae is
described in Section 2, our findings on novae as sources of Li in the Milky
Way presented Section 3, and commentary on the significance and uncertainty in
our findings presented in Section 4. We conclude in Section 5. Supplementary
material can be accessed here: DOI:10.5281/zenodo.6644271

##  2 Methodology

###  2.1 Nova treatment

Figure 1 shows a nova delay time (upper panel) and ejecta-delay time (lower
panel) distribution coloured by the WD composition. The delay time
distribution tracks the time from star formation to each nova eruption, where
each nova eruption is weighted equally. The ejecta-delay time distribution
instead weights each nova eruption by the total mass ejected into the
interstellar medium, tracking the mass ejection profile of novae for each
burst of star formation.

Figure 1: Delay time (upper panel, tracking when the nova eruptions occur
relative to star formation) and ejecta-delay time (lower panel, tracking when
mass is ejected relative to star formation) distributions for $Z$=$10^{-3}$.
Both distributions are normalised per solar mass of star forming material $\rm
M_{\rm\odot SFM}$, (for details, see Kemp et al.,, 2021, 2022).

A metallicity-dependent grid of these pre-computed ejecta-delay time
distributions is taken as a direct input into our galactic chemical evolution
models. The underlying binary population synthesis models are a subset
($Z=10^{-4}$, $10^{-3}$, $5\times 10^{-3}$, 0.01, 0.02) of those presented in
Kemp et al., (2022), and further details about the process of simulating each
nova eruption in binary_c (Izzard et al.,, 2004, 2006, 2009, 2018) are
available in Kemp et al., (2021).

To make predictions about specific elements or isotopes, these ejecta-delay
time distributions must be combined with nova yield tables. We can divide our
nova population according to the white dwarf mass $M_{\rm WD}$ and accretion
rate $\dot{M}$ at the time of eruption by using an array of ejecta-delay time
distributions that correspond to different regions of $M_{\rm
WD}$-$\dot{M}$parameter space. This allows us to map physics-dependent yields
from theoretical models (José and Hernanz,, 1998; Starrfield et al.,, 2009,
2020; Rukeya et al.,, 2017; José et al.,, 2020) to their relevant ejecta-delay
time distributions. This treatment innately accounts for ejecta mass and delay
time variation in different nova systems, the metallicity dependence of nova-
system formation and evolution, and physics-dependant aspects of nova
nucleosynthesis (Iliadis,, 2015).

###  2.2 Nova yield sets

We have compiled five theoretical nova nucleosynthesis yield profiles. The
implementation of each of these yield profiles is described below:

  1. 1.

J1998: Makes use of 50% pre-enriched C/O WD nucleosynthesis yields for $M_{\rm
WD}$ = 0.8 and 1.0 M⊙, and 50% pre-enriched O/Ne WD yields for $M_{\rm WD}$ =
1.15, 1.25, and 1.35 M⊙ (José and Hernanz,, 1998).

  2. 2.

S2009/2020: Makes use of 50% pre-enriched C/O WD nucleosynthesis yields for
$M_{\rm WD}$ = 0.6, 0.8, 1.0 and 1.15 M⊙ (Starrfield et al.,, 2020), and 50%
pre-enriched O/Ne WD yields for $M_{\rm WD}$ = 1.25, 1.35 M⊙ (Starrfield et
al.,, 2009).

  3. 3.

J2020: Makes use of C/O WD nucleosynthesis yields for $M_{\rm WD}$ = 1.0 M⊙
and O/Ne WD yields for $M_{\rm WD}$ = 1.15 M⊙ (José et al.,, 2020). Rather
than assuming a pre-enrichment fraction to account for mixing during the
eruption, these yields come from models which instead simulate the eruption by
combining 3D (FLASH; Fryxell et al.,, 2000) and 1D (SHIVA; José and Hernanz,,
1998) methods to model the conditions at outburst.

  4. 4.

R2017: Makes use of 50 percent pre-enriched Li yields for C/O WDs at $M_{\rm
WD}$ = 0.51, 0.6, 0.7, 0.8, 0.9, 1.0, and 50% pre-enriched O/Ne WD yields for
$M_{\rm WD}$ = 1.1, 1.2, 1.3, and 1.34 M⊙ (Rukeya et al.,, 2017). This yield
profile is notable for also providing yields for a range of accretion rates
for each $M_{\rm WD}$, rather than sampling only at a fixed accretion rate of
$2\times 10^{-10}$ M${}_{\odot}\,\textrm{yr}^{-1}$ as in the previously
described J1998, S2009/2020, and J2020 sets. A figure demonstrating the
breakdown of this grid is provided in the supplementary material.

  5. 5.

R2017simple: Makes use of 50% pre-enriched Li yields for C/O WDs at $M_{\rm
WD}$= 0.8 and 1.0 M⊙, and O/Ne WD yields for $M_{\rm WD}$= 1.15, 1.25, and
1.35 M⊙ (Rukeya et al.,, 2017). This yield set is, by design, identical to the
parameter space set out in the J1998 set. It is useful as it can be directly
compared with the R2017 set to assess the impact of only sampling the
accretion rate space at $2\times 10^{-10}$ M${}_{\odot}\,\textrm{yr}^{-1}$.

Note that all of the underlying models informing the above theoretical yield
profiles assume solar composition material.

Figure 2: Observationally-derived A(${}^{7}{\rm B}$e), corrected for
${}^{7}{\rm B}$e decay since eruption. Decay-corrected A(${}^{7}{\rm B}$e)
values are taken from: Molaro et al., (2020) table 1 (V339 Del, V5668 Sgr,
V2944 Oph, V407 Lup, V357 Mus, V906 Car), Molaro et al., (2022) table 4 (V6595
Sgr, V838 Her, V5669 Sgr), and Izzo et al., (2022) (ASASSN-19qv, ASASSN-20ni).

Figure 2 presents all available observationally-derived ${}^{7}{\rm B}$e
abundances A(${}^{7}{\rm B}$e), corrected for ${}^{7}{\rm B}$e decay (Molaro
et al.,, 2020, 2022). Over-plotted are the mean and 1-$\sigma$ error bars for
this data computed using all available observations (A(${}^{7}{\rm B}$e) =
$7.12\pm 0.71$, grey). The H mass fraction $X$ of the ejecta is required to
convert this information into a useful mass fraction of ${}^{7}{\rm B}$e, a
quantity dependent on the fraction of core material mixed into the burning
zone, which we must assume. We assess the impact of this assumption by
considering two cases for the mixing fraction, 50% ($X=0.35$) and 25%
($X=0.5$), with the associated $X$ values for these two cases based on the
simulations of José and Hernanz, (1998) and Starrfield et al., (2009, 2020).

Significant scatter exists in these measurements. The novae V407 Lup, V6595
Sgr, and V838 Her are classified as O/Ne novae due to the detection of bright
Ne lines in their late nebula phases (table 4, Molaro et al.,, 2022). Of these
novae, only V407 Lup and V838 Her have measurements notably above average. The
dusty nova V5668 Sgr has the highest value (A(${}^{7}{\rm B}$e$)=8.1$), but
V357 Mus and V906 Car appear typical despite also being dusty (Gordon et al.,,
2021, and references therein).

###  2.3 Milky Way chemical evolution model

Our chemical evolution model is computed using the two-zone GCE code OMEGA+
(Côté et al.,, 2018). The model structure consists of a star-forming galactic
component coupled with a circum-galactic halo which functions as a hot gas
reservoir and facilitates galactic recycling. The central galactic component
is simulated using OMEGA (Côté et al.,, 2017), a one-zone GCE code accounting
for the chemical yields of different stellar populations.

Our baseline Milky Way model, to which we add nova contributions, uses an
exponential Galactic inflow rate and calculates outflows as a function of star
formation. Stellar yields for type Ia supernovae, asymptotic giant branch
(AGB), and massive stars are taken from Thielemann et al., (1986), Karakas,
(2010), and Kobayashi et al., (2006) respectively. Contributions to Li
production from cosmic rays are approximated as a function of [Fe/H] using
data from Figure 16 of Prantzos, (2012).

Further details and figures used to validate our baseline Milky Way model can
be found in the supplementary material.

##  3 Results

###  3.1 Lithium source comparison

Figure 3: Li mass contributions from different sources as a function of
Galactic age. See main text for discussion.

Figure 3 presents the mass of Li produced by different sources as a function
of Galactic age. The total mass of Li for the full J1998 GCE model is shown
for comparison, which includes primordial material being introduced through
galactic inflows as well as AGB, massive stellar, Galactic cosmic ray (GCR),
and nova contributions using the previously described J1998 yield profile.

Two different models relying on different observationally-derived yield sets
are shown (dotted lines + shaded 1-$\sigma$ error bar): assuming 50% mixing of
underlying core material (grey); and assuming 25% mixing (blue). Reducing the
mixing fraction from 50% to 25% can be seen to increase the average
noticeably, although the change is small relative to the observational
uncertainty. Despite the large error bars resulting from the scatter in
observational Li determinations, it is clear that even in the most pessimistic
case novae are expected to produce far more Li than any other stellar source
over the Galaxy’s lifetime.

The S2009/2020 yield set results in the most Li produced by novae when
theoretical nova yields are used, but even this yield set produces almost an
order of magnitude less Li by $t=10$ Gyr than the lowest error bound of the
observational yield sets. The disagreement between theoretical models of Li
production in novae and observations of Li in the ejecta is at least a factor
of 5.

Comparing the different theoretical nova yield profiles, the profiles relying
on pre-mixing – J1998, S2009/2020, R2017, R2017simple – have broadly
comparable productivity, overtaking GCR contributions after roughly 2 Gyr of
evolution but never overproducing relative to AGB yields by more than a factor
of 2 (S2009/2020). Note that the R2017 and R2017simple models are almost
identical in Figure 3, but there is discernible variation between the Li
masses produced by yield profiles from different research groups. This
strongly implies that neglecting to resolve the accretion-rate dependency of
nova nucleosynthesis is of secondary importance. A figure presenting the
distribution of Li mass production in $M_{\rm WD}$-$\dot{M}$ space according
to the R2017 model is included in the supplementary material.

The only theoretical yield set that does not rely on pre-mixing – J2020 –
produces three orders of magnitude less Li than the theoretical pre-enriched
yield sets. José et al., (2020) attribute this to the longer timescale over
which the thermonuclear runaway develops in these models, which allows far
more ${}^{7}{\rm B}$e to be destroyed before it can be advected to cooler
regions. A more detailed discussion is presented in José et al., (2020).

###  3.2 Lithium in the Milky Way

Figure 4 presents the Li abundance versus [Fe/H] in each of our Galactic
models, overlaid with the proto-solar Li abundance (yellow) and Li abundances
of stars in GALAH DR3 (Buder et al.,, 2021, grey-scale). The GALAH dataset
used is comprised of the 85490 main sequence stars (logg $>3.7$) that had
acceptable signal-to-noise ratios (snr_c3_iraf $>30$), stellar parameters
(flag_sp $=0$), and Fe/H (flag_fe_h $=0$) and Li/Fe (flag_Li_fe $=0$)
abundances. Unlike Figure 3, which only plots the contributions specifically
from novae in the curves labelled with nova yield profiles, each of the models
in Figure 4 includes the specified nova yield profile in addition to all non-
nova Li sources.

Figure 4 demonstrates that models making use of observationally-derived nova
Li yields can account for the proto-solar Li abundance. Our model relying upon
observational Li yield observational yield with the 50% mixing assumption
passes cleanly through the proto-solar Li observation. The Li mass fraction
for novae used in this GCE model is approximately $4.8\times 10^{-5}$, which
we present as our empirical solution for the average Li mass fraction in nova
ejecta. However, note that the model assuming 25% mixing produces a Li
abundance at [Fe/H] = 0 that lies within 0.1 dex of the proto-solar
observation, well within the 1-$\sigma$ uncertainties accounting for scatter
in the observationally-derived A(${}^{7}{\rm B}$e) data.

Conversely, all models relying on theoretical yield profiles are
indistinguishable from each other or the baseline model without novae. These
predict Li abundances an order of magnitude below the proto-solar Li
abundance. Despite being non-negligible Li producers (Figure 3), according to
these yield-profiles novae simply do not produce enough Li to account for the
solar Li abundance or, to a lesser extent, the upper envelope of the GALAH
data.

It has been proposed (Grisoni et al.,, 2019) that the metallicity-dependence
of novae could explain observations of declining Li abundances at high
metallicities, a feature detected in the Milky Way by multiple groups (Mena et
al.,, 2015; Guiglion et al.,, 2016; Bensby and Lind,, 2018; Fu et al.,, 2018;
Buder et al.,, 2018, 2021). We report that none of our models show signs of a
Li decline at high metallicity, despite accounting for metallicity-dependent
variation in our nova ejecta-delay time distributions.

Figure 4: Li abundance versus [Fe/H] in our Galactic models, overlaid with the
proto-solar Li abundance (Lodders et al.,, 2009) (yellow cross-hair) and Li
abundance data from GALAH DR3 (Buder et al.,, 2021). All models include
asymptotic giant branch, massive stellar, and galactic cosmic ray sources in
addition to the designated nova yield set.

##  4 Discussion

GCE models incorporate, albeit often indirectly, a vast amount of physics. The
adopted yield sets and productivities of different stellar phenomena (eg. AGB
stars, massive stars, type Ia supernovae, neutron star mergers, etc.) all
affect how a simulated galaxy evolves, and each of these prescriptions has its
own physical assumptions. Additionally, modelling choices for galactic
processes such as the star formation efficiency and inflow and outflow rates
are vital. This modelling complexity renders assessment of uncertainty
difficult.

Our baseline model of the Milky Way satisfactorily reproduces a range of
observables (see supplementary material for supporting figures), including key
elemental abundance trends. This does not necessarily make the model correct;
rather, it is intended to provide a representative baseline model with which
we can assess the importance of novae. In this work, we are concerned with Li,
an element notoriously sensitive to stellar modelling choices. The two classes
of assumptions most likely to affect our results are the adopted non-nova
stellar yields and the assumed binary physics behind our ejecta-delay time
distributions. We summarise here the results of a brief exploration of both
these uncertainties.

We find that changing the adopted AGB yields from Karakas, (2010) yields to
NuGrid yields (Pignatari et al.,, 2016) has minimal effect on the ability of
our models using observational nova yield profiles to reproduce the solar Li
abundance, despite the NuGrid yields producing roughly twice as much Li from
AGB stars by [Fe/H]=0. Assessing the extreme lower bound, removing all non-
nova sources of Li results in a 0.25 dex negative shift relative to the proto-
solar abundance. This is well within the 1-$\sigma$ bounds associated with the
observational data. Preliminary results of an investigation into the
statistical effects of binary interactions on solar metallicity AGB yields –
an effect neglected in contemporary GCE codes – indicate at most a 15%
reduction in Li (Osborn et al., in prep). We therefore conclude that our
results are likely robust against uncertainties in non-nova Li yields.

Removing the metallicity dependence of the nova ejecta-delay time
distributions and instead only using ejecta-delay time distributions computed
at $Z=0.02$ results in a 35% reduction in Li production from novae. This
reduces the Li abundance at [Fe/H]=0 predicted by our GCE models using
observationally-derived yield sets by 0.3-0.4 dex. The proto-solar Li
abundance remains comfortably within the 1-$\sigma$ error bars associated with
the observationally-derived yield profiles. We therefore conclude that our
results are robust to the metallicity-resolution of our array of nova ejecta-
delay time distributions.

Figure 5: As Figures 3 and 4 when common-envelope parameter $\lambda_{\rm CE}$
is set to 0.5, rather than the Wang et al., (2016) prescription used in our
baseline case. This physics case doubles the expected nova rate in M31 Kemp et
al., (2021), and we see this reflected in a doubling in Li production). The
ability of observationally-derived nova yield profiles to reproduce the proto-
solar Li abundance, and the inability of theoretical yield profiles, appears
robust.

A full investigation of binary uncertainty is beyond the scope of this work.
However, as an indicative example, Figure 5 presents the results of using an
alternative set of ejecta-delay time distributions computed using a different
set of binary physics. In producing Figure 5, we have replaced the Wang et
al., (2016) prescription for the common-envelope parameter $\lambda_{\rm CE}$
(Kemp et al.,, 2021) with a constant value of 0.5. This physics set was found
in previous work (Kemp et al.,, 2021) to approximately double the predicted
nova rate in M31, and represents an extreme case of variation due to binary
physics (Kemp et al.,, 2021). We find that the total mass of Li produced by
novae approximately doubles under this assumption, implying roughly a factor
of 2 uncertainty in our results from physical uncertainty in binary stellar
physics.

We also investigated the effect of replacing the nova accretion efficiency
prescription of Wang, (2018) with a constant to assess the potential impact
that nova-specific modelling choices could have on our results. Setting the
accretion efficiency to 0.01, meaning that in each nova eruption 99% of the
accreted material is lost to the WD per eruption, has a negligible effect on
our results.

##  5 Conclusions

In this work, we assess the importance of novae to the synthesis of Li in the
Milky Way using the galactic chemical evolution code OMEGA+. Previous galactic
chemical evolution models have either relied on simplified treatments of novae
or neglected them altogether. In this work, we employ metallicity-dependent
arrays of nova ejecta-delay time distributions computed using the binary
population synthesis code binary_c to model the Galactic nova population.

We assess the viability of novae in the context of five different theoretical
yield profiles (José and Hernanz,, 1998; Starrfield et al.,, 2009; Rukeya et
al.,, 2017; José et al.,, 2020; Starrfield et al.,, 2020) in addition to
observationally-derived Li yields (Molaro et al.,, 2020, 2022; Izzo et al.,,
2022). We find that our Galactic chemical evolution models which make use of
observationally-derived Li yields account for the proto-solar Li abundance
very well, while the models which make use of theoretical yield profiles
universally fail to reproduce this observation by an order of magnitude.

We find this result to be robust to all physical uncertainties which were
included in our exploratory analysis, including the choice of asymptotic giant
branch yields, the metallicity-resolution of our array of nova ejecta-delay
time distributions, common-envelope physics, and nova accretion efficiencies.
Further, physical uncertainties appear to be of secondary importance when
compared to the large amount of scatter present in observations of ${}^{7}{\rm
B}$e in nova ejecta.

The authors wish to thank the anonymous referee for their helpful feedback. A.
R. C. is supported in part by the Australian Research Council through a
Discovery Early Career Researcher Award (DE190100656). B. C. acknowledges
support from the National Science Foundation (NSF, USA) under grant No.
PHY-1430152 (JINA Center for the Evolution of the Elements). R. G. I. thanks
the STFC for funding, in particular Rutherford fellowship ST/L003910/1 and
consolidated grant ST/R000603/1. Parts of this research were supported by the
Australian Research Council Centre of Excellence for All Sky Astrophysics in 3
Dimensions (ASTRO 3D), through project number CE170100013.

##  Appendix A Key features of the nova population: ejecta mass

Most of the key features of our nova populations are discussed in Kemp et al.,
(2021) and Kemp et al., (2022), which discuss which binary systems produce the
most novae. However, the key features and distributions of nova properties
when considering which systems produce the most nova ejecta requires a brief
description.

The relative importance of different WD masses in terms of the number of novae
they produce is a balance between the initial mass function, which disfavors
high-mass WD systems, and the increased rate at which these systems can
produce novae because of their higher surface gravity and lower critical
ignition masses (the mass of accreted material required for a nova eruption to
occur).

However, reducing the critical ignition mass also reduces the mass of ejecta,
which is further complicated by the question of the accretion efficiency (the
fraction of accreted material that is retained by the WD beyond the outburst).
Increasing the WD mass can lead to higher accretion efficiencies (Wang,,
2018), further reducing the ejecta mass per eruption, but this effect is of
secondary importance to the reduced critical ignition mass.

As shown in Figure 1, comparing the ejecta-delay time distributions to their
delay time counterparts reveals that late time ejecta-mass contributions are
significantly higher than when considering the raw nova rate. This is because
the more massive O/Ne WDs contribute far less ejecta per event than their low-
mass C/O WD counterparts, reducing the relative importance of their
contributions to the distribution.

In general, nova systems which require high ($M_{\rm ig}$$\ \gtrsim$ $5\times
10^{-5}$ M⊙) critical ignition masses for outburst – typically characterised
by low mass WDs ($M_{\rm WD}$$\ <$ 0.8 M⊙) without extremely high accretion
rates ($\dot{M}$$\ \lesssim$ $10^{-7}$ M${}_{\odot}\,\textrm{yr}^{-1}$) –
dominate in terms of the mass of nova processed material. High mass ($M_{\rm
WD}$$\ >$ 1 M⊙) WD systems are significantly less important when considering
the ejecta mass. However, this does not preclude high mass WDs contributing at
a level comparable or even higher than low mass WDs when it comes to the
nucleosynthesis of specific isotopes. The higher temperatures obtainable in
nova eruptions on massive WDs offer unique nucleosynthetic pathways leading to
far more efficient synthesis of elements with mass-numbers higher than F, an
effect able to outweigh their lower ejecta masses and system birth rates.

## References

  * Arai et al., (2021) Arai, A., Tajitsu, A., Kawakita, H., and Shinnaka, Y. (2021).  Detection of 7Be II in the Classical Nova V5669 Sgr (Nova Sagittarii 2015 No.3).  ApJ, 916(1):44.
  * Arnould and Norgaard, (1975) Arnould, M. and Norgaard, H. (1975).  The Explosive Thermonuclear Formation of 7Li and 11B.  A&A, 42:55.
  * Asplund et al., (2009) Asplund, M., Grevesse, N., Sauval, A. J., and Scott, P. (2009).  The Chemical Composition of the Sun.  ARA&A, 47(1):481–522.
  * Bath and Shaviv, (1978) Bath, G. T. and Shaviv, G. (1978).  The space density, recurrence rate and classification of novae.  MNRAS, 183:515–522.
  * Bensby and Lind, (2018) Bensby, T. and Lind, K. (2018).  Exploring the production and depletion of lithium in the Milky Way stellar disk.  A&A, 615:A151.
  * Buder et al., (2018) Buder, S., Asplund, M., Duong, L., Kos, J., Lind, K., Ness, M. K., Sharma, S., Bland-Hawthorn, J., Casey, A. R., de Silva, G. M., D’Orazi, V., Freeman, K. C., Lewis, G. F., Lin, J., Martell, S. L., Schlesinger, K. J., Simpson, J. D., Zucker, D. B., Zwitter, T., Amarsi, A. M., Anguiano, B., Carollo, D., Casagrande, L., Čotar, K., Cottrell, P. L., da Costa, G., Gao, X. D., Hayden, M. R., Horner, J., Ireland, M. J., Kafle, P. R., Munari, U., Nataf, D. M., Nordlander, T., Stello, D., Ting, Y.-S., Traven, G., Watson, F., Wittenmyer, R. A., Wyse, R. F. G., Yong, D., Zinn, J. C., Žerjal, M., and Galah Collaboration (2018).  The GALAH Survey: second data release.  MNRAS, 478(4):4513–4552.
  * Buder et al., (2021) Buder, S., Sharma, S., Kos, J., Amarsi, A. M., Nordlander, T., Lind, K., Martell, S. L., Asplund, M., Bland-Hawthorn, J., Casey, A. R., de Silva, G. M., D’Orazi, V., Freeman, K. C., Hayden, M. R., Lewis, G. F., Lin, J., Schlesinger, K. J., Simpson, J. D., Stello, D., Zucker, D. B., Zwitter, T., Beeson, K. L., Buck, T., Casagrande, L., Clark, J. T., Čotar, K., da Costa, G. S., de Grijs, R., Feuillet, D., Horner, J., Kafle, P. R., Khanna, S., Kobayashi, C., Liu, F., Montet, B. T., Nandakumar, G., Nataf, D. M., Ness, M. K., Spina, L., Tepper-García, T., Ting, Y.-S., Traven, G., Vogrinčič, R., Wittenmyer, R. A., Wyse, R. F. G., Žerjal, M., and GALAH Collaboration (2021).  The GALAH+ survey: Third data release.  MNRAS, 506(1):150–201.
  * Cameron and Fowler, (1971) Cameron, A. G. W. and Fowler, W. A. (1971).  Lithium and the s-PROCESS in Red-Giant Stars.  ApJ, 164:111.
  * Cescutti and Molaro, (2019) Cescutti, G. and Molaro, P. (2019).  7Li evolution in the thin and thick discs of the Milky Way.  MNRAS, 482(4):4372–4382.
  * Côté et al., (2017) Côté, B., O’Shea, B. W., Ritter, C., Herwig, F., and Venn, K. A. (2017).  The Impact of Modeling Assumptions in Galactic Chemical Evolution Models.  ApJ, 835(2):128.
  * Côté et al., (2018) Côté, B., Silvia, D. W., O’Shea, B. W., Smith, B., and Wise, J. H. (2018).  Validating Semi-analytic Models of High-redshift Galaxy Formation Using Radiation Hydrodynamical Simulations.  ApJ, 859(1):67.
  * D’Antona and Matteucci, (1991) D’Antona, F. and Matteucci, F. (1991).  Galactic evolution of lithium.  A&A, 248:62.
  * Fryxell et al., (2000) Fryxell, B., Olson, K., Ricker, P., Timmes, F. X., Zingale, M., Lamb, D. Q., MacNeice, P., Rosner, R., Truran, J. W., and Tufo, H. (2000).  FLASH: An Adaptive Mesh Hydrodynamics Code for Modeling Astrophysical Thermonuclear Flashes.  ApJS, 131(1):273–334.
  * Fu et al., (2018) Fu, X., Romano, D., Bragaglia, A., Mucciarelli, A., Lind, K., Delgado Mena, E., Sousa, S. G., Randich, S., Bressan, A., Sbordone, L., Martell, S., Korn, A. J., Abia, C., Smiljanic, R., Jofré, P., Pancino, E., Tautvaišienė, G., Tang, B., Magrini, L., Lanzafame, A. C., Carraro, G., Bensby, T., Damiani, F., Alfaro, E. J., Flaccomio, E., Morbidelli, L., Zaggia, S., Lardo, C., Monaco, L., Frasca, A., Donati, P., Drazdauskas, A., Chorniy, Y., Bayo, A., and Kordopatis, G. (2018).  The Gaia-ESO Survey: Lithium enrichment histories of the Galactic thick and thin disc.  A&A, 610:A38.
  * Gordon et al., (2021) Gordon, A. C., Aydi, E., Page, K. L., Li, K.-L., Chomiuk, L., Sokolovsky, K. V., Mukai, K., and Seitz, J. (2021).  Surveying the X-Ray Behavior of Novae as They Emit $\gamma$-Rays.  ApJ, 910(2):134.
  * Grisoni et al., (2019) Grisoni, V., Matteucci, F., Romano, D., and Fu, X. (2019).  Evolution of lithium in the Milky Way halo, discs, and bulge.  MNRAS, 489(3):3539–3546.
  * Guiglion et al., (2016) Guiglion, G., de Laverny, P., Recio-Blanco, A., Worley, C. C., De Pascale, M., Masseron, T., Prantzos, N., and Mikolaitis, Š. (2016).  The AMBRE project: Constraining the lithium evolution in the Milky Way.  A&A, 595:A18.
  * Iliadis, (2015) Iliadis, C. (2015).  Nuclear physics of stars.  John Wiley & Sons.
  * Izzard et al., (2006) Izzard, R. G., Dray, L. M., Karakas, A. I., Lugaro, M., and Tout, C. A. (2006).  Population nucleosynthesis in single and binary stars. I. Model.  A&A, 460(2):565–572.
  * Izzard et al., (2009) Izzard, R. G., Glebbeek, E., Stancliffe, R. J., and Pols, O. R. (2009).  Population synthesis of binary carbon-enhanced metal-poor stars.  A&A, 508(3):1359–1374.
  * Izzard et al., (2018) Izzard, R. G., Preece, H., Jofre, P., Halabi, G. M., Masseron, T., and Tout, C. A. (2018).  Binary stars in the Galactic thick disc.  MNRAS, 473(3):2984–2999.
  * Izzard et al., (2004) Izzard, R. G., Tout, C. A., Karakas, A. I., and Pols, O. R. (2004).  A new synthetic model for asymptotic giant branch stars.  Monthly Notices of the Royal Astronomical Society, 350(2):407–426.
  * Izzo et al., (2015) Izzo, L., Della Valle, M., Mason, E., Matteucci, F., Romano, D., Pasquini, L., Vanzi, L., Jordan, A., Fernandez, J. M., Bluhm, P., Brahm, R., Espinoza, N., and Williams, R. (2015).  Early Optical Spectra of Nova V1369 Cen Show the Presence of Lithium.  ApJ, 808(1):L14.
  * Izzo et al., (2022) Izzo, L., Molaro, P., Cescutti, G., Aydi, E., Selvelli, P., Harvey, E., Agnello, A., Bonifacio, P., Della Valle, M., Guido, E., and Hernanz, M. (2022).  Detection of 7Be II in the Small Magellanic Cloud.  MNRAS, 510(4):5302–5314.
  * José and Hernanz, (1998) José, J. and Hernanz, M. (1998).  Nucleosynthesis in classical novae: Co versus one white dwarfs.  The Astrophysical Journal, 494(2):680.
  * José et al., (2020) José, J., Shore, S. N., and Casanova, J. (2020).  123-321 models of classical novae.  A&A, 634:A5.
  * Karakas, (2010) Karakas, A. I. (2010).  Updated stellar yields from asymptotic giant branch models.  MNRAS, 403(3):1413–1425.
  * Kemp et al., (2021) Kemp, A. J., Karakas, A. I., Casey, A. R., Izzard, R. G., Ruiter, A. J., Agrawal, P., Broekgaarden, F. S., and Temmink, K. D. (2021).  Population synthesis of accreting white dwarfs: rates and evolutionary pathways of H and He novae.  MNRAS, 504(4):6117–6143.
  * Kemp et al., (2022) Kemp, A. J., Karakas, A. I., Casey, A. R., Kobayashi, C., and Izzard, R. G. (2022).  The impact of metallicity on nova populations.  MNRAS, 509(1):1175–1193.
  * Kobayashi et al., (2006) Kobayashi, C., Umeda, H., Nomoto, K., Tominaga, N., and Ohkubo, T. (2006).  Galactic Chemical Evolution: Carbon through Zinc.  ApJ, 653(2):1145–1171.
  * Lodders et al., (2009) Lodders, K., Palme, H., and Gail, H. P. (2009).  Abundances of the Elements in the Solar System.  Landolt Börnstein, 4B:712.
  * Matteucci et al., (1995) Matteucci, F., D’Antona, F., and Timmes, F. X. (1995).  News ($\nu$s) on the Galactic evolution of lithium.  A&A, 303:460.
  * Mena et al., (2015) Mena, E. D., de Lis, S. B., Adibekyan, V. Z., Sousa, S., Figueira, P., Mortier, A., Hernández, J. G., Tsantaki, M., Israelian, G., and Santos, N. (2015).  Li abundances in f stars: planets, rotation, and galactic evolution.  Astronomy & Astrophysics, 576:A69.
  * Molaro et al., (2020) Molaro, P., Izzo, L., Bonifacio, P., Hernanz, M., Selvelli, P., and della Valle, M. (2020).  Search for 7Be in the outbursts of four recent novae.  MNRAS, 492(4):4975–4985.
  * Molaro et al., (2022) Molaro, P., Izzo, L., D’Odorico, V., Aydi, E., Bonifacio, P., Cescutti, G., Harvey, E. J., Hernanz, M., Selvelli, P., and della Valle, M. (2022).  7Be in the outburst of the ONe nova V6595 Sgr.  MNRAS, 509(3):3258–3267.
  * Molaro et al., (2016) Molaro, P., Izzo, L., Mason, E., Bonifacio, P., and Della Valle, M. (2016).  Highly enriched 7Be in the ejecta of Nova Sagittarii 2015 No. 2 (V5668 Sgr) and the Galactic 7Li origin.  MNRAS, 463(1):L117–L121.
  * Pignatari et al., (2016) Pignatari, M., Herwig, F., Hirschi, R., Bennett, M., Rockefeller, G., Fryer, C., Timmes, F. X., Ritter, C., Heger, A., Jones, S., Battino, U., Dotter, A., Trappitsch, R., Diehl, S., Frischknecht, U., Hungerford, A., Magkotsios, G., Travaglio, C., and Young, P. (2016).  NuGrid Stellar Data Set. I.Stellar Yields from H to Bi for Stars with Metallicities Z = 0.02 and Z = 0.01.  ApJS, 225(2):24.
  * Prantzos, (2012) Prantzos, N. (2012).  Production and evolution of Li, Be, and B isotopes in the Galaxy.  A&A, 542:A67.
  * Romano et al., (2001) Romano, D., Matteucci, F., Ventura, P., and D’Antona, F. (2001).  The stellar origin of 7Li. Do AGB stars contribute a substantial fraction of the local Galactic lithium abundance?  A&A, 374:646–655.
  * Rukeya et al., (2017) Rukeya, R., Lü, G., Wang, Z., and Zhu, C. (2017).  Novae Contribution to the Galactic Lithium Enhancement.  PASP, 129(977):074201.
  * Selvelli et al., (2018) Selvelli, P., Molaro, P., and Izzo, L. (2018).  Absorption and emission features of 7Be II in the outburst spectra of V838 Her (Nova Her 1991).  MNRAS, 481(2):2261–2272.
  * Starrfield et al., (2020) Starrfield, S., Bose, M., Iliadis, C., Hix, W. R., Woodward, C. E., and Wagner, R. M. (2020).  Carbon-Oxygen Classical Novae Are Galactic 7Li Producers as well as Potential Supernova Ia Progenitors.  ApJ, 895(1):70.
  * Starrfield et al., (2009) Starrfield, S., Iliadis, C., Hix, W. R., Timmes, F. X., and Sparks, W. M. (2009).  The Effects of the pep Nuclear Reaction and Other Improvements in the Nuclear Reaction Rate Library on Simulations of the Classical Nova Outburst.  ApJ, 692(2):1532–1542.
  * Starrfield et al., (2000) Starrfield, S., Sparks, W., Truran, J., and Wiescher, M. (2000).  The effects of new nuclear reaction rates and opacities on hydrodynamic simulations of the nova outburst.  The Astrophysical Journal Supplement Series, 127(2):485.
  * Starrfield et al., (1978) Starrfield, S., Truran, J. W., and Sparks, W. M. (1978).  CNO abundances and hydrodynamic studies of the nova outburst. V. 1.00 M sun models with small mass envelopes.  ApJ, 226:186–202.
  * Tajitsu et al., (2015) Tajitsu, A., Sadakane, K., Naito, H., Arai, A., and Aoki, W. (2015).  Explosive lithium production in the classical nova V339 Del (Nova Delphini 2013).  Nature, 518(7539):381–384.
  * Tajitsu et al., (2016) Tajitsu, A., Sadakane, K., Naito, H., Arai, A., Kawakita, H., and Aoki, W. (2016).  The 7Be II Resonance Lines in Two Classical Novae V5668 Sgr and V2944 Oph.  ApJ, 818(2):191.
  * Thielemann et al., (1986) Thielemann, F. K., Nomoto, K., and Yokoi, K. (1986).  Explosive nucleosynthesis in carbon deflagration models of Type I supernovae.  A&A, 158(1-2):17–33.
  * Wang, (2018) Wang, B. (2018).  Mass-accreting white dwarfs and type ia supernovae.  Research in Astronomy and Astrophysics, 18(5):049.
  * Wang et al., (2016) Wang, C., Jia, K., and Li, X.-D. (2016).  The binding energy parameter for common envelope evolution.  Research in Astronomy and Astrophysics, 16(8):126.