# Superconductivity in the doped polymerized fullerite clathrate from first
principles

Jorge Laranjeira<EMAIL_ADDRESS>Departamento de Física and CICECO,
Universidade de Aveiro, 3810-193 Aveiro, Portugal  Ion Errea  Fisika Aplikatua
Saila, Gipuzkoako Ingeniaritza Eskola, University of the Basque Country
(UPV/EHU), Europa Plaza 1, 20018 Donostia/San Sebastián, Spain  Centro de
Física de Materiales (CSIC-UPV/EHU), Manuel de Lardizabal Pasealekua 5, 20018
Donostia/San Sebastián, Spain  Donostia International Physics Center (DIPC),
Manuel de Lardizabal Pasealekua 4, 20018 Donostia/San Sebastián, Spain  Đorđe
Dangić  Fisika Aplikatua Saila, Gipuzkoako Ingeniaritza Eskola, University of
the Basque Country (UPV/EHU), Europa Plaza 1, 20018 Donostia/San Sebastián,
Spain  Centro de Física de Materiales (CSIC-UPV/EHU), Manuel de Lardizabal
Pasealekua 5, 20018 Donostia/San Sebastián, Spain  Leonel Marques
Departamento de Física and CICECO, Universidade de Aveiro, 3810-193 Aveiro,
Portugal  Manuel Melle-Franco  Departamento de Química and CICECO,
Universidade de Aveiro, 3810-193 Aveiro, Portugal  Karol Strutyński
Departamento de Química and CICECO, Universidade de Aveiro, 3810-193 Aveiro,
Portugal

###### Abstract

Superconductivity in the new polymerized $\mathrm{C}_{60}$ fullerite clathrate
doped with simple metals was investigated through density functional theory
methods. The phonon dispersion curves were systematically computed for the
clathrate structures containing the guest dopants (Li, Na, K, Be, Mg, Ca, Al,
Ga, Ge) in one of the two distinct cages, fullerene-like $\mathrm{C}_{60}$ and
sodalite-like $\mathrm{C}_{24}$, exhibited by this structure. Only four of
these structures, Li@$\mathrm{C}_{24}$, Na@$\mathrm{C}_{24}$,
Ga@$\mathrm{C}_{24}$ and Be@$\mathrm{C}_{60}$, are dynamically stable in the
harmonic approximation. They all show superconducting behavior, although their
critical temperatures are predicted to be below 2 K.

DFT Calculations, Carbon Clathrates, Superconductivity

††preprint: APS/123-QED

## Introduction

The achievement of room temperature superconductivity is extremely desirable
for technological applications [revIon]. Very high superconducting critical
temperature ($\mathrm{T}_{c}$), $\sim$200 K, was initially predicted from
first principles density functional theory (DFT) for the
$\mathrm{H}_{3}\mathrm{S}$ compound [hs3_theoric2014], soon followed by its
experimental observation [hs3_experimental2015] under a pressure of 150 GPa.
These exciting results fostered huge efforts, led again by computational
methods, to find new materials with even higher $\mathrm{T}_{c}$ by exploring
hydrogen bearing compounds. A new lanthanum hydride, LaH10, in which the
hydrogen atoms form sodalite-like cages, was found at 140-220 GPa and
displaying Tc of approximately 250 K [lah10_exp, drozdov2019]. This near-room
temperature superconducting phase is only stable under megabar pressures and
cannot be quenched to room conditions, thus precluding practical applications.
The focus of investigations is now directed to the search for superconducting
materials that can operate at room pressure. One strategy to attain this goal
is to consider other types of clathrate structures that still use a light
element in its building blocks, a known key ingredient for high
$\mathrm{T}_{c}$ conventional superconductivity [revIon], while being able to
retain the clathrate structure at ambient pressure. Carbon clathrates are
notable candidates since, besides containing a light element such as carbon,
they exhibit strong covalent bonds that lead to large phonon frequencies,
which is another key ingredient for high $\mathrm{T}_{c}$ superconductivity
[dicataldo, sodalite].

Unfortunately, experimental carbon clathrates are yet to be synthesized,
although an important step towards its realization was done recently with the
high-pressure synthesis of mixed carbon-boron clathrates, containing guest
metallic atoms [strobel2020, Zhu2020]. Initial predictions based on DFT
methods were again crucial to guide the investigations towards their
experimental realization [tao2015]. The mixed carbon-boron clathrate doped
with strontium guest-atoms was hypothesized as a potential candidate for
phonon-mediated superconductivity at ambient conditions, since its electronic
structure showed the coexistence of steep and flat bands close to the Fermi
level [Zhu2020]. Indeed ab initio DFT simulations performed on the strontium
and barium doped carbon-boron clathrates yield Tc’s of 40 K and 43 K,
respectively, at room pressure [srbac3b3]. Furthermore, an increase in Tc up
to 88 K was predicted in this class of compounds for the potassium-lead
binary-guest clathrate [dicataldo, nisha_zurek].

A significant number of superconducting doped carbon allotropes have already
been synthesized. Boron doped diamond yields a $\mathrm{T}_{c}$ of 4 K for a
doping level of 2.5% [ekimov_bdiamond] and 11.4 K for a doping level of 5%
[Bdopeddiamond]. Graphite intercalation compounds such as Yb$\mathrm{C}_{6}$
and Ca$\mathrm{C}_{6}$, display $\mathrm{T}_{c}$’s of 6.5 K and 11.5 K,
respectively [Yc6]. Also, the alkali-metal fulleride
RbCs${}_{2}\mathrm{C}_{60}$ displays a $\mathrm{T}_{c}$ of 33 K at room
pressure. The compound Cs${}_{3}\mathrm{C}_{60}$ presents a higher
$\mathrm{T}_{c}$ 40 K, nevertheless, it requires a pressure of 1.5 GPa to
stabilize the cubic superconducting phase [gunnarsson, PALSTRA]. A theoretical
study of a sodalite-like carbon clathrate, with a structure identical to that
of the mixed carbon-boron clathrate, revealed that it could achieve high
$\mathrm{T}_{c}$, 116 K, at room pressure when doped with sodium [sodalite].

Very recently, we have proposed a new carbon clathrate based on the
polymerized fullerite, which is expected to be synthesized by subjecting
fullerite $\mathrm{C}_{60}$ to high-pressure and high-temperature (HPHT)
treatment. Some experimental high-pressure $\mathrm{C}_{60}$ phases found in
the literature have interfullerene distances identical to that of the
polymerized fullerite clathrate structure, showing evidence that it could be
synthesized once the proper thermodynamic path is found [LARANJEIRA2022].
Interestingly, its electronic structure has a narrow bandgap of 0.68 eV and
its density of states (DOS), shown in Figure S1 of the Supporting Information
(SI), displays a strong peak above ($\sim$0.7 eV) the Fermi level. We found
that this peak is due to the three nearly-flat $\pi^{*}$ bands from the
remaining sp2 carbons. The new clathrate also presents large phonon
frequencies with the highest mode lying at $\sim$1350 cm-1, which suggests
that it could be a good candidate for displaying superconductivity once
properly doped. Here we explore this possibility by performing a systematic
DFT study of the doping effect on the fullerite $\mathrm{C}_{60}$ clathrate
structure. Nine simple metals were tested as guest-atoms being inserted in one
of the two cages, fullerene-like $\mathrm{C}_{60}$ and sodalite-like
$\mathrm{C}_{24}$, exhibited by this clathrate structure. We compute the
electronic structure and phonon dispersion curves and find that only four
doped fullerite clathrates are dynamically stable, all of them showing
metallic behavior. In addition, they display superconductivity but their
$\mathrm{T}_{c}$’s, computed using Migdal-Eliashberg theory, are predicted to
be quite low.

## Methods

For each distinct doped fullerite clathrate we optimized its structure using
the Perdew-Burke-Ernzerhof (PBE) exchange correlation functional [i2, i3] as
implemented in the ultrasoft pseudopotentials provided with the Quantum
Espresso (QE) package [qe_Gionnozzi_2009, qe_Giannozzi_2017]. A k-space grid
of $8\times 8\times 8$ was used and the self-consistent-field cycle was
stopped once the energy difference between consecutive steps was smaller than
$10^{-8}$ Ry. We also used a Methfessel-Paxton [mp] smearing of 0.02 Ry, a
kinetic energy cutoff of 60 Ry and a charge density cutoff of 600 Ry. Phonon
dispersion and electron-phonon coupling calculations were performed with
Density Functional Perturbation Theory (DFPT) [baroni2001phonons] as
implemented in QE. A q-grid of $2\times 2\times 2$ was used for the phonon
dispersion calculation, when stability was found the q-space mesh was
increased to $4\times 4\times 4$ prior to the electron-phonon matrix elements
calculation. These parameters were converged within 1 cm-1. To calculate the
electron-phonon matrix elements we have used a smearing of 0.008 Ry for the
double Dirac delta over the Fermi surface and a k-space mesh of $24\times
24\times 24$. To compute $\mathrm{T}_{c}$, we solved isotropic Eliashberg
equations [eliashberg]. We also evaluated $\mathrm{T}_{c}$ with the Allen-
Dynes modified McMillan equation [dynes].

## Results and Discussion

The fullerite clathrate has a simple cubic cell, with a lattice constant of
6.21 Å, and belongs to the Pm$\bar{3}$ space group. The structure has three
independent atoms, one at 6f and two at 12k Wyckoff positions
[LARANJEIRA2022]. This structure exhibits two different cages, sodalite-like
C24 and fullerene-like C60. Thus, doping may be achieved by inserting guest
atoms in either one or in both cages. Here we only studied the effect of
doping in only one cage. Figure 1 a) shows the clathrate structure resulting
from doping on the 1a Wyckoff position at the center of the $\mathrm{C}_{60}$
cage, hereafter denoted M@$\mathrm{C}_{60}$ (M = guest atom), while Figure 1
b) shows the clathrate structure resulting from doping on the 1b Wyckoff
position at the center of the $\mathrm{C}_{24}$ cage and hereafter denoted
M@$\mathrm{C}_{24}$. Note that the resulting stoichiometry,
M@$\mathrm{C}_{30}$, is the same for both insertion types. The optimized
lattice constants of the differently doped structures and their relative
stabilities are given in Tables S1 and S2.

Figure 1: Doped fullerite clathrate unit cells (top panels) and $2\times
2\times 2$ supercell depicting the $\mathrm{C}_{60}$ cages (bottom panels)
with guest atoms at: a) $\mathrm{C}_{60}$ cages, 1a Wyckoff positions; b)
$\mathrm{C}_{24}$ cages, the 1b Wyckoff positions. The grey atoms are carbons
and the green ones are the dopants. Figure 2: Electronic band structure (left
panels) and density of states (right panels) for the stable compounds: a)
Li@$\mathrm{C}_{24}$; b) Na@$\mathrm{C}_{24}$; c) Ga@$\mathrm{C}_{24}$; d)
Be@$\mathrm{C}_{60}$. The red line indicates the Fermi level.

In an attempt to push the Fermi level to the strong DOS peak observed in the
conduction band that arises from the three nearly-flat $\pi^{*}$ bands, we
initially considered doping with trivalent ions, yttrium and scandium, at the
$\mathrm{C}_{60}$ cages, thus forming the Y@$\mathrm{C}_{60}$ and
Sc@$\mathrm{C}_{60}$ clathrates. Inserting these guest atoms keeps the valence
and conduction bands almost unchanged, while the Fermi level moves almost to
the top of the DOS peak (see the left and middle panels of Figure S2). This
rigid-band picture indicates a nearly-complete charge transfer from the metal
guest to the carbon framework. The number of states at the Fermi level is
around 0.3 and 0.2 states/eV/spin/atom for Sc@$\mathrm{C}_{60}$ and
Y@$\mathrm{C}_{60}$ respectively, a significant number that is an important
prerequisite for the observation of superconductivity. Unfortunately, filling
the antibonding $\pi^{*}$ states destabilizes the carbon framework and leads
to large imaginary frequencies in the phonon dispersion curves, indicating
that these doped clathrates are dynamically unstable (see the right panels of
Figure S2). Moreover, we also found that they have spin-polarized ground
states. The dynamical instabilities are an indication that these systems are
prone to the formation of a charge-density wave that lowers the DOS at the
Fermi level and prevents a magnetic ground state.

Figure 3: Phonon dispersion curves (left panels), projected phonon density of
states (middle panels), Eliashberg spectral function $\alpha^{2}F(\omega)$,
blue line, and integrated electron-phonon coupling $\lambda(\omega)$, red
line, (right panels) for the doped clathrate structures: a)
Li@$\mathrm{C}_{24}$; b) Na@$\mathrm{C}_{24}$; c) Ga@$\mathrm{C}_{24}$; d)
Be@$\mathrm{C}_{60}$.

Given the unfavorable results just described, the search for superconductivity
was pursued through a systematic search of doped clathrate structures
employing dopants from groups I, II and III (Li, Na, K, Be, Mg, Ca, Al, Ga)
and also from group IV (Ge). As already mentioned, in the doped clathrates
considered, the guest atoms are inserted in only one of the two cages
available in the structure. The electronic bands, DOS and phonon dispersion
curves for all the doped clathrates being studied are given in the SI,
sections D and E, respectively. The electronic structures across the
M@$\mathrm{C}_{60}$ series essentially follow a rigid-band model, in
particular for the monovalent doped structures, the guest atom donates most of
its charge to the carbon framework (see section D of the SI, Figures S3 and
S4). The most noticeable exceptions are Be@$\mathrm{C}_{60}$ and
Mg@$\mathrm{C}_{60}$, shown in Figure S3, where the metal states hybridize
with the carbon states, thus changing deeply the electronic structure around
the Fermi level. An interesting case is Ca@$\mathrm{C}_{60}$ where the charge
transfer shifts the Fermi level to the DOS peak, inducing a high density of
states (see the last panel of Figure S3). Concerning the electronic structure
of the M@$\mathrm{C}_{24}$ series (also shown in section D of the SI, Figures
S5 and S6) the rigid-band model is not followed, with the exceptions of the
monovalent doped compounds (M=Li,Na,K) and Ca@$\mathrm{C}_{24}$. The insertion
of the dopant in the smaller $\mathrm{C}_{24}$ cage leads to the hybridization
of the dopant and carbon framework states changing deeply the band structure.

The phonon dispersion curves for the studied doped fullerite clathrates are
shown in the SI, section E. Only four doped clathrate structures do not
display imaginary phonon modes and are dynamically stable, these structures
being Li@$\mathrm{C}_{24}$, Na@$\mathrm{C}_{24}$, Ga@$\mathrm{C}_{24}$ and
Be@$\mathrm{C}_{60}$. The imaginary phonon modes of unstable doped clathrates
are either originated from the carbon framework or from mixed carbon-dopant
vibrations. In the first case, the charge transfer from the guest atom to the
carbon framework structure is observed to be nearly completed, while it is
clearly uncompleted in the latter case.

Superconducting behavior was explored for the four stable clathrates. As
referred above, the Be@$\mathrm{C}_{60}$ clathrate shows a very different
electronic band structure from that of the pristine fullerite clathrate (see
Figure 2 d). In particular, the band that crosses the Fermi level is the
result from the hybridization of beryllium and carbon framework states.
Indeed, the incomplete charge transfer from the dopant to the carbon structure
is confirmed from the integrated local DOS contours and the projected DOS
(PDOS) around the Fermi level (see Figures S11 and S12). The PDOS curve for
the Be@$\mathrm{C}_{60}$ shows electronic density from the dopant at the Fermi
level, while no density is observed for the Li@$\mathrm{C}_{24}$,
Na@$\mathrm{C}_{24}$, indicating in the latter case a complete charge transfer
to the framework carbon structure. Ga@$\mathrm{C}_{24}$ presents an
intermediate case where $\sim 1$ electron is donated to the carbon framework
but there is still a very low amount of states due to the dopant at the Fermi
level. This picture is corroborated by the computed Bader charges given in
Table S3.

The phonon dispersion curves shown in the left panels of Figure 3 present two
distinct zones separated around 1000 cm-1. A second gap around 1200 cm-1 is
also observed for Li@$\mathrm{C}_{24}$ and Be@$\mathrm{C}_{60}$ compounds. The
middle panels in Figure 3 present the projected phonon density of states
(PHDOS) showing that the dopant atoms contribute only to lower frequencies,
below 400 cm-1. The calculated Eliashberg spectral function,
$\alpha^{2}$F$(\omega)$, and the electron-phonon coupling constant,
$\lambda(\omega)$, are also shown in Figure 3 (right panels). The strongest
peaks in the Eliashberg spectral function arise at low energy, between 100 and
300 cm-1, for Be@$\mathrm{C}_{60}$ and thus these are the phonon frequencies
that will mostly contribute to the electron-phonon coupling ($\sim 80\%$) and
concomitantly to the superconducting Tc, see Figure 3 d). For the other
systems, the contribution to the Eliashberg spectral function is more evenly
distributed over all frequencies leading to a smother growth of the integrated
electron-phonon coupling curve, Figures 3 a), b) and c). Hence, beryllium is
the only dopant that contributes significantly to the Eliashberg spectral
function, wich is in agreement with the fact that it is the sole dopant whose
electronic states strongly hybridize with the carbon states. The integrated
electron-phonon coupling constant is always lower than 0.5, as indicated in
Table 1, considerably smaller than the 2.92 reported for the sodium doped
sodalite-like carbon clathrate [sodalite], and thus the Tc’s should not be
high. Indeed, using the Allen-Dynes modified McMillan equation with a Coulomb
screening potential $\mu^{*}$=0.1 the Tc’s obtained are quite low: 1.81 K,
0.23 K, 0.99 K and 0.88 K, respectively for Li@$\mathrm{C}_{24}$,
Na@$\mathrm{C}_{24}$, Ga@$\mathrm{C}_{24}$ and Be@$\mathrm{C}_{60}$
clathrates. We also solved the isotropic Eliashberg equations obtaining the
superconducting gaps for all the stable structures, shown in Figure 4. The
calculated Tc’s are 2.03 K, 0.40 K, 1.29 K and 1.14 K for
Li@$\mathrm{C}_{24}$, Na@$\mathrm{C}_{24}$, Ga@$\mathrm{C}_{24}$ and
Be@$\mathrm{C}_{60}$, respectively, in agreement with those referred above.

Figure 4: Superconducting gap for the doped $\mathrm{C}_{60}$ clathrate
structure: Li@$\mathrm{C}_{24}$ (blue curve); Na@$\mathrm{C}_{24}$ (orange
curve); Ga@$\mathrm{C}_{24}$ (yellow curve); Be@$\mathrm{C}_{60}$ (purple
curve).

The electron-phonon coupling constants, for each high-symmetry q-point, are
given in Table 1. The Na@$\mathrm{C}_{60}$ clathrate has been included in this
list because, despite being unstable, it has a higher electron-phonon coupling
constant than all the stable doped clathrates at X and M q-points ( excluding
the X point for Be@$\mathrm{C}_{60}$ clathrate). In addition, it has, by far,
the highest density of states at the Fermi level, N(EF). If this unstable
structure could be turned dynamically stable, by accounting anharmonic
[LaH10_Ion, ion_nature, ion_prl] and/or pressure effects a serious increase in
the Tc would then be expected. Considering that its softest phonon is barely
imaginary, $56i$ cm-1, it is possible that such effects will be sufficient to
turn the structure stable, as was observed for the superconductor LaH10
[LaH10_Ion] and for the ferroelectric SnTe [SnTe_Ion].

Table 1: Superconducting properties of a few selected systems. The electron-
phonon coupling constant at R q-point is not present for Na@C60 due to the
existence of imaginary modes. The number of density of states at the Fermi
level N(EF) is given in state per spin per meV and per atom in the last
column.

| $\lambda$ | $\lambda$ | $\lambda$ | $\lambda$ | $\omega_{log}$ | TC | N(EF)
---|---|---|---|---|---|---|---
| X q-point | M q-point | R q-point | integrated | cm-1 | K | state/spin/eV/atom
Li@$\mathrm{C}_{24}$ | 0.3740 | 0.3368 | 0.2801 | 0.41 | 254.69 | 1.81 | 63.0
Na@$\mathrm{C}_{24}$ | 0.3170 | 0.2326 | 0.1597 | 0.30 | 342.87 | 0.23 | 63.3
Ga@$\mathrm{C}_{24}$ | 0.3716 | 0.2263 | 0.3158 | 0.36 | 289.45 | 0.99 | 56.4
Be@$\mathrm{C}_{60}$ | 0.9176 | 0.1835 | 0 | 0.39 | 178.34 | 0.88 | 21.4
Na@$\mathrm{C}_{60}$ | 0.6142 | 0.7207 | - | - | - | - | 105.7

## Conclusion

The stability and superconducting properties of the fullerite clathrate
structures doped with simple metals has been explored. Only four structures
were found to be stable: Li@$\mathrm{C}_{24}$, Na@$\mathrm{C}_{24}$,
Ga@$\mathrm{C}_{24}$ and Be@$\mathrm{C}_{60}$. They all show superconducting
critical temperatures bellow 4 K, due to the weak electron-phonon coupling and
the small number of states at the Fermi level. These two parameters,
particularly the latter, are higher in the barely unstable
Na@$\mathrm{C}_{60}$, indicating that the stabilization of this compound would
result in a superconductor with Tc much higher than those reported here. We
are currently exploring this possibility by explicitly considering anharmonic
and pressure effects.

###### Acknowledgements.

This work was developed within the scope of the project CICECO-Aveiro
Institute of Materials, UIDB/50011/2020, UIDP/50011/2020 & LA/P/0006/2020,
financed by national funds through the FCT/MCTES (PIDDAC) and IF/00894/2015
finances by FCT. J. Laranjeira acknowledges a PhD grant from FCT
(SFRH/BD/139327/2018). I.E. and Đ.D. acknowledge funding from the European
Research Council (ERC) under the European Union’s Horizon 2020 research and
innovation program (Grant Agreement No. 802533) and the Department of
Education, Universities and Research of the Eusko Jaurlaritza and the
University of the Basque Country UPV/EHU (Grant No. IT1527-22). The work has
been performed under the project HPC-EUROPA3 (INFRAIA-2016-1-730897), with the
support of the EC Research Innovation Action under the H2020 Programme; in
particular, the author gratefully acknowledges the support of the computer
resources and technical support provided by BSC.


# End-to-end Temporal Action Detection with Transformer

Xiaolong Liu1 Qimeng Wang1 Yao Hu2 Xu Tang2 Song Bai1 Xiang Bai1
1Huazhong University of Science and Technology 2Alibaba Group
{liuxl, qimengwang<EMAIL_ADDRESS><EMAIL_ADDRESS>
{yaoohu<EMAIL_ADDRESS>Work done during an internship at Alibaba
Group.

###### Abstract

Temporal action detection (TAD) aims to determine the semantic label and the
boundaries of every action instance in an untrimmed video. It is a fundamental
and challenging task in video understanding and significant progress has been
made. Previous methods involve multiple stages or networks and hand-designed
rules or operations, which fall short in efficiency and flexibility. In this
paper, we propose an end-to-end framework for TAD upon Transformer, termed
TadTR, which maps a set of learnable embeddings to action instances in
parallel. TadTR is able to adaptively extract temporal context information
required for making action predictions, by selectively attending to a sparse
set of snippets in a video. As a result, it simplifies the pipeline of TAD and
requires lower computation cost than previous detectors, while preserving
remarkable detection performance. TadTR achieves state-of-the-art performance
on HACS Segments (+3.35% average mAP). As a single-network detector, TadTR
runs 10$\times$ faster than its comparable competitor. It outperforms existing
single-network detectors by a large margin on THUMOS14 (+5.0% average mAP) and
ActivityNet (+7.53% average mAP). When combined with other detectors, it
reports 54.1% mAP at IoU=0.5 on THUMOS14, and 34.55% average mAP on
ActivityNet-1.3. Our code will be released at https://github.com/xlliu7/TadTR.

##  1 Introduction

Video understanding has become more important than ever as the rapid growth of
media prompts the generation, sharing, and consumption of videos. As a
fundamental task in video understanding, temporal action detection (TAD) aims
to predict the semantic label, the start time, and the end time of every
action instance in an untrimmed and possibly long video. For its wide range of
applications, including security surveillance, home care, video editing, and
so on, temporal action detection has gained increasing attention from the
community in recent years [31, 37, 25, 28, 4, 20].

Figure 1:  Comparison of recent temporal action detection methods on THUMOS14,
in terms of both performance (average mAP) and speed. Our method state-of-the-
art performance while running more than 3$\times$ faster. When combined with a
two-stage method P-GCN [43], it achieves 51.6% average mAP, and is still
faster than many of the compared methods.

Many previous TAD methods have complicated pipelines. As shown in Fig. 2,
these pipelines involve multiple steps, such as segment scoring, proposal
generation, action classification, and post-processing. While these methods
achieve promising performance on standard benchmarks, there are several
limitations. Firstly, the multiple steps may increase the time cost since they
need to be carried out sequentially. Some stages, such as the proposal
classification stage, are time-consuming when there is a large number of
candidates ($10^{3}\sim 10^{4}$). Secondly, the hand-designed operations, such
as anchor setting and ground truth assignment, restrict the flexibility of a
method. Thirdly, they can not enjoy end-to-end training. For example, some
multi-stage methods, such as BSN, BMN, and G-TAD, need to train at least two
different networks to get action detection results. Although some methods
unify all steps in a single network, they require non-differentiable steps
such as non-maximal suppression (NMS) and grouping. As a consequence, multiple
stages may accumulate errors and lead to sub-optimal results.

In this paper, we propose a simple end-to-end temporal action detector. The
detector adopts a set-prediction pipeline that directly maps a set of
learnable embeddings to action instances in parallel. This is different from
previous methods that achieve action detection by classifying pre-defined
anchors or proposals. In these methods, it is straightforward to extract
features in certain time intervals according to the locations of the anchors
or proposals. The set-prediction pipeline, however, requires to extract
sufficient temporal information before knowing which interval an action falls
in. In other words, the network should learn where to look at when predicting
actions. To do that, we design a Transformer architecture that adaptively
extracts the temporal context required by each action prediction and updates
the initial embeddings with the extracted context. The core is the temporal
deformable attention module that dynamically attends to a sparse set of key
snippets in the video, similar to the way human finds actions in a video. It
is more flexible than previous works that extract context information by
increasing the receptive field by a fixed ratio [9, 8]. Furthermore, it
requires less computation cost than the vanilla self-attention module [33]
that attends to all locations in an input sequence. Upon each embedding
enhanced with video context, two feed-forward networks (FFNs) predict the
class and the segment of the corresponding action prediction. To better rank
the actions, an actionness regression head is attached to predict the
actionness (confidence score, similar to objectness in object detection) of
these action predictions from the context aligned with the corresponding
segment. During training, a segment matching module dynamically determines a
one-to-one ground truth assignment. Owing to this, our detector avoids
duplicate detections and NMS is unnecessary. It produces a very sparse set of
action detections ($10\sim 10^{2}$), orders of fewer than previous methods.

The temporal action detection Transformer, dubbed as TadTR, achieves
remarkable performance with low computation cost. Without bells and whistles,
TadTR achieves state-of-the-art performance on HACS Segments and THUMOS14.
When combined with existing methods, it reaches 51.6% average mAP on THUMOS14
[15] and 34.55% average mAP on ActivityNet-1.3 [5], outperforming strong
competitors such as G-TAD [37] and P-GCN [43]. In terms of efficiency, it
takes only 155 ms per video with a single network, which is more than
3$\times$ faster than recent state-of-the-art methods, as shown in Fig. 1.

The contributions of this work are three-fold:

  * •

We propose an end-to-end set-prediction-based detector that simplifies the
pipeline for temporal action detection.

  * •

We propose a Transformer architecture that is able to adaptively capture
temporal context for temporal action detection. To the best of our knowledge,
this is the first work that explores adaptive context for TAD.

  * •

Our method achieves state-of-the-art or highly competitive performance on
standard datasets and requires lower computation cost than its competitors.

Figure 2:  Comparison of different pipelines of temporal action detection. (a)
Multi-stage pipeline in [31, 37] _etc_. (b) Two-stage pipeline in [36, 8]. (c)
Top-down one-stage pipeline in [17], (d) Bottom-up one-stage pipeline in [42]
(e) An end-to-end pipeline in this work.

##  2 Related Works

###  2.1 Temporal Action Detection Methods

Previous methods for temporal action detection can be roughly categorized into
groups depending on the pipeline.

(a) Multi-stage methods [37, 16, 41]. They first generate candidate segments
and train a binary classifier that associates each segment with a confidence
score, resulting in proposals. Those proposals with high scores are fed to a
multi-class classifier to classify the actions. The candidate segments are
generated by dense uniform sampling [31, 12] or grouping local frames that may
contain actions [18]. Some methods [11, 21] combine multiple schemes for
complementarity. (b) Two-stage methods [36, 8, 10, 3] simplify the multi-stage
pipeline by adopting a one-stage proposal generator, which directly predicts
the scores and boundaries of pre-defined multi-scale anchors associated with
each temporal location. These methods need to manually set multiple anchor
scales, which restricts the flexibility. (c) Top-down one-stage methods [17,
22] can be seen as the class-aware variant of the one-stage proposal
generator. Recently, [38] augment the anchor-based one-stage detector with an
anchor-free branch that makes predictions based on centers of actions. (d) A
bottom-up one-stage method proposed in [42] first predicts the action and
boundary probabilities and then groups frames with maximal structured sum as
actions. We note that all the above methods require post-processing steps such
as NMS or grouping, which prevent end-to-end learning.

An early work by [39] proposes an end-to-end temporal action detector based on
recurrent neural networks (RNN) and reinforcement learning (RL). It learns
action detection by training an agent that iteratively picking an observation
location and deciding whether to emit or refine a candidate action after
observation. In this work, an end-to-end Transformer-based detector
significantly different from [39] is proposed. While [39] is restricted to a
manner of one-by-one prediction due to the recurrent nature of RNN, our method
can decode all action instances in parallel. In addition, our method is fully
differentiable, whereas the RL-based method is not.

All the methods discussed above are fully supervised methods. There are also
some weakly supervised methods [24, 19, 26, 27, 30, 34, 40], which are beyond
the scope of this paper.

###  2.2 Transformers for Video Understanding

Transformers [33] have achieved great success in natural language processing.
The core of Transformer is the self-attention mechanism that aggregates non-
local cues through a weighted sum of features at attended locations. Compared
with convolutions, self-attention can capture long-range context and
dynamically adjust weights according to the input. Recently, many works have
revealed the great potential of Transformers in video understanding tasks. For
example, VideoBert [32] and ActBert [49] utilize Transformers to learn an
joint representation for video and text. TimeSformer [2] decouples spatial and
temporal self-attention for video classification. Zhou  _et al_. [48] capture
the temporal dependency with Transformer for video captioning. Girdhar  _et
al_. [13] apply Transformer to model the relationship between spatial
proposals for spatio-temporal action detection.

In this paper, Transformer is used to capture temporal context information for
temporal action detection. Specifically, We employ attention modules to model
the relationship between video snippets, the relationship between actions and
frames, and the relationship between actions. Different from many previous
works where an attention module attends to all locations in the input
sequence, we introduce a temporal deformable attention module that adaptively
attends to a sparse set of key frames instead of all frames. Therefore,
context information can be extracted without excessive computation cost.
Besides adaptive context, we also employ an actionness regression head to
extract aligned context in order to assign reliable confidence scores to the
detections by Transformer.

##  3 TadTR

Figure 3: The architecture of TadTR. It takes the video features extracted
with a CNN and position embedding as inputs and decodes a set of action
predictions in parallel via a Transformer. The segment matching module
determines a one-to-one ground truth assignment during training. The
actionness regression head estimates the IoU with its best-matched ground
truth for each prediction. The actionness score is used for ranking and
filtering.

TadTR is constructed on video features encoded with a pre-trained video
classification network (e.g. I3D [7]). Fig. 3 shows the overall architecture
of TadTR. TadTR takes the video features and the temporal position encoding as
input and outputs a set of action predictions. It consists of a Transformer
encoder to model the interactions between frames, a Transformer decoder to
predict action segments, and an extra actionness regression head to estimate
the confidence score of the predicted segments. A segment matching module is
used for ground truth assignments during training.

###  3.1 Architecture

Encoder. Let $X_{V}\in\mathbb{R}^{T_{S}\times C_{V}}$ denotes the video
feature sequence, where $T_{S}$ and $C_{V}$ are the length and dimension,
respectively. Each frame in the feature sequence is a feature vector extracted
from a certain snippet in the video. Here, a snippet means a sequence of a few
(e.g., 8) consecutive frames. A 1D convolutional layer with kernel size 1 is
applied to reduce its dimension to $C$. The encoder models the relation
between different snippets and outputs a feature sequence
$X_{E}\in\mathbb{R}^{T_{S}\times C}$ enhanced with temporal context. As
depicted in Fig. 3, it consists of $L_{E}$ Transformer encoder layers of the
homogeneous architecture, where the key component is the attention module.

The primitive dense attention module in [33] attends to all locations in an
input feature sequence, suffering from high computation cost and slow
convergence. To mitigate these issues, we employ a temporal deformable
attention (TDA) module that adaptively attends to a sparse set of temporal
locations around a reference location in the feature sequence.

Let $z_{q}\in\mathbb{R}^{C}$ be denote the feature of query $q$ and
$t_{q}\in[0,1]$ be the normalized coordinate of the corresponding reference
point. Given an input feature sequence $X\in\mathbb{R}^{T_{S}\times C}$, the
output $h_{m}\in\mathbb{R}^{T_{S}\times(C/M)}$ of the $m$-th
($m\in\\{1,2,...,M\\}$) head of a TDA module is computed by an weighted sum of
a set of key elements sampled from $X$:

$h_{m}=\sum_{k=1}^{K}a_{mqk}\mathbf{W}^{V}_{m}X(t_{q}+\Delta t_{mqk}),$ (1)

where $K$ is the number of sampling points, $a_{mqk}\in[0,1]$ is the
normalized attention weight, and $\Delta t_{mqk}\in[0,1]$ is the sampling
offset relative to $t_{q}$. $X(t_{q}+\Delta t_{mqk})$ is the linear
interpolated feature at $t_{q}+\Delta t_{mqk}$ as it is fractional. Following
[50], the attention weight $a_{mqk}$ and the sampling offset $\Delta t_{mqk}$
are predicted from the query feature $z_{q}$ by linear projection. We
normalize the attention weight with softmax to make $\sum_{k=1}^{K}a_{mqk}=1$.
$\mathbf{W}^{V}_{m}\in\mathbb{R}^{C\times(C/M)}$ is a learnable weight. The
output of TDA is computed by a linear combination of the outputs of different
heads:

$TDA(z_{q},t_{q},X)=\mathbf{W}^{O}Concat(h_{1},h_{2},...,h_{m}),$ (2)

where $\mathbf{W}^{O}\in\mathbb{R}^{C\times C}$ is a learnable weight.

When computing the $\tau$-th frame in the output sequence, the query and the
reference point are both the $\tau$-th frame in the input sequence. Therefore,
we refer to TDA in the encoder as temporal deformable self-attention (TDSA).
The query feature is the summation of the feature of that frame and the
position embedding at that location. The position embedding is used to
differentiate different locations in the input sequence. In this paper, we use
the sinusoidal position embedding following [33]:

$X_{P}(\tau,\gamma)=\begin{cases}\sin{\frac{\tau}{10000^{\gamma/C}}}&\gamma~{}\text{is
even}\\\ \cos{\frac{\tau}{10000^{(\gamma-1)/C}}}&\gamma~{}\text{is
odd}\end{cases}.$ (3)

Decoder. The decoder takes as inputs the encoder features
$X_{E}\in\mathbb{R}^{T_{S}\times C}$ and $N_{q}$ action queries
$Q=\\{q_{i}\\}_{i=1}^{N_{q}}$ with learnable embeddings
$\\{\hat{z}_{q_{i}}\\}_{i=1}^{N_{q}}$ and transforms these embeddings to
$N_{q}$ action predictions $\hat{Y}=\\{\hat{y}_{i}\\}$. As illustrated in Fig.
3, the decoder consists of $L_{D}$ sequential decoder layers. In each decoder
layer, self-attention and temporal deformable cross-attention (TDCA) are both
used. The self-attention module models the relation between action queries and
updates their embeddings. The motivation here is that multiple actions in one
video are often related. For example, a cricket shot action often appears
after a cricket bowling action. To make an action prediction, each query needs
to extract relevant context information from the video. To do that, we pass
both the encoder features $X_{E}$ and the query feature
$\hat{z}_{q_{i}}\in\mathbb{R}^{C}$ to the TDCA module. The output is
formulated as $TDA(\hat{z}_{q_{i}},\hat{t}_{q_{i}},X_{E})$. Here,
$\hat{t}_{q_{i}}$ is the coordinate of the reference point in $X_{E}$. It is
predicted from the query embedding by a linear layer and normalized by sigmoid
to make $\hat{t}_{q_{i}}\in[0,1]$. The reference point can be seen as the
initial estimation of the center of the corresponding action segment.

Different from TDSA in the encoder, the query embedding $\hat{z}_{q_{i}}$ and
the reference point is learnable. This allows the network to learn the global
distribution of the action locations in the training dataset, which is more
flexible than hand-crafted anchor setting or proposal sampling. An analysis is
given in Sec. 4.4.

Based on the output of each decoder layer, we can apply two feed-forward
networks to predict the classification probabilities $\hat{p_{i}}$ and the
temporal segment $\hat{s}_{i}=(\hat{t}_{i},\hat{d}_{i})$ of the detection
$\hat{y}_{i}$ mapped from each query $q_{i}$. $\hat{t}_{i}$ and $\hat{d}_{i}$
are the coordinate of the center and the normalized length of the detection,
respectively. To make the boundaries of the detection more accurate, a segment
refinement is proposed, which is detailed below.

Segment Refinement. Transformer is able to capture long-range context
information. However, the predicted action boundaries might be unsatisfactory
for lack of locality. Therefore, we introduce a refinement mechanism to
improve the localization performance. Based on the action segment
$\hat{s}_{i}^{(l)}=(\hat{t}_{i}^{(l)},\hat{d}_{i}^{(l)})$ predicted by the
$l$-th ($1\leq l\leq L_{D}-1$) decoder layer, the $(l+1)$-th decoder layer
refine its locations to make a new segment
$\hat{s}_{i}^{(l+1)}=(\hat{t}_{i}^{(l+1)},\hat{d}_{i}^{(l+1)})$:

$\displaystyle\hat{t}_{i}^{(l+1)}$ $\displaystyle=\sigma(\Delta
t_{i}^{(l+1)}+\sigma^{-1}(\hat{t}_{i}^{(l)}))$ (4)
$\displaystyle\hat{d}_{i}^{(l+1)}$ $\displaystyle=\sigma(\Delta
d_{i}^{(l+1)}+\sigma^{-1}(\hat{d}_{i}^{(l)})),$ (5)

where $\hat{t}_{i}^{(l)}$ and $\hat{d}_{i}^{(l)}$ are the coordinate of the
center and the length of $\hat{s}_{i}^{(l)}$, respectively. $\Delta
t_{i}^{(l+1)}$ and $\Delta d_{i}^{(l+1)}$ are the predicted offsets.
$\sigma(\cdot)$ and $\sigma^{-1}(\cdot)$ are the sigmoid and the inverse
sigmoid function, respectively. The center of the predicted segment at the
$l$-th decoder layer is used as the reference point of the next decoder layer.
In this way, the decoder learns to pay more attention to the locations around
the estimated segment gradually, so that the boundaries can be more accurate.

Actionness Regression. One challenge of temporal action detection is the
extreme scale (duration) variation of action instances, ranging from a few
seconds to several minutes. Although the attention module can dynamically
attend features in different ranges, it cannot always perfectly align the
receptive field with the span of target actions. As a result, the
classification score of the detections, predicted with adaptive context, may
be unreliable for ranking. An example is shown in Fig. 4.

Figure 4: The Transformer may generate unreliable confidence scores. Here, a
prediction with lower overlap with the curling action has a higher score than
a more accurate prediction (0.50  _vs_. 0.39).

To mitigate this issue, we propose to regress the actionness of a detection
with hindsight. We employ an actionness regression head that extracts context
aligned with the interval of a predicted segment and predicts an actionness
score upon it. Given the encoder feature sequence $X_{E}$ and a predicted
segment $\hat{s}_{i}$ by the decoder, we first apply temporal RoIAlign [14]
upon $X_{E}$ to obtain the aligned features
$X_{s_{i}}\in\mathbb{R}^{T_{R}\times C}$ within the interval defined by
$s_{i}$ from $X_{E}$. Here, $T_{R}$ is the number of bins for RoIAlign. As
pointed out in previous works [46, 8, 9], the context information outside the
segment is useful for assessing whether a segment is complete. Therefore, we
slightly expand the segment by a factor of $\epsilon$ when applying RoIAlign.
The expanded segment can be expressed as $(\hat{t}_{i},\epsilon\hat{d}_{i})$.
Then, a feed-forward network is used to predict the actionness score
$\hat{g_{i}}$ from the aligned feature. $\hat{g_{i}}$ is supervised by the
maximal IoU $g_{i}$ (intersection over union) between $s_{i}$ and all ground
truth actions. In this way, the prediction with the highest IoU is expected to
be ranked first among all predictions.

One may argue that the actionness regression head makes TadTR a two-stage
detector. In fact, TadTR is significantly different from traditional two-stage
methods. First, TadTR can already produce complete action detections without
actionness regression. The actionness regression head is simply for more
reliable confidence scores. In contrast, a two-stage detector is unable to
make complete predictions without the second stage. Second, TadTR produces
very sparse action predictions. It does require a preliminary stage to
evaluate dense candidates as two-stage methods do. Therefore it requires low
computation cost.

For convenience, we denote TadTR without actionness regression and segment
refinement as TadTR-base.

###  3.2 Training and Inference

Segment Matching. The segment matching module determines the targets assigned
to each detection during training. Inspired by DETR [6] in object detection,
we frame it as a set-to-set bipartite matching problem and solve it with the
Hungarian algorithm.

Let $Y=\\{y_{j}\\}_{j=1}^{N_{q}}$ be a set of ground truth actions padded with
$\varnothing$ (no action) and $\pi$ be the permutation that assigns each
target $y_{j}$ to the corresponding detection $\hat{y}_{\pi({y_{j}})}$.
Bipartie matching aims to find a permutation that minimizes the overall
matching cost:

$\hat{\pi}=\arg\min\sum_{j=1}^{N_{q}}\mathcal{C}(y_{j},\hat{y}_{\pi(j)}).$ (6)

The matching cost considers the classification probabilities and the distance
between ground truth and predicated segments. Specifically,
$\mathcal{C}(y_{j},\hat{y}_{\pi(j)})$ is defined as

$-\mathbbm{1}_{c_{j}\neq\varnothing}[(\hat{p}_{\pi(j)}(c_{j})+\mathcal{L}_{seg}(s_{j},\hat{s}_{\pi(j)})],$
(7)

where $c_{j}$ and $s_{j}$ are the class label and the temporal segment of
$y_{j}$. $\mathcal{L}_{seg}(s_{j},\hat{s}_{\pi(j)})$ is the distance between
the predicted location and the ground truth location, defined as

$\lambda_{iou}\mathcal{L}_{iou}(s_{j},\hat{s}_{\pi(j)})+\lambda_{coord}\mathcal{L}_{L1}(s_{j},\hat{s}_{\pi(j)}),$
(8)

where $\mathcal{L}_{L1}$ is the $L_{1}$ distance and $\mathcal{L}_{iou}$ is
the IoU loss. IoU loss is defined as the the opposite number of the IoU.
$\lambda_{iou}$ and $\lambda_{coord}$ are hyper-parameters.

Through the set-based segment matching, each ground truth will be assigned to
only one prediction, thus avoiding duplicate predictions. This brings two
merits. First, TadTR does not rely on the non-differentiable non-maximal
suppression (NMS) for post-processing and enjoys end-to-end training. Second,
we can make sparse predictions with limited queries (_e.g_. 10) instead of
dense predictions in many previous works (_e.g_. tens of thousands for BMN
[16] and G-TAD [37]), which saves the computation cost.

In a way, the segment matching module performs a learnable NMS. The matching
cost takes the classification scores of the detections into account. In this
way, those detections with lower scores are more likely to be assigned with a
non-action target. As a result, their classification scores will be suppressed
in the training process.

Loss Functions. Once the ground truth assignment is determined, we optimize
the network by minimizing the following multi-part loss functions:

$\displaystyle\mathcal{L}=\sum_{j=1}^{N_{q}}[-\log\hat{p}_{\hat{\pi}(j)}(c_{j})+\mathbbm{1}_{c_{j}\neq\varnothing}\mathcal{L}_{seg}(s_{j},\hat{s}_{\hat{\pi}(j)})$
(9)
$\displaystyle+\lambda_{act}\mathcal{L}_{L1}(\hat{g}_{\hat{\pi}(j)},g_{\hat{\pi}(j)})],$

where the first two items optimize the detections from the decoder and the
last optimizes the outputs of actionness regression. $\hat{\pi}$ is the
solution of Equation 6. $\lambda_{act}$ is a hyper-parameter.

Inference. During inference, we ignore the action predictions from all but the
last decoder layer. The production of the actionness score and the
classification probabilities predicted by the decoder is used for detection
ranking.

Table 1: Comparison with state-of-the-art methods on THUMOS14. Run time is the average inference time per video, including post-processing operations, such as NMS. For proposal generation methods (BSN, MGG, BMN, G-TAD, MR, BC-GNN) in the first group, the run time of the classification stage is not included as the classification models are not available. E2E: end-to-end. TS: two-stream. * Results copied from [38]. ** Our implementation. Method | Feature | Single-network | E2E |  0.3 |  0.4 |  0.5 |  0.6 |  0.7 |  Avg. | Run time (ms) | FLOPs (G)
---|---|---|---|---|---|---|---|---|---|---|---
Yeung  _et al_. [39] | VGG16 | ✓ | ✓ |  36.0 |  26.4 |  17.1 |  - |  - |  - | 195 | 3.9
Yuan  _et al_. [42] | TS | - | - |  36.5 |  27.8 |  17.8 |  - |  - |  - | - | -
SSAD [17] | TS | ✓ | - |  43.0 |  35.0 |  24.6 |  - |  - |  - | - | -
R-C3D [36] | C3D | ✓ | - |  44.8 |  35.6 |  28.9 |  - |  - |  - | - | -
SSN [47] | TS | - | - |  51.9 |  41.0 |  29.8 |  - |  - |  - | - | -
TAL-Net [8] | I3D | ✓ | - |  53.2 |  48.5 |  42.8 |  33.8 |  20.8 |  39.8 | - | -
BSN [18] | TS | - | - |  53.5 |  45.0 |  36.9 |  28.4 |  20.0 |  36.8 | 2065 | 3.4
MGG [21] | TS | - | - |  53.9 |  46.8 |  37.4 |  29.5 |  21.3 |  37.8 | - | -
BMN [16] | TS | - | - |  56.0 |  47.4 |  38.8 |  29.7 |  20.5 |  38.5 | 483 | 171.0
BC-GNN [1] | TS | - | - |  57.1 |  49.1 |  40.4 |  31.2 |  23.1 |  40.2 | - | -
G-TAD [37] | TS | - | - |  54.5 |  47.6 |  40.2 |  30.8 |  23.4 |  39.3 | 4440 | 639.8
BMN* (I3D) [16] | I3D | - | - |  56.4 |  47.9 |  39.2 |  30.2 |  21.2 |  39.0 | - | -
G-TAD** (I3D) [37] | I3D | - | - |  58.7 |  52.7 |  44.9 |  33.6 |  23.8 |  42.7 | 3552 | 368.9
MR [45] | I3D | - | - |  53.9 |  50.7 |  45.4 |  38.0 |  28.5 |  43.3 | 644 | 36.8
A2Net [38] | I3D | ✓ | - |  58.6 |  54.1 |  45.5 |  32.5 |  17.2 |  41.6 | 1554 | 30.4
P-GCN [43] | I3D | - | - |  63.6 |  57.8 |  49.1 |  - |  - |  - | 7298 | 4.4
P-GCN** [43] | I3D | - | - |  64.9 |  59.0 |  49.4 |  36.7 |  22.6 |  46.5 | 7298 | 4.4
TadTR (Ours) | I3D | ✓ | ✓ |  62.4 |  57.4 |  49.2 |  37.8 |  26.3 |  46.6 | 155 | 0.75
TadTR+P-GCN (Ours) | I3D | - | - |  70.8 |  63.5 |  54.1 |  41.7 |  28.0 |  51.6 | 1647 | 1.0

##  4 Experiments

###  4.1 Experimental Setup

Datasets and Evaluation Metrics. We conduct experiments on THUMOS14 [15], HACS
Segments [44], and ActivityNet-1.3 [5]. THUMOS14 is built on videos from 20
sports action classes. It contains 200 and 213 untrimmed videos for training
and testing. There are 3007 and 3358 action instances on the two sets.
ActivityNet-1.3 and HACS Segments share the same 200 classes of daily
activities. Both datasets are split into three sets: training, validation, and
testing. The numbers of videos in these sets are 10024, 4926, and 5044
respectively on ActivityNet-1.3, and 37613, 5981, and 5987 on HACS Segments.
On both datasets, the annotations on the testing set are reserved by the
organizers. Therefore, we evaluate on the validation set.

Following conventions, the mean average precision (mAP) at different IoU
thresholds is used for performance evaluation. On THUMOS14, the IoU thresholds
for computing mAPs are $[0.3:0.7:0.1]$. On the other two datasets, we report
mAPs at the thresholds $\\{0.5,0.75,0.95\\}$ and the average mAP at the
thresholds $[0.5:0.95:0.05]$.

Video Feature Extraction. Most TAD methods are based on offline extracted
video features. For easier comparison with them, we also use video features as
the input of our method. For experiments on HACS Segments, we directly use the
official I3D features111http://hacs.csail.mit.edu/hacs_segments_features.zip,
which are extracted with I3D trained on Kinetics at 2FPS. On the other
datasets, we use the commonly used features in previous works. On THUMOS14,
the two-stream I3D [7] networks pre-trained on Kinetics [7] are taken as the
video encoder, and the features are extracted every 8 frames. On
ActivityNet-1.3, we use the two-stream TSN [35] features extracted at 5FPS.
Following previous works [16, 37], we resize the video features to a fixed
length of 100 via linear interpolation on ActivityNet-1.3 and HACS Segments.

Implementation Details. The dimension of the feed-forward network is 2048. $C$
is set to 256. $L_{E}$ and $L_{D}$ are set to 2 and 4, respectively. The loss
weights $\lambda_{iou}$, $\lambda_{coord}$ and $\lambda_{act}$ are set to 2, 5
and 5 respectively. The numbers of attention heads $M$ and sampling points $K$
are set to 8 and 4, respectively. The parameters of the linear layers that
predict attention weights are initialized to zero. We initialize the linear
layers that predict sampling offsets to make $\\{\Delta
p_{mqk}\\}_{m=1}^{8}=(k,0,-k,0,k,0,-k,0)$ at initialization. The expanding
factor $\epsilon$ and the number of bins $T_{R}$ for RoIAlign in the
actionness regression head are 1.5 and 16 respectively.

TadTR is trained using AdamW [23] optimizer. The initial learning rate is
$2\times 10^{-4}$ and scaled by a factor of 0.1 after training for a certain
number of epochs. The learning rates of the linear projection layers for
predicting attention weights and sampling offsets are multiplied by 0.1. We
train the models for 30, 15, and 20 epochs and decrease the learning rate
after 25, 12, and 15 epochs on THUMOS14, ActivityNet-1.3, and HACS Segments
respectively. The batch size is set to 16.

The experiments are conducted on a workstation with a single Tesla P100 GPU
card, and Intel(R) Xeon(R) CPU E5-2682 v4 @ 2.50GHz. It takes around 40
minutes, 1.5 hours, and 7 hours to finish training on THUMOS14,
ActivityNet-1.3, and HACS Segments, respectively.

###  4.2 Main Results

THUMOS14. Table 1 demonstrates the temporal action detection performance and
run time comparison on the testing set of THUMOS14. We measure the run time of
these methods with publicly available implementations under the same
environment (a single P100 GPU). We run methods on the full testing set with
batch size set to 1 and report the average time and FLOPs per video. The
average length of videos on THUMOS14 is 217 seconds. BMN [16] and G-TAD [37]
use two-stream TSN [35] features originally. For fair comparison, we also
report their performance with I3D features. We observe that:

1) Compared with previous methods, TadTR achieves the best performance in
terms of average mAP.

2) Compared with the strongest single-network method A2Net [38], TadTR runs
10$\times$ faster with 5.0% higher average mAP. A2Net is slow due to the heavy
network and a large number of anchors.

3) Compared with the second best method P-GCN [43], TadTR achieves slightly
better performance with 46$\times$ faster inference speed. We note that P-GCN
has relatively lower FLOPs than other methods, as it contains no 1D
convolution layers for temporal modelling. However, the graph structure
restricts its speed. It needs to extract the features of extra 24 neighboring
proposals to compute the detection score of a proposal.

4) The performance of TadTR can be further boosted when it is combined with
P-GCN. Specifically, we pass the detections from TadTR as proposals to the
proposal classification stage of P-GCN and use the detection scores by P-GCN.
Notably, TadTR achieves an mAP of 54.1% at IoU=0.5, which is 5.1% higher than
primitive P-GCN. The result reveals the great potential of TadTR. As for run
time, TadTR+P-GCN is much faster than primitive P-GCN, as it has fewer
detections than its original proposal generator.

The above results indicate that TadTR is both accurate and efficient. We also
note that many other methods are composed of multiple networks for different
functions, while TadTR is able to perform action detection with only a single
unified network.

We note that our method is not directly comparable with [39], since it is a
very early work and uses different features. The goal of this paper is not to
compete against it. Instead, we aim to explore an alternative end-to-end
framework with adaptive context and validate whether it can achieve
competitive performance over more complicated and well optimized methods. The
computation cost of TadTR is also not comparable with those methods that
directly take image as input, such as R-C3D. Most previous works use video
features as input and focus on the design of detection network.

HACS Segments. This is a recently introduced dataset, therefore the results of
many early methods on this dataset are not available. We report the
performance of TadTR, SSN [46], and the state-of-the-art method G-TAD [37] in
Table 2. Our method achieves an average mAP of 30.83%, which clearly
outperforms SSN (+11.86% mAP) and G-TAD (+3.35% mAP). Besides, our method
requires 455$\times$ less computation cost than G-TAD. As for run time, the
network inference and post-processing step of G-TAD take 33 ms and 908 ms per
video, respectively. The total run time is 941 ms, 49.5$\times$ that of TadTR
(19 ms). The results again illustrate the superiority of TadTR.

Table 2:  Comparison of different methods and different variants of TadTR on the validation set of HACS Segments. The results of SSN are from [44].  Method |  0.5 |  0.75 |  0.95 |  Avg. | FLOPs (M)
---|---|---|---|---|---
SSN [47] |  28.82 |  18.80 |  5.32 |  18.97 | -
G-TAD [37] |  41.08 |  27.59 |  8.34 |  27.48 | 45725.9
Ours
TadTR |  45.16 |  30.70 |  11.78 |  30.83 | 100.5
TadTR w/o encoder |  39.65 |  26.99 |  9.08 |  26.94 | 95.3
TadTR w/o instance-level context |  43.11 |  29.97 |  10.43 |  29.70 | 66.8
TadTR with a 1D CNN encoder |  40.96 |  28.00 |  9.73 |  27.95 | 134.8
TadTR with dense attention |  22.76 |  12.52 |  4.19 |  13.58 | 564.5
TadTR w/o actionness regression |  42.10 |  28.44 |  10.23 |  28.51 | 99.4
TadTR w/o segment refinement |  39.89 |  28.03 |  9.54 |  27.65 | 99.8

ActivityNet-1.3. In Table 3, we compare temporal action detection performance
of different methods on the validation set of ActivityNet-1.3. Some methods,
such as G-TAD, only implement action proposal generation and cannot produce
action detections without external action classifiers. We divide the methods
into two groups according to whether external action classifiers are used.
Being simple and end-to-end trainable, TadTR achieves an average mAP of
27.75%, which is stronger than more complicated methods such as TAL-Net [8]
and P-GCN [43]. Compared with the previous best single-network detector TAL-
Net, we improve the performance by 7.53%.

To enhance the detection performance, we also try combining TadTR with an
ensemble of action classifiers trained by [47] following previous works [38,
37]. To be concrete, we pass the detections by TadTR to the classifiers and
fuse the classification scores of TadTR and the classifiers by multiplication.
When fused with [46], TadTR enjoys a significant performance boost to an
average mAP of 32.27%. The results are better than recent methods such as MR
[45] and A2Net [38] that are also combined with [47].

To make a comparison with highly optimized two-stage methods, such as BMN [16]
and G-TAD [37], we also develop a two-stage variant TadTR+BMN. This is
implemented by combining the encoder of TadTR and the detection head of BMN.
Owing to the adaptive context captured by the Transformer encoder, TadTR+BMN
achieves an improvement of 0.7% over BMN, reaching 34.55% average mAP. It also
outperforms the recent method G-TAD. This indicates the great potential of
Transformer in temporal action detection. The result is also much higher than
vanilla TadTR. We conjecture that the performance bottleneck in ActivityNet is
the confidence prediction of detections. Two-stage methods have orders of
magnitude more candidates, providing sufficient training samples for the
confidence prediction head. In the future, we plan to investigate better
strategies to train the confidence prediction head of TadTR.

Table 3: Comparison of different methods on ActivityNet-1.3. Methods in the second group are combined with an ensemble of action classifiers [47]. The computation costs (in FLOPs) of the action classifiers are not included. Method |  Single-network |  E2E |  0.5 |  0.75 |  0.95 |  Avg. | FLOPs (G)
---|---|---|---|---|---|---|---
Self-contained methods
CDC [29] |  - |  - |  43.83 |  25.88 |  0.21 |  22.77 | -
R-C3D [36] |  ✓ |  - |  26.8 |  - |  - |  - | -
SSN [46] |  - |  - |  39.12 |  23.48 |  5.49 |  23.98 | -
TAL-Net [8] |  ✓ |  - |  38.23 |  18.30 |  1.30 |  20.22 | -
P-GCN [43] |  - |  - |  42.90 |  28.14 |  2.47 |  26.99 | 5.0
TadTR (Ours) |  ✓ |  ✓ |  40.85 |  28.44 |  7.84 |  27.75 | 0.038
Combined with an ensemble of action classifiers [47]
P-GCN [43] |  - |  - |  48.26 |  33.16 |  3.27 |  31.11 | 5.0
MR [45] |  - |  - |  43.47 |  33.91 |  9.21 |  30.12 | -
A2Net [38] |  - |  - |  43.55 |  28.69 |  3.70 |  27.75 | 1.2
BMN [16] |  - |  - |  50.07 |  34.78 |  8.29 |  33.85 | 45.6
G-TAD [37] |  - |  - |  50.36 |  34.60 |  9.02 |  34.09 | 45.7
TadTR (Ours) |  - |  - |  49.08 |  32.58 |  8.49 |  32.27 | 0.038
TadTR+BMN (Ours) |  - |  - |  50.51 |  35.35 |  8.18 |  34.55 | 45.6

###  4.3 Ablation Study

The importance of context information. The key of Transformer is the self-
attention mechanism that incorporates the context in a video sequence. In
TadTR, we leverage two kinds of context, inter-snippet context from all
snippets and inter-action context from all action queries, which are captured
by Transformer encoder and the self-attention module in Transformer decoder
respectively. By removing Transformer encoder, we get a variant “TadTR w/o
encoder”. By removing the self-attention module in the decoder, we get a
variant “TadTR w/o inter-action context”. We report the performance of the two
variants in Table 2. It is observed that removing the encoder leads to a
significant performance drop of 3.93% average mAP. It indicates that the
Transformer encoder is crucial for our model, as the decoder requires long-
range and adaptive context to reason the relations between the actions and the
video. Removing instance-level context, the average mAP drops by 1.13%. We
conclude that the context information from other action instances is also
helpful for action detection.

Transformer encoder v.s. CNN encoder. We try replacing the Transformer encoder
with a 1D CNN encoder, which is common for temporal modeling in previous TAD
methods. The 1D CNN encoder is composed of two 1D convolutional layers with
256 filters of kernel size 3 and ReLU activation. This variant is denoted as
“TadTR with a 1D CNN encoder”. As can be observed in Table 2, the performance
of this variant is worse than TadTR (-2.92% average mAP), even with higher
computation cost. We also explore deeper CNNs and larger convolution kernels,
but no improvement is observed. The results show that 1D CNN is inferior to
Transformer encoder.

Dense attention v.s. temporal deformable attention. We try replacing all
temporal deformable attention modules in TadTR with vanilla dense attention
modules. This variant is called “TadTR with dense attention”. As depicted in
Table 2, the performance of this variant is far behind TadTR (13.58% v.s.
30.83% average mAP). Besides, it has 5.7$\times$ higher computation cost than
that of TadTR. The main reason is that the dense attention lacks induced bias.
A dense attention module is initialized to nearly take an average of all
locations in the input sequence approximately. It is hard to learn to focus on
specific locations. Therefore, temporal deformable attention is a better
choice.

Effects of actionness regression and segment refinement. We study the effects
of iterative segment refinement and actionness regression by removing them
from TadTR individually, resulting in 2 different model variants. Their
results on HACS Segments are presented in Table 2. Comparing TadTR w/o
actionness regression and TadTR, we observe that actionness regression leads
to improvements of all metrics. Specifically, the improvements are 3.06%,
2.26%, 1.55%, and 2.32% in terms of mAP at IoU 0.5, 0.75, 0.95 and the average
mAP. Several qualitative examples are presented in Fig. 5 to show how
actionness regression helps. It is able to produce more reliable confidence
scores for the action predictions by Transformer, owing to explicit feature
alignment. Comparing TadTR w/o segment refinement and TadTR, we find this
component helpful for improving the localization accuracy. It improves the mAP
at the strict threshold of 0.95 by a large margin of 2.24%. The mAPs at other
thresholds are also consistently improved.

Besides the detection performance, another important aspect is the computation
cost of the detector. In terms of FLOPs, adding or removing the two components
has little impact, which is shown in Table 2. In terms of run time, we keep
the same environment used in Fig. 1 and Table 1 for run time measurement. The
average time cost per video is 130 ms for TadTR-base. Adding the rescoring
module will increase the average time cost to 141 ms. With segment refinement
enabled, the average time cost per video becomes 155 ms, which is still very
efficient compared with state-of-the-art methods.

Table 4: Effect of numbers of encoder layers and decoder layers. The detection performance is evaluated on HACS Segments. $L_{E}$ |  $L_{D}$ | 0.5 | 0.75 | 0.95 | Avg. | FLOPs (M)
---|---|---|---|---|---|---
2 |  4 | 45.16 | 30.70 | 11.78 | 30.83 | 100.5
4 |  4 | 44.63 | 30.39 | 10.76 | 30.39 | 105.6
6 |  4 | 40.55 | 27.55 | 9.88 | 27.63 | 110.8
2 |  2 | 42.10 | 29.05 | 9.57 | 28.84 | 79.6
2 |  4 | 45.16 | 30.70 | 11.78 | 30.83 | 100.5
2 |  6 | 45.20 | 30.82 | 10.67 | 30.74 | 121.3

Effects of the number of action queries. Table 5 compares the performance of
TadTR using different number of action queries ($N_{q}$). The best performance
is achieved at $N_{q}=40$ on THUMOS14, $N_{q}=30$ on HACS Segments and
$N_{q}=10$ on ActivityNet-1.3. The results are reasonable, as the average
number of action instances per video on THUMOS14 (15.4) is larger than that on
ActivityNet-1.3 (1.5) and HACS Segments (2.8). We note that TadTR also
achieves strong performance on other choices of $N_{q}$. Adjusting $N_{q}$ is
also much easier than manually setting a series of anchor scales in
traditional methods.

Table 5: Effect of the number of action queries on different datasets. The average mAPs are reported. # queries | 10 | 20 | 30 | 40 | 50
---|---|---|---|---|---
THUMOS14 | 43.52 | 45.27 | 45.75 | 46.63 | 45.49
HACS Segments | 29.63 | 30.73 | 30.83 | 29.99 | -
ActivityNet-1.3 | 27.75 | 26.47 | 26.27 | 26.29 | -

Effects of the numbers of encoder layers and decoder layers. We evaluate TadTR
with different numbers of encoder layers ($L_{E}$) and decoder layers
($L_{D}$) and report results in Table 4. With $L_{D}$ fixed, the best
performance is achieved when $L_{E}$ is 2. Larger $L_{E}$ gives inferior
results probably due to the difficulty of training. Therefore, we set $L_{E}$
to 2. With $L_{E}$ fixed, the average mAP increases by 2.03% when $L_{D}$
increases from 2 to 4. Larger $L_{D}$ gives a slightly lower performance.
Therefore we suggest setting $L_{D}$ to 4.

Table 6: Impact of the number of sampling points $K$ on the performance and computation cost on HACS Segments. The models are trained for half of the full training cycle (15 epochs). # points | 0.5 | 0.75 | 0.95 | Avg. | FLOPs (M)
---|---|---|---|---|---
1 | 39.22 | 27.07 | 9.80 | 27.03 | 99.1
2 | 40.22 | 27.62 | 10.15 | 27.61 | 99.6
4 | 41.20 | 28.52 | 10.63 | 28.49 | 100.5
8 | 39.77 | 27.15 | 9.86 | 27.20 | 102.4

Effect of the number of sampling points. Table 6 compares the performance and
computation cost using different numbers of sampling points $K$ in TDA modules
on HACS Segments. We observe that moderately increasing $K$ improves the
performance, as more temporal details can be captured. The best performance is
achieved at $K=4$. The number of sampling points has little impact on the
computation cost.

Table 7: Effect of the number of attention heads $M$ on the performance and computation cost on HACS Segments. The models are trained for half of the full training cycle (15 epochs). # head | 0.5 | 0.75 | 0.95 | Avg. | FLOPs (M)
---|---|---|---|---|---
1 | 38.62 | 25.25 | 7.94 | 25.57 | 100.2
2 | 40.30 | 26.37 | 8.09 | 26.52 | 100.2
4 | 40.74 | 28.41 | 10.51 | 28.30 | 100.3
8 | 41.20 | 28.52 | 10.63 | 28.49 | 100.5
16 | 41.07 | 28.35 | 10.23 | 28.31 | 100.9

Effect of the number of attention heads. Table 7 compares the performance and
computation cost using different number of attention heads $M$ on HACS
Segments. It is observed that that moderately increasing $M$ boosts the
performance, as more diverse features can be learned. The performance is
saturated at $M=8$. Similar to the number of sampling points, the number of
attention heads has little impact on the computation cost.

Figure 5: actionness regression improves the ranking of detections. In each of
the above cases, the more accurate obtains a lower score than the less
accurate one before rescoring. With the new scores (red color) applied, the
ranking order turns satisfactory.

###  4.4 Analysis

Visualization of action queries. Fig. 6 illustrates the distribution of
locations and scales (lengths) of output actions associated with each action
query. We observe that each query produces action predictions in certain
locations and scales. Different locations and scales are covered by a small
number of queries. It means that the detector learns the distribution of
actions in the training dataset. This is more flexible and efficient than the
handcrafted anchor design in previous methods.

Figure 6: Distribution of all output actions on all videos from HACS Segments
validation set for 30 query slots with action queries. In each 1-by-1 square,
an action prediction predicted from the corresponding action query is
represented as a point, and the horizontal and vertical coordinates are the
coordinate of the center and the length normalized by the video length. For
example, we observe that the first and the second queries are responsible for
actions near the ending of a video and actions near the starting of a video,
respectively.

Visualization of attention. Fig. 7 visualizes attention of the last encoder
layer and the last decoder layer. We use different colors to distinguish
different attention heads. We observe that: (1) Different attention heads
focus on different temporal regions and scales. The red and the pink points
are sparely distributed on the left side of a reference point. The blue and
the purple points are sparsely distributed on the right. Other points are near
the referent points. (2) Sampling points in the decoder almost cover the same
temporal extent of an action prediction, providing a very large receptive
field. Differently, sampling points in the encoder have a more flexible
distribution, capturing a moderate amount of context.

(a) Futsal (b) Diving (c) High jump

Figure 7: Visualization of attention. Each subfigure is composed of three
rows. The first row is uniformly sampled video frames. The second row
visualizes the attention at two randomly picked reference points in the last
encoder layer. The third row visualizes the attention for the predicted action
in the last decoder layer. Sampling points in different attention heads are in
different colors. We separate the sampling points in different heads
vertically for better readability. Best viewed in color.

##  5 Conclusion

We propose TadTR, a simple end-to-end framework for temporal action detection
based on Transformer. It views the TAD task as a direct set-prediction problem
and maps a series of learnable embeddings to action instances in parallel by
adaptively extracting temporal context in the video. It simplifies the
pipeline of TAD and removes hand-crafted designs such as anchor setting and
post-processing. Extensive experiments are conducted to validate the
performance and efficiency of TadTR and the effectiveness of different
components. TadTR achieves state-of-the-art or competitive performance on HACS
Segments, THUMOS14, and ActivityNet-1.3 with significantly lower computation
costs. We hope that our work could trigger the development of efficient models
and vision Transformers for temporal action detection.

## References

  * [1] Yueran Bai, Yingying Wang, Yunhai Tong, Yang Yang, Qiyue Liu, and Junhui Liu.  Boundary content graph neural network for temporal action proposal generation.  In ECCV, pages 121–137, 2020.
  * [2] Gedas Bertasius, Heng Wang, and Lorenzo Torresani.  Is space-time attention all you need for video understanding?  In ICML, July 2021.
  * [3] Shyamal Buch, Victor Escorcia, Chuanqi Shen, Bernard Ghanem, and Juan Carlos Niebles.  Sst: Single-stream temporal action proposals.  In CVPR, pages 6373–6382, 2017.
  * [4] Fabian Caba Heilbron, Juan Carlos Niebles, and Bernard Ghanem.  Fast temporal activity proposals for efficient detection of human actions in untrimmed videos.  In CVPR, pages 1914–1923, 2016.
  * [5] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles.  Activitynet: A large-scale video benchmark for human activity understanding.  In CVPR, pages 961–970, 2015.
  * [6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In ECCV, pages 213–229, 2020.
  * [7] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In CVPR, pages 4724–4733, 2017.
  * [8] Yu-Wei Chao, Sudheendra Vijayanarasimhan, Bryan Seybold, David A Ross, Jia Deng, and Rahul Sukthankar.  Rethinking the faster r-cnn architecture for temporal action localization.  In CVPR, pages 1130–1139, 2018.
  * [9] Xiyang Dai, Bharat Singh, Guyue Zhang, Larry S Davis, and Yan Qiu Chen.  Temporal context network for activity localization in videos.  In ICCV, pages 5727–5736, 2017.
  * [10] Victor Escorcia, Fabian Caba Heilbron, Juan Carlos Niebles, and Bernard Ghanem.  Daps: Deep action proposals for action understanding.  In ECCV, pages 768–784, 2016.
  * [11] Jiyang Gao, Kan Chen, and Ram Nevatia.  Ctap: Complementary temporal action proposal generation.  In ECCV, September 2018.
  * [12] Jiyang Gao, Zhenheng Yang, Chen Sun, Kan Chen, and Ram Nevatia.  Turn tap: Temporal unit regression network for temporal action proposals.  In ICCV, pages 3648–3656, 2017.
  * [13] Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zisserman.  Video action transformer network.  In CVPR, pages 244–253, 2019.
  * [14] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In ICCV, pages 2961–2969, 2017.
  * [15] YG Jiang, Jingen Liu, A Roshan Zamir, G Toderici, I Laptev, Mubarak Shah, and Rahul Sukthankar.  Thumos challenge: Action recognition with a large number of classes, 2014\.
  * [16] Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen.  Bmn: Boundary-matching network for temporal action proposal generation.  In ICCV, pages 3889–3898, 2019.
  * [17] Tianwei Lin, Xu Zhao, and Zheng Shou.  Single shot temporal action detection.  In ACM MM, pages 988–996, 2017.
  * [18] Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang.  Bsn: Boundary sensitive network for temporal action proposal generation.  In ECCV, September 2018.
  * [19] Daochang Liu, Tingting Jiang, and Yizhou Wang.  Completeness modeling and context separation for weakly supervised temporal action localization.  In CVPR, pages 1298–1307, 2019.
  * [20] Xiaolong Liu, Yao Hu, Song Bai, Fei Ding, Xiang Bai, and Philip H. S. Torr.  Multi-shot temporal event localization: A benchmark.  In CVPR, pages 12596–12606, June 2021.
  * [21] Yuan Liu, Lin Ma, Yifeng Zhang, Wei Liu, and Shih-Fu Chang.  Multi-granularity generator for temporal action proposal.  In CVPR, pages 3604–3613, 2019.
  * [22] Fuchen Long, Ting Yao, Zhaofan Qiu, Xinmei Tian, Jiebo Luo, and Tao Mei.  Gaussian temporal awareness networks for action localization.  In CVPR, pages 344–353, 2019.
  * [23] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In ICLR, 2017.
  * [24] Fan Ma, Linchao Zhu, Yi Yang, Shengxin Zha, Gourab Kundu, Matt Feiszli, and Zheng Shou.  Sf-net: Single-frame supervision for temporal action localization.  In ECCV, pages 420–437. Springer, 2020.
  * [25] Shugao Ma, Leonid Sigal, and Stan Sclaroff.  Learning activity progression in lstms for activity detection and early detection.  In CVPR, June 2016.
  * [26] Phuc Nguyen, Ting Liu, Gautam Prasad, and Bohyung Han.  Weakly supervised action localization by sparse temporal pooling network.  In CVPR, pages 6752–6761, 2018.
  * [27] Sujoy Paul, Sourya Roy, and Amit K. Roy-Chowdhury.  W-talc: Weakly-supervised temporal activity localization and classification.  In ECCV, September 2018.
  * [28] Alexander Richard and Juergen Gall.  Temporal action detection using a statistical language model.  In CVPR, pages 3131–3140, 2016.
  * [29] Zheng Shou, Jonathan Chan, Alireza Zareian, Kazuyuki Miyazawa, and Shih-Fu Chang.  Cdc: Convolutional-de-convolutional networks for precise temporal action localization in untrimmed videos.  In ICCV, pages 1417–1426, 2017.
  * [30] Zheng Shou, Hang Gao, Lei Zhang, Kazuyuki Miyazawa, and Shih-Fu Chang.  Autoloc: Weakly-supervised temporal action localization in untrimmed videos.  In ECCV, pages 154–171, 2018.
  * [31] Zheng Shou, Dongang Wang, and Shih-Fu Chang.  Temporal action localization in untrimmed videos via multi-stage cnns.  In CVPR, pages 1049–1058, 2016.
  * [32] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid.  Videobert: A joint model for video and language representation learning.  In ICCV, pages 7464–7473, 2019.
  * [33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NIPS, pages 5998–6008, 2017.
  * [34] Limin Wang, Yuanjun Xiong, Dahua Lin, and Luc Van Gool.  Untrimmednets for weakly supervised action recognition and detection.  In CVPR, pages 4325–4334, 2017.
  * [35] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool.  Temporal segment networks: Towards good practices for deep action recognition.  In ECCV, pages 20–36, 2016.
  * [36] Huijuan Xu, Abir Das, and Kate Saenko.  R-c3d: region convolutional 3d network for temporal activity detection.  In ICCV, pages 5794–5803, 2017.
  * [37] Mengmeng Xu, Chen Zhao, David S Rojas, Ali Thabet, and Bernard Ghanem.  G-TAD: Sub-graph localization for temporal action detection.  In CVPR, pages 10156–10165, 2020.
  * [38] Le Yang, Houwen Peng, Dingwen Zhang, Jianlong Fu, and Junwei Han.  Revisiting anchor mechanisms for temporal action localization.  IEEE Transactions on Image Processing, 29:8535–8548, 2020.
  * [39] Serena Yeung, Olga Russakovsky, Greg Mori, and Li Fei-Fei.  End-to-end learning of action detection from frame glimpses in videos.  In CVPR, pages 2678–2687, 2016.
  * [40] Tan Yu, Zhou Ren, Yuncheng Li, Enxu Yan, Ning Xu, and Junsong Yuan.  Temporal structure mining for weakly supervised action detection.  In ICCV, pages 5522–5531, 2019.
  * [41] Jun Yuan, Bingbing Ni, Xiaokang Yang, and Ashraf A Kassim.  Temporal action localization with pyramid of score distribution features.  In CVPR, pages 3093–3102, 2016.
  * [42] Ze-Huan Yuan, Jonathan C Stroud, Tong Lu, and Jia Deng.  Temporal action localization by structured maximal sums.  In CVPR, volume 2, page 7, 2017.
  * [43] Runhao Zeng, Wenbing Huang, Mingkui Tan, Yu Rong, Peilin Zhao, Junzhou Huang, and Chuang Gan.  Graph convolutional networks for temporal action localization.  In ICCV, pages 7094–7103, 2019.
  * [44] Hang Zhao, Antonio Torralba, Lorenzo Torresani, and Zhicheng Yan.  HACS: human action clips and segments dataset for recognition and temporal localization.  In ICCV, pages 8667–8677, 2019.
  * [45] Peisen Zhao, Lingxi Xie, Chen Ju, Ya Zhang, Yanfeng Wang, and Qi Tian.  Bottom-up temporal action localization with mutual regularization.  In ECCV, 2020.
  * [46] Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin.  Temporal action detection with structured segment networks.  ICCV, pages 2914–2923, 2017.
  * [47] Yue Zhao, Bowen Zhang, Zhirong Wu, Shuo Yang, Lei Zhou, Sijie Yan, Limin Wang, Yuanjun Xiong, Wang Yali, Dahua Lin, Yu Qiao, and Xiaoou Tang.  CUHK & ETHZ & SIAT submission to ActivityNet challenge 2017\.  arXiv preprint arXiv:1710.08011, pages 20–24, 2017.
  * [48] Luowei Zhou, Yingbo Zhou, Jason J Corso, Richard Socher, and Caiming Xiong.  End-to-end dense video captioning with masked transformer.  In CVPR, pages 8739–8748, 2018.
  * [49] Linchao Zhu and Yi Yang.  Actbert: Learning global-local video-text representations.  In CVPR, pages 8746–8755, 2020.
  * [50] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  In ICLR, 2021.