September 8, 2022

NuSmear: Fast Simulation of Energy Smearing and Angular Smearing for Neutrino-
Nucleon Scattering Events in the GENIE Event Generator

Ishaan Vohra

_Phillips Exeter Academy_

_Exeter, NH, USA_

Complete Monte Carlo (MC) simulation of a neutrino experiment typically
involves the lengthy and CPU-intensive process of integrating models of
incoming neutrino fluxes, event generation, and detector setup. We describe a
fast, geometry-independent, GENIE-based system known as NuSmear which provides
preliminary simulation of energy smearing and angular smearing for neutrino-
nucleon interactions. We discuss NuSmear’s simulation methodology, explaining
its calculation of reconstructed values through its model-based resolution
computation, smearing distributions, and particle detection dependencies. We
go on to validate NuSmear’s performance as a fast simulation system through a
series of internal and external comparisons testing its predictive accuracy
and model-adherence. Finally, we briefly explore the potential for future
user-customization in NuSmear.

PRESENTED AT

Fourteenth Conference on the Intersections

of Particle and Nuclear Physics

Lake Buena Vista, FL, USA

August 29 – September 4, 2022

##  1 Introduction

In the realm of modern neutrino physics, the ability of Monte Carlo methods to
provide detailed simulations of neutrino-nucleon interactions plays an
essential role in both data analysis and the planning of future experiments
[1, 2, 3]. For the sake of generating accurate predictions, however, complete
Monte Carlo simulation of a neutrino experiment often requires the meticulous
modeling of incoming neutrino fluxes, event generation, and detector setup,
resulting in necessarily long computing times and high CPU loads to generate a
statistically significant sample of events [4, 5].

In recent years, the solution to this problem has appeared in the form of fast
Monte Carlo methods, which provide preliminary simulations of experiments
centered around balancing the needs of predictive accuracy and computational
speed [6, 7]. In the area of detector response smearing simulations, various
existing tools have found the comfortable middle ground between these two
requirements, ranging from multipurpose collider frameworks such as DELPHES to
experiment-specific frameworks such as EIC-Smear and the ATLAS Fast Track
Simulation Project [8, 9, 10, 11, 12, 13]. Within the neutrino physics
community, however, while there are tools offering fast simulations for
specific experimental setups, few dedicated systems currently exist to provide
rapid preliminary smearing simulations for generic neutrino-nucleon scattering
events [14].

In this work, we propose NuSmear – a novel software system providing fast,
generic, geometry-independent simulations of energy smearing and angular
smearing via parameterized model-based presets. For ease of user integration,
NuSmear111NuSmear’s code is publicly available at https://github.com/GENIE-
MC/Generator/pull/222 is written as a contribution package built directly onto
the GENIE Monte Carlo event generator, which is used by the majority of
neutrino experiments worldwide to simulate interactions between all flavors of
neutrinos and nuclear targets within the MeV to PeV energy scales [15].

For a given input data set of neutrino-nucleon scattering events, NuSmear
performs smearing in four major steps:

  1. 1.

Smearing model selection

  2. 2.

Computation of resolution

  3. 3.

Application of smearing distribution

  4. 4.

Consideration of particle detection dependency (energy smearing operation
only)

First, the user selects one of NuSmear’s two current smearing model presets to
perform either the energy smearing or the angular smearing operation. The
first smearing model is derived from the Deep Underground Neutrino
Experiment’s Conceptual Design Report (DUNE-CDR) – it incorporates energy
resolution functions, angular resolution values, and particle detection
dependencies simulating the preliminary far detector response at the future
DUNE experiment [16, 17, 18]. The second smearing model is the NuSmear Default
model, which incorporates more straightforward energy resolution functions,
angular resolution values, and particle detection dependencies. Once a model
is chosen, particles are indexed by type and assigned to one of 72 energy
resolution functions or angular resolution values, from which a
resolution222Unlike the common usage of the term _resolution_ , in the context
of this paper, an increase in resolution value corresponds to a greater degree
of smearing, and vice versa. is computed or obtained. Subsequently, according
to the chosen smearing parameter of energy or angle, either a log-normal or
Gaussian distribution is generated and applied to the true value of the
smearing parameter, producing a reconstructed value. In the case of angular
smearing, further dependencies are considered negligible and the reconstructed
value is simply returned to the user as an output. In the case of energy
smearing, however, further particle detection dependencies may be considered
in reference to the type of particle and the chosen smearing model. In the
DUNE-CDR model, for example, a neutron with momentum below a 1 GeV/c threshold
has a non-zero probability of entirely escaping detection and returning zero
reconstructed energy. Once consideration of particle detection dependencies
has been completed, the final reconstructed value of the smearing parameter is
returned to the user as an output. Figure 1 provides a flowchart
representation of NuSmear’s complete simulation process.

Figure 1: Flowchart diagram depicting NuSmear’s smearing simulation process.
The turquoise arrows represent decisions made by the user.

##  2 Simulation methodology

###  2.1 Computation of resolution

####  2.1.1 Energy resolution

NuSmear’s DUNE-CDR energy resolution functions include up to three particle
parameters to compute an energy resolution: total energy, kinetic energy, and
magnitude of momentum. Kinetic energy and magnitude of momentum are compared
to a variety of particle-specific numerical thresholds, returning an energy
resolution that is either a constant value or dependent on the total energy
[17, 18]. Moreover, if a particle does not pass a function’s minimum kinetic
energy threshold, it is considered undetected, and NuSmear will return zero
reconstructed energy to the user. As NuSmear is inherently a geometry-
independent smearing system, all DUNE-CDR geometric resolution dependencies
are omitted and approximated with numerical values. Table 1 summarizes the
calculations performed by NuSmear’s DUNE-CDR energy resolution functions.

Particle type | Function calculations | Omitted dependencies
---|---|---
$\pi^{\pm}$ | if KE $\geq$ 100 MeV, return 15% | track length,
| showering,
| contained/exiting track
$\gamma$, $e^{\pm}$ | if KE $\geq$ 30 MeV, return |
2% $\oplus$ 15%/$\sqrt{E}$[GeV] |
$p$ | if KE $\geq$ 50 MeV, return |
10% if $|$p$|$ $<$ 400 MeV/c, else |
return 5% $\oplus$ 30%/$\sqrt{E}$[GeV] |
$\mu^{\pm}$ | if KE $\geq$ 30 MeV, return 15% | track length,
| contained/exiting track
$n$ | if KE $\geq$ 50 MeV, |
return 40%/$\sqrt{E}$[GeV] |
other | if KE $\geq$ 50 MeV, return |
5% $\oplus$ 5%/$\sqrt{E}$[GeV] |
Table 1: Summary of the calculations performed by NuSmear’s energy resolution
functions in the DUNE-CDR model. $E$, KE, and $|$p$|$ denote total energy,
kinetic energy, and magnitude of momentum respectively.

NuSmear’s Default model energy resolution functions involve simpler
calculations than that of the DUNE-CDR model. A single comparison is made to
check if the particle kinetic energy passes a standard numerical threshold of
50 MeV. If not, the particle is considered undetected and zero reconstructed
energy is returned to the user. If so, the particle is assigned a constant
energy resolution value according to its type, as summarized in Table 2.

Particle type | Energy resolution
---|---
$\pi^{\pm}$,$\pi^{0}$ | 15%
$K^{\pm}$, $K^{0}$/$\bar{K^{0}}$ | 20%
$\gamma$ | 30%
$e^{\pm}$ | 40%
$p$ | 40%
$\mu^{\pm}$ | 15%
$n$ | 50%
other | 30%
Table 2: Summary of the energy resolution values assigned to each particle
type in the Default model.

####  2.1.2 Angular resolution

To obtain an angular resolution for a given particle, NuSmear assigns the
particle to a constant angular resolution value determined purely by particle
type. The DUNE-CDR model and Default model angular resolution values are
itemized in Table 3 below.

Particle type | Angular resolution
---|---
DUNE-CDR model | Default model
$\pi^{\pm}$ | 1∘ | 2∘
$\pi^{0}$ | 5∘ | 8∘
$K^{\pm}$ | 5∘ | 2∘
$K^{0}/\bar{K^{0}}$ | 5∘ | 8∘
$\gamma$, $e^{\pm}$ | 1∘ | 3∘
$p$ | 5∘ | 8∘
$\mu^{\pm}$ | 1∘ | 2∘
$n$ | 5∘ | 10∘
other | 5∘ | 8∘
Table 3: Summary of the angular resolution values assigned to each particle
type in the DUNE-CDR and Default models.

###  2.2 Smearing distributions

####  2.2.1 Energy smearing distribution

Although particle energies are often smeared using a Gaussian distribution,
this poses the problem of how one should exclude negative values of energy
[12]. The common solution of truncating the Gaussian distribution reduces the
simulation’s accuracy to a real detector at low energies, hence NuSmear makes
use of the log-normal distribution instead, as shown in Figure 2 [10, 19].

Figure 2: Example log-normal and truncated Gaussian smearing distributions
representing the same true energy and resolution values.

The log-normal distribution takes the form

$f(x)={\frac{1}{x\sigma{\sqrt{2\pi}}}}\
\exp\left(-{\frac{\left(\ln\left(x\right)-\mu\right)^{2}}{2\sigma^{2}}}\right)$,

in which the distribution parameters $\mu$ and $\sigma$ are given in terms of
the mean and variance of the smearing distribution, $m$ and $Var[X]$, by

$\mu=\ln{\left(\frac{m^{2}}{\sqrt{Var[X]+m^{2}}}\right)}$,

$\sigma^{2}=\ln{\left(1+\frac{Var[X]}{m^{2}}\right)}$.

Furthermore, $m$ and $Var[X]$ are physically related to the particle’s true
energy and calculated energy resolution, $E_{true}$ and $R_{E}$, via

$m=E_{true}$,

$Var[X]=(R_{E}E_{true})^{2}$.

Once a particle’s smearing distribution is created, NuSmear makes use of the
Mersenne Twister pseudo-random number generator (PRNG) to generate the
reconstructed energy according to the distribution [20].

####  2.2.2 Angular smearing distribution

In NuSmear, a particle’s true outgoing angle with respect to the incident
neutrino is represented by $\theta$, as illustrated in Figure 3.

Figure 3: Example charged-current (CC) interaction between an incident muon
neutrino and target neutron, producing an outgoing muon at angle $\theta$ with
respect to the incident neutrino.

For each particle, angular smearing is performed using a Gaussian distribution
with a mean set to the true angle, $\theta$, and a standard deviation set to
the particle’s angular resolution, ${R_{A}}$, as exemplified in Figure 4.

Figure 4: Example Gaussian angular smearing distribution with a true angle
value of 5∘ and an angular resolution of 2∘.

Alike NuSmear’s application of energy smearing distributions, once a
particle’s angular smearing distribution is created, the Mersenne Twister is
used to generate the reconstructed angle according to the distribution [20].

###  2.3 Particle detection dependency

In addition the minimum kinetic energy thresholds incorporated into NuSmear’s
energy resolution functions, further particle detection dependencies are
implemented into the energy smearing operation in order to more accurately
simulate unobserved particles in real detectors. Particle detection
dependencies are present in both the DUNE-CDR and Default energy smearing
models.

Within NuSmear’s DUNE-CDR energy smearing model, neutrons with momentum less
than 1 GeV/c have a 10% probability of escaping detection, and for neutrons
that are detected, only 60% of the energy generated by the smearing
distribution is returned to the user as the final reconstructed energy [18].

In the Default energy smearing model, photons and neutrons have a 50%
probability of escaping detection and returning zero reconstructed energy.

##  3 Validation of smearing performance

To demonstrate the efficacy of NuSmear in providing fast smearing simulations
which both maintain predictive accuracy and adhere to their input models, we
generate sets of neutrino-nucleon scattering events and perform a series of
comparisons between multiple NuSmear simulations, as well as between NuSmear
and independent complete Monte Carlo simulations [21, 22]. Within each
comparison, we construct and juxtapose multiple smearing matrices, analyzing
each simulation through distributions representing the reconstructed smearing
parameter in relation to the true smearing parameter.

###  3.1 Energy smearing

####  3.1.1 Complete MC comparison

Generating events using an incoming electron neutrino flux derived from the
the OPERA experiment [23], we evaluate NuSmear’s reproduction of the OPERA
detector’s energy smearing response by constructing a neutrino energy smearing
matrix and comparing it to the smearing matrix constructed by the OPERA
Collaboration’s complete Monte Carlo simulation[21]. Both matrices (see Figure
5 below) are filtered to contain only events involving charged-current (CC)
interactions.

Figure 5: NuSmear Default (left) compared to complete Monte Carlo simulation
(right) electron neutrino CC energy smearing matrices for the OPERA detector
in the CNGS beam.

The overall agreement in the structure and shape of the distributions between
NuSmear’s Default model matrix and the complete Monte Carlo simulation’s
matrix indicates that NuSmear provides a reasonably accurate approximation for
the OPERA detector’s energy smearing response.

####  3.1.2 Deconstructed by particle type

Using an example data set of muon neutrinos incident on an Argon-40 target, we
construct energy smearing matrices (illustrated in Figures 6 and 7) with
NuSmear’s DUNE-CDR model and Default model, deconstructing them into multiple
smearing matrices according to final state particle type (as itemized in Table
1 and 2 of Section 4.1.1, respectively).

Figure 6: NuSmear DUNE-CDR energy smearing matrices deconstructed by final
state particle type for muon neutrinos on an Argon-40 target – $\pi^{\pm}$
(1); $\gamma$, $e^{\pm}$ (2); $p$ (3); $\mu^{\pm}$ (4); $n$ (5); other (6).

The DUNE-CDR energy smearing matrices portray significant agreement with the
resolution functions and particle detection dependencies of the model.
Photons, electrons, and positrons, for example, appear smeared to a lesser
extent due to their on average smaller resolutions, while protons appear
smeared to a greater extent due to their on average greater resolutions.
Moreover, in producing the reconstructed energies of neutrons, the true
energies appear to have been reduced by a constant multiplicative factor – a
result which follows from the fact that only 60% of smeared neutron energy is
finally reconstructed in the DUNE-CDR model.

Figure 7: NuSmear Default energy smearing matrices deconstructed by final
state particle type for muon neutrinos on an Argon-40 target –
$\pi^{\pm}$,$\pi^{0}$ (1); $K^{\pm}$, $K^{0}$/$\bar{K^{0}}$ (2); $\gamma$ (3);
$e^{\pm}$ (4); $p$ (5); $\mu^{\pm}$ (6); $n$ (7); other (8).

The Default energy smearing matrices also portray significant agreement with
their resolution functions and further dependencies. Particles with on average
smaller energy resolutions such as pions and kaons are smeared to a
comparatively lesser extent, while particles with on average greater energy
resolutions such as protons and neutrons are smeared to a comparatively
greater extent. Furthermore, while in the DUNE-CDR model only few data points
in the neutron smearing matrix lie along the x-axis, a more significant
portion of the data points in the Default model photon and neutron matrices
lie along the x-axis, which predictably results from the fact that both have a
50% chance of not being observed at all.

###  3.2 Angular smearing

####  3.2.1 Complete MC comparison

A set of events is generated using an incoming electron neutrino flux obtained
from T2K experiment data, and a comparison is made between the electron
angular smearing matrices constructed by NuSmear’s Default model and the
complete Monte Carlo simulation of the T2K Collaboration[22, 24]. The two
matrices (see Figure 8 below), which simulate the angular smearing of the
ND280 near detector, are filtered to contain only events involving charged-
current (CC) interactions.

Figure 8: NuSmear Default model (left) compared to complete Monte Carlo
simulation (right) electron angular smearing matrices for electron neutrino CC
interactions at the T2K experiment’s ND280 near detector.

There is overall strong agreement between the main distributions of NuSmear’s
Default model matrix and the complete Monte Carlo simulation’s matrix.
Although some data points spread further from the main distribution are not
reproduced to the same extent in NuSmear, we expect that this is a result of
more complex reconstruction efficiencies and detector dependencies beyond the
scope of NuSmear’s geometry-independent smearing capabilities.

####  3.2.2 Deconstructed by particle type

Using an example data set of muon neutrinos incident on an Argon-40 target, we
construct angular smearing matrices (illustrated in Figure 9) with both
NuSmear’s DUNE-CDR model and Default model, deconstructing them into multiple
smearing matrices according to final state particle type (as itemized in Table
3, Section 4.1.2). Only the charged pion, proton, neutron, and muon smearing
matrices are displayed below – the other angular smearing matrices can be
found in the Appendix.

(a) DUNE-CDR angular smearing matrices

(b) Default angular smearing matrices

Figure 9: NuSmear DUNE-CDR and Default angular smearing matrices deconstructed
by final state particle type for muon neutrinos on an Argon-40 target –
$\pi^{\pm}$ (1); $p$ (2); $n$ (3); $\mu^{\pm}$ (4).

Both the DUNE-CDR and Default angular smearing matrices portray significant
agreement with the resolution values of the models. The charged pion and muon
matrices exhibit less smearing, reflecting their smaller resolution values,
while the proton and neutron matrices exhibit more smearing, reflecting their
greater resolution values. In addition, each of the Default smearing matrices
exhibit a greater degree of smearing than each of the corresponding DUNE-CDR
smearing matrices, reflecting the Default model’s overall greater resolution
values for the displayed particle types.

##  4 Conclusion

In this paper, we describe NuSmear – a generic, fast, parameterized system for
simulating energy smearing and angular smearing in neutrino-nucleon scattering
events. After one of NuSmear’s smearing model presets (the DUNE-CDR model or
the Default model) is selected, each input particle is assigned to an energy
resolution function or angular resolution value, from which a particle-
specific resolution is computed or obtained. NuSmear then generates a smearing
distribution parameterized by the calculated resolution as well as the
particle’s true energy or angle, from which a reconstructed value is produced.
For the angular smearing operation, the reconstructed value is returned to the
user immediately, whereas in the energy smearing operation, particle detection
dependencies are considered before the reconstructed value is returned to the
user.

We go on to demonstrate the smearing performance of NuSmear through a series
of internal and external comparisons of smearing matrices. Both NuSmear’s
energy smearing and angular smearing operations are validated with respect to
their strong adherence to the input models, as well as their accurate
reproduction of results from independent complete Monte Carlo simulations.

Hence, we conclude that NuSmear effectively satisfies the requirements of
predictive accuracy and speed of computation, thereby offering itself as a
robust and efficient alternative for the rapid simulation of energy and
angular smearing in neutrino-nucleon scattering events.

##  5 Future Prospects

By virtue of its open access smearing software and straightforwardly
adjustable smearing models, NuSmear naturally lends itself to user
customization and the implementation of custom smearing configurations. Such
customization could range from simply tweaking values of the numerical
thresholds within resolution functions and particle detection dependencies, to
applying one’s own smearing models with theoretically limitless complexity. In
this spirit, NuSmear users are encouraged to incorporate their own ideas into
their NuSmear smearing systems, providing greater control and more precise
simulation capabilities over a wide range of parameters.

##  6 Acknowledgements

This research project was undertaken as part of an internship at the
University of Liverpool in the area of computational particle physics. I would
like to sincerely thank my mentor, Dr. Marco Roda, for guiding me through this
project and for offering helpful advice on my simulations and data analysis. I
am also grateful to Professor Costas Andreopoulos for supervising my
internship and providing me with the opportunity to conduct particle physics
research as a high school student. Lastly, I would like to thank the Oliver
Lodge Laboratory for hosting me and providing me with a comfortable
environment to pursue my research.

## References

  * [1] M. H. Seymour and M. Marx, “Monte carlo event generators,” in LHC Phenomenology, pp. 287–319, Springer, 2015.

  * [2] M. Tanabashi et al., “Particle data group,” Phys. Rev. D, vol. 98, no. 3, p. 030001, 2018.
  * [3] C. Bozzi, “Monte carlo simulation in particle physics,” November 2002.
  * [4] G. Iaselli, “A fast monte carlo event generator for particle physics,” Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, vol. 248, no. 2, pp. 488–490, 1986.
  * [5] J. Y. Araz, B. Fuks, and G. Polykratis, “Simplified fast detector simulation in madanalysis 5,” The European Physical Journal C, vol. 81, no. 4, pp. 1–24, 2021.
  * [6] T. Wlodek, “Monte carlo methods in high energy physics,” June 2002.
  * [7] B. Siddi, “Development and deployment of a fast monte carlo simulation in lhcb,” Il nuovo cimento C, vol. 41, no. 1-2, pp. 1–2, 2018.
  * [8] E. Aschenauer, T. Burton, R. Darienzo, and A. Kiselev, “Eic detector simulations,” 2013.
  * [9] A. Accardi, J. Albacete, M. Anselmino, N. Armesto, E. Aschenauer, A. Bacchetta, D. Boer, W. Brooks, T. Burton, N.-B. Chang, et al., “Electron-ion collider: The next qcd frontier,” The European Physical Journal A, vol. 52, no. 9, pp. 1–100, 2016.
  * [10] C. Delaere, “” delphes: Framework for fast simulation of a generic collider experiment,” in MC4BSM 2013-Monte Carlo Tools for Physics Beyond the Standard Model, 2013.
  * [11] L. Koch, “A response-matrix-centred approach to presenting cross-section measurements,” Journal of Instrumentation, vol. 14, no. 09, p. P09013, 2019\.
  * [12] A. Buckley, D. Kar, and K. Nordström, “Fast simulation of detector effects in rivet,” SciPost Physics, vol. 8, no. 2, p. 025, 2020.
  * [13] S. Hamilton, K. Sliwa, S. Zimmermann, W. Lukas, J. Wetter, E. Ritsch, E. Kneringer, S. Todorova, and A. Salzburger, “The atlas fast track simulation project,” tech. rep., ATL-COM-SOFT-2010-070, 2011.
  * [14] P. Huber, M. Lindner, and W. Winter, “Simulation of long-baseline neutrino oscillation experiments with globes:(general long baseline experiment simulator),” Computer Physics Communications, vol. 167, no. 3, pp. 195–202, 2005.
  * [15] C. Andreopoulos, A. Bell, D. Bhattacharya, F. Cavanna, J. Dobson, S. Dytman, H. Gallagher, P. Guzowski, R. Hatcher, P. Kehayias, et al., “The genie neutrino monte carlo generator,” Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, vol. 614, no. 1, pp. 87–104, 2010.
  * [16] R. Acciarri, M. Acero, M. Adamowski, C. Adams, P. Adamson, S. Adhikari, Z. Ahmad, C. Albright, T. Alion, E. Amador, et al., “Long-baseline neutrino facility (lbnf) and deep underground neutrino experiment (dune) conceptual design report, volume 4 the dune detectors at lbnf,” arXiv preprint arXiv:1601.02984, 2016.
  * [17] B. Abi, R. Acciarri, M. A. Acero, G. Adamov, D. Adams, M. Adinolfi, Z. Ahmad, J. Ahmed, T. Alion, S. A. Monsalve, et al., “Deep underground neutrino experiment (dune), far detector technical design report, volume ii: Dune physics,” arXiv preprint arXiv:2002.03005, 2020.
  * [18] T. Alion, J. Back, A. Bashyal, M. Bass, M. Bishai, D. Cherdack, M. Diwan, Z. Djurcic, J. Evans, E. Fernandez-Martinez, et al., “Experiment simulation configurations used in dune cdr,” arXiv preprint arXiv:1606.09550, 2016.
  * [19] L. Lyons, “Statistical issues in particle physics,” in Particle Physics Reference Library, pp. 645–692, Springer, 2020.
  * [20] M. Matsumoto and T. Nishimura, “Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator,” ACM Transactions on Modeling and Computer Simulation (TOMACS), vol. 8, no. 1, pp. 3–30, 1998.
  * [21] N. Agafonova, A. Aleksandrov, A. Anokhina, S. Aoki, A. Ariga, T. Ariga, A. Bertolin, C. Bozza, R. Brugnera, A. Buonaura, et al., “Final results of the search for $\nu$$\mu$→ $\nu$e oscillations with the opera detector in the cngs beam,” Journal of High Energy Physics, vol. 2018, no. 6, pp. 1–15, 2018.
  * [22] K. Abe, N. Akhlaq, R. Akutsu, A. Ali, C. Alt, C. Andreopoulos, L. Anthony, M. Antonova, S. Aoki, A. Ariga, et al., “Measurement of the charged-current electron (anti-) neutrino inclusive cross-sections at the t2k off-axis near detector nd280,” Journal of High Energy Physics, vol. 2020, no. 10, pp. 1–43, 2020.
  * [23] R. Acquafredda, T. Adam, N. Agafonova, P. A. Sanchez, M. Ambrosio, A. Anokhina, S. Aoki, A. Ariga, T. Ariga, L. Arrabito, et al., “The opera experiment in the cern to gran sasso neutrino beam,” Journal of Instrumentation, vol. 4, no. 04, p. P04018, 2009.
  * [24] K. Abe, N. Abgrall, H. Aihara, Y. Ajima, J. Albert, D. Allan, P.-A. Amaudruz, C. Andreopoulos, B. Andrieu, M. Anerella, et al., “The t2k experiment,” Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, vol. 659, no. 1, pp. 106–135, 2011.

##  7 Appendix

###  7.1 Additional NuSmear angular smearing matrices

Figure 10: NuSmear DUNE-CDR angular smearing matrices deconstructed by final
state particle type for muon neutrinos on an Argon-40 target – $\pi^{0}$ (1);
$K^{\pm}$ (2); $K^{0}/\bar{K^{0}}$ (3); $\gamma$, $e^{\pm}$ (4); other (5).
Figure 11: NuSmear Default angular smearing matrices deconstructed by final
state particle type for muon neutrinos on an Argon-40 target – $\pi^{0}$ (1);
$K^{\pm}$ (2); $K^{0}/\bar{K^{0}}$ (3); $\gamma$, $e^{\pm}$ (4); other (5).


Kevin<EMAIL_ADDRESS>Dima
<EMAIL_ADDRESS>Michael<EMAIL_ADDRESS>School
of Computer Science
University of Bristol
Bristol, UK Learning Temporal .. From Narrated EgoVideos

# Learning Temporal Sentence Grounding From Narrated EgoVideos

###### Abstract

The onset of long-form egocentric datasets such as Ego4D and EPIC-Kitchens
presents a new challenge for the task of Temporal Sentence Grounding (TSG).
Compared to traditional benchmarks on which this task is evaluated, these
datasets offer finer-grained sentences to ground in notably longer videos. In
this paper, we develop an approach for learning to ground sentences in these
datasets using only narrations and their corresponding rough narration
timestamps. We propose to artificially merge clips to train for temporal
grounding in a contrastive manner using text-conditioning attention. This Clip
Merging (Cli$\mathcal{M}$er) approach is shown to be effective when compared
with a high performing TSG method—e.g. mean R@1 improves from 3.9 to 5.7 on
Ego4D and from 10.7 to 13.0 on EPIC-Kitchens. Code and data splits available
from: https://github.com/keflanagan/CliMer

Figure 1: Left: We generate training examples from videos with rough
timestamps of narrations by artificially merging clips to provide a
contrastive signal. Right: At test time, Cli$\mathcal{M}$er can perform
temporal grounding in long, dense videos.

##  1 Introduction

Suppose you have recorded a video of your birthday party using a wearable
device, and you wish to find the exact moment you blew out the candles, or
ripped open a present with reckless abandon. Searching through video data to
retrieve the temporal extents of meaningful moments can be an arduous task. A
method that can take in a sentence describing the intended moment and return
that moment, as a short temporal segment, is known as Temporal Sentence
Grounding [Anne Hendricks et al.(2017)Anne Hendricks, Wang, Shechtman, Sivic,
Darrell, and Russell, Gao et al.(2017)Gao, Sun, Yang, and Nevatia]. This
problem is commonly explored within datasets with videos of a couple of
minutes duration, typically grounding clips 10s of seconds long [Krishna et
al.(2017)Krishna, Hata, Ren, Fei-Fei, and Niebles, Gao et al.(2017)Gao, Sun,
Yang, and Nevatia, Anne Hendricks et al.(2017)Anne Hendricks, Wang, Shechtman,
Sivic, Darrell, and Russell, Regneri et al.(2013)Regneri, Rohrbach, Wetzel,
Thater, Schiele, and Pinkal].

In this work, we explore Temporal Sentence Grounding (TSG) for long-form
egocentric video datasets where searches may correspond to segments of only a
few seconds in videos that are up to an hour in length (See Table 1). Recent
works in TSG have explored long-form videos [Barrios et al.(2023)Barrios,
Soldan, Caba Heilbron, Ceballos-Arroyo, and Ghanem, Hou et al.(2022)Hou,
Zhong, Ji, Gao, Yan, Chan, Ngo, Shou, and Duan], yet still require full
supervision in the form of start/end times of clips. In contrast, rough
timestamps from video narrations have previously been used for learning other
tasks such as recognition [Moltisanti et al.(2019)Moltisanti, Fidler, and
Damen], detection [Li et al.(2021)Li, Abu Farha, and Gall, Ma et al.(2020)Ma,
Zhu, Yang, Zha, Kundu, Feiszli, and Shou, Rahaman et al.(2022)Rahaman,
Singhania, Thiery, and Yao], and spatio-temporal localisation [Mettes et
al.(2016)Mettes, Van Gemert, and Snoek]. In [Ma et al.(2020)Ma, Zhu, Yang,
Zha, Kundu, Feiszli, and Shou], Ma  _et al_ found that narration timestamps
roughly located at or near the relevant action require $6\times$ less
annotation effort than labelling with start/end times. The improved efficiency
is crucial for event-rich datasets with long videos. This proposes the unique
problem that we investigate within this paper: how do we train a successful
model for TSG on long videos with only narrations and their rough timestamps
as supervision.

Our proposed method, Cli$\mathcal{M}$er, artificially generates training
examples from long-form narrated videos by merging clips together to form a
merged segment, Figure 1 (left). The features from the merged segment can be
conditioned with the narrations, thus providing hard boundaries for
Cli$\mathcal{M}$er to learn from in a contrastive learning set-up. Because of
this training setup, at inference time our method is able to perform TSG over
long-form videos with a high density of annotated sentences, Figure 1 (right).

To summarise, (i) we propose Cli$\mathcal{M}$er for Temporal Sentence
Grounding, only using narrations and their rough timestamps as supervision.
Cli$\mathcal{M}$er merges clips as a supervisory signal to use for contrastive
learning. (ii) We explore for the first time using Ego4D and EPIC-Kitchens for
TSG providing train-test splits for this task. (iii) We show that
Cli$\mathcal{M}$er outperforms the baseline VSLNet [Zhang et al.(2021)Zhang,
Sun, Jing, Zhen, Zhou, and Goh] on these two datasets, ablating our design
decisions.

##  2 Related Work

|  | Total Vid
---
Duration
| Avg. Vid
---
Duration
| Avg. Mom.
---
Duration
| Annotations
---
/ Video
| Total
---
Annotations
| Avg.
---
Coverage $\downarrow$
ANet-Captions [Krishna et al.(2017)Krishna, Hata, Ren, Fei-Fei, and Niebles] | 487.6h | 2.0min | 37.1s | 4.9 | 72k | 30.90%
Charades-STA [Gao et al.(2017)Gao, Sun, Yang, and Nevatia] | 57.1h | 0.5min | 8.1s | 2.3 | 16k | 27.00%
DiDeMo [Anne Hendricks et al.(2017)Anne Hendricks, Wang, Shechtman, Sivic, Darrell, and Russell] | 88.7h | 0.5min | 6.5s | 3.9 | 41k | 21.70%
TACoS [Regneri et al.(2013)Regneri, Rohrbach, Wetzel, Thater, Schiele, and Pinkal] | 10.1h | 4.8min | 27.9s | 143.6 | 18k | 9.70%
Ego4D [Grauman et al.(2022)Grauman, Westbury, Byrne, Chavis, Furnari, Girdhar, Hamburger, Jiang, Liu, Liu, Martin, Nagarajan, Radosavovic, Ramakrishnan, Ryan, Sharma, Wray, Xu, Xu, Zhao, Bansal, Batra, Cartillier, Crane, Do, Doulaty, Erapalli, Feichtenhofer, Fragomeni, Fu, Fuegen, Gebreselasie, Gonzalez, Hillis, Huang, Huang, Jia, Khoo, Kolar, Kottur, Kumar, Landini, Li, Li, Li, Mangalam, Modhugu, Munro, Murrell, Nishiyasu, Price, Puentes, Ramazanova, Sari, Somasundaram, Southerland, Sugano, Tao, Vo, Wang, Wu, Yagi, Zhu, Arbelaez, Crandall, Damen, Farinella, Ghanem, Ithapu, Jawahar, Joo, Kitani, Li, Newcombe, Oliva, Park, Rehg, Sato, Shi, Shou, Torralba, Torresani, Yan, and Malik] | 234.9h | 17.3min | 2.0s | 214.1 | 223k | 0.19%
EPIC-Kitchens [Damen et al.(2022)Damen, Doughty, Farinella, Furnari, Ma, Kazakos, Moltisanti, Munro, Perrett, Price, and Wray] | 73.4h | 8.9min | 3.1s | 134.1 | 67k | 0.58%

Table 1: Temporal Sentence Grounding Datasets—comparative statistics. We
compare previously used datasets (top) to the egocentric ones we use (bottom).
Avg. Coverage showcases the ratio of the ground-truth moment duration to the
video duration. Ego4D and EPIC-Kitchens have the shortest avg. moment duration
with significantly lower coverage ($<1\%$).

TSG Datasets Table 1 shows a comparison between datasets previously used for
Temporal Sentence Grounding and the two EgoVideo datasets we explore in this
work, Ego4D [Grauman et al.(2022)Grauman, Westbury, Byrne, Chavis, Furnari,
Girdhar, Hamburger, Jiang, Liu, Liu, Martin, Nagarajan, Radosavovic,
Ramakrishnan, Ryan, Sharma, Wray, Xu, Xu, Zhao, Bansal, Batra, Cartillier,
Crane, Do, Doulaty, Erapalli, Feichtenhofer, Fragomeni, Fu, Fuegen,
Gebreselasie, Gonzalez, Hillis, Huang, Huang, Jia, Khoo, Kolar, Kottur, Kumar,
Landini, Li, Li, Li, Mangalam, Modhugu, Munro, Murrell, Nishiyasu, Price,
Puentes, Ramazanova, Sari, Somasundaram, Southerland, Sugano, Tao, Vo, Wang,
Wu, Yagi, Zhu, Arbelaez, Crandall, Damen, Farinella, Ghanem, Ithapu, Jawahar,
Joo, Kitani, Li, Newcombe, Oliva, Park, Rehg, Sato, Shi, Shou, Torralba,
Torresani, Yan, and Malik] and EPIC-Kitchens [Damen et al.(2022)Damen,
Doughty, Farinella, Furnari, Ma, Kazakos, Moltisanti, Munro, Perrett, Price,
and Wray]. Avg. Coverage (Avg. of Moment Duration/Video Duration) in
particular displays the distinction between these datasets. Previous datasets
aim to retrieve a moment $\geq 10\%$ of the video length on average. Ego4D and
EPIC-Kitchens have a significantly smaller coverage ($0.19\%$ and $0.58\%$
respectively), due to their fine-grained sentences with much shorter average
moment duration. Additionally, biases within previous TSG datasets ([Gao et
al.(2017)Gao, Sun, Yang, and Nevatia, Krishna et al.(2017)Krishna, Hata, Ren,
Fei-Fei, and Niebles]) have been explored in [Otani et al.(2020), Yuan et
al.(2021)Yuan, Lan, Wang, Chen, Wang, and Zhu]. Specifically, [Yuan et
al.(2021)Yuan, Lan, Wang, Chen, Wang, and Zhu] finds that the evaluated
methods “fail to utilize the video temporal relation or vision language
interaction” and was shown to be even more catastrophic for weakly supervised
approaches.

Our motivation to explore datasets with significantly smaller coverage is
related to the seminal dataset of movie descriptions, MAD [Soldan et
al.(2022)Soldan, Pardo, Alcázar, Caba, Zhao, Giancola, and Ghanem]. In MAD,
movies up to 3 hours long were explored for temporal grounding. Movies offer
different challenges including plot understanding and they typically contain
scene boundaries. In this work, we focus on EgoVideos that are unedited with
finer-grained narrations.

Fully Supervised Grounding Fully supervised approaches make use of exact start
and end times for each sentence during training. Approaches are divided
between proposal-based and proposal-free. Proposal-based methods [Anne
Hendricks et al.(2017)Anne Hendricks, Wang, Shechtman, Sivic, Darrell, and
Russell, Cao et al.(2021)Cao, Chen, Shou, Zhang, and Zou, Gao et al.(2017)Gao,
Sun, Yang, and Nevatia, Ge et al.(2019)Ge, Gao, Chen, and Nevatia, Jiang et
al.(2019)Jiang, Huang, Yang, and Yuan, Liu et al.(2018)Liu, Wang, Nie, Tian,
Chen, and Chua, Liu et al.(2021)Liu, Qu, Dong, Zhou, Cheng, Wei, Xu, and Xie,
Liu et al.(2022b)Liu, Qu, Zhou, and Liu, Xu et al.(2019)Xu, He, Plummer,
Sigal, Sclaroff, and Saenko, Chen et al.(2018)Chen, Chen, Ma, Jie, and Chua,
Qu et al.(2020)Qu, Tang, Zou, Cheng, Dong, Zhou, and Xu, Wang et
al.(2020)Wang, Ma, and Jiang, Zhang et al.(2019)Zhang, Peng, Fu, and Luo]
generate a set of candidate segments within the video and rank these for a
given sentence. Proposal-free methods [Chen et al.(2020b)Chen, Jiang, Liu, and
Jiang, Chen and Jiang(2020), Chen et al.(2020a)Chen, Lu, Tang, Xiao, Zhang,
Tan, and Li, Chen et al.(2021)Chen, Tsai, and Yang, Hao et al.(2022)Hao, Sun,
Ren, Wang, Qi, and Liao, Liu et al.(2022a)Liu, Qu, Di, Cheng, Xu, and Zhou, Lu
et al.(2019a)Lu, Chen, Tan, Li, and Xiao, Nan et al.(2021)Nan, Qiao, Xiao,
Liu, Leng, Zhang, and Lu, Rodriguez et al.(2020)Rodriguez, Marrese-Taylor,
Saleh, Li, and Gould, Yuan et al.(2019)Yuan, Mei, and Zhu, Zhang et
al.(2021)Zhang, Sun, Jing, Zhen, Zhou, and Goh, Zhao et al.(2020)Zhao, Xie,
Ju, Zhang, Wang, and Tian] directly predict the start and end times. A small
number of works have tackled the task of fully supervised long video grounding
[Hou et al.(2022)Hou, Zhong, Ji, Gao, Yan, Chan, Ngo, Shou, and Duan, Barrios
et al.(2023)Barrios, Soldan, Caba Heilbron, Ceballos-Arroyo, and Ghanem].
These have taken the approach of using a secondary module to first pick out
segments from the full video before applying models better suited to short
video grounding within these segments.

Weakly Supervised Grounding Weakly supervised approaches [Chen et
al.(2020c)Chen, Ma, Luo, Tang, and Wong, Chen et al.(2022)Chen, Luo, Zhang,
and Ma, Cui et al.(2022)Cui, Qian, Peng, Daskalaki, Chen, Guo, Sun, and Jiang,
Lin et al.(2020)Lin, Zhao, Zhang, Wang, and Liu, Mithun et al.(2019)Mithun,
Paul, and Roy-Chowdhury, Wang et al.(2021)Wang, Deng, Zhou, and Li, Yang et
al.(2021)Yang, Zhang, Zhang, and Wu, Xu et al.(2022)Xu, Wei, Yang, and Deng,
Zheng et al.(2022b)Zheng, Huang, Chen, Peng, and Liu, Zheng et
al.(2022a)Zheng, Huang, Chen, and Liu] do not have access to start and end
times for each sentence during training. Instead, videos are paired with
sentences which can be grounded somewhere within them. Recent approaches
include CPL [Zheng et al.(2022b)Zheng, Huang, Chen, Peng, and Liu], which
generates content-dependent proposals and mines hard negative samples from
within the same video, and LCNet [Yang et al.(2021)Yang, Zhang, Zhang, and
Wu], which extracts a hierarchical feature representation for video and text
and models the local correspondences between these. Similar to our work, [Chen
et al.(2022)Chen, Luo, Zhang, and Ma] also creates merged videos, yet their
merging approach assumes a dataset with high coverage by sampling a random
clip from the video to act as the ground truth for a sentence. In our method,
we merge clips based on the rough timestamp supervision and use hard negative
mining to ensure a trainable signal for Ego4D and EPIC-Kitchens, which have
much lower coverage. Other recent approaches [Cui et al.(2022)Cui, Qian, Peng,
Daskalaki, Chen, Guo, Sun, and Jiang, Xu et al.(2022)Xu, Wei, Yang, and Deng]
have made use of “glance”/“point” annotations, which are similar to the rough
timestamp we use, but both works make the assumption that the “glance”/“point”
is within the ground truth segment which can be unrealistic given potential
annotation noise, either human or automatic [Miech et al.(2019)Miech, Zhukov,
Alayrac, Tapaswi, Laptev, and Sivic].

##  3 Method

We first introduce the task of Temporal Sentence Grounding (TSG) from
Narration Timestamp Supervision in Sec. 3.1. Next, we provide an overview of
the method in Sec. 3.2. Finally, we give details of sampling sentences and
creating the merged segments in Sec. 3.3 followed by text conditioning in Sec.
3.4, as well as training losses and inference in Sec. 3.5.

###  3.1 Task Formulation: TSG from Narration Timestamps

Temporal Sentence Grounding is the problem of finding moments of a video that
ground a sentence. Formally, given a sentence $c_{i}$ and a video $x$111For
simplicity we drop the video index as Temporal Sentence Grounding assumes
oracle knowledge of the video that the sentence corresponds to., we wish to
find the start and end times—given by $t^{s}_{i}$ and $t^{e}_{i}$
respectively—within $x$ which ground the sentence $c_{i}$.

In the fully supervised setting, during both training and testing, methods use
the tuple $(x,c_{i},t^{s}_{i},t^{e}_{i})$ to first train and later evaluate
the model by comparing predicted times ($\hat{t}^{s}_{i}$ and
$\hat{t}^{e}_{i}$) with the ground truth times. In this work, we explore a
weaker form of supervision in which only rough timestamps are available for
each sentence during training. These represent a narration’s single timestamp
$t_{i}$ that roughly corresponds to the sentence as labelled by a narrator.
Thus, in this setting, models are trained using $(x,c_{i},t_{i})$ but are
still evaluated on their ability to ground the sentence to its annotated
start/end times. Naturally, this leads to a more difficult training regime,
yet has the benefit of requiring much less annotation time [Ma et al.(2020)Ma,
Zhu, Yang, Zha, Kundu, Feiszli, and Shou].

###  3.2 Cli$\mathcal{M}$er Overview

An overview of the approach can be seen in Figure 2, where from an untrimmed
video two sentences $c_{i}$ (yellow) and $c_{j}$ (green) are sampled. We
additionally sample a third sentence from the same video, $c_{k}$ (red), as a
negative sentence in order to train the model to distinguish video segments
that do not contain the corresponding grounding. As discussed in the previous
section, during training we only have access to single rough timestamps from
the narrations as supervision, _e.g_ $t_{i}$/$t_{j}$, for sentence
$c_{i}$/$c_{j}$. We first generate rough clips, $v_{i}$ and $v_{j}$ around
$t_{i}$ and $t_{j}$.

Next, we create a new input $S_{ij}$ for training, which we term the merged
segment, by merging the clips $v_{i}$ and $v_{j}$. This is beneficial for two
reasons: firstly, the artificial boundaries between clips act as a supervision
signal; and secondly, contrasting multiple sentences ensures that the method
learns to discriminate and ground individual sentences. As we artificially
perform the merging, we use the merging signal as supervision for learning to
ground.

We condition each sentence against $S_{ij}$, using cross-attentional text
conditioning (Figure 2 (b)) and train the model with three losses which
contrast the merged segment with the three sentences and regularise the
predictions. We detail our method next.

Figure 2: Overview of Cli$\mathcal{M}$er. (a) We generate a merged segment
($S_{ij}$) from two clips ($v_{i}$/$v_{j}$) of one long video to contrast
against sentences $c_{i}$/$c_{j}$/$c_{k}$. The video features are conditioned
using extracted text features and trained with 3 losses
$L_{adv},L_{sep},L_{neg}$. (b) Cross-Attentional Text Conditioning uses $n$
blocks of self- and cross-attention to condition the text.

###  3.3 Sampling Sentences and Merging Segments

We now explain how to sample the sentences, from a given video, and create a
merged segment on which to train. We avoid neighbouring clips, by ensuring
_e.g_ $|i-j|\geq 3$. Additionally, we check that all three sentences $c_{i}$,
$c_{j}$, and $c_{k}$ are semantically distinct, by ensuring the main action
(the verb) and/or the main object (the noun) are distinct. _E.g_ given the
sentence “The person closes the bin”, the sentence “The person opens the bin”
is considered distinct due to the main verb being different. However, “The
person shuts the trash can” is considered indistinct due to the main verb
(closes/shuts) and noun (bin/trash can) both being semantically equivalent.

Next, we segment the rough clips $v_{i}$ and $v_{j}$. To compensate for the
lack of temporal bounds, we use the timestamps of surrounding narrations,
$t_{i+1}$ and $t_{i-1}$ (resp. $t_{j+1}$ and $t_{j-1}$), as upper and lower
bounds for the clips. Note that there are many possible ways of initialising
these bounds (_e.g_ in [Lin et al.(2022)Lin, Wang, Soldan, Wray, Yan, XU, Gao,
Tu, Zhao, Kong, et al.]) which we explore further in Table 1 of the
supplementary material.

To merge the clips, we randomly sample two variables $\alpha,\beta$ such that
$0\leq\alpha\leq 1$ is the position at which we position the first clip and
$0<\beta\leq(1-\alpha)$ is its duration. From these, we define a merging
template $y$ as a rectangular function, so that $y(t)=1$ if
$\alpha<t<\alpha+\beta$, and $y(t)=0$ otherwise. This template merges the two
clips $v_{i}$ and $v_{j}$ such that:

$S_{ij}(t)=\begin{cases}v_{i}((t-\alpha)/\beta)&y(t)=1\\\
v_{j}(t/(1-\beta))&y(t)=0\quad\&\quad t<\alpha\\\
v_{j}((t-\beta)/(1-\beta))&y(t)=0\quad\&\quad t>\alpha\end{cases}$ (1)

where $0\leq t\leq 1$ represents the time along the normalised length of any
clip and $v_{i}(t)$ represents the clip $v_{i}$ at time $t$. The template $y$
also defines the supervisory signal ($y_{i}$ and $y_{j}$) for both sentences.
For the sentence $c_{i}$, $y_{i}=y$ is a binary vector set to 1 for all
locations where $S_{ij}$ contains parts of the clip $v_{i}$ and 0 otherwise.
Analogously $y_{j}=\mathbbm{1}_{{|y|}}-y$, where $\mathbbm{1}_{{|y|}}$ is a
vector of 1s of length $|y|$.

Note that the above definition assumes continuous time $t$. Instead, we pre-
extract visual video features $F$ at a fixed temporal rate from the video.
This rate is then used to discretise the merging operation noted above.

###  3.4 Text Conditioning

We condition the visual features $F_{ij}$ of the merged segment $S_{ij}$ on
the sentences $c_{i}$ and $c_{j}$, by matching to the corresponding sentence
and contrasting from the other sentences. A projection layer, $g$, is applied
to extracted word features to project the sentence embeddings into the same
space as the visual feature embeddings, resulting in final text features
$\hat{c}_{i}$ with $L\times d$ dimensions where $L$ is the length of the
sentence:

$\hat{c}_{i}=[g(f(w_{i,1})),...,g(f(w_{i,L}))]$ (2)

where $w_{i,l}$ represents the $l$th word in the sentence $c_{i}$ and $f$
represents the text feature extractor. To condition the visual features on
$\hat{c}_{i}$, we propose a combination of both self-attention and cross-
attention within successive transformer layers to learn correlations both
within and between the modalities, see Figure 2(b). We follow a similar
approach to [Lu et al.(2019b)Lu, Batra, Parikh, and Lee] where first visual
and text features undergo self-attention, before being concatenated and passed
through a transformer layer using cross-attention:

$P=\mathcal{A}^{s}_{v}(F_{ij})\quad;\quad\quad\quad
Q_{i}=\mathcal{A}^{s}_{t}(\hat{c}_{i})$ (3)

where $P=(p_{1},...,p_{M})$ and $Q_{i}=(q_{i,1},...,q_{i,L})$ are the outputs
of the self-attention layers and $A^{s}$ represents a self-attention layer.

$H_{i}^{\prime}=\mathcal{A}^{c}(H_{i})$ (4)

where $H_{i}=[p_{1},...,p_{m},...,p_{M},q_{i,1},...,q_{i,l},...,q_{i,L}]$ (the
concatenated video and textual outputs from the previous layer) and
$\mathcal{A}^{C}$ is a cross-attention transformer layer in which keys and
values are passed as inputs to the opposing modality. The combination of self-
attention and cross-attention layers, _i.e_ equations 3 and 4, are repeated
$n$ times. We then include a residual connection from the input visual
features. This is followed by two fully connected layers and a sigmoid
function that estimates the probability of each visual input feature being the
grounding for the sentence $c_{i}$:

$\hat{Y}_{i}=\mathcal{M}(F_{ij},c_{i})=\sigma\big{(}e(H^{\prime}_{i})+F_{ij}\big{)}$
(5)

where $\mathcal{M}$ defines the model Cli$\mathcal{M}$er, $e$ represents two
fully connected layers and $\hat{Y}_{i}$ is the vector of estimated
probabilities that the visual features ground the sentence $c_{i}$.

###  3.5 Training and Inference

During training, we randomly sample clips for merging within each video in the
dataset. We restrict the size of the merged clip $S_{ij}$ to a fixed size of
$M$ features for batching purposes. We next describe the losses used to train
Cli$\mathcal{M}$er.

The main contrasting loss, denoted collectively as $\mathcal{L}_{sep}$, is
applied between the model’s predicted probabilities $\hat{Y}$ and the merged
clips supervisory signals $y_{i}$, $y_{j}$.

$\mathcal{L}_{sep}=(1-\beta)\cdot\text{BCE}(\hat{Y}_{i},y_{i})+\beta\cdot\text{BCE}(\hat{Y}_{j},y_{j})$
(6)

where BCE is the Binary Cross Entropy Loss, balanced by the proportion of the
other sentence in the merged segment $S_{ij}$ using $\beta$ from Sec. 3.3 and
$\hat{Y}_{i}$ is the vector of estimated grounding probabilities from Eq 5.

$\mathcal{L}_{neg}$ is applied to the negative sentence $c_{k}$ to prevent the
method overfitting to the sampled boundaries within the merged segment
$S_{ij}$. This is then applied across video features that have been
conditioned on $c_{k}$ where resulting probabilities should be $0$:

$\mathcal{L}_{neg}=\text{BCE}(\mathcal{M}(S_{ij},c_{k}),0\cdot\mathbbm{1}_{|S_{ij}|})$
(7)

A further regularisation loss, which we term the adversarial loss
$\mathcal{L}_{adv}$, is used on the non-conditioned visual features (i.e.
without any text conditioning). The adversarial loss enforces that
Cli$\mathcal{M}$er uses both the textual query and the visual frames to
retrieve moments. This follows related works which find that temporal sentence
grounding models can perform well with no visual inputs [Otani et al.(2020),
Yuan et al.(2021)Yuan, Lan, Wang, Chen, Wang, and Zhu]. Using this loss,
visual features with no text are conditioned to produce a vector with maximal
uncertainty (i.e. $0.5\cdot\mathbbm{1}_{|S_{ij}|})$).

$\mathcal{L}_{adv}=\text{BCE}(\mathcal{M}(S_{ij}),0.5\cdot\mathbbm{1}_{|S_{ij}|})$
(8)

The final loss to train the model is given as:

$\mathcal{L}=\mathcal{L}_{sep}+\lambda_{neg}\cdot\mathcal{L}_{neg}+\lambda_{adv}\cdot\mathcal{L}_{adv}$
(9)

where $\lambda_{\\{neg,adv\\}}$ are weights used to balance the different
losses.

At inference, the model considers a single sentence and is applied to the full
untrimmed length of a single video, no longer restricting the number of
features. The model outputs the probability of grounding the sentence across
the full extent of the video, not merged segments. To predict the grounding,
we normalise the predictions between $0$ and $1$ and consider a threshold
$\epsilon$ to convert predictions into a grounding such that
${\forall\hat{t}:\hat{t}^{s}_{i}\leq\hat{t}\leq\hat{t}^{e}_{i}\iff\hat{Y}_{i}(\hat{t})>\epsilon}$.
Predicted groundings ($\hat{t}^{s}_{i}$, $\hat{t}^{e}_{i}$) are ranked by each
one’s maximum prediction value and returned as a ranked list. We then
calculate the temporal IoU of the highest ranked grounding against the ground
truth ($t^{s}_{i}$, $t^{e}_{i}$) to report performance.

##  4 Experiments

Metrics We use the standard metric for TSG, Recall@K (R@K) with Intersection
Over Union (IoU) at threshold $\theta$ (IoU=$\theta$). As in [Zhang et
al.(2021)Zhang, Sun, Jing, Zhen, Zhou, and Goh], we evaluate
${\theta\in\\{0.1,0.3,0.5\\}}$ and K=1. We also report Mean Recall@1 (mR@1)
metric introduced in [Barrios et al.(2023)Barrios, Soldan, Caba Heilbron,
Ceballos-Arroyo, and Ghanem] as the average across $\theta$.

Baselines We select VSLNet [Zhang et al.(2021)Zhang, Sun, Jing, Zhen, Zhou,
and Goh] and its implementation in [Grauman et al.(2022)Grauman, Westbury,
Byrne, Chavis, Furnari, Girdhar, Hamburger, Jiang, Liu, Liu, Martin,
Nagarajan, Radosavovic, Ramakrishnan, Ryan, Sharma, Wray, Xu, Xu, Zhao,
Bansal, Batra, Cartillier, Crane, Do, Doulaty, Erapalli, Feichtenhofer,
Fragomeni, Fu, Fuegen, Gebreselasie, Gonzalez, Hillis, Huang, Huang, Jia,
Khoo, Kolar, Kottur, Kumar, Landini, Li, Li, Li, Mangalam, Modhugu, Munro,
Murrell, Nishiyasu, Price, Puentes, Ramazanova, Sari, Somasundaram,
Southerland, Sugano, Tao, Vo, Wang, Wu, Yagi, Zhu, Arbelaez, Crandall, Damen,
Farinella, Ghanem, Ithapu, Jawahar, Joo, Kitani, Li, Newcombe, Oliva, Park,
Rehg, Sato, Shi, Shou, Torralba, Torresani, Yan, and Malik] as our baseline.
For fair comparison, we use the same Omnivore visual features and BERT text
features for training this baseline as we use for Cli$\mathcal{M}$er. This is
one of two baselines used for the NLQ task in Ego4D and has shown high
performance on grounding tasks in previous datasets. We train VSLNet using
artificially generated start and end times based on the rough timestamps. We
also include a random baseline which samples a random segment of any length
from the video to demonstrate the challenge of the test sets.

Datasets We use two datasets: Ego4D [Grauman et al.(2022)Grauman, Westbury,
Byrne, Chavis, Furnari, Girdhar, Hamburger, Jiang, Liu, Liu, Martin,
Nagarajan, Radosavovic, Ramakrishnan, Ryan, Sharma, Wray, Xu, Xu, Zhao,
Bansal, Batra, Cartillier, Crane, Do, Doulaty, Erapalli, Feichtenhofer,
Fragomeni, Fu, Fuegen, Gebreselasie, Gonzalez, Hillis, Huang, Huang, Jia,
Khoo, Kolar, Kottur, Kumar, Landini, Li, Li, Li, Mangalam, Modhugu, Munro,
Murrell, Nishiyasu, Price, Puentes, Ramazanova, Sari, Somasundaram,
Southerland, Sugano, Tao, Vo, Wang, Wu, Yagi, Zhu, Arbelaez, Crandall, Damen,
Farinella, Ghanem, Ithapu, Jawahar, Joo, Kitani, Li, Newcombe, Oliva, Park,
Rehg, Sato, Shi, Shou, Torralba, Torresani, Yan, and Malik] and EPIC-
Kitchens-100 [Damen et al.(2022)Damen, Doughty, Farinella, Furnari, Ma,
Kazakos, Moltisanti, Munro, Perrett, Price, and Wray]. Both datasets have
published a mapping from open vocabulary to closed vocabulary, using manual
and automatic clustering. For each dataset, we consider all verbs/nouns within
the same ‘closed vocabulary’ cluster to be semantically equivalent (i.e. pan
and saucepan are both contained within the pan cluster so are considered
semantically equivalent). These are used to measure the semantic equivalence
of sentences.

Ego4D: We consider a subset of videos from [Grauman et al.(2022)Grauman,
Westbury, Byrne, Chavis, Furnari, Girdhar, Hamburger, Jiang, Liu, Liu, Martin,
Nagarajan, Radosavovic, Ramakrishnan, Ryan, Sharma, Wray, Xu, Xu, Zhao,
Bansal, Batra, Cartillier, Crane, Do, Doulaty, Erapalli, Feichtenhofer,
Fragomeni, Fu, Fuegen, Gebreselasie, Gonzalez, Hillis, Huang, Huang, Jia,
Khoo, Kolar, Kottur, Kumar, Landini, Li, Li, Li, Mangalam, Modhugu, Munro,
Murrell, Nishiyasu, Price, Puentes, Ramazanova, Sari, Somasundaram,
Southerland, Sugano, Tao, Vo, Wang, Wu, Yagi, Zhu, Arbelaez, Crandall, Damen,
Farinella, Ghanem, Ithapu, Jawahar, Joo, Kitani, Li, Newcombe, Oliva, Park,
Rehg, Sato, Shi, Shou, Torralba, Torresani, Yan, and Malik] covering various
scenarios: {cooking, carpenter, scooter mechanic, car mechanic, gardener,
farmer}. Annotations from the hand-object interaction challenge are used as
ground truth in the val/test splits, as these videos contain human-labelled
start and end times with corresponding sentences. Due to the nature of the
Ego4D videos, there are often a number of repeated or semantically similar
sentences within the videos. Currently, TSG approaches assume there is only
one positive grounding moment for each sentence. To align with the grounding
task, any repeated or semantically indistinct sentences were excluded from the
val/test sets. This resulted in a dataset with 462/96/223 videos and
197k/1.9k/4.4k sentences in train/val/test. When using the val/test sets, only
full videos are used, no clip merging takes place.

EPIC-Kitchens-100: We use all 495 videos in the EPIC-Kitchens-100 train set to
make up our train/val/test splits. We use the narration annotations for
training and the manually labelled start-end times for evaluation. We
similarly remove semantically equivalent sentences, and the resulting dataset
has 332/50/110 videos and 45k/1.8k/3.8k sentences in train/val/test
respectively.

Features BERT [Devlin et al.(2019)Devlin, Chang, Lee, and Toutanova] is used
to generate features for the sentences, trained on BookCorpus [Zhu et
al.(2015)Zhu, Kiros, Zemel, Salakhutdinov, Urtasun, Torralba, and Fidler] and
English Wikipedia, generating 768-length features for each word. Visual
features are obtained from the video head of the Omnivore [Girdhar et
al.(2022)Girdhar, Singh, Ravi, van der Maaten, Joulin, and Misra] model (SwinL
for Ego4D, SwinB for EPIC-Kitchens) trained on Kinetics-400 and ImageNet-1K
jointly and are of length 1536 for Ego4D and 1024 for EPIC-Kitchens. Each
visual feature represents 32 frames of the video, and these are generated with
a stride of 16 frames. As Ego4D videos have an FPS of 30, the feature rate is
$\sim$1.87 per second. EPIC-Kitchens videos are downsampled to 30 FPS to match
this.

| IoU=0.1 | IoU=0.3 | IoU=0.5 | mR
---|---|---|---|---
Random Baseline | 0.47 | 0.09 | 0.02 | 0.19
VSLNet [Zhang et al.(2021)Zhang, Sun, Jing, Zhen, Zhou, and Goh] | 7.32 | 3.16 | 1.35 | 3.94
Cli$\mathcal{M}$er | 9.68 | 5.03 | 2.24 | 5.65

Table 2: Results for R@1 against baselines on Ego4D [Grauman et
al.(2022)Grauman, Westbury, Byrne, Chavis, Furnari, Girdhar, Hamburger, Jiang,
Liu, Liu, Martin, Nagarajan, Radosavovic, Ramakrishnan, Ryan, Sharma, Wray,
Xu, Xu, Zhao, Bansal, Batra, Cartillier, Crane, Do, Doulaty, Erapalli,
Feichtenhofer, Fragomeni, Fu, Fuegen, Gebreselasie, Gonzalez, Hillis, Huang,
Huang, Jia, Khoo, Kolar, Kottur, Kumar, Landini, Li, Li, Li, Mangalam,
Modhugu, Munro, Murrell, Nishiyasu, Price, Puentes, Ramazanova, Sari,
Somasundaram, Southerland, Sugano, Tao, Vo, Wang, Wu, Yagi, Zhu, Arbelaez,
Crandall, Damen, Farinella, Ghanem, Ithapu, Jawahar, Joo, Kitani, Li,
Newcombe, Oliva, Park, Rehg, Sato, Shi, Shou, Torralba, Torresani, Yan, and
Malik]

| IoU=0.1 | IoU=0.3 | IoU=0.5 | mR
---|---|---|---|---
Random Baseline | 0.78 | 0.13 | 0.03 | 0.31
VSLNet [Zhang et al.(2021)Zhang, Sun, Jing, Zhen, Zhou, and Goh] | 19.32 | 8.76 | 3.90 | 10.66
Cli$\mathcal{M}$er | 22.20 | 11.57 | 5.25 | 13.01

Table 3: Results for R@1 against baselines on EPIC-Kitchens [Damen et
al.(2022)Damen, Doughty, Farinella, Furnari, Ma, Kazakos, Moltisanti, Munro,
Perrett, Price, and Wray]

Figure 3: Examples from Ego4D comparing Cli$\mathcal{M}$er and VSLNet with
ground truth.

Model The textual features and visual features are first projected to the
common size of $2048$ dimensions. For Ego4D, the number of blocks of self and
cross-attention before the final self-attention block is set to $n=2$. For
EPIC-Kitchens, we switch the text attenuation module with a simpler one. In
this case, the visual features undergo two self-attention layers before a
Hadamard product is used to attenuate them with mean-pooled text features
instead of a cross attention layer. We ablate this design choice in Table 7
and show it is more suited to the simpler sentences within EPIC-Kitchens.

All attention layers contain 6 heads. The number of visual features during
training is set to $M=20$. We use the Adam optimiser with a learning rate of
$10^{-5}$ and batch size of 32. We use a dropout value of 0.3 and a value of
0.5 for the text feature projection. The model is trained for 100 epochs and
the epoch with the best validation result is selected to report performance on
the test set (_i.e_ we do not optimise on the test set). We set the weights
$\lambda_{neg}=\lambda_{adv}=1$ and ablate these losses. The prediction
threshold, $\epsilon$ is empirically chosen as 0.8 for Ego4D and 0.6 for EPIC-
Kitchens (see Figure 4).

###  4.1 Results

We first show results of Cli$\mathcal{M}$er on Ego4D in Table 3. The random
baseline shows the challenge of this dataset. Although VSLNet has been adapted
for long videos as described in [Grauman et al.(2022)Grauman, Westbury, Byrne,
Chavis, Furnari, Girdhar, Hamburger, Jiang, Liu, Liu, Martin, Nagarajan,
Radosavovic, Ramakrishnan, Ryan, Sharma, Wray, Xu, Xu, Zhao, Bansal, Batra,
Cartillier, Crane, Do, Doulaty, Erapalli, Feichtenhofer, Fragomeni, Fu,
Fuegen, Gebreselasie, Gonzalez, Hillis, Huang, Huang, Jia, Khoo, Kolar,
Kottur, Kumar, Landini, Li, Li, Li, Mangalam, Modhugu, Munro, Murrell,
Nishiyasu, Price, Puentes, Ramazanova, Sari, Somasundaram, Southerland,
Sugano, Tao, Vo, Wang, Wu, Yagi, Zhu, Arbelaez, Crandall, Damen, Farinella,
Ghanem, Ithapu, Jawahar, Joo, Kitani, Li, Newcombe, Oliva, Park, Rehg, Sato,
Shi, Shou, Torralba, Torresani, Yan, and Malik], Cli$\mathcal{M}$er
outperforms it across all metrics.

Table 3 includes the results on EPIC-Kitchens. Random performance continues to
be low for EPIC-Kitchens, and both methods perform considerably better than
it. Once again, Cli$\mathcal{M}$er achieves higher performance across all
metrics. We believe the contrastive learning enabled by our clip merging
method allows the model to learn to locate corresponding video segments more
effectively when given an input sentence.

Figure 3 shows qualitative examples comparing Cli$\mathcal{M}$er with VSLNet
[Zhang et al.(2021)Zhang, Sun, Jing, Zhen, Zhou, and Goh]. VSLNet often over-
predicts start/end times leading to poor IoU scores when compared to the
ground truth. An extreme example of this can be seen in the top right
predicting a $\sim$40s clip for a $\sim$2s ground truth segment. We find that
Cli$\mathcal{M}$er is able to match the ground truth more closely, despite the
fine-grained nature of the sentences. In the bottom right example,
Cli$\mathcal{M}$er continues to predict the part of the video where the tissue
is visible but is no longer being adjusted.

$c_{j}$ | $\mathcal{L}_{neg}$ | $\mathcal{L}_{adv}$ | IoU=0.1 | IoU=0.3 | IoU=0.5 | mR
---|---|---|---|---|---|---
$\checkmark$ | $\checkmark$ | - | 9.57 | 4.46 | 2.13 | 5.39
$\checkmark$ | - | $\checkmark$ | 8.61 | 3.62 | 1.58 | 4.60
- | $\checkmark$ | $\checkmark$ | 8.88 | 4.35 | 1.90 | 5.04
$\checkmark$ | $\checkmark$ | $\checkmark$ | 9.68 | 5.03 | 2.24 | 5.65

Table 4: Impact of removing each loss during training on Cli$\mathcal{M}$er.

$\lambda_{neg}$ | $\lambda_{adv}$ | IoU=0.1 | IoU=0.3 | IoU=0.5 | mR
---|---|---|---|---|---
0.5 | 1 | 8.93 | 4.12 | 1.95 | 5.00
1 | 0.5 | 8.88 | 4.49 | 2.08 | 5.15
1 | 0 | 9.57 | 4.46 | 2.13 | 5.39
0 | 1 | 8.61 | 3.62 | 1.58 | 4.60
1 | 1 | 9.68 | 5.03 | 2.24 | 5.65

Table 5: Effect of changing the loss weights on Cli$\mathcal{M}$er.

Merged Segment | Hard Negatives | IoU=0.1 | IoU=0.3 | IoU=0.5 | mR
---|---|---|---|---|---
$\checkmark$ | - | 7.28 | 3.09 | 1.08 | 3.82
- | $\checkmark$ | 7.23 | 4.42 | 2.20 | 4.62
$\checkmark$ | $\checkmark$ | 9.68 | 5.03 | 2.24 | 5.65

Table 6: Performance on Ego4D with ablations on merging clips to produce the
supervision signal and sampling clips from the same video as hard negatives.
(Left inset) We show how each row corresponds to the merged segment creation.

###  4.2 Ablation Studies

Losses In Table 4 we showcase that each loss contributes to the model’s
performance, with performance decreasing when any are omitted. Specifically,
removing the loss of the second sentence ($c_{j}$) from $\mathcal{L}_{sep}$
removes the contrastive learning aspect of the model, leading to performance
decreases, albeit still being able to achieve reasonable performance via
grounding the single sentence only. Removing $\mathcal{L}_{adv}$ has the least
impact, being a regularisation loss. Removing $\mathcal{L}_{neg}$ leads to the
largest performance drop, as the model otherwise assumes that all sentences
can be grounded resulting in an increase of false positives.

We also adjust the weights of the negative and adversarial losses
($\lambda_{neg}$ and $\lambda_{adv}$ from Eq. 9) in the loss function to
examine the effect this has on model performance for Ego4D. The results in
Table 5 show that decreasing either weights results in reduced performance
across all metrics. This demonstrates the importance of each loss having equal
weighting to the main separation loss, providing complementary signals during
training.

Clip Merging We investigate the impact of merging segments. We replace merging
with using a single clip with surrounding background. The background is
selected to make up between 25% and 75% of the video at random, retaining the
value of $M=20$ for the number of features. The losses are retained as before
however the weighting of loss $\mathcal{L}_{sep}$ is doubled to compensate for
the lack of contribution from $c_{j}$.

From results in Table 6 (row 2) without merging, performance drops. The
decrease at IoU=0.1 in particular shows that the model without merging
struggles to learn the general alignment between sentences and video, likely
due to the lack of contrastive learning and the decreased visual diversity
found in continuous segments compared to merged segments. The model trained
without merging frequently misses the ground truth moment entirely. The
results show that clip merging increases robustness to changes in clip
positions and lengths in full videos.

Hard Negative Sampling Table 6 also compares clips sampled as hard negatives
from the same video against when they are sampled from random videos. A
significant performance drop is shown in the latter case. Hard negatives force
the model to distinguish between the actions occurring within the videos as
the visual environments of the two clips are similar.

| Ego4D |  | EPIC-Kitchens
---|---|---|---
| IoU=0.1 | IoU=0.3 | IoU=0.5 | mR |  | IoU=0.1 | IoU=0.3 | IoU=0.5 | mR
Hadamard Prod. Cross Att. | 9.29 | 4.30 | 2.04 | 5.21 |  | 22.20 | 11.57 | 5.25 | 13.01
Learned Cross Att. | 9.68 | 5.03 | 2.24 | 5.65 |  | 19.79 | 10.58 | 5.46 | 11.94

Table 7: Ablation comparing the type of text attenuation used: Cross-attention
as in Figure 2(b) or using a Hadamard Product (shown right).

Figure 4: Cli$\mathcal{M}$er performance as the prediction threshold,
$\epsilon$, is varied on Ego4D (Left) and EPIC-Kitchens (Right). The vertical
lines indicate our thresholds of 0.8 and 0.6.

Text Attenuation Table 7 evaluates the choice of text conditioning module,
comparing using Cross Attention (see Section 3.4) with the Hadamard product.
The simpler module which we use for EPIC-Kitchens works best due to the
smaller dataset and shorter sentences (avg. $3.0$ words) vs. Ego4D (avg. $8.4$
words).

Prediction Threshold Figure 4 displays the effect of varying the value of the
prediction threshold $\epsilon$. The threshold $\epsilon$ has more impact on
Ego4D, however the figure demonstrates that the model is reasonably robust to
the choice of $\epsilon>0.5$.

##  5 Conclusion

In this work we have explored Temporal Sentence Grounding for long-form
egocentric datasets Ego4D and EPIC-Kitchens. To overcome the need for full
supervision, we propose to merge clips from the same video from rough
narration timestamps. Our method, Cli$\mathcal{M}$er, trains using contrastive
learning with text-conditioning over sentences. Results demonstrate the
ability of Cli$\mathcal{M}$er to tackle long videos at test time, grounding
fine-grained sentences in egocentric videos which can be over an hour in
length.

#### Acknowledgments.

K Flanagan is supported by UKRI (Grant ref EP/S022937/1) CDT in Interactive AI
& Qinetiq Ltd via studentship CON11954. D Damen is supported by EPSRC
Fellowship UMPIRE (EP/T004991/1) & EPSRC Program Grant Visual AI
(EP/T028572/1).

## References

  * [Anne Hendricks et al.(2017)Anne Hendricks, Wang, Shechtman, Sivic, Darrell, and Russell] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell.  Localizing moments in video with natural language.  In _ICCV_ , 2017.
  * [Barrios et al.(2023)Barrios, Soldan, Caba Heilbron, Ceballos-Arroyo, and Ghanem] Wayner Barrios, Mattia Soldan, Fabian Caba Heilbron, Alberto M. Ceballos-Arroyo, and Bernard Ghanem.  Localizing moments in long video via multimodal guidance.  _ICCV_ , 2023.
  * [Cao et al.(2021)Cao, Chen, Shou, Zhang, and Zou] Meng Cao, Long Chen, Mike Zheng Shou, Can Zhang, and Yuexian Zou.  On pursuit of designing multi-modal transformer for video grounding.  In _EMNLP_ , 2021.
  * [Chen et al.(2022)Chen, Luo, Zhang, and Ma] Jiaming Chen, Weixin Luo, Wei Zhang, and Lin Ma.  Explore inter-contrast between videos via composition for weakly supervised temporal sentence grounding.  In _AAAI_ , 2022.
  * [Chen et al.(2018)Chen, Chen, Ma, Jie, and Chua] Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jie, and Tat-Seng Chua.  Temporally grounding natural sentence in video.  In _EMNLP_ , 2018.
  * [Chen et al.(2020a)Chen, Lu, Tang, Xiao, Zhang, Tan, and Li] Long Chen, Chujie Lu, Siliang Tang, Jun Xiao, Dong Zhang, Chilie Tan, and Xiaolin Li.  Rethinking the bottom-up framework for query-based video localization.  In _AAAI_ , 2020a.
  * [Chen and Jiang(2020)] Shaoxiang Chen and Yu-Gang Jiang.  Hierarchical visual-textual graph for temporal activity localization via language.  In _ECCV_ , 2020.
  * [Chen et al.(2020b)Chen, Jiang, Liu, and Jiang] Shaoxiang Chen, Wenhao Jiang, Wei Liu, and Yu-Gang Jiang.  Learning modality interaction for temporal sentence localization and event captioning in videos.  In _ECCV_ , 2020b.
  * [Chen et al.(2021)Chen, Tsai, and Yang] Yi-Wen Chen, Yi-Hsuan Tsai, and Ming-Hsuan Yang.  End-to-end multi-modal video temporal grounding.  _NeurIPS_ , 2021.
  * [Chen et al.(2020c)Chen, Ma, Luo, Tang, and Wong] Zhenfang Chen, Lin Ma, Wenhan Luo, Peng Tang, and Kwan-Yee K Wong.  Look closer to ground better: Weakly-supervised temporal grounding of sentence in video.  _CoRR abs/2001.09308_ , 2020c.
  * [Cui et al.(2022)Cui, Qian, Peng, Daskalaki, Chen, Guo, Sun, and Jiang] Ran Cui, Tianwen Qian, Pai Peng, Elena Daskalaki, Jingjing Chen, Xiaowei Guo, Huyang Sun, and Yu-Gang Jiang.  Video moment retrieval from text queries via single frame annotation.  In _SIGIR_ , 2022.
  * [Damen et al.(2022)Damen, Doughty, Farinella, Furnari, Ma, Kazakos, Moltisanti, Munro, Perrett, Price, and Wray] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray.  Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100.  _IJCV_ , 2022.
  * [Devlin et al.(2019)Devlin, Chang, Lee, and Toutanova] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HLT_ , 2019.
  * [Gao et al.(2017)Gao, Sun, Yang, and Nevatia] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia.  Tall: Temporal activity localization via language query.  In _ICCV_ , 2017.
  * [Ge et al.(2019)Ge, Gao, Chen, and Nevatia] Runzhou Ge, Jiyang Gao, Kan Chen, and Ram Nevatia.  Mac: Mining activity concepts for language-based temporal localization.  In _WACV_ , 2019.
  * [Girdhar et al.(2022)Girdhar, Singh, Ravi, van der Maaten, Joulin, and Misra] Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra.  Omnivore: A single model for many visual modalities.  In _CVPR_ , 2022.
  * [Grauman et al.(2022)Grauman, Westbury, Byrne, Chavis, Furnari, Girdhar, Hamburger, Jiang, Liu, Liu, Martin, Nagarajan, Radosavovic, Ramakrishnan, Ryan, Sharma, Wray, Xu, Xu, Zhao, Bansal, Batra, Cartillier, Crane, Do, Doulaty, Erapalli, Feichtenhofer, Fragomeni, Fu, Fuegen, Gebreselasie, Gonzalez, Hillis, Huang, Huang, Jia, Khoo, Kolar, Kottur, Kumar, Landini, Li, Li, Li, Mangalam, Modhugu, Munro, Murrell, Nishiyasu, Price, Puentes, Ramazanova, Sari, Somasundaram, Southerland, Sugano, Tao, Vo, Wang, Wu, Yagi, Zhu, Arbelaez, Crandall, Damen, Farinella, Ghanem, Ithapu, Jawahar, Joo, Kitani, Li, Newcombe, Oliva, Park, Rehg, Sato, Shi, Shou, Torralba, Torresani, Yan, and Malik] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Christian Fuegen, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Giovanni Maria Farinella, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik.  Ego4d: Around the World in 3,000 Hours of Egocentric Video.  In _CVPR_ , 2022.
  * [Hao et al.(2022)Hao, Sun, Ren, Wang, Qi, and Liao] Jiachang Hao, Haifeng Sun, Pengfei Ren, Jingyu Wang, Qi Qi, and Jianxin Liao.  Query-aware video encoder for video moment retrieval.  _Neurocomputing_ , 2022.
  * [Hou et al.(2022)Hou, Zhong, Ji, Gao, Yan, Chan, Ngo, Shou, and Duan] Zhijian Hou, Wanjun Zhong, Lei Ji, Difei Gao, Kun Yan, Wing-Kwong Chan, Chong-Wah Ngo, Zheng Shou, and Nan Duan.  Cone: An efficient coarse-to-fine alignment framework for long video temporal grounding.  _ECCVW_ , 2022.
  * [Jiang et al.(2019)Jiang, Huang, Yang, and Yuan] Bin Jiang, Xin Huang, Chao Yang, and Junsong Yuan.  Cross-modal video moment retrieval with spatial and language-temporal attention.  In _ICMR_ , pages 217–225, 2019.
  * [Krishna et al.(2017)Krishna, Hata, Ren, Fei-Fei, and Niebles] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles.  Dense-captioning events in videos.  In _ICCV_ , 2017.
  * [Li et al.(2021)Li, Abu Farha, and Gall] Zhe Li, Yazan Abu Farha, and Jurgen Gall.  Temporal action segmentation from timestamp supervision.  In _CVPR_ , 2021.
  * [Lin et al.(2022)Lin, Wang, Soldan, Wray, Yan, XU, Gao, Tu, Zhao, Kong, et al.] Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Z XU, Difei Gao, Rong-Cheng Tu, Wenzhe Zhao, Weijie Kong, et al.  Egocentric video-language pretraining.  _NeurIPS_ , 2022.
  * [Lin et al.(2020)Lin, Zhao, Zhang, Wang, and Liu] Zhijie Lin, Zhou Zhao, Zhu Zhang, Qi Wang, and Huasheng Liu.  Weakly-supervised video moment retrieval via semantic completion network.  In _AAAI_ , 2020.
  * [Liu et al.(2021)Liu, Qu, Dong, Zhou, Cheng, Wei, Xu, and Xie] Daizong Liu, Xiaoye Qu, Jianfeng Dong, Pan Zhou, Yu Cheng, Wei Wei, Zichuan Xu, and Yulai Xie.  Context-aware biaffine localizing network for temporal sentence grounding.  In _CVPR_ , 2021.
  * [Liu et al.(2022a)Liu, Qu, Di, Cheng, Xu, and Zhou] Daizong Liu, Xiaoye Qu, Xing Di, Yu Cheng, Zichuan Xu, and Pan Zhou.  Memory-guided semantic learning network for temporal sentence grounding.  In _AAAI_ , 2022a.
  * [Liu et al.(2022b)Liu, Qu, Zhou, and Liu] Daizong Liu, Xiaoye Qu, Pan Zhou, and Yang Liu.  Exploring motion and appearance information for temporal sentence grounding.  In _AAAI_ , 2022b.
  * [Liu et al.(2018)Liu, Wang, Nie, Tian, Chen, and Chua] Meng Liu, Xiang Wang, Liqiang Nie, Qi Tian, Baoquan Chen, and Tat-Seng Chua.  Cross-modal moment localization in videos.  In _MM_ , 2018.
  * [Lu et al.(2019a)Lu, Chen, Tan, Li, and Xiao] Chujie Lu, Long Chen, Chilie Tan, Xiaolin Li, and Jun Xiao.  Debug: A dense bottom-up grounding approach for natural language video localization.  In _EMNLP-IJCNLP_ , 2019a.
  * [Lu et al.(2019b)Lu, Batra, Parikh, and Lee] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _NeurIPS_ , 2019b.
  * [Ma et al.(2020)Ma, Zhu, Yang, Zha, Kundu, Feiszli, and Shou] Fan Ma, Linchao Zhu, Yi Yang, Shengxin Zha, Gourab Kundu, Matt Feiszli, and Zheng Shou.  Sf-net: Single-frame supervision for temporal action localization.  In _ECCV_ , 2020.
  * [Mettes et al.(2016)Mettes, Van Gemert, and Snoek] Pascal Mettes, Jan C Van Gemert, and Cees GM Snoek.  Spot on: Action localization from pointly-supervised proposals.  In _ECCV_ , 2016.
  * [Miech et al.(2019)Miech, Zhukov, Alayrac, Tapaswi, Laptev, and Sivic] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic.  Howto100m: Learning a text-video embedding by watching hundred million narrated video clips.  In _ICCV_ , 2019.
  * [Mithun et al.(2019)Mithun, Paul, and Roy-Chowdhury] Niluthpol Chowdhury Mithun, Sujoy Paul, and Amit K Roy-Chowdhury.  Weakly supervised video moment retrieval from text queries.  In _CVPR_ , 2019.
  * [Moltisanti et al.(2019)Moltisanti, Fidler, and Damen] Davide Moltisanti, Sanja Fidler, and Dima Damen.  Action recognition from single timestamp supervision in untrimmed videos.  In _CVPR_ , 2019.
  * [Nan et al.(2021)Nan, Qiao, Xiao, Liu, Leng, Zhang, and Lu] Guoshun Nan, Rui Qiao, Yao Xiao, Jun Liu, Sicong Leng, Hao Zhang, and Wei Lu.  Interventional video grounding with dual contrastive learning.  In _CVPR_ , 2021.
  * [Otani et al.(2020)] Mayu Otani et al.  Uncovering hidden challenges in query-based video moment retrieval.  In _BMVC_ , 2020.
  * [Qu et al.(2020)Qu, Tang, Zou, Cheng, Dong, Zhou, and Xu] Xiaoye Qu, Pengwei Tang, Zhikang Zou, Yu Cheng, Jianfeng Dong, Pan Zhou, and Zichuan Xu.  Fine-grained iterative attention network for temporal language localization in videos.  In _MM_ , 2020.
  * [Rahaman et al.(2022)Rahaman, Singhania, Thiery, and Yao] Rahul Rahaman, Dipika Singhania, Alexandre Thiery, and Angela Yao.  A generalized and robust framework for timestamp supervision in temporal action segmentation.  In _ECCV_ , 2022.
  * [Regneri et al.(2013)Regneri, Rohrbach, Wetzel, Thater, Schiele, and Pinkal] Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal.  Grounding action descriptions in videos.  _TACL_ , 2013.
  * [Rodriguez et al.(2020)Rodriguez, Marrese-Taylor, Saleh, Li, and Gould] Cristian Rodriguez, Edison Marrese-Taylor, Fatemeh Sadat Saleh, Hongdong Li, and Stephen Gould.  Proposal-free temporal moment localization of a natural-language query in video using guided attention.  In _WACV_ , 2020.
  * [Soldan et al.(2022)Soldan, Pardo, Alcázar, Caba, Zhao, Giancola, and Ghanem] Mattia Soldan, Alejandro Pardo, Juan León Alcázar, Fabian Caba, Chen Zhao, Silvio Giancola, and Bernard Ghanem.  Mad: A scalable dataset for language grounding in videos from movie audio descriptions.  In _CVPR_ , 2022.
  * [Wang et al.(2020)Wang, Ma, and Jiang] Jingwen Wang, Lin Ma, and Wenhao Jiang.  Temporally grounding language queries in videos by contextual boundary-aware prediction.  In _AAAI_ , 2020.
  * [Wang et al.(2021)Wang, Deng, Zhou, and Li] Yuechen Wang, Jiajun Deng, Wengang Zhou, and Houqiang Li.  Weakly supervised temporal adjacent network for language grounding.  _TMM_ , 2021.
  * [Xu et al.(2019)Xu, He, Plummer, Sigal, Sclaroff, and Saenko] Huijuan Xu, Kun He, Bryan A Plummer, Leonid Sigal, Stan Sclaroff, and Kate Saenko.  Multilevel language and vision integration for text-to-clip retrieval.  In _AAAI_ , 2019.
  * [Xu et al.(2022)Xu, Wei, Yang, and Deng] Zhe Xu, Kun Wei, Xu Yang, and Cheng Deng.  Point-supervised video temporal grounding.  _TMM_ , 2022.
  * [Yang et al.(2021)Yang, Zhang, Zhang, and Wu] Wenfei Yang, Tianzhu Zhang, Yongdong Zhang, and Feng Wu.  Local correspondence network for weakly supervised temporal sentence grounding.  _TIP_ , 2021.
  * [Yuan et al.(2019)Yuan, Mei, and Zhu] Yitian Yuan, Tao Mei, and Wenwu Zhu.  To find where you talk: Temporal sentence localization in video with attention based location regression.  In _AAAI_ , 2019.
  * [Yuan et al.(2021)Yuan, Lan, Wang, Chen, Wang, and Zhu] Yitian Yuan, Xiaohan Lan, Xin Wang, Long Chen, Zhi Wang, and Wenwu Zhu.  A closer look at temporal sentence grounding in videos: Dataset and metric.  In _HuMA_ , 2021.
  * [Zhang et al.(2021)Zhang, Sun, Jing, Zhen, Zhou, and Goh] Hao Zhang, Aixin Sun, Wei Jing, Liangli Zhen, Joey Tianyi Zhou, and Rick Siow Mong Goh.  Natural language video localization: A revisit in span-based question answering framework.  _TPAMI_ , 2021.
  * [Zhang et al.(2019)Zhang, Peng, Fu, and Luo] Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo.  Learning 2d temporal adjacent networks for moment localization with natural language.  In _AAAI_ , 2019.
  * [Zhao et al.(2020)Zhao, Xie, Ju, Zhang, Wang, and Tian] Peisen Zhao, Lingxi Xie, Chen Ju, Ya Zhang, Yanfeng Wang, and Qi Tian.  Bottom-up temporal action localization with mutual regularization.  In _ECCV_ , 2020.
  * [Zheng et al.(2022a)Zheng, Huang, Chen, and Liu] Minghang Zheng, Yanjie Huang, Qingchao Chen, and Yang Liu.  Weakly supervised video moment localization with contrastive negative sample mining.  In _AAAI_ , 2022a.
  * [Zheng et al.(2022b)Zheng, Huang, Chen, Peng, and Liu] Minghang Zheng, Yanjie Huang, Qingchao Chen, Yuxin Peng, and Yang Liu.  Weakly supervised temporal sentence grounding with gaussian-based contrastive proposal learning.  In _ICCV_ , 2022b.
  * [Zhu et al.(2015)Zhu, Kiros, Zemel, Salakhutdinov, Urtasun, Torralba, and Fidler] Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.  In _ICCV_ , pages 19–27, 2015.