Evangelos<EMAIL_ADDRESS>Jaesung
<EMAIL_ADDRESS>Arsha<EMAIL_ADDRESS>Andrew<EMAIL_ADDRESS>Dima<EMAIL_ADDRESS>Dept.
of Computer Science
University of Bristol
Bristol, UK Visual Geometry Group
University of Oxford,
Oxford, UK With a Little Help from my Temporal Context

# With a Little Help from my Temporal Context: Multimodal Egocentric Action
Recognition

###### Abstract

In egocentric videos, actions occur in quick succession. We capitalise on the
action’s temporal context and propose a method that learns to attend to
surrounding actions in order to improve recognition performance. To
incorporate the temporal context, we propose a transformer-based multimodal
model that ingests video and audio as input modalities, with an explicit
language model providing action sequence context to enhance the predictions.
We test our approach on EPIC-KITCHENS and EGTEA datasets reporting state-of-
the-art performance. Our ablations showcase the advantage of utilising
temporal context as well as incorporating audio input modality and language
model to rescore predictions. Code and models at:
https://github.com/ekazakos/MTCN.

22footnotetext: Now at Google Research.

##  1 Introduction

Figure 1: Egocentric video demonstrating two temporal context windows (pink,
green), centred around the action to be recognised. We can infer ‘wash
aubergine’ with higher accuracy if we know that the tap was turned on before
and turned off afterwards.

Action recognition in egocentric video streams from sources like EPIC-KITCHENS
poses a number of challenges that differ substantially from those of
conventional third-person action recognition – where training and evaluation
is on 10 second video clips and classes are quite high-level [Kay et
al.(2017)Kay, Carreira, Simonyan, Zhang, Hillier, Vijayanarasimhan, Viola,
Green, Back, Natsev, Suleyman, and Zisserman]. Actions are fine-grained (e.g.
‘open bottle’) and noticeably short, often one second or shorter. Along with
the challenge, the footage offers an under-explored opportunity, as actions
are captured in long untrimmed videos of well-defined and at-times predictable
sequences. For example the action ‘wash aubergine’ can be part of the
following sequence – you first ‘take the aubergine’, ‘turn on the tap’, ‘wash
the aubergine’ and finally ‘turn off the tap’ (Fig. 1). Furthermore, the
objects (the aubergine and tap in this case) are persistent over some of the
neighbouring actions.

In this work, we investigate utilising not only the action’s temporal context
in the data stream, but also the temporal context from the labels of
neighbouring actions. We propose a model that attends to neighbouring
actions111Note that these action start/end times are readily available in
labelled datasets for untrimmed videos and do not require additional labels.
We just leverage these.. Concretely, we use the attention mechanism of a
multimodal transformer architecture to take account of the context in both the
data and labels, using three modalities: vision, audio and language. We are
motivated by previous works that demonstrate the significance of audio in
recognising egocentric actions [Kazakos et al.(2019)Kazakos, Nagrani,
Zisserman, and Damen, Wang et al.(2020a)Wang, Tran, and Feiszli, Xiao et
al.(2020b)Xiao, Lee, Grauman, Malik, and Feichtenhofer], and thus include the
auditory temporal context in addition to the visual clips. We also utilise
context further, by training a language model on the sequence of action
labels, inspired by the success of language models [Devlin et al.(2019)Devlin,
Chang, Lee, and Toutanova, Yang et al.(2019)Yang, Dai, Yang, Carbonell,
Salakhutdinov, and Le, Brown et al.(2020)Brown, Mann, Ryder, Subbiah, Kaplan,
Dhariwal, Neelakantan, Shyam, Sastry, Askell, Agarwal, Herbert-Voss, Krueger,
Henighan, Child, Ramesh, Ziegler, Wu, Winter, Hesse, Chen, Sigler, Litwin,
Gray, Chess, Clark, Berner, McCandlish, Radford, Sutskever, and Amodei] in re-
scoring model outputs for speech recognition [Chan et al.(2016)Chan, Jaitly,
Le, and Vinyals, Mikolov et al.(2010)Mikolov, Karafiát, Burget, Černockỳ, and
Khudanpur, Hannun et al.(2014)Hannun, Case, Casper, Catanzaro, Diamos, Elsen,
Prenger, Satheesh, Sengupta, Coates, and Ng] and machine translation [Gulcehre
et al.(2017)Gulcehre, Firat, Xu, Cho, and Bengio] (Fig. 2).

Our main contributions are summarised as follows: First, we formulate temporal
context as a sequence of actions surrounding the action in a sliding window.
Second, we propose a novel framework able to model multimodal temporal
context. It consists of a transformer encoder that uses vision and audio as
input context, and a language model as output context operating on the action
labels. Third, we obtain state-of-the-art performance on two datasets: (i) the
large-scale egocentric dataset EPIC-KITCHENS, outperforming high-capacity end-
to-end transformer models; and (ii) the EGTEA dataset. Finally, we include an
ablation study analysing the importance of the extent of the temporal context
and of the various modalities.

##  2 Related Work

Action Recognition. There is a rich literature of seminal works in action
recognition innovating temporal sampling [Wang et al.(2016a)Wang, Xiong, Wang,
Qiao, Lin, Tang, and Van Gool, Zhou et al.(2018)Zhou, Andonian, Oliva, and
Torralba, Kazakos et al.(2019)Kazakos, Nagrani, Zisserman, and Damen],
multiple streams [Feichtenhofer et al.(2019)Feichtenhofer, Fan, Malik, and
He], spatio-temporal modelling [Tran et al.(2018)Tran, Wang, Torresani, Ray,
LeCun, and Paluri, Lin et al.(2019)Lin, Gan, and Han, Girdhar et
al.(2019)Girdhar, Carreira, Doersch, and Zisserman] or modelling actions as
transformations from initial to final states [Wang et al.(2016b)Wang, Farhadi,
and Gupta]. Our work is related to the more recent transformer-based
approaches [Girdhar et al.(2019)Girdhar, Carreira, Doersch, and Zisserman,
Neimark et al.(2021)Neimark, Bar, Zohar, and Asselmann, Bertasius et
al.(2021)Bertasius, Wang, and Torresani, Arnab et al.(2021)Arnab, Dehghani,
Heigold, Sun, Lucic, and Schmid, Bulat et al.(2021)Bulat, Perez-Rua,
Sudhakaran, Martínez, and Tzimiropoulos, Patrick et al.(2021)Patrick,
Campbell, Asano, Metze, Feichtenhofer, Vedaldi, and Henriques, Nagrani et
al.(2021)Nagrani, Yang, Arnab, Schmid, and Sun]. We compare our model with
four recent works [Arnab et al.(2021)Arnab, Dehghani, Heigold, Sun, Lucic, and
Schmid, Bulat et al.(2021)Bulat, Perez-Rua, Sudhakaran, Martínez, and
Tzimiropoulos, Patrick et al.(2021)Patrick, Campbell, Asano, Metze,
Feichtenhofer, Vedaldi, and Henriques, Nagrani et al.(2021)Nagrani, Yang,
Arnab, Schmid, and Sun]. [Arnab et al.(2021)Arnab, Dehghani, Heigold, Sun,
Lucic, and Schmid] investigated spatio-temporal attention factorisation
schemes, while in [Bulat et al.(2021)Bulat, Perez-Rua, Sudhakaran, Martínez,
and Tzimiropoulos] the authors propose full-attention within a temporal
window. [Patrick et al.(2021)Patrick, Campbell, Asano, Metze, Feichtenhofer,
Vedaldi, and Henriques] proposes temporal attention along trajectories with
learnt tokens. [Nagrani et al.(2021)Nagrani, Yang, Arnab, Schmid, and Sun]
proposes fusion bottlenecks for cross-modal attention. These approaches
require training on large-scale datasets and strong data augmentation for
generalisation. Our model operates on pre-extracted features which are
processed with a lightweight transformer and outperforms these works by
relying on multimodal temporal context.

In egocentric action recognition, researchers propose a range of techniques to
address its unique challenges [Sudhakaran et al.(2019)Sudhakaran, Escalera,
and Lanz, Baradel et al.(2018)Baradel, Neverova, Wolf, Mille, and Mori, Li et
al.(2021b)Li, Nagarajan, Xiong, and Grauman, Kazakos et al.(2019)Kazakos,
Nagrani, Zisserman, and Damen, Munro and Damen(2020)]. [Sudhakaran et
al.(2019)Sudhakaran, Escalera, and Lanz] considered long-term understanding
through an LSTM with attention to focus on relevant spatio-temporal regions,
but the approach operates within the action clip solely. [Baradel et
al.(2018)Baradel, Neverova, Wolf, Mille, and Mori] shows that modelling hand-
object interactions is beneficial, while in [Li et al.(2021b)Li, Nagarajan,
Xiong, and Grauman] the authors pre-train egocentric networks by distilling
egocentric signals from large-scale third-person datasets. [Kazakos et
al.(2019)Kazakos, Nagrani, Zisserman, and Damen] shows that audio is key to
egocentric action recognition due to sounds produced from interactions with
objects. In [Munro and Damen(2020)], the authors propose a multimodal
unsupervised domain adaptation approach to tackle the distribution shift
between environments. We build on these works but propose the first approach
to incorporate both audio-visual and language-model predictions.

Temporal Context for Video Recognition Tasks. A few works have considered
temporal context to improve the models’ performance for action anticipation
[Sener et al.(2020)Sener, Singhania, and Yao, Furnari and Farinella(2019)],
action recognition [Wu et al.(2019)Wu, Feichtenhofer, Fan, He, Krahenbuhl, and
Girshick, Zhang et al.(2021)Zhang, Gupta, and Zisserman, Ng and
Fernando(2019), Cartas et al.(2021)Cartas, Radeva, and Dimiccoli], and object
detection [Beery et al.(2020)Beery, Wu, Rathod, Votel, and Huang, Bertasius
and Torresani(2020)]. [Sener et al.(2020)Sener, Singhania, and Yao] proposes a
model that operates on multi-scale past temporal context for action
anticipation while they also test on action recognition by modifying the
architecture to consider both past and future context. In [Zhang et
al.(2021)Zhang, Gupta, and Zisserman], a set of learnable query vectors
attends to dense temporal context to identify events in untrimmed videos. In
[Ng and Fernando(2019)], an encoder-decoder LSTM is proposed for classifying
sequences of human actions, effectively attending to the relevant temporal
context of each action. Closest to our work is [Wu et al.(2019)Wu,
Feichtenhofer, Fan, He, Krahenbuhl, and Girshick], where long-term features
from the past and the future of an untrimmed video are utilised, to improve
the recognition performance of the ongoing action. A key difference is that we
exploit both the temporal bounds and predicted labels of neighbouring actions.

Multimodal transformers. The self attention mechanism of transformers provides
a natural bridge to connect multimodal signals. Applications include audio
enhancement [Ephrat et al.(2018)Ephrat, Mosseri, Lang, Dekel, Wilson,
Hassidim, Freeman, and Rubinstein, Tzinis et al.(2021)Tzinis, Wisdom, Jansen,
Hershey, Remez, Ellis, and Hershey], speech recognition [Harwath et
al.(2016)Harwath, Torralba, and Glass], image segmentation [Ye et al.(2019)Ye,
Rochan, Liu, and Wang, Tzinis et al.(2021)Tzinis, Wisdom, Jansen, Hershey,
Remez, Ellis, and Hershey], cross-modal sequence generation [Gan et
al.(2020)Gan, Huang, Chen, Tenenbaum, and Torralba, Li et al.(2021a)Li, Yang,
Ross, and Kanazawa, Li et al.(2020)Li, Yin, Chu, Zhou, Wang, Fidler, and Li],
video retrieval [Gabeur et al.(2020)Gabeur, Sun, Alahari, and Schmid] and
image/video captioning/classification [Lu et al.(2019a)Lu, Batra, Parikh, and
Lee, Sun et al.(2019b)Sun, Myers, Vondrick, Murphy, and Schmid, Sun et
al.(2019a)Sun, Baradel, Murphy, and Schmid, Li et al.(2019)Li, Zhu, Liu, and
Yang, Iashin and Rahtu(2020), Jaegle et al.(2021)Jaegle, Gimeno, Brock,
Zisserman, Vinyals, and Carreira]. A common paradigm (which we also adapt) is
to use the output representations of single modality convolutional networks as
inputs to the transformer [Lee et al.(2021)Lee, Yu, Kim, Breuel, Kautz, and
Song, Gabeur et al.(2020)Gabeur, Sun, Alahari, and Schmid]. Unlike these
works, we use transformers to combine modalities in two specific ways – we
first combine audio and visual inputs to predict actions based on neighbouring
context, and then re-score these predictions with the help of a language model
applied on the outputs.

Language models for action detection. There are a few works that incorporate a
language model in action detection. The most relevant work is [Richard and
Gall(2016)], which utilises a statistical n-gram language model along with a
length model and an action classifier. [Lin et al.(2017)Lin, Inoue, and
Shinoda] combines the Connectionist Temporal Classification (CTC) [Graves et
al.(2006)Graves, Fernández, Gomez, and Schmidhuber] with a language model to
learn relationships between actions. Both of these works improve the results
by considering the contextual structure of the sequence of actions, albeit
relying on a statistical language model, whereas in this work we utilise a
neural language model which has shown better performance [Bengio et
al.(2003)Bengio, Ducharme, Vincent, and Janvin, Doval and Gómez-
Rodríguez(2019), Kim et al.(2016)Kim, Jernite, Sontag, and Rush].

##  3 Multimodal Temporal Context Network (MTCN)

Given a long video, we predict the action in a video segment by leveraging the
_temporal context_ around it. We define the temporal context as the sequence
of neighbouring actions that precede and succeed the action, and aim to
leverage that information, when useful, through learnt attention. We utilise
multimodal temporal context both at the input and the output of our model. An
audio-visual transformer ingests a temporally-ordered sequence of visual
inputs, along with the corresponding sequence of auditory inputs. We use
modality-independent positional encodings as well as modality-specific
encodings. The language model, acting on the output of the transformer learns
the prior temporal context of actions, i.e. the probability of the sequence of
actions, using a learnt text embedding space.

Inspired by similar approaches [Wray et al.(2019)Wray, Larlus, Csurka, and
Damen], and instead of using a single summary embedding as in prior works for
image [Dosovitskiy et al.(2021)Dosovitskiy, Beyer, Kolesnikov, Weissenborn,
Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit, and Houlsby]
and action classification [Arnab et al.(2021)Arnab, Dehghani, Heigold, Sun,
Lucic, and Schmid], the audio-visual model utilises two separate summary
embeddings to attend to the action (i.e. verb) class and the object (i.e.
noun) class. This allows the model to attend independently to the temporal
context of verbs vs objects. For example, the object is likely to be the same
in neighbouring actions while the possible sequences of verbs can be
independent of objects (e.g. ‘take’ $\rightarrow$ ‘put’). Each summary
embedding uses a different learnt classification token, and the classifier
predicts a verb and a noun from the summary embeddings. The predictions of the
audio-visual transformer are then enhanced by filtering out improbable
sequences using the language model. We term the proposed model Multimodal
Temporal Context Network (MTCN).

In the next three subsections, we detail the architectural components of MTCN,
as well as our training strategy. An overview of MTCN is depicted in Fig. 2.

###  3.1 Audio-Visual Transformer

Let $X_{v}\in\mathbb{R}^{{w}\times d_{v}}$ be the sequence of visual inputs
from a video, and $X_{a}\in\mathbb{R}^{w\times d_{a}}$ the corresponding audio
inputs222Video and audio inputs/features are synchronised, therefore they have
the same length $w$., for $w$ consecutive actions in the video (i.e. the
temporal context window), with $d_{v}\text{, }d_{a}$ being the input
dimensions of the two modalities respectively. $X_{v}$ and $X_{a}$ correspond
to features extracted from visual and auditory networks, respectively. Our
temporal window is centred around an action $b_{i}$ with surrounding action
segments, excluding any background frames. That is, each action $b_{j}$ within
the window, $i-\frac{w-1}{2}\leq j\leq i+\frac{w-1}{2}$ is part of the
transformer’s input.

Figure 2: The Multimodal Temporal Context Network (MTCN). Visual and auditory
tokens are tagged with positional and modality encodings. An audio-visual
transformer encoder attends to the sequence. Verb and noun summary embeddings
with independent positional encodings, predict the action at the centre of the
window (‘take washing liquid’). The classifier also predicts the sequence of
actions to train an auxiliary loss that enhances the prediction of the centre
action. The language model filters out improbable sequences.

Encoding layer. Our model first projects the inputs $X_{v}\text{, }X_{a}$ to a
lower dimension $D$ and tags each with positional and modality encodings.
Then, an audio-visual encoder performs self-attention on the sequence to
aggregate relevant audio-visual temporal context from neighbouring actions.
Because all self-attention operations in a transformer are permutation
invariant, we use positional encodings to retain information about the
ordering of actions in the sequence. We use $w$ learnt absolute positional
encodings, shared between audio-visual features to model corresponding inputs
from the two modalities. Modality encodings, $m_{v}\text{,
}m_{a}\in\mathbb{R}^{D}$, are learnt vectors added to discriminate between
audio and visual tokens.

A classifier predicts the action $b_{i}$, using two summary embeddings, acting
on the learnt verb/noun tokens. We use the standard approach of appending
learnable classification tokens to the end of the sequence but use two tokens,
one for verbs and one for nouns, denoted as $\text{CLS}_{\text{V}}\text{,
}\text{CLS}_{\text{N}}\in\mathbb{R}^{D}$, with unique positional encodings. To
summarise, the encoding layer transforms the inputs $X_{v}$ and $X_{a}$ as
follows:

$\displaystyle X_{v_{j}}^{e}=g_{v}(X_{v_{j}})+p_{j}+m_{v}\qquad
X_{a_{j}}^{e}=g_{a}(X_{a_{j}})+p_{j}+m_{a}\qquad\forall j\in[1,...,w]$ (1)
$\displaystyle\text{CLS}_{\text{V}}^{e}=\text{CLS}_{\text{V}}+p_{w+1}\qquad\text{CLS}_{\text{N}}^{e}=\text{CLS}_{\text{N}}+p_{w+2}\qquad
X^{e}=[X_{v}^{e};X_{a}^{e};\text{CLS}_{\text{V}}^{e};\text{CLS}_{\text{N}}^{e}]\text{,
}$ (2)

where $[;]$ denotes input concatenation and $p\in\mathbb{R}^{(w+2)\times D}$
are the positional encodings.
$g_{v}(\cdot):\mathbb{R}^{d_{v}}\mapsto\mathbb{R}^{D}$ and
$g_{a}(\cdot):\mathbb{R}^{d_{a}}\mapsto\mathbb{R}^{D}$, are fully-connected
layers projecting the visual and audio features, respectively, to a lower
dimension $D$. The input to the transformer is
${X^{e}\in\mathbb{R}^{(2w+2)\times D}}$. In appendix E, we compare the
absolute positional encodings with relative [Shaw et al.(2018)Shaw, Uszkoreit,
and Vaswani] and Fourier feature positional encodings [Jaegle et
al.(2021)Jaegle, Gimeno, Brock, Zisserman, Vinyals, and Carreira].

Transformer and classifier. We use a transformer encoder $f(\cdot)$ to process
sequential audio-visual inputs, $Z=f(X^{e})$. We share the weights of the
transformer encoder layer-wise. In appendix E, we compare this to a version
without weight sharing. Weight sharing uses $2.7\times$ less parameters with
comparable results. A two-head classifier $h(\cdot)$ for verbs and nouns then
predicts the sequence of $w$ actions from both the transformed visual and
audio tokens $\hat{Y}=h(Z_{1:2w})$, and the action $b_{i}$ from the summary
embeddings $\hat{y}=h(Z_{2w:2w+2})$.

Loss function. Recall that our goal is to classify $b_{i}$, the action
localised at the centre of our temporal context. Nevertheless, we can leverage
the ground-truth of neighbouring actions within $w$ for additional supervision
to train the audio-visual transformer. Our loss is composed of two terms, the
main loss for training the model to classify the action at the centre of our
temporal context $i=\frac{w}{2}$, and an auxiliary loss to predict all actions
in the sequence:

$\displaystyle L_{\text{m}}$
$\displaystyle=CE(Y_{i}^{\text{V}},\hat{y}^{\text{V}})+CE(Y_{i}^{\text{N}},\hat{y}^{\text{N}})$
(3) $\displaystyle L_{\text{a}}$
$\displaystyle=\sum_{j=1}^{w}\big{(}CE(Y_{j}^{\text{V}},\hat{Y}_{j}^{\text{V}})+CE(Y_{j}^{\text{V}},\hat{Y}_{|w|+j}^{\text{V}})+CE(Y_{j}^{\text{N}},\hat{Y}_{j}^{\text{N}})+CE(Y_{j}^{\text{N}},\hat{Y}_{|w|+j}^{\text{N}})\big{)}$
(4) $\displaystyle loss$ $\displaystyle=\beta
L_{\text{m}}+(1-\beta)L_{\text{a}}\text{,}$ (5)

where $CE()$ is a cross-entropy loss, and $Y=(Y_{1},...,Y_{w})$ is the ground-
truth of the sequence, while $\hat{Y}_{1},...,\hat{Y}_{w}$ and
$\hat{Y}_{w+1},...,\hat{Y}_{2w}$ correspond to predictions from the
transformed visual and auditory inputs respectively. We use $\beta$ to weight
the importance of the auxiliary loss.

###  3.2 Language Model

In addition to input context from the visual and audio domains, we introduce
output context using a language model. Language modelling, commonly applied to
predict the probability of a sequence of words, is a fundamental task in NLP
research [Peters et al.(2018)Peters, Neumann, Iyyer, Gardner, Clark, Lee, and
Zettlemoyer, Devlin et al.(2019)Devlin, Chang, Lee, and Toutanova, Brown et
al.(2020)Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal, Neelakantan, Shyam,
Sastry, Askell, Agarwal, Herbert-Voss, Krueger, Henighan, Child, Ramesh,
Ziegler, Wu, Winter, Hesse, Chen, Sigler, Litwin, Gray, Chess, Clark, Berner,
McCandlish, Radford, Sutskever, and Amodei, Yang et al.(2019)Yang, Dai, Yang,
Carbonell, Salakhutdinov, and Le]. Our language model predicts the probability
of a sequence of actions. We use the language model to improve the predictions
of the audio-visual transformer by filtering out improbable sequences.

We adopt the popular Masked Language Model (MLM), introduced in BERT [Devlin
et al.(2019)Devlin, Chang, Lee, and Toutanova]. We train this model
independently from the audio-visual transformer. Specifically, given a
sequence of actions $Y=(Y_{i-\frac{w-1}{2}},...,Y_{i+\frac{w-1}{2}})$, we
randomly mask any action $Y_{j}$ and train the model to predict it. For
example, an input sequence to the model (for $w=5$) would be: (‘turn on tap’,
‘wash hands’, <MASK>, ‘pick up towel’, ‘dry hands’). Without any visual or
audio input, the language model is tasked to learn a high prior probability
for ‘turn off tap’, which is masked. Note that the model is trained using the
ground-truth sequence of actions. For input representation, we split the
action into verb and noun tokens (e.g. ‘dry hands’ $\rightarrow$ ‘dry’ and
‘hand’), convert them to one-hot vectors and input them into separate word-
embedding layers333Learning a word embedding outperforms pretrained
embeddings.. MLM takes as input the concatenation of verb and noun embeddings.
The outputs are scores for verb and noun classes using a two-head classifier,
and the model is trained with a cross-entropy loss per output.

###  3.3 Inference

Given the the scores of the sequence,
$\hat{S}=(\hat{y}_{i-\frac{w-1}{2}},...,\hat{y}_{i+\frac{w-1}{2}})$, from the
audio-visual model444These are temporally ordered predictions from the summary
embeddings, and thus different from $\hat{Y}$., we apply a beam-search of size
$K$ to find the $K$ most probable action sequences $\hat{S}_{b}$. Therefore,
$\hat{S}_{b}$ is of size $K\times w$. In inference, the trained language model
takes as input $\hat{S}_{b}$, i.e. it operates on sequences predicted from the
audio-visual transformer.

For each sequence $l$, we calculate the probability of the sequence
$p_{LM}(\hat{S}_{b}^{l})$ from the language model by utilising the method
introduced in [Shin et al.(2019)Shin, Lee, and Jung]. We mask actions, one at
a time, and predict their probability. $p_{LM}(\hat{S}_{b}^{l})$ is the sum of
log probabilities of all actions in $l$. We also calculate the probability
$p_{AV}(\hat{S}_{b}^{l})$ by summing the log probabilities of all predicted
actions in $l$ by the audio-visual model. Then, we combine the probabilities
of sequences of the audio-visual and language models:

$p(\hat{S}_{b}^{l})=\lambda
p_{LM}(\hat{S}_{b}^{l})+(1-\lambda)p_{AV}(\hat{S}_{b}^{l})\text{.}$ (6)

Sequences are then sorted in descending order by $p(\hat{S}_{b}^{l})$. The
score of the centre action from the sequence with the highest probability, is
used as the final prediction.

##  4 Experiments

###  4.1 Implementation Details

Datasets. EPIC-KITCHENS-100 [Damen et al.(2021)Damen, Doughty, Farinella, ,
Furnari, Ma, Kazakos, Moltisanti, Munro, Perrett, Price, and Wray] is the
largest egocentric audio-visual dataset, containing unscripted daily
activities, thus offering naturally variable sequences of actions. There are
on average 129 actions per video (std 163 actions/video and maximum of 940
actions/video). This makes the dataset ideal for exploring temporal context.
The length of sequences of $w=9$ actions (this is our default window length)
is 34.4 seconds of video on average with an std of 27.8 seconds555minimum of
3.4 seconds to a maximum of 720.2 seconds.. EGTEA [Li et al.(2018b)Li, Li, and
Vasconcelos] is another video-only egocentric dataset. There are 28 hours of
untrimmed cooking activities with 10K annotated action segments. Although the
dataset does not contain audio, it has sequential actions annotated within
long videos, and we use it to train part of our approach (vision and
language).

Visual features. For EPIC-KITCHENS, we extract visual features with SlowFast
[Feichtenhofer et al.(2019)Feichtenhofer, Fan, Malik, and He], using the
public model and code from [Damen et al.(2021)Damen, Doughty, Farinella, ,
Furnari, Ma, Kazakos, Moltisanti, Munro, Perrett, Price, and Wray]. We first
train the model with slightly different hyperparameters, where we sample a
clip of 2s from an action segment, do not freeze batch normalisation layers,
and warm-up training during the first epoch starting from a learning rate of
0.001. We note that this gave us better performance. All unspecified
hyperparameters remain unchanged. For feature extraction, 10 clips of 1s each
are uniformly sampled for each action segment, with a center crop per clip.
The resulting features have a dimensionality of $d_{v}=2304$. The SlowFast
visual features are used for all the results in this paper, apart for the
comparison with the state of the art in Table 3 where we additionally
experiment with features from Mformer-HR [Patrick et al.(2021)Patrick,
Campbell, Asano, Metze, Feichtenhofer, Vedaldi, and Henriques]. These are
extracted from the EPIC-KITCHENS pretrained model using a single crop per
clip. The resulting features have a dimensionality of $d_{v}$=768. For EGTEA,
see appendix F.

Auditory features. We use Auditory SlowFast [Kazakos et al.(2021)Kazakos,
Nagrani, Zisserman, and Damen] for audio feature extraction when present.
Similarly to the visual features, we extract 10 clips of 1s each uniformly
spaced for each action segment, with average pooling and concatenation of the
features from the Slow and Fast streams, and the resulting features have the
same dimensionality, $d_{a}=2304$.

Architectural details. Both the audio-visual transformer encoder and the
language model consist of 4 layers with shared weights, 8 attention heads and
a hidden unit dimension of 512. In the audio-visual transformer,
positional/modality encodings as well as verb/noun tokens have also
dimensionality $D=512$ and are initialised to $\mathcal{N}(0,0.001)$. The
layers $g_{v}(\cdot)$ and $g_{a}(\cdot)$ reduce the features to the common
dimension $D=512$. In the encoding layer, dropout is applied at the inputs of
$g_{v}(\cdot)$ and $g_{a}(\cdot)$ as well as at $X^{e}$. In the language
model, both verb and noun word-embedding layers have a dimension of 256, and
positional encodings have a dimension of 512, while dropout is applied to its
inputs.

Scheduled sampling. We modify the scheduled sampling from [Bengio et
al.(2015)Bengio, Vinyals, Jaitly, and Shazeer] to train the language model. At
each training iteration, we randomly mask an action, predict it, and replace
the corresponding ground-truth with the prediction.

Train / Val details. For EPIC-KITCHENS, the audio-visual transformer is
trained using SGD, a batch size of 32 and a learning rate of 0.01 for 100
epochs. Learning rate is decayed by a factor of 0.1 at epochs 50 and 75. In
the loss function, we set $\beta=0.9$. For regularisation, a weight decay of
0.0005 is used and a dropout 0.5 and 0.1 for the encoding layers and
transformer layers respectively. We use mixup data augmentation [Zhang et
al.(2018)Zhang, Cissé, Dauphin, and Lopez-Paz] with $\alpha=0.2$. The language
model is trained for the same number of epochs with a batch size of 64, Adam
optimiser with initial learning rate of 0.001 and the learning rate is
decreased by a factor of 0.1 when validation accuracy saturates for over 10
epochs. The values of dropout and weight decay are the same as those of the
audio-visual model. For inference, we tune $\lambda$ in Equation 6 on the
validation set with grid-search from the set
$\\{0,0.05,0.1,0.15,0.2,0.25,0.3\\}$, and we use a beam size of $K=10$. For
training the audio-visual transformer, we randomly sample 1 out of 10 features
per action. For testing, we feed all 10 features per action to the transformer
and we share the positional encoding corresponding to an action with all 10
features. We also tried single feature per action followed by averaging 10
predictions but observed no difference in performance. For the train/val
details in EGTEA, please see appendix F.

Evaluation metrics. For EPIC-KITCHENS-100, we follow [Damen et al.(2021)Damen,
Doughty, Farinella, , Furnari, Ma, Kazakos, Moltisanti, Munro, Perrett, Price,
and Wray] and report top-1 and top-5 accuracy for the validation and test sets
separately (Test results and results on EPIC-KITCHENS-55 in appendix A an B).
We also follow [Damen et al.(2021)Damen, Doughty, Farinella, , Furnari, Ma,
Kazakos, Moltisanti, Munro, Perrett, Price, and Wray] and report results for
two subsets within val/test: unseen participants and tail classes. For EGTEA,
we follow [Min and Corso(2021), Lu et al.(2019b)Lu, Liao, and Li, Kapidis et
al.(2019)Kapidis, Poppe, van Dam, Noldus, and Veltkamp] and report top-1
accuracy and mean class accuracy using the first train/test split.

###  4.2 Analysis of temporal context length

| Overall | Unseen Participants | Tail-classes
---|---|---|---
| Top-1 Accuracy (%) | Top-5 Accuracy (%) | Top-1 Accuracy (%) | Top-1 Accuracy (%)
$w$ | Verb | Noun | Action | Verb | Noun | Action | Verb | Noun | Action | Verb | Noun | Action
1 | 67.93 | 52.29 | 41.30 | 90.53 | 76.47 | 61.52 | 61.13 | 44.60 | 32.58 | 42.05 | 27.42 | 21.48
3 | 69.80 | 55.24 | 43.52 | 91.30 | 79.04 | 63.25 | 61.41 | 46.48 | 33.71 | 39.09 | 32.58 | 23.06
5 | 70.38 | 56.16 | 45.13 | 91.67 | 79.47 | 64.14 | 61.97 | 46.95 | 34.74 | 43.12 | 32.53 | 24.54
7 | 70.43 | 56.19 | 45.01 | 91.23 | 79.13 | 63.52 | 62.63 | 47.14 | 34.84 | 41.31 | 32.79 | 24.12
9 | 70.60 | 56.26 | 45.48 | 91.14 | 79.06 | 63.06 | 63.76 | 47.14 | 35.87 | 41.36 | 32.84 | 24.70
11 | 70.55 | 55.74 | 44.68 | 91.18 | 79.23 | 63.02 | 62.91 | 46.57 | 34.74 | 41.82 | 33.58 | 24.44

Table 1: Analysis of temporal context extent for MTCN on EPIC-KITCHENS.

We first analyse the importance of the temporal context extent by varying the
size of $w$, i.e. the length of window of actions that our model observes. We
use the validation set of EPIC-KITCHENS for this analysis as well as for the
ablations in Sec. 4.3, and report EGTEA analysis in appendix G. We perform
this analysis both for MTCN containing all modalities as shown in Table 1, as
well as for the language model as shown in Table 2. Varying the length of the
window has a big impact on the model’s accuracy showcasing that MTCN
successfully utilises temporal context. Using temporal context outperforms
$w=1$, i.e. no temporal context. As the window length increases the
performance also increases. Overall top-1 accuracy increases up to $w=9$ while
top-5 up to a window $w=5$. Performance on unseen participants and tail
classes also increases up to $w=9$.

| Overall
---|---
| Top-1 Accuracy (%)
$w$ | Verb | Noun | Action
1 | 19.32 | 3.74 | 0.82
3 | 38.08 | 45.56 | 23.85
5 | 42.15 | 50.36 | 29.48
7 | 42.93 | 50.35 | 29.91
9 | 43.06 | 50.22 | 29.41
11 | 41.89 | 49.96 | 29.14

Table 2: Analysis of the temporal context on LM.

Figure 3: Effect of temporal context on verb (left) and noun (right) accuracy
of individual modalities and for MTCN on EPIC-KITCHENS. Y-axis cut to
emphasise details.

In Table 2, experiments are conducted by masking the centre action and
measuring how well the model predicts it. We use ground-truth for the other
actions, as in this experiment we are interested in the maximum possible
performance of the language model, i.e. assuming correct predictions from the
audio-visual model. Here $w=1$ corresponds to a language model that randomly
guesses the masked action without any context. When varying the length of the
window for the language model, verb performance increases when enlarging the
temporal context from $w=3$ to $w=9$ while for nouns optimal temporal context
is $w=5$, and $w=7$ for actions. Interestingly, the language model is
performing particularly well for nouns, since the same object is often used
for the entire sequence.

Finally, Fig. 2 shows the effect of temporal context extent on verb and noun
accuracy of the individual modalities as well as of MTCN. For verbs, visual
modality performance increases up to 7 actions and then decreases while for
nouns, it steadily increases up to 11 actions, possibly because larger context
is able to resolve ambiguities due to occlusion. Audio can better capture
temporal context for verbs than the language model, showcasing that the
progression of sounds conveys more useful information about the action. For
nouns, the language model outperforms audio. Moreover, the language model
performs better on nouns because of repetitions of the same object in the
sequence. MTCN utilising all modalities (A-V-LM) benefits the most from larger
temporal context, particularly for verbs. With this analysis completed, we fix
$w$ to 9 in all subsequent experiments.

###  4.3 Results and Ablations

We compare our approach with the state-of-the-art (SOTA) approaches on EPIC-
KITCHENS-100 as shown in Table 3. MTCN significantly outperforms convolutional
approaches [Wang et al.(2016a)Wang, Xiong, Wang, Qiao, Lin, Tang, and Van
Gool, Lin et al.(2019)Lin, Gan, and Han, Kazakos et al.(2019)Kazakos, Nagrani,
Zisserman, and Damen, Feichtenhofer et al.(2019)Feichtenhofer, Fan, Malik, and
He]. We outperform the audio-visual TBN [Kazakos et al.(2019)Kazakos, Nagrani,
Zisserman, and Damen] by 8% on top-1 actions, and SlowFast [Feichtenhofer et
al.(2019)Feichtenhofer, Fan, Malik, and He] by 6% (using the same visual
features). We also outperform very recent transformer-based approaches [Arnab
et al.(2021)Arnab, Dehghani, Heigold, Sun, Lucic, and Schmid, Bulat et
al.(2021)Bulat, Perez-Rua, Sudhakaran, Martínez, and Tzimiropoulos, Patrick et
al.(2021)Patrick, Campbell, Asano, Metze, Feichtenhofer, Vedaldi, and
Henriques, Nagrani et al.(2021)Nagrani, Yang, Arnab, Schmid, and Sun],
reporting published results (only top-1 accuracy). Note, that MTCN consists of
a lightweight transformer that operates on pre-extracted features, while
[Arnab et al.(2021)Arnab, Dehghani, Heigold, Sun, Lucic, and Schmid, Bulat et
al.(2021)Bulat, Perez-Rua, Sudhakaran, Martínez, and Tzimiropoulos, Patrick et
al.(2021)Patrick, Campbell, Asano, Metze, Feichtenhofer, Vedaldi, and
Henriques, Nagrani et al.(2021)Nagrani, Yang, Arnab, Schmid, and Sun] are
high-capacity models trained end-to-end.

Additionally, when we employ visual features from Mformer-HR [Patrick et
al.(2021)Patrick, Campbell, Asano, Metze, Feichtenhofer, Vedaldi, and
Henriques], MTCN improves over all transformer-based approaches, including
audio-visual fusion [Nagrani et al.(2021)Nagrani, Yang, Arnab, Schmid, and
Sun], by 3.5% on top-1 nouns and 5% on actions. We attribute the boost on
nouns to the enhanced object recognition performance of the ViT backbone
[Dosovitskiy et al.(2021)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai,
Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit, and Houlsby] in
Mformer-HR and its large-scale pretraining. These results, however, further
demonstrate the potential to boost other methods and features by utilising
multimodal temporal context.

| Overall | Unseen Participants | Tail-classes
---|---|---|---
| Top-1 Accuracy (%) | Top-5 Accuracy (%) | Top-1 Accuracy (%) | Top-1 Accuracy (%)
Model | Verb | Noun | Action | Verb | Noun | Action | Verb | Noun | Action | Verb | Noun | Action
TSN [Wang et al.(2016a)Wang, Xiong, Wang, Qiao, Lin, Tang, and Van Gool] | 60.2 | 46.0 | 33.2 | 89.6 | 72.9 | 55.1 | 47.4 | 38.0 | 23.5 | 30.5 | 19.4 | 13.9
TBN [Kazakos et al.(2019)Kazakos, Nagrani, Zisserman, and Damen] | 66.0 | 47.2 | 36.7 | 90.5 | 73.8 | 57.7 | 59.4 | 38.2 | 29.5 | 39.1 | 24.8 | 19.1
TSM [Lin et al.(2019)Lin, Gan, and Han] | 67.9 | 49.0 | 38.3 | 91.0 | 75.0 | 60.4 | 58.7 | 39.6 | 29.5 | 36.6 | 23.4 | 17.6
SlowFast [Feichtenhofer et al.(2019)Feichtenhofer, Fan, Malik, and He] | 65.6 | 50.0 | 38.5 | 90.0 | 75.6 | 58.6 | 56.4 | 41.5 | 29.7 | 36.2 | 23.3 | 18.8
ViViT-L/16x2 [Arnab et al.(2021)Arnab, Dehghani, Heigold, Sun, Lucic, and Schmid] | 66.4 | 56.8 | 44.0 | - | - | - | - | - | - | - | - | -
X-ViT (16x) [Bulat et al.(2021)Bulat, Perez-Rua, Sudhakaran, Martínez, and Tzimiropoulos] | 68.7 | 56.4 | 44.3 | - | - | - | - | - | - | - | - | -
Mformer-HR [Patrick et al.(2021)Patrick, Campbell, Asano, Metze, Feichtenhofer, Vedaldi, and Henriques] | 67.0 | 58.5 | 44.5 | - | - | - | - | - | - | - | - | -
MBT [Nagrani et al.(2021)Nagrani, Yang, Arnab, Schmid, and Sun] | 64.8 | 58.0 | 43.4 | - | - | - | - | - | - | - | - | -
MTCN - v.f. SlowFast [Feichtenhofer et al.(2019)Feichtenhofer, Fan, Malik, and He] | 70.6 | 56.3 | 45.5 | 91.1 | 79.1 | 63.1 | 63.8 | 47.1 | 35.9 | 41.4 | 32.8 | 24.7
MTCN - v.f. Mformer-HR [Patrick et al.(2021)Patrick, Campbell, Asano, Metze, Feichtenhofer, Vedaldi, and Henriques] | 70.7 | 62.1 | 49.6 | 90.7 | 83.1 | 68.6 | 63.7 | 50.9 | 38.9 | 41.9 | 39.2 | 27.7

Table 3: Comparison with SOTA on EPIC-KITCHENS-100 using two visual features
(‘v.f.’)

|  |  |  | Overall | Unseen Participants | Tail-classes
---|---|---|---|---|---|---
|  |  |  | Top-1 Accuracy (%) | Top-5 Accuracy (%) | Top-1 Accuracy (%) | Top-1 Accuracy (%)
V | A | LM | Aux | Verb | Noun | Action | Verb | Noun | Action | Verb | Noun | Action | Verb | Noun | Action
✓ | ✗ | ✗ | ✓ | 67.10 | 53.54 | 41.49 | 90.62 | 78.22 | 62.32 | 58.87 | 43.29 | 30.99 | 40.97 | 30.47 | 22.35
✓ | ✗ | ✓ | ✓ | 67.84 | 54.08 | 42.05 | 90.63 | 78.20 | 60.68 | 59.06 | 44.23 | 31.36 | 39.77 | 31.32 | 22.38
✓ | ✓ | ✗ | ✓ | 70.23 | 55.82 | 45.00 | 91.13 | 79.06 | 64.58 | 63.29 | 46.38 | 35.02 | 41.76 | 32.26 | 24.41
✓ | ✓ | ✓ | ✗ | 69.31 | 55.46 | 43.81 | 91.19 | 79.76 | 62.35 | 61.13 | 46.01 | 33.90 | 39.55 | 30.74 | 22.74
✓ | ✓ | ✓ | ✓ | 70.60 | 56.26 | 45.48 | 91.14 | 79.06 | 63.06 | 63.76 | 47.14 | 35.87 | 41.36 | 32.84 | 24.70
✓ | ✓ | ${\dagger}$ | ✓ | 71.33 | 63.56 | 50.32 | 92.05 | 83.16 | 67.85 | 62.25 | 52.68 | 37.56 | 41.53 | 44.16 | 29.89

Table 4: Ablation on multimodal temporal context and auxiliary loss in EPIC-
KITCHENS. ${\dagger}$: upper bound using ground-truth knowledge as input to
the language model.

Method | Top-1 (%) | MC(%)
---|---|---
Li et al. [Li et al.(2018a)Li, Liu, and Rehg] | - | 53.30
Ego-RNN [Sudhakaran and Lanz(2018)] | 62.17 | -
Kapidis et al. [Kapidis et al.(2019)Kapidis, Poppe, van Dam, Noldus, and Veltkamp] | 68.99 | 61.40
Lu et al. [Lu et al.(2019b)Lu, Liao, and Li] | 68.60 | 60.54
SlowFast [Feichtenhofer et al.(2019)Feichtenhofer, Fan, Malik, and He] | 70.43 | 61.92
MCN [Huang et al.(2020)Huang, Cai, Li, Lu, and Sato] | 55.63 | -
Min et al. [Min and Corso(2021)] | 69.58 | 62.84
MTCN (V) (Ours) | 72.55 | 64.86
MTCN (V+LM) (Ours) | 73.59 | 65.87

Table 5: Comparative results on EGTEA. MC: Mean Class

We compare MTCN without audio with the state-of-the-art on EGTEA, in Table 5.
This model uses $w=3$. Our model improves over previous approaches by 3% in
both top-1 and mean class accuracy. Note that MTCN outperforms SlowFast
[Feichtenhofer et al.(2019)Feichtenhofer, Fan, Malik, and He] which we used to
extract features. In appendix G, we ablate the temporal context and language
model, where we showcase that the language model provides a bigger boost in
performance, possibly due to the absence of the audio modality (also shown in
Table 5: ‘V’ vs ‘V+LM’).

Figure 4: Qualitative results of attention weights along with the predictions
of our model in EPIC-KITCHENS. Green and purple edges represent attention from
the noun embedding to visual and auditory tokens, respectively. Thickness
indicates attention weights magnitude to centre (bordered) and temporal
context actions.

Multimodality Ablation. We offer an ablation to identify the performance
impact of the components of our MTCN. Results are shown in Table 4. We remove
audio and language modalities from our model’s input and output respectively
to assess their importance. Multimodal context is important, as our proposed
model enjoys considerable margins compared with the model trained only with
visual context (line 1 in Table 4). Audio is beneficial, confirming the
finding of prior works.

Although the language model provides smaller boost in performance than audio,
it showcases that it is useful to model prior temporal context at the output
level, and that its benefits are complementary to audio. We also include upper
bound performance improvement from the addition of the language model, where
it takes as input the ground-truth preceding/succeeding actions rather than
the predictions from the audio-visual transformer (last line in Table 4),
effectively a language infilling problem. These results demonstrate the
potential margin for improvement, particularly for nouns and accordingly
action accuracy as well as tail classes. In appendix C, we report statistical
significance over multiple runs for the language model, and compare it to
alternative baselines.

Auxiliary loss. Also from Table 4, training MTCN with auxiliary loss boosts
its performance almost in all metrics, confirming that utilising supervision
from neighbouring actions can improve the performance of the action of
interest, i.e. the action at the centre of the window.

While we focus on offline action recognition, we can evaluate our model for
online recognition by predicting the last action in the sequence instead of
the centre one. Results are included in appendix D for different values of
$w$, where $w=7$ provided best results for EPIC-KITCHENS. Best top-1 action
performance using past context solely is 42.96% compared to 45.48% using
surrounding context. However, we demonstrate that our model can leverage
multimodal temporal context in this setting.

In Fig. 4, we visualise the auditory and visual temporal context attention on
a correctly recognised sequence (left) – where subsequent actions of pouring
and closing are particularly informative, and an incorrectly (right) predicted
sequence – where both attention weights are high on actions containing the
egg, causing the model to incorrectly predict the noun as egg. More
qualitative examples are included in appendix H.

##  5 Conclusion

We formulate temporal context as a sequence of actions, and utilise past and
future context to enhance the prediction of the centre action in the sequence.
We propose MTCN, a model that attends to vision and audio as input modality
context, and language as output modality context. We train MTCN with
additional supervision from neighbouring actions. Our results showcase the
importance of audio and language as additional modality context. We report
SOTA results on two egocentric video datasets: EPIC-KITCHENS and EGTEA. An
extension to MTCN would incorporate an actionness score of neighbouring
frames, to distinguish background frames and learn from action sequences in
untrimmed videos without temporal bounds during testing. This would bridge the
problems of recognition and detection, utilising multimodality and temporal
context. We will be exploring this in future work.

#### Acknowledgements.

This work used public datasets and is supported by EPSRC Doctoral Training
Program, EPSRC UMPIRE (EP/T004991/1) and the EPSRC Programme Grant VisualAI
(EP/T028572/1). Jaesung Huh is funded by a Global Korea Scholarship.

## References

  * [Arnab et al.(2021)Arnab, Dehghani, Heigold, Sun, Lucic, and Schmid] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lucic, and Cordelia Schmid.  Vivit: A video vision transformer.  In _Proceedings of International Conference on Computer Vision (ICCV)_ , 2021.
  * [Baradel et al.(2018)Baradel, Neverova, Wolf, Mille, and Mori] Fabien Baradel, Natalia Neverova, Christian Wolf, Julien Mille, and Greg Mori.  Object level visual reasoning in videos.  In _Proceedings of European Conference on Computer Vision (ECCV)_ , 2018.
  * [Beery et al.(2020)Beery, Wu, Rathod, Votel, and Huang] Sara Beery, Guanhang Wu, Vivek Rathod, Ronny Votel, and Jonathan Huang.  Context r-cnn: Long term temporal context for per-camera object detection.  In _Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [Bengio et al.(2015)Bengio, Vinyals, Jaitly, and Shazeer] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer.  Scheduled sampling for sequence prediction with recurrent neural networks.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2015.
  * [Bengio et al.(2003)Bengio, Ducharme, Vincent, and Janvin] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin.  A neural probabilistic language model.  _Journal of Machine Learning Research (JMLR)_ , 3:1137–1155, 2003.
  * [Bertasius and Torresani(2020)] Gedas Bertasius and Lorenzo Torresani.  Classifying, segmenting, and tracking object instances in video with mask propagation.  In _Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [Bertasius et al.(2021)Bertasius, Wang, and Torresani] Gedas Bertasius, Heng Wang, and Lorenzo Torresani.  Is space-time attention all you need for video understanding?  In _Proceedings of International Conference on Machine Learning (ICML)_ , 2021.
  * [Brown et al.(2020)Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal, Neelakantan, Shyam, Sastry, Askell, Agarwal, Herbert-Voss, Krueger, Henighan, Child, Ramesh, Ziegler, Wu, Winter, Hesse, Chen, Sigler, Litwin, Gray, Chess, Clark, Berner, McCandlish, Radford, Sutskever, and Amodei] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.
  * [Bulat et al.(2021)Bulat, Perez-Rua, Sudhakaran, Martínez, and Tzimiropoulos] Adrian Bulat, Juan-Manuel Perez-Rua, Swathikiran Sudhakaran, Brais Martínez, and Georgios Tzimiropoulos.  Space-time mixing attention for video transformer.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021.
  * [Cartas et al.(2021)Cartas, Radeva, and Dimiccoli] Alejandro Cartas, Petia Radeva, and Mariella Dimiccoli.  Modeling long-term interactions to enhance action recognition.  In _Proceedings of International Conference on Pattern Recognition (ICPR)_ , 2021.
  * [Chan et al.(2016)Chan, Jaitly, Le, and Vinyals] William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals.  Listen, attend and spell: A neural network for large vocabulary conversational speech recognition.  In _Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2016.
  * [Damen et al.(2018)Damen, Doughty, Farinella, Fidler, Furnari, Kazakos, Moltisanti, Munro, Perrett, Price, and Wray] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray.  Scaling egocentric vision: The epic-kitchens dataset.  In _European Conference on Computer Vision (ECCV)_ , 2018.
  * [Damen et al.(2021)Damen, Doughty, Farinella, , Furnari, Ma, Kazakos, Moltisanti, Munro, Perrett, Price, and Wray] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, , Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray.  Rescaling egocentric vision: Collection pipeline and challenges for epic-kitchens-100.  _International Journal of Computer Vision (IJCV)_ , 2021.
  * [Devlin et al.(2019)Devlin, Chang, Lee, and Toutanova] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of North American Chapter of the Association for Computational Linguistics (NAACL)_ , 2019.
  * [Dosovitskiy et al.(2021)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit, and Houlsby] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _Proceedings of International Conference on Learning Representations (ICLR)_ , 2021.
  * [Doval and Gómez-Rodríguez(2019)] Yerai Doval and Carlos Gómez-Rodríguez.  Comparing neural-and n-gram-based language models for word segmentation.  _Journal of the Association for Information Science and Technology_ , 70:187–197, 2019.
  * [Ephrat et al.(2018)Ephrat, Mosseri, Lang, Dekel, Wilson, Hassidim, Freeman, and Rubinstein] Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein.  Looking to listen at the cocktail party: a speaker-independent audio-visual model for speech separation.  _ACM Transactions on Graphics (TOG)_ , 37:1–11, 2018.
  * [Feichtenhofer et al.(2019)Feichtenhofer, Fan, Malik, and He] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  SlowFast Networks for Video Recognition.  In _Proceedings of International Conference on Computer Vision (ICCV)_ , 2019.
  * [Furnari and Farinella(2019)] Antonino Furnari and Giovanni Maria Farinella.  What would you expect? anticipating egocentric actions with Rolling-Unrolling LSTMs and modality attention.  In _Proceedings of International Conference on Computer Vision (ICCV)_ , 2019.
  * [Gabeur et al.(2020)Gabeur, Sun, Alahari, and Schmid] Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid.  Multi-modal transformer for video retrieval.  In _Proceedings of European Conference on Computer Vision (ECCV)_ , 2020.
  * [Gan et al.(2020)Gan, Huang, Chen, Tenenbaum, and Torralba] Chuang Gan, Deng Huang, Peihao Chen, Joshua B Tenenbaum, and Antonio Torralba.  Foley music: Learning to generate music from videos.  In _Proceedings of European Conference on Computer Vision (ECCV)_ , 2020.
  * [Girdhar et al.(2019)Girdhar, Carreira, Doersch, and Zisserman] Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zisserman.  Video action transformer network.  In _Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [Graves et al.(2006)Graves, Fernández, Gomez, and Schmidhuber] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber.  Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks.  In _Proceedings of International Conference on Machine Learning (ICML)_ , 2006.
  * [Gulcehre et al.(2017)Gulcehre, Firat, Xu, Cho, and Bengio] Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, and Yoshua Bengio.  On integrating a language model into neural machine translation.  _Computer Speech & Language_, 45:137–148, 2017.
  * [Hannun et al.(2014)Hannun, Case, Casper, Catanzaro, Diamos, Elsen, Prenger, Satheesh, Sengupta, Coates, and Ng] Awni Y. Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, and Andrew Y. Ng.  Deep speech: Scaling up end-to-end speech recognition.  _CoRR_ , abs/1412.5567, 2014.
  * [Harwath et al.(2016)Harwath, Torralba, and Glass] David Harwath, Antonio Torralba, and James Glass.  Unsupervised learning of spoken language with visual context.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2016.
  * [Huang et al.(2020)Huang, Cai, Li, Lu, and Sato] Yifei Huang, Minjie Cai, Zhenqiang Li, Feng Lu, and Yoichi Sato.  Mutual context network for jointly estimating egocentric gaze and action.  _IEEE Transactions on Image Processing (TIP)_ , 29:7795–7806, 2020.
  * [Iashin and Rahtu(2020)] Vladimir Iashin and Esa Rahtu.  Multi-modal dense video captioning.  In _Proceedings of Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2020.
  * [Jaegle et al.(2021)Jaegle, Gimeno, Brock, Zisserman, Vinyals, and Carreira] Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and João Carreira.  Perceiver: General perception with iterative attention.  _CoRR_ , abs/2103.03206, 2021.
  * [Kapidis et al.(2019)Kapidis, Poppe, van Dam, Noldus, and Veltkamp] Georgios Kapidis, Ronald Poppe, Elsbeth van Dam, Lucas Noldus, and Remco Veltkamp.  Multitask learning to improve egocentric action recognition.  In _Proceedings of International Conference on Computer Vision Workshops (ICCVW)_ , 2019.
  * [Kay et al.(2017)Kay, Carreira, Simonyan, Zhang, Hillier, Vijayanarasimhan, Viola, Green, Back, Natsev, Suleyman, and Zisserman] Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman.  The kinetics human action video dataset.  _CoRR_ , abs/1705.06950, 2017.
  * [Kazakos et al.(2019)Kazakos, Nagrani, Zisserman, and Damen] Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen.  Epic-fusion: Audio-visual temporal binding for egocentric action recognition.  In _Proceedings of International Conference on Computer Vision (ICCV)_ , 2019.
  * [Kazakos et al.(2021)Kazakos, Nagrani, Zisserman, and Damen] Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen.  Slow-fast auditory streams for audio recognition.  In _Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021.
  * [Kim et al.(2016)Kim, Jernite, Sontag, and Rush] Yoon Kim, Yacine Jernite, David Sontag, and Alexander Rush.  Character-aware neural language models.  In _Proceedings of AAAI Conference on Artificial Intelligence_ , 2016\.
  * [Lee et al.(2021)Lee, Yu, Kim, Breuel, Kautz, and Song] Sangho Lee, Youngjae Yu, Gunhee Kim, Thomas Breuel, Jan Kautz, and Yale Song.  Parameter Efficient Multimodal Transformers for Video Representation Learning.  In _Proceedings of International Conference on Learning Representations (ICLR)_ , 2021.
  * [Li et al.(2019)Li, Zhu, Liu, and Yang] Guang Li, Linchao Zhu, Ping Liu, and Yi Yang.  Entangled transformer for image captioning.  In _Proceedings of International Conference on Computer Vision (ICCV)_ , 2019.
  * [Li et al.(2020)Li, Yin, Chu, Zhou, Wang, Fidler, and Li] Jiaman Li, Yihang Yin, Hang Chu, Yi Zhou, Tingwu Wang, Sanja Fidler, and Hao Li.  Learning to generate diverse dance motions with transformer.  _CoRR_ , abs/2008.08171, 2020.
  * [Li et al.(2021a)Li, Yang, Ross, and Kanazawa] Ruilong Li, Shan Yang, David A. Ross, and Angjoo Kanazawa.  Ai choreographer: Music conditioned 3d dance generation with aist++.  In _Proceedings of International Conference on Computer Vision (ICCV)_ , 2021a.
  * [Li et al.(2021b)Li, Nagarajan, Xiong, and Grauman] Yanghao Li, Tushar Nagarajan, Bo Xiong, and Kristen Grauman.  Ego-exo: Transferring visual representations from third-person to first-person videos.  In _Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021b.
  * [Li et al.(2018a)Li, Liu, and Rehg] Yin Li, Miao Liu, and James M. Rehg.  In the eye of beholder: Joint learning of gaze and actions in first person video.  In _Proceedings of European Conference on Computer Vision (ECCV)_ , 2018a.
  * [Li et al.(2018b)Li, Li, and Vasconcelos] Yingwei Li, Yi Li, and Nuno Vasconcelos.  Resound: Towards action recognition without representation bias.  In _Proceedings of European Conference on Computer Vision (ECCV)_ , 2018b.
  * [Lin et al.(2019)Lin, Gan, and Han] Ji Lin, Chuang Gan, and Song Han.  TSM: Temporal shift module for efficient video understanding.  In _Proceedings of International Conference on Computer Vision (ICCV)_ , 2019.
  * [Lin et al.(2017)Lin, Inoue, and Shinoda] Mengxi Lin, Nakamasa Inoue, and Koichi Shinoda.  Ctc network with statistical language modeling for action sequence recognition in videos.  In _Proceedings of the on Thematic Workshops of ACM Multimedia_ , 2017\.
  * [Lu et al.(2019a)Lu, Batra, Parikh, and Lee] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019a.
  * [Lu et al.(2019b)Lu, Liao, and Li] Minlong Lu, Danping Liao, and Ze-Nian Li.  Learning spatiotemporal attention for egocentric action recognition.  In _Proceedings of International Conference on Computer Vision Workshops (ICCVW)_ , 2019b.
  * [Mikolov et al.(2010)Mikolov, Karafiát, Burget, Černockỳ, and Khudanpur] Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černockỳ, and Sanjeev Khudanpur.  Recurrent neural network based language model.  In _Proceedings of Annual Conference of the International Speech Communication Association (INTERSPEECH)_ , 2010.
  * [Min and Corso(2021)] Kyle Min and Jason J. Corso.  Integrating human gaze into attention for egocentric activity recognition.  In _Proceedings of Winter Conference on Applications of Computer Vision (WACV)_ , 2021.
  * [Munro and Damen(2020)] Jonathan Munro and Dima Damen.  Multi-modal domain adaptation for fine-grained action recognition.  In _Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * [Nagrani et al.(2021)Nagrani, Yang, Arnab, Schmid, and Sun] Arsha Nagrani, Shan Yang, Anurag Arnab, Cordelia Schmid, and Chen Sun.  Attention bottlenecks for multimodal fusion.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021.
  * [Neimark et al.(2021)Neimark, Bar, Zohar, and Asselmann] Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann.  Video transformer network.  _CoRR_ , abs/2102.00719, 2021.
  * [Ng and Fernando(2019)] Yan Bin Ng and Basura Fernando.  Human action sequence classification.  _CoRR_ , abs/1910.02602, 2019.
  * [Patrick et al.(2021)Patrick, Campbell, Asano, Metze, Feichtenhofer, Vedaldi, and Henriques] Mandela Patrick, Dylan Campbell, Yuki M. Asano, Ishan Misra Florian Metze, Christoph Feichtenhofer, Andrea Vedaldi, and João F. Henriques.  Keeping your eye on the ball: Trajectory attention in video transformers.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021.
  * [Peters et al.(2018)Peters, Neumann, Iyyer, Gardner, Clark, Lee, and Zettlemoyer] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.  Deep contextualized word representations.  In _Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_ , 2018.
  * [Richard and Gall(2016)] Alexander Richard and Juergen Gall.  Temporal action detection using a statistical language model.  In _Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * [Sener et al.(2020)Sener, Singhania, and Yao] Fadime Sener, Dipika Singhania, and Angela Yao.  Temporal aggregate representations for long-range video understanding.  In _Proceedings of European Conference on Computer Vision (ECCV)_ , 2020.
  * [Shaw et al.(2018)Shaw, Uszkoreit, and Vaswani] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani.  Self-attention with relative position representations.  In _Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_ , 2018.
  * [Shin et al.(2019)Shin, Lee, and Jung] Joonbo Shin, Yoonhyung Lee, and Kyomin Jung.  Effective sentence scoring method using bert for speech recognition.  In _Proceedings of Asian Conference on Machine Learning (ACML)_ , 2019\.
  * [Sudhakaran and Lanz(2018)] Swathikiran Sudhakaran and Oswald Lanz.  Attention is all we need: Nailing down object-centric attention for egocentric activity recognition.  In _Proceedings of British Machine Vision Conference (BMVC)_ , 2018\.
  * [Sudhakaran et al.(2019)Sudhakaran, Escalera, and Lanz] Swathikiran Sudhakaran, Sergio Escalera, and Oswald Lanz.  Lsta: Long short-term attention for egocentric action recognition.  In _Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [Sun et al.(2019a)Sun, Baradel, Murphy, and Schmid] Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid.  Learning video representations using contrastive bidirectional transformer.  _CoRR_ , abs/1906.05743, 2019a.
  * [Sun et al.(2019b)Sun, Myers, Vondrick, Murphy, and Schmid] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid.  Videobert: A joint model for video and language representation learning.  In _Proceedings of International Conference on Computer Vision (ICCV)_ , 2019b.
  * [Tran et al.(2018)Tran, Wang, Torresani, Ray, LeCun, and Paluri] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri.  A closer look at spatiotemporal convolutions for action recognition.  In _Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * [Tzinis et al.(2021)Tzinis, Wisdom, Jansen, Hershey, Remez, Ellis, and Hershey] Efthymios Tzinis, Scott Wisdom, Aren Jansen, Shawn Hershey, Tal Remez, Daniel PW Ellis, and John R Hershey.  Into the wild with audioscope: Unsupervised audio-visual separation of on-screen sounds.  In _Proceedings of International Conference on Learning Representations (ICLR)_ , 2021.
  * [Wang et al.(2016a)Wang, Xiong, Wang, Qiao, Lin, Tang, and Van Gool] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool.  Temporal segment networks: Towards good practices for deep action recognition.  In _Proceedings of European Conference on Computer Vision (ECCV)_ , 2016a.
  * [Wang et al.(2020a)Wang, Tran, and Feiszli] Weiyao Wang, Du Tran, and Matt Feiszli.  What makes training multi-modal classification networks hard?  In _Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020a.
  * [Wang et al.(2020b)Wang, Tran, and Feiszli] Weiyao Wang, Du Tran, and Matt Feiszli.  What makes training multi-modal classification networks hard?  In _Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020b.
  * [Wang et al.(2016b)Wang, Farhadi, and Gupta] Xiaolong Wang, Ali Farhadi, and Abhinav Gupta.  Actions ~ transformations.  In _Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016b.
  * [Wray et al.(2019)Wray, Larlus, Csurka, and Damen] Michael Wray, Diane Larlus, Gabriela Csurka, and Dima Damen.  Fine-grained action retrieval through multiple parts-of-speech embeddings.  In _Proceedings of International Conference on Computer Vision (ICCV)_ , 2019.
  * [Wu et al.(2019)Wu, Feichtenhofer, Fan, He, Krahenbuhl, and Girshick] Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick.  Long-term feature banks for detailed video understanding.  In _Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [Xiao et al.(2020a)Xiao, Lee, Grauman, Malik, and Feichtenhofer] Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, and Christoph Feichtenhofer.  Audiovisual slowfast networks for video recognition.  _arXiv preprint arXiv:2001.08740_ , 2020a.
  * [Xiao et al.(2020b)Xiao, Lee, Grauman, Malik, and Feichtenhofer] Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, and Christoph Feichtenhofer.  Audiovisual slowfast networks for video recognition.  _CoRR_ , abs/2001.08740, 2020b.
  * [Yang et al.(2019)Yang, Dai, Yang, Carbonell, Salakhutdinov, and Le] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le.  XLNet: Generalized autoregressive pretraining for language understanding.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019.
  * [Ye et al.(2019)Ye, Rochan, Liu, and Wang] Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang.  Cross-modal self-attention network for referring image segmentation.  In _Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * [Zhang et al.(2021)Zhang, Gupta, and Zisserman] Chuhan Zhang, Ankush Gupta, and Andrew Zisserman.  Temporal query networks for fine-grained video understanding.  In _Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * [Zhang et al.(2018)Zhang, Cissé, Dauphin, and Lopez-Paz] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  In _Proceedings of International Conference on Learning Representations ICLR_ , 2018.
  * [Zhou et al.(2018)Zhou, Andonian, Oliva, and Torralba] Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba.  Temporal relational reasoning in videos.  In _Proceedings of European Conference on Computer Vision (ECCV)_ , 2018.

## Appendix

##  Appendix A EPIC-KITCHENS-100: Results on the Test Set

| Overall | Unseen Participants | Tail-classes
---|---|---|---
| Top-1 Accuracy (%) | Top-5 Accuracy (%) | Top-1 Accuracy (%) | Top-1 Accuracy (%)
Model | Verb | Noun | Action | Verb | Noun | Action | Verb | Noun | Action | Verb | Noun | Action
TSN [Wang et al.(2016a)Wang, Xiong, Wang, Qiao, Lin, Tang, and Van Gool] | 59.03 | 46.78 | 33.57 | 87.55 | 72.10 | 53.89 | 53.11 | 42.02 | 27.37 | 26.23 | 14.73 | 11.43
TBN [Kazakos et al.(2019)Kazakos, Nagrani, Zisserman, and Damen] | 62.72 | 47.59 | 35.48 | 88.77 | 73.08 | 56.34 | 56.69 | 43.65 | 29.27 | 30.97 | 19.52 | 14.10
TSM [Lin et al.(2019)Lin, Gan, and Han] | 65.32 | 47.80 | 37.39 | 89.16 | 73.95 | 57.89 | 59.68 | 42.51 | 30.61 | 30.03 | 16.96 | 13.45
SlowFast [Feichtenhofer et al.(2019)Feichtenhofer, Fan, Malik, and He] | 63.79 | 48.55 | 36.81 | 88.84 | 74.49 | 56.39 | 57.66 | 42.55 | 29.27 | 29.65 | 17.11 | 13.45
Ego-Exo [Li et al.(2021b)Li, Nagarajan, Xiong, and Grauman] | 66.07 | 51.51 | 39.98 | 89.39 | 76.31 | 60.68 | 59.83 | 45.50 | 32.63 | 33.92 | 22.91 | 16.96
MTCN - v.f. SlowFast [Feichtenhofer et al.(2019)Feichtenhofer, Fan, Malik, and He] | 68.44 | 55.41 | 44.10 | 88.74 | 78.04 | 61.69 | 61.82 | 47.62 | 34.94 | 34.77 | 28.60 | 20.45
MTCN - v.f. Mformer-HR [Patrick et al.(2021)Patrick, Campbell, Asano, Metze, Feichtenhofer, Vedaldi, and Henriques] | 67.88 | 60.02 | 46.83 | 88.69 | 81.84 | 66.48 | 61.07 | 55.21 | 38.98 | 35.16 | 34.70 | 22.79

Table 6: Results on the test set of EPIC-Kitchens-100.

| Top-1 Accuracy (%) | Top-5 Accuracy (%)
---|---|---
Model | Verb | Noun | Action | Verb | Noun | Action
LFB [Wu et al.(2019)Wu, Feichtenhofer, Fan, He, Krahenbuhl, and Girshick] | 60.0 | 45.0 | 32.7 | 88.4 | 71.8 | 55.3
G-Blend [Wang et al.(2020b)Wang, Tran, and Feiszli] | 66.7 | 48.5 | 37.1 | 88.9 | 71.4 | 56.2
AV-SlowFast [Xiao et al.(2020a)Xiao, Lee, Grauman, Malik, and Feichtenhofer] | 65.7 | 46.4 | 35.9 | 89.5 | 71.7 | 57.8
Ego-Exo [Li et al.(2021b)Li, Nagarajan, Xiong, and Grauman] | 65.97 | 47.99 | 37.09 | 90.32 | 70.72 | 56.32
MTCN (Ours) | 69.12 | 51.30 | 40.77 | 90.18 | 73.53 | 59.15

Table 7: Comparison with SOTA on the Seen split (S1) of EPIC-KITCHENS-55.

In Table 3 of the main paper, we compare to published works on the validation
set of EPIC-KITCHENS-100. Unfortunately, most works do not report on the
leaderboard test set. In Table 6, we provide results on the test set comparing
our model to baselines from [Damen et al.(2021)Damen, Doughty, Farinella, ,
Furnari, Ma, Kazakos, Moltisanti, Munro, Perrett, Price, and Wray], as well as
Ego-Exo [Li et al.(2021b)Li, Nagarajan, Xiong, and Grauman] that distills
knowledge from a much larger training set. MTCN outperforms all other methods,
including the competitive method of [Li et al.(2021b)Li, Nagarajan, Xiong, and
Grauman], showcasing that multimodal temporal context from consecutive actions
is more beneficial than pretraining large models (ResNet101) using egocentric
signals from third-person datasets.

##  Appendix B EPIC-KITCHENS-55 Results

We also compare our model to works that report on the earlier version of this
dataset, namely EPIC-KITCHENS-55 [Damen et al.(2018)Damen, Doughty, Farinella,
Fidler, Furnari, Kazakos, Moltisanti, Munro, Perrett, Price, and Wray] in
Table 7. We report results for the Seen split (S1). We opted to include these
in the appendix to avoid confusion in the main paper as the results are not
comparable across these two dataset versions. We compare MTCN with two audio-
visual approaches [Wang et al.(2020b)Wang, Tran, and Feiszli] and [Xiao et
al.(2020a)Xiao, Lee, Grauman, Malik, and Feichtenhofer], as well as [Wu et
al.(2019)Wu, Feichtenhofer, Fan, He, Krahenbuhl, and Girshick] which was one
of the first works to utilise temporal context. We also report the performance
of [Li et al.(2021b)Li, Nagarajan, Xiong, and Grauman] which evaluates their
method on both EPIC-KITCHENS-55 and EPIC-KITCHENS-100. Our MTCN outperforms
all approaches.

##  Appendix C Language model analysis and baselines

In this section, we assess the statistical significance of our language model
and compare the performance of our MTCN to variants using baseline language
models. All the experiments in this section as well as in Sections D and E and
the visualisations in Section H are performed on the validation set of EPIC-
KITCHENS-100 and using the SlowFast visual features.

Statistical significance of LM. We train 10 audio-visual transformers and 10
corresponding language models with different random seeds. Table 8 shows the
mean and standard deviation top-1 and top-5 accuracy without and with the
language model. Utilising the language model improves performance on average
with a low std, demonstrating that the improvement from the language model is
statistically significant. We further showcase that by conducting T-tests on
verb, noun and action top-1 accuracies, obtaining a p-value of $3.6e-2$,
$6.0e-4$, $9.7e-4$, respectively.

| Top-1 Accuracy (%) | Top-5 Accuracy (%)
---|---|---
LM | Verb | Noun | Action | Verb | Noun | Action
✗ | 70.26 $\pm$ 0.27 | 55.70 $\pm$ 0.22 | 44.90 $\pm$ 0.20 | 91.12 $\pm$ 0.13 | 79.03 $\pm$ 0.18 | 64.79 $\pm$ 0.17
✓ | 70.52 $\pm$ 0.25 | 56.08 $\pm$ 0.21 | 45.25 $\pm$ 0.18 | 91.13 $\pm$ 0.13 | 79.03 $\pm$ 0.18 | 64.58 $\pm$ 0.18

Table 8: Mean and standard deviation of multiple runs both w. & w/o language
model in the validation set of EPIC-KITCHENS-100.

| Overall | Unseen Participants | Tail-classes
---|---|---|---
| Top-1 Accuracy (%) | Top-5 Accuracy (%) | Top-1 Accuracy (%) | Top-1 Accuracy (%)
Model | Verb | Noun | Action | Verb | Noun | Action | Verb | Noun | Action | Verb | Noun | Action
No LM | 70.23 | 55.82 | 45.00 | 91.13 | 79.06 | 64.58 | 63.29 | 46.38 | 35.02 | 41.76 | 32.26 | 24.41
N-gram | 70.23 | 55.84 | 45.02 | 91.13 | 79.06 | 64.49 | 63.29 | 46.38 | 35.02 | 41.76 | 32.26 | 24.41
BiLSTM | 70.57 | 55.97 | 45.09 | 91.14 | 79.06 | 64.55 | 63.29 | 46.76 | 35.31 | 40.68 | 32.47 | 24.15
Transformer enc. (proposed) | 70.60 | 56.26 | 45.48 | 91.14 | 79.06 | 63.06 | 63.76 | 47.14 | 35.87 | 41.36 | 32.84 | 24.70

Table 9: Performance of MTCN in the validation set of EPIC-KITCHENS-100 using
different language models.

Baselines comparison. We compare our MTCN that uses a transformer based
language model to two baselines, N-gram and Bi-directional LSTM (BiLSTM). For
N-gram, we follow a similar procedure to natural language processing. In
particular, from all action sequences of length 9 in the training set, we
derive the heuristic probability of occurrence of the centre action given the
preceding and succeeding actions. We train a BiLSTM with 3 layers and a hidden
size of 512. The rest hyperparameters are the same as the transformer encoder.

Results are shown in Table 9. It turns out that only a few preceding-
succeeding action sequences in the training set also appear in the validation
set, resulting in no difference in performance when N-gram is added comparing
to not using a language model. Our transformer-based Masked Language Model
(MLM) outperforms both the N-gram and BiLSTM, showcasing that it is beneficial
to use a deep neural network language model over a heuristic prior and that
MLM with transformers outperforms recurrent architectures in this problem.

##  Appendix D Online recognition

The focus of this work is to leverage both past and future context to predict
an action. In this section however, we explore the performance of our model in
online recognition, i.e. using only the preceding actions as context to
predict the current action. This approach can be used to recognise actions in
an online fashion for streaming videos. For this setting, we train the audio-
visual transformer to predict the last action in the sequence. We do not train
a new language model for this task; we simply mask and predict the last action
in the sequence instead of the centre one.

Results are demonstrated in Table 10 by varying $w$. Our model can also
utilise temporal context in this setting, as performance improves for $w>1$
with optimal top-1 accuracy at $w=7$ and optimal accuracy on tail-classes at
$w=9$. Compared to our original proposal that utilises also future context
(see Table 1 on main paper), the overall performance degrades, indicating that
leveraging future context is beneficial.

| Overall | Unseen Participants | Tail-classes
---|---|---|---
| Top-1 Accuracy (%) | Top-5 Accuracy (%) | Top-1 Accuracy (%) | Top-1 Accuracy (%)
$w$ | Verb | Noun | Action | Verb | Noun | Action | Verb | Noun | Action | Verb | Noun | Action
1 | 67.93 | 52.29 | 41.30 | 90.53 | 76.47 | 61.52 | 61.13 | 44.60 | 32.58 | 42.05 | 27.42 | 21.48
3 | 68.42 | 54.15 | 42.59 | 91.20 | 78.52 | 61.10 | 61.69 | 44.41 | 32.11 | 40.11 | 31.26 | 22.58
5 | 68.58 | 54.04 | 42.75 | 90.96 | 78.27 | 62.04 | 59.81 | 43.94 | 32.11 | 39.49 | 31.11 | 22.48
7 | 68.88 | 54.31 | 42.96 | 90.89 | 77.87 | 62.39 | 61.41 | 43.38 | 32.02 | 40.51 | 32.00 | 23.61
9 | 68.77 | 54.28 | 42.77 | 90.66 | 77.72 | 62.44 | 60.38 | 45.07 | 31.83 | 40.80 | 32.68 | 23.86
11 | 67.83 | 54.04 | 42.13 | 90.63 | 78.85 | 62.10 | 57.46 | 43.94 | 31.46 | 36.88 | 30.74 | 21.96

Table 10: Online action recognition results by varying temporal context length
in the validation set of EPIC-KITCHENS-100.

|  | Overall | Unseen Participants | Tail-classes
---|---|---|---|---
|  | Top-1 Accuracy (%) | Top-5 Accuracy (%) | Top-1 Accuracy (%) | Top-1 Accuracy (%)
Layers | Shared | Verb | Noun | Action | Verb | Noun | Action | Verb | Noun | Action | Verb | Noun | Action
1 | - | 69.58 | 55.04 | 43.71 | 91.27 | 79.02 | 63.96 | 61.03 | 46.01 | 33.33 | 42.10 | 32.42 | 24.09
2 | ✗ | 69.49 | 55.41 | 43.94 | 91.13 | 78.96 | 63.86 | 62.72 | 46.57 | 34.37 | 41.42 | 32.32 | 23.90
4 | ✗ | 71.01 | 56.55 | 46.04 | 90.98 | 79.28 | 63.97 | 62.35 | 47.61 | 35.96 | 39.94 | 31.89 | 24.22
6 | ✗ | 69.58 | 55.68 | 44.89 | 90.28 | 78.17 | 63.37 | 61.31 | 45.63 | 34.46 | 38.75 | 32.21 | 23.90
2 | ✓ | 69.82 | 55.37 | 43.81 | 91.09 | 78.97 | 64.26 | 61.50 | 44.32 | 32.68 | 42.05 | 32.58 | 23.74
4 | ✓ | 70.60 | 56.26 | 45.48 | 91.14 | 79.06 | 63.06 | 63.76 | 47.14 | 35.87 | 41.36 | 32.84 | 24.70
6 | ✓ | 69.58 | 55.68 | 44.89 | 90.28 | 78.17 | 63.37 | 61.31 | 45.63 | 34.46 | 38.75 | 32.21 | 23.90

Table 11: Analysis of performance using different number of layers, both w.
and w/o weight sharing. Results are shown in the validation set of EPIC-
KITCHENS-100.

##  Appendix E Architecture ablations

| Overall | Unseen Participants | Tail-classes
---|---|---|---
| Top-1 Accuracy (%) | Top-5 Accuracy (%) | Top-1 Accuracy (%) | Top-1 Accuracy (%)
Pos. enc. | Verb | Noun | Action | Verb | Noun | Action | Verb | Noun | Action | Verb | Noun | Action
Fourier PE | 69.60 | 56.13 | 44.63 | 90.65 | 78.86 | 63.31 | 63.29 | 45.63 | 35.12 | 38.86 | 32.53 | 23.09
Relative PE | 70.32 | 56.30 | 45.37 | 91.01 | 79.35 | 64.04 | 61.41 | 46.67 | 33.99 | 41.42 | 33.74 | 24.96
Absolute PE (Proposed) | 70.60 | 56.26 | 45.48 | 91.14 | 79.06 | 63.06 | 63.76 | 47.14 | 35.87 | 41.36 | 32.84 | 24.70

Table 12: Comparison of different positional encodings (PE) using the
validation set of EPIC-KITCHENS-100.

In Table 11, we explore different number of layers in MTCN, both without and
with (layer-wise) weight sharing, and compare each case with a single layer.
Note that we use the same number of layers and sharing strategy for both AV
and LM. We use bold to indicate best performance within each group rather than
overall. Best results are obtained using four layers in most metrics, both
without & with weight sharing. These outperform a single layer, demonstrating
that is beneficial to use a multi-layered transformer. Although MTCN without
weight sharing performs slightly better, our proposed model has $2.7\times$
less parameters with only a minor drop in performance.

In Table 12, we compare the effect of different types of positional encodings.
Particularly, we replace our chosen absolute learnt positional encoding with
relative positional encodings [Shaw et al.(2018)Shaw, Uszkoreit, and Vaswani]
and Fourier feature positional encodings [Jaegle et al.(2021)Jaegle, Gimeno,
Brock, Zisserman, Vinyals, and Carreira]. Fourier feature positional encodings
replace our learnable absolute positional encodings with non-learnable ones
represented as a vector of log-linearly spaced frequency bands up to a maximum
frequency. Relative positional encodings replace our absolute positional
encodings of the inputs, with positional encodings representing distances
between tokens and placed within the self-attention layers. As shown in the
table our proposed absolute learnable positional encodings outperform Fourier
feature positional encodings in all metrics (except top-5 action accuracy).
Comparing to relative positional encodings, our positional encodings are
slightly better in top-1 verbs and actions, as well as in unseen participants,
while relative positional encodings perform slightly better in top-5 accuracy
and tail classes. Overall, there are no notable differences between the
different choices of positional encodings.

##  Appendix F EGTEA Implementation Details

Visual features. For EGTEA, we train SlowFast [Feichtenhofer et
al.(2019)Feichtenhofer, Fan, Malik, and He] using the EPIC-KITCHENS pre-
trained model, by sampling a clip of 2s from an action segment similar to
EPIC-KITCHENS. We use a learning rate of 0.001, no warm-up, and we keep the
batch normalisation layers frozen. All unspecified hyperparameters remain
unchanged. For feature extraction, we follow the same procedure as EPIC-
KITCHENS, except that we use clips of 2s rather than 1s.

Train/Val Details. Here, we discuss differences in the architecture for
training/evaluating EGTEA. Remember that for EGTEA we train only vision and
language as EGTEA does not contain audio. First, as there is no audio input to
the transformer, we do not use modality encodings either. Second, following
previous methods [Li et al.(2018a)Li, Liu, and Rehg, Min and Corso(2021), Lu
et al.(2019b)Lu, Liao, and Li, Sudhakaran and Lanz(2018), Sudhakaran et
al.(2019)Sudhakaran, Escalera, and Lanz, Huang et al.(2020)Huang, Cai, Li, Lu,
and Sato] that train using a single head for actions and report only action
accuracy, we use a single summary embedding for actions, rather than verb/noun
embeddings. Accordingly, the language model utilises a single word-embedding
for actions, with a dimension of 512. For training the visual-only
transformer, we use a learning rate of 0.001, train the model for 50 epochs
and decay the learning rate at epochs 25 and 38, while keeping all other
hyperparameters unchanged. We use same hyperparameters for the language model.
For evaluation, differently than EPIC-KITCHENS, we average the predictions of
the 10 clips per action, rather than feeding all 10 clips in the transformer.

##  Appendix G Ablation of temporal context and language model in EGTEA

| Visual | Visual + LM
---|---|---
$w$ | Top-1(%) | Mean Class (%) | Top-1(%) | Mean Class (%)
1 | 72.26 | 64.98 | 72.26 | 64.98
3 | 72.55 | 64.86 | 73.59 | 65.87
5 | 73.10 | 65.42 | 73.49 | 65.57
7 | 72.26 | 64.38 | 73.19 | 65.31
9 | 72.55 | 64.86 | 73.44 | 66.02

Table 13: Ablation of temporal context extent and language model in the first
test split of EGTEA. Figure 5: Additional qualitative results of attention
weights along with the predictions of our model. Green and purple edges
represent attention to visual and auditory tokens, respectively, from the noun
summary embedding. Thickness indicates attention weights magnitude to centre
(bordered) and temporal context actions.

We study the effect of the temporal context length both with and without the
language model on the first test split of EGTEA. Results are shown in Table
13. For the visual only model, top-1 accuracy increases when we increase the
length of temporal context from $w=1$ to 5, and optimal results for both top-1
and mean class accuracy are obtained for $w=5$. When the language model is
incorporated top-1 accuracy increases from $w=1$ to 3 and then decreases while
best mean class accuracy is obtained at $w=9$. These findings showcase that
our model successfully utilises context in this dataset as well. The language
model is helpful for EGTEA, and provides a bigger boost in performance than
EPIC-KITCHENS, possibly due to the absence of audio modality. Finally, it is
worth noting that after the addition of the language model best performance is
obtained at a shorter temporal context, showing that shorter sequences of
actions provide a stronger prior in this dataset.

##  Appendix H Attention Visualisation

In Fig. 5, we show additional qualitative examples, similar to Fig. 4 in the
main paper. These demonstrate how our model attends to temporal context. In
the first three examples, the model predicts the centre action correctly,
while in the last one it gives incorrect predictions. In the ‘wash chopping
board’ example, the model particularly attends to actions containing the
chopping board. For ‘open cupboard’, the model has high audio-visual attention
to the centre action, and high attention to the audio of the previous action
(‘insert coffee maker’), showing that at times audio provides useful temporal
context. The importance of audio is also apparent in the third example. A
source of error in the model results from confusing the centre action with
another action in the sequence; in the fourth example ‘close dishwasher’ is
predicted as ‘take chopping board’ which corresponds to the first and last
actions in the temporal context.