# Activity Graph Transformer for Temporal Action Localization

Megha Nawhal1, Greg Mori1,2
1 Simon Fraser University, Burnaby, Canada
2 Borealis AI, Vancouver, Canada

###### Abstract

We introduce Activity Graph Transformer, an end-to-end learnable model for
temporal action localization, that receives a video as input and directly
predicts a set of action instances that appear in the video. Detecting and
localizing action instances in untrimmed videos requires reasoning over
multiple action instances in a video. The dominant paradigms in the literature
process videos temporally to either propose action regions or directly produce
frame-level detections. However, sequential processing of videos is
problematic when the action instances have non-sequential dependencies and/or
non-linear temporal ordering, such as overlapping action instances or re-
occurrence of action instances over the course of the video. In this work, we
capture this non-linear temporal structure by reasoning over the videos as
non-sequential entities in the form of graphs. We evaluate our model on
challenging datasets: THUMOS14, Charades, and EPIC-Kitchens-100. Our results
show that our proposed model outperforms the state-of-the-art by a
considerable margin.

##  1 Introduction

Figure 1: Main Idea. Given an untrimmed human activity video, we directly
predict the set of action instances (label, start time, end time) that appear
in the video. We observe that human activity videos contain non-sequential
dependencies (illustrated by the ground truth instances as colored bars). In
this work, we propose Activity Graph Transformer that captures this non-
sequential structure by reasoning over such videos as graphs. Overall, the
network receives a video and directly infers a set of action instances. The
network achieves this by transforming a set of graph-structured abstract
queries into contextual embeddings which are then used to provide predictions
of action instances. It is trained end-to-end using classification and
regression losses.

Visual understanding of human activities in untrimmed videos involves
reasoning over multiple action instances with varying temporal extents. This
problem has been formally studied in the setup of temporal action
localization, _i.e_., given a human activity video, the goal is to predict a
set of action labels and their corresponding start and end timestamps
indicating their occurrence in the video. Reasoning over untrimmed human
activity videos for action localization is particularly challenging due to the
idiosyncrasies of the videos such as: (1) overlap - the action instances may
have overlaps in their temporal extents indicating non-sequential temporal
ordering of the instances; (2) non-sequential dependencies - some action
instances may have temporal dependencies but are separated by other unrelated
action instances and/or durations of no action; and (3) re-occurrence -
instances belonging to same category may appear more than once over the course
of the video. In this work, we propose a novel end-to-end learnable model for
temporal action localization that receives a video as an input and directly
predicts the set of action instances that appear in the video.

Existing approaches for the task of temporal action localization predominantly
fall into two paradigms. First is the local-then-global paradigm where the
video-level predictions are obtained by postprocessing of local (_i.e_. frame-
level or snippet-level) predictions using sequence modeling techniques such as
recurrent neural networks, temporal convolutions and temporal pooling [41, 65,
11, 47, 32, 63, 28, 39, 25, 39]. Second is the proposal-then-classification
paradigm which involves generation of a sparse set of class agnostic segment
proposals from the overall video followed by classification of the action
categories for each proposal using either two-stage learning [4, 3, 18, 12,
44, 42, 69, 68] or end-to-end learning [8, 14, 7, 61, 66].

The local-then-global paradigm does not utilize the overall temporal context
provided by the activity in the video as the local predictions are solely
based on visual information confined to the frame or the snippet. For
instance, consider the example in Figure 1, these approaches would miss out on
important relevant information provided by ‘mix pasta’ in predicting ‘put
spoon’ or may produce imprecise predictions when the temporal extents of
instances ‘take colander’ or ‘open cupboard’ overlap.

Alternatively, the proposal-then-classification paradigm generates a subset of
proposals by processing the video as a sequence. As a result, these approaches
suffer from limited receptive field for incorporating temporal information,
and do not capture non-sequential temporal dependencies effectively. This
problem is further aggravated in the case of overlapping action instances. For
instance, in the example in Figure 1, ‘open cupboard’ and ‘close cupboard’
share information but are separated by other, potentially overlapping, action
instances such as ‘take colander’. Due to such ordering, when generating
proposals corresponding to ‘close cupboard’, these approaches are unlikely to
capture the dependency with the visual information pertaining to ‘open
cupboard’. Furthermore, these approaches use heuristics to perform non-maximal
suppression of proposals that might result in imprecise localization outcomes
when the action instances vary widely in their temporal extents.

As such, both these types of approaches process videos sequentially to either
generate direct local predictions or action proposals and are problematic when
action instances reoccur, overlap, or have non-sequential dependencies. These
observations suggest that although a video has a linear ordering of frames,
the reasoning over the video need not be sequential. We argue that modeling
the non-linear temporal structure is a key requirement for effective reasoning
over untrimmed human activity videos. In this work, we seek a temporal action
localization model that: (1) captures the temporal structure in complex human
activity videos, (2) does not rely on heuristics or postprocessing of the
predictions, and (3) is trained end-to-end.

Towards this goal, we formulate temporal action localization as a direct set
prediction task. We propose a novel temporal action localization model,
Activity Graph Transformer (AGT), an end-to-end learnable model that receives
a video as input and predicts the set of action instances that appear in the
video. In order to capture the non-linear temporal structure in videos, we
reason over videos as non-sequential entities, specifically, learnable graph
structures. Particularly, we map the input video to graph-structured
embeddings using an encoder-decoder transformer architecture that operates
using graph attention. A final feed forward network then uses these embeddings
to directly predict the action instances. Thus, we propose a streamlined end-
to-end training process that does not require any heuristics.

To summarize, our contributions are as follows: (1) we propose an encoder-
decoder transformer based model Activity Graph Transformer that reasons over
videos as graphs and can be trained end-to-end, and (2) we achieve state-of-
the-art performance on the task of temporal action localization on challenging
human activity datasets, namely, THUMOS14 [23], Charades [45], and EPIC-
Kitchens100 [10].

##  2 Related Work

In this section, we discuss the prior work relevant to temporal action
localization and graph based modeling in videos.

Temporal Action Localization. Early methods for temporal action localization
use temporal sliding windows and design hand-crafted features to classify
action within each window [64, 13, 22, 50, 35]. However, these approaches are
computationally inefficient as they apply classifiers on windows of all
possible sizes and locations in the entire video.

With the advances in convolutional neural networks, recent approaches fall
into two dominant paradigms: (1) local-then-global, (2) proposal-then-
classification. The methods following the local-then-global paradigm rely on
obtaining temporal boundaries of actions based on local (_i.e_. frame-level or
snippet-level) predictions and perform video-level reasoning using temporal
modeling techniques such as explicit modeling of action durations or
transitions [41, 65], recurrent neural networks [11, 47, 32, 63], temporal
pooling [25], temporal convolutions [28, 39], and temporal attention [39].
However, these approaches does not utilize the overall temporal context of the
videos as local predictions are computed using only the frame/snippet
information.

The methods based on proposal-then-classification paradigm formulate temporal
action localization as the mirror problem of object detection in the temporal
domain. Inspired by the progress in object detection [16] techniques, some
methods employ a two-stage training framework [4, 3, 18, 12, 44, 42, 69, 68] –
they generate a set of class-agnostic segment proposals in the first stage and
predict an action label for each proposal in the second stage. Most recent
methods in this direction focus on improving the proposal generation stage [4,
3, 18, 12, 29, 30, 68, 1], while a few propose a more accurate classification
stage [42, 69].

Recently, some end-to-end trainable architectures have also been proposed [8,
14, 7, 61, 66, 62]. However, these methods also process the video as a
sequence and, thus, have limited receptive field for capturing temporal
information. They do not capture non-sequential temporal dependencies in
action instances. Moreover, these approaches use heuristics during training
(_e.g_. intersection-over-union thresholds) to perform non-maximal suppression
in the set of proposals. This might lead to poor localization performance when
the action instances vary widely in their temporal extents as it might skip
some highly overlapping proposals. To address these problems in object
detection, [5] propose a transformer based end-to-end learnable architecture
that implicitly learns the non-max suppression and perform object detection
using proposals as abstract encodings.

In contrast to the above approaches, we formulate temporal action localization
as a direct set prediction task. We propose to reason over untrimmed videos as
non-sequential entities (_i.e_. graphs) as opposed to existing methods that
perform sequential reasoning. Our approach is inspired by [5] in that we
propose an end-to-end learnable transformer based model for direct set
prediction. But unlike [5], the transformer model in our approach operates
graphs.

Additionally, there are other realms of work on temporal action localization
in weakly supervised setting [43, 58, 21] and spatio-temporal action
localization [48, 24, 17, 15]. These are beyond the scope of this paper.

Action Recognition. Action recognition methods operate on short video clips
that are trimmed such that a single action instance spans the video duration
and, hence, are not suitable for untrimmed videos containing multiple actions.
Nonetheless, models pretrained for the task of action recognition provide
effective feature representations for tasks related to untrimmed videos. A
wide variety of action recognition approaches have been proposed ranging from
earlier methods based on hand-crafted features [27, 9, 55] to convolutional
models such as I3D [46], 3D-CNN [51] through to advanced temporal modeling
[57, 59, 67] and graph modeling [60, 20] techniques. In this paper, we use I3D
[46] pretrained on the Kinetics dataset [6] for feature extraction.

Graph-based Modeling for Videos. The advances in graph convolutional networks
(GCNs) [26] have inspired several recent approaches for video based tasks [36,
60, 34]. Most of the graph based approaches for videos represent either the
input space (_i.e_. videos or derived visual information) as graphs [36, 60,
34, 19] or the output space (_i.e_. labels) as graphs [52]. In contrast, we
design our model based on the insight that both the input space (_i.e_.
features derived from videos) and the output space (_i.e_. labels and
timestamps for the action) are graph-structured for the task of temporal
action localization. Specifically, we propose an encoder-decoder transformer
architecture to learn the mapping between the input and output space.
Furthermore, GCNs require the information pertaining to the nodes and edges a
priori. In contrast, we learn the graph structure (_i.e_. both nodes and
edges) from the data itself using self-attention.

##  3 Proposed Approach

Figure 2: Model Overview. Activity Graph Transformer (AGT) receives a video as
input and directly predicts a set of action instances that appear in the
video. The input video is fed into a backbone network to obtain a compact
representation (Section 3.1.1). Then, the encoder network (Section 3.1.2)
receives the compact video-level representation from the backbone network and
encodes it to a latent graph representation context graph. The decoder network
(Section 3.1.3) receives the context graph along with graph-structured
abstract query encodings action query graph. The decoder transforms the action
query graph to a graph-structured set of embeddings. Each node embedding of
the decoder output is fed into a prediction head (Section 3.1.4). The network
is trained end-to-end (Section 3.1.5) using classification and regression
losses for the action labels and timestamps of the action instances
respectively.

In this section, we present the problem formulation and provide a detailed
description of our proposed approach.

Problem Formulation. The task of temporal action localization involves
prediction of the category labels as well as start and end timestamps of the
actions that occur in a given video. In this work, we formulate this task as a
direct set prediction problem wherein each element in the predicted set
denotes an action instance in a video. Specifically, given a video $V$, the
goal is to predict a set $\mathcal{A}$ where the $i$-th element
$a^{(i)}=(c^{(i)},t^{(i)}_{s},t^{(i)}_{e})$ denotes an action instance in the
video depicting action category $c^{(i)}$ that starts at time $0\leq
t^{(i)}_{s}\leq T$, ends at time $0\leq t^{(i)}_{e}\leq T$, for
$i\in\\{1,2,\ldots,|\mathcal{A}|\\}$. Here, $|\mathcal{A}|$ is the number of
action instances present in the video and $T$ is the duration of the video.
Thus, $|\mathcal{A}|$ and $T$ vary based on the input video.

Towards this goal, we propose Activity Graph Transformer (AGT), an end-to-end
learnable model that receives a video as input and directly infers the set of
action instances (label, start time, end time) in the video. Our approach
consists of: (1) a network that predicts a set of action instances in a single
forward pass; and (2) a loss function to train the network by obtaining a
unique alignment between the predicted and ground truth action instances. We
contend that effective reasoning over untrimmed human activity videos requires
modeling the non-linear temporal structure in the videos. In our approach, we
seek to capture this structure by employing graphs. Specifically, we propose a
novel encoder-decoder transformer network that leverages graph based self-
attention to reason over the videos. We describe the details of our approach
below.

###  3.1 Activity Graph Transformer

As shown in Figure 2, Activity Graph Transformer (AGT) consists of three
components: (1) backbone network to obtain a compact representation of the
input video; (2) transformer network consisting of an encoder network and a
decoder network that operates over graphs; and (3) prediction heads for the
final prediction of action instances of the form (label, start time, end
time). The encoder network receives the compact video-level representation
from the backbone network and encodes it to a latent graph representation,
referred to as context graph. The decoder network receives graph-structured
abstract query encodings (referred to as action query graph) as input along
with the context graph. The decoder uses the context graph to transform the
action query graph to a graph-structured set of embeddings. Each node
embedding of this decoder output is fed into a feed forward network to obtain
predictions of action instances. The whole AGT network is trained end-to-end
using a combination of classification and regression losses for the action
labels and timestamps respectively. Refer to Algorithm 1 for an overview of
one training iteration of AGT. We provide detailed description of the
components below.

####  3.1.1 Backbone

To obtain a compact representation for the input video $V$ containing $T$
frames, any 3D convolutional network can be used to extract the features. In
our implementation, we chunk the videos into short overlapping segments of 8
frames and use an I3D model pretrained on the Kinetics[6] dataset to extract
features of dimension $C$ ($=2048$) from the segments, resulting in video-
level feature
$\mathbf{v}=[\mathbf{v}^{(1)},\mathbf{v}^{(2)}\ldots\mathbf{v}^{(N_{v})}]$
where $N_{v}$ is the number of chunks used in the feature extraction.

####  3.1.2 Transformer Encoder

The backbone simply provides a sequence of local features and does not
incorporate the overall context of the video or the temporal structure in the
video. Therefore, we use an encoder network that receives the video-level
feature as input and encodes this video representation to a graph (referred to
as the context graph). Intuitively, the encoder is designed to model the
interactions among the local features using self-attention modules.

The context graph is initialized with video-level feature $\mathbf{v}^{(i)}$
(of dimension $C=2048$) as the $i$-th node for $i\in\\{1,2,\ldots,N_{v}\\}$.
Usually, transformer networks use fixed positional encoding [37] to provide
position information of each element in the input sequence. In contrast, in
our setting, we contend that the video features have a non-linear temporal
structure. Thus, we provide the positional information using learnable
positional encodings $\mathbf{p}_{v}$ as additional information to the video
feature $\mathbf{v}$. The positional encoding $\mathbf{p}_{v}^{(i)}$
corresponds to the $i$-th node in the graph and is of the same dimension as
the node. Next, the graph nodes are from the same video and hence, they are
related to each other. However, their connection information (edges) is not
known a priori. Thus, we model the interactions among these nodes as learnable
edge weights. This is enabled by the graph self-attention module (described
below).

We design the transformer encoder network $\mathbf{E}$ as a sequence of
$L_{e}$ blocks, wherein, an encoder block $\mathbf{E}_{\ell}$ for
$\ell\in\\{1,2,\ldots,L_{e}\\}$ consists of a graph self-attention module
followed by a feed forward network. The output of the encoder network is the
context graph
$\mathbf{h}_{L_{e}}=[\mathbf{h}^{(1)}_{L_{e}},\mathbf{h}^{(2)}_{L_{e}}\ldots\mathbf{h}^{(N_{v})}_{L_{e}}]$
where $\mathbf{h}^{(i)}_{L_{e}}$ is the $i$-th node and is of dimension $d$
(same for each block). The output of the $\ell$-th encoder block
$\mathbf{h}_{\ell}$ and the final output of the encoder $\mathbf{h}_{L_{e}}$
are defined as:

$\begin{split}\mathbf{h}_{0}&=\mathbf{v}\\\
\mathbf{h}_{\ell}&=\mathbf{E}_{\ell}(\mathbf{h}_{\ell-1},\mathbf{p}_{v})\\\
\mathbf{h}_{L_{e}}&=\mathbf{E}_{L_{e}}\circ\cdots\circ\mathbf{E}_{1}(\mathbf{v},\mathbf{p}_{v}).\end{split}$
(1)

Graph Self-Attention. This module aims to model interactions among graph
structured variables along with learnable edge weights. Here, we describe the
graph self-attention module in the context of the $(\ell+1)$-th encoder block
$\mathbf{E}_{\ell+1}$. For simplicity of notation, let $\mathbf{x}$ be the
output of the $\ell$-th block of the encoder, _i.e_.,
$\mathbf{x}=\mathbf{E}_{\ell}(\mathbf{h}_{\ell-1},\mathbf{p}_{v})$.
$\mathbf{x}$ is a graph contains $N_{v}$ nodes
$\mathbf{x}^{(1)},\mathbf{x}^{(2)},\ldots,\mathbf{x}^{(N_{v})}$ which are
connected using learnable edge weights. The graph self-attention module first
performs graph message passing (as described in [54]) to produce the output
$\mathbf{x}^{\prime}$, with the $i$-th node of the output defined as

$\mathbf{x^{\prime}}^{(i)}=\mathbf{x}^{(i)}+\Big{|}\Big{|}_{k=1}^{K}\sigma\Big{(}\sum_{j\in\mathcal{N}_{i}}\alpha_{ij}^{k}\mathbf{W}_{g}^{k}\mathbf{x}^{(j)}\Big{)},$
(2)

where $\Big{|}\Big{|}$ represents concatenation operator, $K$ is the number of
parallel heads in the self-attention module, $\sigma$ is a non-linear function
(leaky ReLU in our case), $\mathcal{N}_{i}$ represents the set of neighbours
of the $i$-th node, $\mathbf{W}_{g}^{k}$ is the learnable transformation
weight matrix. $\alpha_{ij}^{k}$ are the self attention coefficients computed
by the $k$-th attention head described as:

$\alpha_{ij}^{k}=\frac{\exp(f(\mathbf{w}_{a,k}^{T}[\mathbf{W}_{g}^{k}\mathbf{x}^{(i)}||\mathbf{W}_{g}^{k}\mathbf{x}^{(j)}]))}{\sum_{m\in\mathcal{N}_{i}}\exp(f(\mathbf{w}_{a,k}^{T}[\mathbf{W}_{g}^{k}\mathbf{x}^{(i)}||\mathbf{W}_{g}^{k}\mathbf{x}^{(m)}]))},$
(3)

where $\cdot^{T}$ represents a transpose operator, $f$ is a non-linear
activation (leaky ReLU in our case) and $\mathbf{w}_{a,k}$ is the attention
coefficients. $\alpha_{ij}^{k}$ is the attention weight and denotes the
strength of the interaction between $i$-th and $j$-th node of the input graph
of the module. Subsequent to the message passing step, we apply batch
normalization and a linear layer. This is then followed by a standard multi-
head self-attention layer (same as in [53]). Overall, the graph self-attention
module models interactions between the nodes, _i.e_., local features derived
from the video.

Algorithm 1 A training iteration of AGT model

1:video $V$ containing $T$ frames; number of action query encodings $N_{o}$;
ground truth action instances
$\mathcal{A}=\\{a^{(i)}\\}_{i=1}^{|\mathcal{A}|}$

2:Backbone initialized to I3D model [6] pretrained on Kinetics dataset;
initialize action query graph $\mathbf{q}$ with $N_{o}$ random encodings

3:compute features using backbone

4:compute context graph using encoder (see Eq. 1)

5:compute output embeddings using decoder (see Eq. 4)

6:for $i\in 1,2,\ldots,N_{o}$ do

7: predict action instance $\tilde{a}^{(i)}$ using prediction head

8:end for

9:compute optimal matching $\hat{\phi}$ between
$\\{a^{(i)}\\}_{i=1}^{|\mathcal{A}|}$ and
$\\{\tilde{a}^{(i)}\\}_{i=1}^{N_{o}}$ using matcher

10:compute final loss $\mathcal{L}_{H}$ between
$\\{a^{(i)}\\}_{i=1}^{|\mathcal{A}|}$ and
$\\{\tilde{a}^{(\hat{\phi}(i))}\\}_{i=1}^{|\mathcal{A}|}$ using Eq. 9

11:backpropagate $\mathcal{L}_{H}$

####  3.1.3 Transformer Decoder

Based on the observation that the action instances in the video have a non-
linear temporal structure, we design the decoder to learn a graph-structured
set of embeddings which would subsequently be used for predicting the action
instances. Intuitively, the output graph provided by the decoder serves as the
latent representation for the set of action instances depicted in the video.

The inputs of the transformer decoder are: (1) a graph-structured abstract
query encodings, referred to as action query graph $\mathbf{q}$, containing
$N_{o}$ nodes wherein each node is a learnable positional encoding of
dimension $d$ (same as the dimension used in the encoder); and (2) the context
graph $\mathbf{h}_{L_{e}}$ containing $N_{v}$ nodes (obtained from the
encoder). We assume that the number of nodes in the action query graph $N_{o}$
is fixed and is sufficiently larger than the maximum number of action
instances per video in the dataset. This idea of using representations of
prediction entities as positional query encodings is inspired from [5].
However, unlike the independent queries in [5], we use graph-structured
encodings for the decoder. To learn the interactions among the graph-
structured query embeddings, we use graph self-attention modules (same module
as used in transformer encoder). Additionally, we use graph-to-graph attention
module (described below) to learn interactions between the context graph,
_i.e_., the latent representation of the input video, and the graph-structured
query embeddings, _i.e_., the latent representations of the action queries.

The overall decoder network $\mathbf{D}$ consists of $L_{d}$ blocks, wherein,
a decoder block $\mathbf{D}_{\ell^{\prime}}$ for
$\ell^{\prime}\in\\{1,2,\ldots,L_{d}\\}$ consists of a graph self-attention
module followed by a graph-to-graph attention module, and then a feed forward
network. The decoder block $\mathbf{D}_{\ell^{\prime}}$ has an output
$\mathbf{y}_{\ell^{\prime}}$ and the final output of the decoder
$\mathbf{y}_{L_{d}}=[\mathbf{y}^{(1)}_{L_{d}},\mathbf{y}^{(2)}_{L_{d}},\ldots,\mathbf{y}^{(N_{o})}_{L_{d}}]$.
They are defined as:

$\begin{split}\mathbf{y}_{0}&=\mathbf{q}\\\
\mathbf{y}_{\ell^{\prime}}&=\mathbf{D}_{\ell^{\prime}}(\mathbf{y}_{\ell^{\prime}-1},\mathbf{h}_{L_{e}})\\\
\mathbf{y}_{L_{d}}&=\mathbf{D}_{L_{d}}\circ\cdots\circ\mathbf{D}_{1}(\mathbf{q},\mathbf{h}_{L_{e}})\end{split}$
(4)

Graph-to-Graph Attention. The graph-to-graph attention module aims to learn
the interactions between two different graphs referred to as a source graph
and a target graph. Here, we describe this module in the context of the
decoder block $\mathbf{D}_{\ell^{\prime}+1}$. The input to this block is the
output $\mathbf{y}_{\ell^{\prime}}$ of the previous decoder block
$\mathbf{D}_{\ell^{\prime}}$. This is fed to the graph self-attention module
in the block $\mathbf{D}_{\ell^{\prime}+1}$, and the output is used as the
target graph for the graph-to-graph attention module. The source graph for
this module (in any decoder block) is the context graph $\mathbf{h}_{L_{e}}$.
For simplicity of notation, let $\mathbf{x}_{s}$ denote the source graph
(_i.e_. $\mathbf{h}_{L_{e}}$) and $\mathbf{x}_{t}$ denote the target graph.
Here, the source and target graphs may contain different number of nodes. In
our case, $\mathbf{x}_{s}$ contains $N_{v}$ nodes and $\mathbf{x}_{t}$
contains $N_{o}$ nodes. The graph-to-graph attention module first performs
message passing from source graph to target graph to provide an output
$\mathbf{x^{\prime}}_{t}$, with the $i$-th node defined as

$\mathbf{x^{\prime}}_{t}^{(i)}=\mathbf{x}_{t}^{(i)}+\Big{|}\Big{|}_{k=1}^{K}\sigma\Big{(}\sum_{j\in\mathcal{N}_{i}}\beta_{ij}^{k}\mathbf{W}_{s}^{k}\mathbf{x}_{s}^{(j)}\Big{)},$
(5)

where $\mathbf{W}_{s}^{k}$ is the learnable transformation weight matrix for
the source graph. Other symbols denote the same entities as in graph self-
attention (Section 3.1.2). $\beta_{ij}^{k}$ is the attention coefficient for
$k$-th attention head between $i$-th node of the source graph and $j$-th node
of the target graph computed as:

$\beta_{ij}^{k}=\frac{\exp(f(\mathbf{w}^{st}{{}_{a,k}^{T}}[\mathbf{W}_{s}^{k}\mathbf{x}_{s}^{(i)}||\mathbf{W}_{t}^{k}\mathbf{x}_{t}^{(j)}]))}{\sum_{m\in\mathcal{N}_{i}}\exp(f(\mathbf{w}^{st}{{}_{a,k}^{T}}[\mathbf{W}_{s}^{k}\mathbf{x}_{s}^{(i)}||\mathbf{W}_{t}^{k}\mathbf{x}_{t}^{(m)}]))}$
(6)

where $\mathbf{w}^{st}_{a,k}$ is the graph-to-graph attention coefficients,
and $\mathbf{W}_{s}^{k}$ and $\mathbf{W}_{t}^{k}$ are the learnable
transformation weight matrices for source and target graphs respectively.
Other symbols denote the same entities as in graph self-attention. Similar to
the transformer encoder, subsequent to the message passing step, we apply
batch normalization and a linear layer. This is then followed by a standard
multi-head self-attention layer (same as in [53]). Overall, the graph-to-graph
attention module models the interactions between the latent representations of
the input video and the action queries.

####  3.1.4 Prediction Heads

The decoder network provides a set of embeddings where the embeddings serve as
the latent representations for the action instances in the video. This output
graph $\mathbf{y}_{L_{d}}$ contains $N_{o}$ nodes. We use these $N_{o}$ node
embeddings to obtain predictions for $N_{o}$ action instances using prediction
heads. The prediction heads consist of a feed forward network (FFN) with ReLU
activation which provides the start time and end time of the action instance
normalized with respect to the overall video duration. Additionally, we use a
linear layer with a softmax function to predict the categorical label
corresponding to the action instance.

Therefore, when provided with the $i$-th node embedding
$\mathbf{y}_{L_{d}}^{(i)}$, the prediction head provides prediction
$\tilde{a}^{(i)}=(\tilde{c}^{(i)},\tilde{t}^{(i)}_{s},\tilde{t}^{(i)}_{e})$
where $\tilde{c}^{(i)}$, $\tilde{t}^{(i)}_{s}$ and $\tilde{t}^{(i)}_{e}$ are
the category label, start time and end time for the $i$-th action instance for
$i\in\\{1,2,\ldots,N_{o}\\}$. Note that the ground truth set would contain a
variable number of action instances, whereas $N_{o}$ is larger than the
maximum number of action instances per video in the dataset. This calls for a
need to suppress irrelevant predictions. We do this by introducing an
additional class label $\varnothing$ indicating no action (similar to [5]). As
such, this non-maximal suppression (typically performed using heuristics in
existing methods [7]) is learnable in our model.

####  3.1.5 Loss functions

To train the overall network, we align the predictions with the ground truth
action instances using a matcher module which optimizes a pair-wise cost
function. This provides a unique matching between the predicted and ground
truth action instances. Subsequently, our model computes losses corresponding
to these matched pairs of predicted and ground truth action instances to train
the overall network end-to-end.

Matcher.  The matcher module finds an optimal matching between the predicted
set of action instances (that contains fixed number of elements for every
video) and the ground truth set of action instances (that contains a variable
number of elements depending on the video). To obtain this matching, we design
a matching cost function and employ the Hungarian algorithm to obtain an
optimal matching between the two sets as described in prior work [49].

Formally, let $\mathcal{A}$ be the ground truth set of action instances
$\mathcal{A}=\\{a^{(i)}\\}^{|\mathcal{A}|}_{i=1}$, where
$a^{(i)}=(c^{(i)},t^{(i)}_{s},t^{(i)}_{e})$ and $\mathcal{\tilde{A}}$ be the
predicted set of action instances
$\mathcal{\tilde{A}}=\\{\tilde{a}^{(i)}\\}^{N_{o}}_{i=1}$ where
$\tilde{a}^{(i)}=(\tilde{c}^{(i)},\tilde{t}^{(i)}_{s},\tilde{t}^{(i)}_{e})$.
In our model, we assume that $N_{o}$ is larger than the number of actions in
any video in the dataset. Therefore, we assume that ground truth set
$\mathcal{A}$ also is a set of size $N_{o}$ by padding the remaining
$(N_{o}-|\mathcal{A}|)$ elements with $\varnothing$ element indicating no
action. The optimal bipartite matching between the two sets reduces to
choosing the permutation of $N_{o}$ elements $\hat{\phi}$ from the set of all
possible permutations $\Phi_{N_{o}}$ that results in lowest value of the
matching cost function $\mathcal{L}_{m}$. Thus,
$\hat{\phi}=\operatorname*{argmin}_{\phi\in\Phi_{N_{o}}}\mathcal{L}_{m}(a^{(i)},\tilde{a}^{(\phi(i))})$,
where $\mathcal{L}_{m}(a^{(i)},\tilde{a}^{(\phi(i))})$ is the matching cost
function between ground truth $a^{(i)}$ and prediction with index $\phi(i)$.
The matching cost function incorporates the class probabilities of the action
instances and the proximity between predicted and ground truth timestamps.
Specifically, we define the cost function as:

$\begin{split}\mathcal{L}_{m}(a^{(i)},\tilde{a}^{(\phi(i))})&=-\mathbbm{1}_{\\{c^{(i)}\neq\varnothing\\}}\tilde{p}_{\phi(i)}(c^{(i)})\\\
&+\mathbbm{1}_{\\{c^{(i)}\neq\varnothing\\}}\mathcal{L}_{s}(s^{(i)},\tilde{s}^{(\phi(i))}),\end{split}$
(7)

where $s^{(i)}=[t^{(i)}_{s},t^{(i)}_{e}]$ and
$\tilde{s}^{(\phi(i))}=[\tilde{t}^{(\phi(i))}_{s},\tilde{t}^{(\phi(i))}_{e}]$,
and $\tilde{p}_{\phi(i)}(c^{(i)})$ is the probability of class $c^{(i)}$ for
prediction $\phi(i)$ and $\mathcal{L}_{s}$ represents segment loss that
measures proximity in the timestamps of the instances. The segment loss is
defined as a weighted combination of an $L_{1}$ loss (sensitive to the
durations of the instances) and an IoU loss (invariant to the durations of the
instances) between the predicted and ground-truth start and end timestamps. It
is expressed as:

$\mathcal{L}_{s}=\lambda_{iou}\mathcal{L}_{iou}(s^{(i)},\tilde{s}^{(\phi(i))})+\lambda_{L1}||s^{(i)}-\tilde{s}^{(\phi(i))}||_{1},$
(8)

where $\lambda_{iou},\lambda_{L1}\in\mathbbm{R}$ are hyperparameters.
Subsequent to obtaining the optimal permutation $\hat{\phi}$, we compute the
Hungarian loss $\mathcal{L}_{H}$ over all the matched pairs as follows:

$\mathcal{L}_{H}=\sum_{i=1}^{N_{o}}\Big{[}-\log\tilde{p}_{\hat{\phi}}(c^{(i)})+\mathbbm{1}_{\\{c^{(i)}\neq\varnothing\\}}\mathcal{L}_{s}(s^{(i)},\tilde{s}^{(\hat{\phi}(i))})\Big{]}.$
(9)

This loss is used to train our AGT model end-to-end. We provide further
implementation details of the model in the supplementary material.

In summary, our proposed Activity Graph Transformer performs temporal action
localization using an encoder-decoder based architecture leveraging graph
based attention modules. We jointly optimize all parameters of our model to
minimize the regression loss for the start and end timestamps of the action
instances and the cross entropy losses for the corresponding action labels.

##  4 Experiments

We conducted several experiments to demonstrate the effectiveness of our
proposed approach. In this section, we report the results of our evaluation.

Datasets. We use three benchmark datasets for evaluation. They vary in their
extent of overlap in action instances, the number of action instances per
video, and the number of action categories in the dataset. Thus, these
datasets together serve as a challenging testbed for our model.

THUMOS14 [23] contains 200 videos in training set and 213 videos in testing
set for the the task of action localization. This dataset has 20 action
categories. The videos contain an average of 15 action instances per video
with an average of 8% overlapping with other instances.

Charades [45] is large scale dataset containing 9848 videos of daily indoor
activities. This dataset has 157 action categories. Videos in the dataset
contain an average of 6 action instances per video with an average of 79% of
overlapping instances in a video. This dataset is challenging because of the
high degree of overlap in the action instances.

EPIC-Kitchens100 [10] contains 700 egocentric videos of daily kitchen
activities. This dataset contains 289 noun and 97 verb classes. Videos in the
dataset contain an average of 128 action instances per video with an average
of 28% overlapping instances in a video.

Table 1: Comparison with state-of-the-art (THUMOS14). We report the mean
average precision at different intersection over union thresholds (mAP@tIoU)
for tIoU$\in\\{0.1,0.2,0.3,0.4,0.5\\}$. $\uparrow$ indicates higher is better.

Method | mAP@tIoU $\uparrow$
---|---
0.1 | 0.2 | 0.3 | 0.4 | 0.5
Oneata _et al_. [35] | 36.6 | 33.6 | 27.0 | 20.8 | 14.4
Wang _et al_. [56] | 18.2 | 17.0 | 14.0 | 11.7 |  08.3
Caba _et al_. [4] | - | - | - | - | 13.5
Richard _et al_. [41] | 39.7 | 35.7 | 30.0 | 23.2 | 15.2
Shou _et al_. [44] | 47.7 | 43.5 | 36.3 | 28.7 | 19.0
Yeung _et al_. [63] | 48.9 | 44.0 | 36.0 | 26.4 | 17.1
Yuan _et al_. [64] | 51.4 | 42.6 | 33.6 | 26.1 | 18.8
Buch _et al_. [3] | - | - | 37.8 | - | 23.0
Shou _et al_. [42] | - | - | 40.1 | 29.4 | 23.3
Yuan _et al_. [65] | 51.0 | 45.2 | 36.5 | 27.8 | 17.8
Buch _et al_. [2] | - | - | 45.7 | - | 29.2
Gao _et al_. [14] | 60.1 | 56.7 | 50.1 | 41.3 | 31.0
Dai _et al_. [8] | - | - | - | 33.3 | 25.6
Xu _et al_. [61] | 54.5 | 51.5 | 44.8 | 35.6 | 28.9
Zhao _et al_. [69] | 66.0 | 59.4 | 51.9 | 41.0 | 29.8
Lin _et al_. [30] | - | - | 53.5 | 45.0 | 36.9
Chao _et al_. [7] | 59.8 | 57.1 | 53.2 | 48.5 | 42.8
Zeng _et al_. [66] | 69.5 | 67.8 | 63.6 | 57.8 | 49.1
Xu _et al_. [62] | 66.1 | 64.2 | 54.5 | 47.6 | 40.2
AGT (Ours) | 72.1 | 69.8 | 65.0 | 58.1 | 50.2

Comparison with state-of-the-art.  We compare the performance of our proposed
AGT with the state-of-the-art methods. We use mean average precision as the
metric to evaluate the model. To ensure fair comparison, we use the same
evaluation protocol as used by state-of-the-art methods for each of the
datasets. Table 1 shows that the our AGT achieves upto 3.5% improvement over
state-of-the-art for THUMOS14 dataset and consistently shows performance
improvement across all IoU thresholds. Table 2 shows the comparisons with
state-of-the-art methods on Charades dataset. Our model achieves 13%
improvement in the Charades dataset. We also perform comparison on recenty
released EPIC-Kitchens100 dataset for classification of verb, noun, and action
(_i.e_. both verb and noun) classes. Table 3 indicates that our model performs
consistently for all three tasks for EPIC-Kitchens100 datasets across all IoU
thresholds. Overall, these results clearly show that our proposed method AGT
outperforms the state-of-the-art methods by a considerable margin.

Table 2: Comparison with state-of-the-art (Charades). We report mean average
precision (mAP) computed using Charades_v1_localize setting in [45].
$\uparrow$: higher is better.

Method | mAP $\uparrow$
---|---
Predictive-corrective (Dave _et al_. [11]) |  08.9
Two-stream (Siggurdson _et al_. [45]) |  08.9
Two-stream + LSTM (Siggurdson _et al_. [45]) |  09.6
R-C3D (Xu _et al_. [61]) | 12.7
SSN (Zhao _et al_. [69]) | 16.4
I3D baseline [38] | 17.2
Super-events (Piergiovanni _et al_. [39]) | 19.4
TGM (Piergiovanni _et al_. [39]) | 22.3
Mavroudi _et al_. [33] | 23.7
3D ResNet-50 + super-events (Piergiovanni _et al_. [40]) | 25.2
AGT (Ours) | 28.6

Table 3: Comparison with state-of-the-art (EPIC-Kitchens100). We report mean
average precision at different intersection over union thresholds (mAP@tIoU)
for tIoU$\in\\{0.1,0.2,0.3,0.4,0.5\\}$. We use the validation split in the
original dataset for testing. $\uparrow$ indicates higher is better.

Method | Task | mAP@tIoU $\uparrow$
---|---|---
0.1 | 0.2 | 0.3 | 0.4 | 0.5
| Verb | 10.51 |  09.24 |  07.67 |  06.40 |  05.12
Damen | Noun | 10.71 |  08.73 |  06.75 |  05.05 |  03.35
_et al_. [10] | Action |  06.78 |  06.03 |  04.94 |  04.04 |  03.35
| Verb | 12.01 | 10.25 |  08.15 |  07.12 |  06.14
AGT | Noun | 11.63 |  09.33 |  07.05 |  06.57 |  03.89
(Ours) | Action |  07.78 |  06.92 |  05.53 |  04.22 |  03.86

Impact of graph based reasoning. To demonstrate the importance of reasoning
over videos as graphs, we conducted ablation studies by removing the graph
based reasoning components from either the encoder or the decoder or both
(_i.e_. overall transformer network) in our model. Specifically, this is
implemented by removing the graph message passing layers from the attention
modules (_i.e_., graph self-attention module and graph-to-graph attention
module) in the encoder and/or decoder blocks in the network. Intuitively, when
the graph message passing module is removed from the whole transformer
network, the transformer encoder treats the input as a sequence and the
transformer decoder treats the action queries as independent. Table 4 shows
the performance of these ablated versions of our model. The results clearly
show that eliminating the graph-based reasoning module hurts the localization
performance. The results also suggest that graph-based modeling is more useful
in the encoder than in the decoder. We believe this is because the graph
reasoning performed by the encoder is more useful in capturing the non-
sequential dependencies as it operates directly on the video features. For
better readability, here, we provide the mAP values averaged over the various
intersection-over-union thresholds (tIoU) for THUMOS14 and EPIC-Kitchens100.
For mAP values at specific thresholds, refer to the supplementary.

Table 4:  Ablation Study (Impact of graph based reasoning). We report
performance of ablated versions of our AGT model. We report mAP for evaluation
performance (higher is better). We remove graph reasoning in the encoder
($\mathbf{E}$) and/or decoder ($\mathbf{D}$) of the transformer. ✓ and ✗
indicates whether a component (encoder or decoder) contains graph message
passing module or not respectively. EPIC(A), EPIC (V), EPIC (N) indicates task
‘Action’, ‘Verb’, ‘Noun’ classification on EPIC-Kitchens100.

Dataset |  $\mathbf{E}$: ✗/ $\mathbf{D}$: ✗ |  $\mathbf{E}$: ✗/ $\mathbf{D}$: ✓ |  $\mathbf{E}$: ✓/ $\mathbf{D}$: ✗ |  $\mathbf{E}$: ✓/ $\mathbf{D}$: ✓
---|---|---|---|---
THUMOS14 | 55.6 | 56.3 | 58.3 | 63.0
Charades | 18.2 | 19.2 | 22.5 | 28.6
EPIC (A) |  03.0 |  03.3 |  05.1 |  05.9
EPIC (V) |  05.7 |  06.1 |  07.4 |  08.7
EPIC (N) |  04.9 |  05.3 |  06.3 |  07.7

Table 5:  Ablation Study (Impact of temporal resolution). Performance of our
AGT for different temporal resolutions of input video. Here, SR indicates
sampling rate of frames for feature extraction, _i.e_., SR=1/$k$ means frames
sampled at 1/$k$ -th factor of the original frame rate. EPIC(A), EPIC (V),
EPIC (N) indicate tasks ‘Action’, ‘Verb’, ‘Noun’ on dataset EPIC-Kitchens100.
We report mAP for evaluation (higher is better).

Dataset | SR = 1/8 | SR=1/4
---|---|---
THUMOS14 | 60.2 | 63.0
Charades | 27.3 | 28.6
EPIC (A) | 4.1 | 5.9
EPIC (V) | 7.2 | 8.7
EPIC (N) | 6.4 | 7.7

Impact of temporal resolution. To evaluate the impact of temporal resolution,
we experimented with different frame rates for the input video. Table 5 shows
the results suggesting higher resolution leads to better performance as the
higher temporal resolution provides more information in the input. However,
our results also show that lower resolution does not lead to any major drop in
performance. For better readability, here, we provide the mAP values averaged
over the various intersection-over-union thresholds (tIoU) for THUMOS14 and
EPIC-Kitchens100. mAP values at specific thresholds are available in the
supplementary.

Qualitative Results. We visualize the predictions of the model on two
different samples in Figure 3. The visualizations indicate that our model is
able to predict the correct number of action instances as well as correct
action categories with minimal errors in start and end timestamps. We believe
this is because video content around the start and end timestamps in some
instances do not contain enough information pertaining to the action. We
provide additional visualizations of predictions in the supplementary.

Additionally, refer to the supplementary for experiments on performance of our
model with varied number of layers and heads in the transformer and ablations
of loss functions.

---
Figure 3: Qualitative Results. Visualization of ground truth and predicted
action instances.

##  5 Conclusion

In this paper, we proposed a novel end-to-end learnable encoder-decoder
transformer model for the task of temporal action localization in untrimmed
human activity videos. Our approach aims to model the non-linear temporal
structure in such videos by reasoning over the videos as graphs using graph
self-attention mechanisms. The experimental evaluation showed that our model
achieves state-of-the-art performance on the task of temporal action
localization on challenging human activity datasets. Overall, this work
highlights the importance of reasoning over videos as non-sequential entities
and shows that graph-based transformers are an effective means to model
complex activity videos.

## References

  * [1] Yueran Bai, Yingying Wang, Yunhai Tong, Yang Yang, Qiyue Liu, and Junhui Liu.  Boundary content graph neural network for temporal action proposal generation.  2020\.
  * [2] Shyamal Buch, Victor Escorcia, Bernard Ghanem, Li Fei-Fei, and Juan Carlos Niebles.  End-to-end, single-stream temporal action detection in untrimmed videos.  In Proceedings of the British Machine Vision Conference (BMVC), 2017\.
  * [3] Shyamal Buch, Victor Escorcia, Chuanqi Shen, Bernard Ghanem, and Juan Carlos Niebles.  Sst: Single-stream temporal action proposals.  In Proceedings of the IEEE International Conference on Computer Vision (CVPR), 2017.
  * [4] Fabian Caba Heilbron, Juan Carlos Niebles, and Bernard Ghanem.  Fast temporal activity proposals for efficient detection of human actions in untrimmed videos.  In Proceedings of the IEEE International Conference on Computer Vision (CVPR), 2016.
  * [5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
  * [6] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [7] Yu-Wei Chao, Sudheendra Vijayanarasimhan, Bryan Seybold, David A Ross, Jia Deng, and Rahul Sukthankar.  Rethinking the faster r-cnn architecture for temporal action localization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [8] Xiyang Dai, Bharat Singh, Guyue Zhang, Larry S Davis, and Yan Qiu Chen.  Temporal context network for activity localization in videos.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
  * [9] Navneet Dalal, Bill Triggs, and Cordelia Schmid.  Human detection using oriented histograms of flow and appearance.  In Proceedings of the European Conference on Computer Vision (ECCV), 2006.
  * [10] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, , Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray.  Rescaling egocentric vision.  CoRR, abs/2006.13256, 2020.
  * [11] Achal Dave, Olga Russakovsky, and Deva Ramanan.  Predictive-corrective networks for action detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [12] Victor Escorcia, Fabian Caba Heilbron, Juan Carlos Niebles, and Bernard Ghanem.  Daps: Deep action proposals for action understanding.  In Proceedings of the European Conference on Computer Vision (ECCV), 2016.
  * [13] Adrien Gaidon, Zaid Harchaoui, and Cordelia Schmid.  Temporal localization of actions with actoms.  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2013.
  * [14] Jiyang Gao, Zhenheng Yang, Kan Chen, Chen Sun, and Ram Nevatia.  Turn tap: Temporal unit regression network for temporal action proposals.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
  * [15] Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zisserman.  Video action transformer network.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [16] Ross Girshick.  Fast r-cnn.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015.
  * [17] Georgia Gkioxari and Jitendra Malik.  Finding action tubes.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [18] Fabian Caba Heilbron, Wayner Barrios, Victor Escorcia, and Bernard Ghanem.  Scc: Semantic context cascade for efficient action detection.  In Proceedings of the IEEE International Conference on Computer Vision (CVPR), 2017.
  * [19] Noureldien Hussein, Efstratios Gavves, and Arnold WM Smeulders.  Videograph: Recognizing minutes-long human activities in videos.  Proceedings of the ICCV Workshop on Scene Graph Representation and Learning (ICCV-W), 2019.
  * [20] Ashesh Jain, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena.  Structural-rnn: Deep learning on spatio-temporal graphs.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [21] Mihir Jain, Amir Ghodrati, and Cees GM Snoek.  Actionbytes: Learning from trimmed videos to localize actions.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [22] Mihir Jain, Jan Van Gemert, Hervé Jégou, Patrick Bouthemy, and Cees GM Snoek.  Action localization with tubelets from motion.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
  * [23] Y.-G. Jiang, J. Liu, A. Roshan Zamir, G. Toderici, I. Laptev, M. Shah, and R. Sukthankar.  THUMOS challenge: Action recognition with a large number of classes.  http://crcv.ucf.edu/THUMOS14/, 2014.
  * [24] Vicky Kalogeiton, Philippe Weinzaepfel, Vittorio Ferrari, and Cordelia Schmid.  Action tubelet detector for spatio-temporal action localization.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
  * [25] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei.  Large-scale video classification with convolutional neural networks.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), 2014.
  * [26] Thomas N. Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  In International Conference on Learning Representations (ICLR), 2017\.
  * [27] Ivan Laptev.  On space-time interest points.  International Journal of Computer Vision(IJCV), 2005.
  * [28] Colin Lea, Michael D Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager.  Temporal convolutional networks for action segmentation and detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [29] Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen.  Bmn: Boundary-matching network for temporal action proposal generation.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019.
  * [30] Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang.  Bsn: Boundary sensitive network for temporal action proposal generation.  In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
  * [31] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  2017\.
  * [32] Shugao Ma, Leonid Sigal, and Stan Sclaroff.  Learning activity progression in lstms for activity detection and early detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [33] Effrosyni Mavroudi, Benjamín Béjar Haro, and René Vidal.  Representation learning on visual-symbolic graphs for video understanding.  In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
  * [34] Tushar Nagarajan, Yanghao Li, Christoph Feichtenhofer, and Kristen Grauman.  Ego-topo: Environment affordances from egocentric video.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [35] Dan Oneata, Jakob Verbeek, and Cordelia Schmid.  Action and event recognition with fisher vectors on a compact feature set.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2013.
  * [36] Boxiao Pan, Haoye Cai, De-An Huang, Kuan-Hui Lee, Adrien Gaidon, Ehsan Adeli, and Juan Carlos Niebles.  Spatio-temporal graph for video captioning with knowledge distillation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [37] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Łukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran.  Image transformer.  Proceedings of the International Conference on Machine Learning (ICML), 2018.
  * [38] AJ Piergiovanni and Michael Ryoo.  Temporal gaussian mixture layer for videos.  In Proceedings of the International Conference on Machine Learning (ICML), 2019.
  * [39] AJ Piergiovanni and Michael S Ryoo.  Learning latent super-events to detect multiple activities in videos.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [40] AJ Piergiovanni and Michael S Ryoo.  Avid dataset: Anonymized videos from diverse countries.  In Advances in Neural Information Processing Systems (NeurIPS), 2020\.
  * [41] Alexander Richard and Juergen Gall.  Temporal action detection using a statistical language model.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [42] Zheng Shou, Jonathan Chan, Alireza Zareian, Kazuyuki Miyazawa, and Shih-Fu Chang.  Cdc: Convolutional-de-convolutional networks for precise temporal action localization in untrimmed videos.  In Proceedings of the IEEE International Conference on Computer Vision (CVPR), 2017.
  * [43] Zheng Shou, Hang Gao, Lei Zhang, Kazuyuki Miyazawa, and Shih-Fu Chang.  Autoloc: Weakly-supervised temporal action localization in untrimmed videos.  In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
  * [44] Zheng Shou, Dongang Wang, and Shih-Fu Chang.  Temporal action localization in untrimmed videos via multi-stage cnns.  In Proceedings of the IEEE International Conference on Computer Vision (CVPR), 2016.
  * [45] Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta.  Hollywood in homes: Crowdsourcing data collection for activity understanding.  In Proceedings of the European Conference on Computer Vision (ECCV), 2016.
  * [46] Karen Simonyan and Andrew Zisserman.  Two-stream convolutional networks for action recognition in videos.  In Advances in Neural Information Processing Systems (NIPS), 2014\.
  * [47] Bharat Singh, Tim K Marks, Michael Jones, Oncel Tuzel, and Ming Shao.  A multi-stream bi-directional recurrent neural network for fine-grained action detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [48] Gurkirt Singh, Suman Saha, Michael Sapienza, Philip HS Torr, and Fabio Cuzzolin.  Online real-time multiple spatiotemporal action localisation and prediction.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
  * [49] Russell Stewart, Mykhaylo Andriluka, and Andrew Y Ng.  End-to-end people detection in crowded scenes.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [50] Kevin Tang, Bangpeng Yao, Li Fei-Fei, and Daphne Koller.  Combining the right features for complex event recognition.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2013.
  * [51] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri.  Learning spatiotemporal features with 3d convolutional networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [52] Yao-Hung Hubert Tsai, Santosh Divvala, Louis-Philippe Morency, Ruslan Salakhutdinov, and Ali Farhadi.  Video relationship reasoning using gated spatio-temporal energy graph.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [53] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in Neural Information Processing Systems (NIPS), 2017\.
  * [54] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio.  Graph attention networks.  In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
  * [55] Heng Wang and Cordelia Schmid.  Action recognition with improved trajectories.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013.
  * [56] Limin Wang, Yu Qiao, and Xiaoou Tang.  Action recognition and detection by combining motion and appearance features.  THUMOS14 Action Recognition Challenge, 2014.
  * [57] Limin Wang, Yu Qiao, and Xiaoou Tang.  Action recognition with trajectory-pooled deep-convolutional descriptors.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  * [58] Limin Wang, Yuanjun Xiong, Dahua Lin, and Luc Van Gool.  Untrimmednets for weakly supervised action recognition and detection.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [59] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool.  Temporal segment networks: Towards good practices for deep action recognition.  In Proceedings of the European Conference on Computer Vision (ECCV), 2016.
  * [60] Xiaolong Wang and Abhinav Gupta.  Videos as space-time region graphs.  In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
  * [61] Huijuan Xu, Abir Das, and Kate Saenko.  R-c3d: Region convolutional 3d network for temporal activity detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [62] Mengmeng Xu, Chen Zhao, David S Rojas, Ali Thabet, and Bernard Ghanem.  G-tad: Sub-graph localization for temporal action detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
  * [63] Serena Yeung, Olga Russakovsky, Greg Mori, and Li Fei-Fei.  End-to-end learning of action detection from frame glimpses in videos.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2678–2687, 2016.
  * [64] Jun Yuan, Bingbing Ni, Xiaokang Yang, and Ashraf A Kassim.  Temporal action localization with pyramid of score distribution features.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [65] Zehuan Yuan, Jonathan C Stroud, Tong Lu, and Jia Deng.  Temporal action localization by structured maximal sums.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [66] Runhao Zeng, Wenbing Huang, Mingkui Tan, Yu Rong, Peilin Zhao, Junzhou Huang, and Chuang Gan.  Graph convolutional networks for temporal action localization.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019.
  * [67] Bowen Zhang, Limin Wang, Zhe Wang, Yu Qiao, and Hanli Wang.  Real-time action recognition with enhanced motion vector cnns.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [68] Peisen Zhao, Lingxi Xie, Chen Ju, Ya Zhang, Yanfeng Wang, and Qi Tian.  Bottom-up temporal action localization with mutual regularization.  In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
  * [69] Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin.  Temporal action detection with structured segment networks.  In Proceedings of the IEEE International Conference on Computer Vision (CVPR), 2017.

##  A Appendix

We report additional quantitative results and qualitative analysis and provide
implementation details of our model. Specifically, this document contains the
following.

  * •

Code provided in the folder agt_code.zip

  * •

Additional quantitative evaluation

    * –

Section A.1.1: Supplemental tables for Table 4 and Table 5 from the main paper
to report mAP values at specific IoU thresholds

    * –

Section A.1.2: Ablation study of loss function (Eq. 9 in the main paper).

    * –

Section A.1.3: Impact of different number of layers in transformer encoder and
decoder.

    * –

Section A.1.4: Impact of different number of heads in attention modules of the
transformer.

    * –

Section A.1.5: Impact of different number of nodes in the action query graph.

  * •

Additional qualitative analysis

    * –

Section A.2.1: Visualization of predictions

    * –

Section A.2.2: Visualization of graphs learned by the model

    * –

Section A.2.3: Analysis of AGT predictions based on the duration of action
instances

  * •

Technical details

    * –

Section A.3.1: Details of the architecture of AGT

    * –

Section A.3.2: Details of initialization, data augmentation, and
hyperparameters.

###  A.1 Additional Quantitative Evaluation

In this section, we report the quantitative evaluation of our proposed AGT
model to supplement the quantitative evaluation in the main paper.

####  A.1.1 Supplemental Tables

In the main paper, we only reported the mAP averaged over different IoU
thresholds for THUMOS14 and EPIC-Kitchens100 dataset (Table 4 and Table 5 in
main paper). For completeness, we report mAP at specific IoU thresholds in
Table T1 and Table T2.

Table T1: Supplemental Tables: Impact of graph based reasoning. We report
performance of ablated versions of our AGT model. We remove graph reasoning in
the encoder ($\mathbf{E}$) and/or decoder ($\mathbf{D}$) of the transformer. ✓
and ✗ indicates whether a component (encoder or decoder) contains graph
message passing module or not respectively. EPIC (A), EPIC (V), EPIC (N)
indicates task ‘Action’, ‘Verb’, ‘Noun’ classification on EPIC-Kitchens100. We
report the mean average precision at different intersection over union
thresholds (mAP@tIoU) for tIoU$\in\\{0.1,0.2,0.3,0.4,0.5\\}$. $\uparrow$
indicates higher is better.

Dataset | Model | mAP@tIoU $\uparrow$
---|---|---
0.1 | 0.2 | 0.3 | 0.4 | 0.5
|  $\mathbf{E}$: ✗/ $\mathbf{D}$: ✗ | 64.6 | 60.8 | 59.1 | 51.2 | 40.3
THUMOS14 |  $\mathbf{E}$: ✗/ $\mathbf{D}$: ✓ | 65.1 | 62.4 | 60.3 | 52.4 | 41.3
|  $\mathbf{E}$: ✓/ $\mathbf{D}$: ✗ | 67.1 | 64.4 | 62.5 | 53.6 | 44.9
|  $\mathbf{E}$: ✓/ $\mathbf{D}$: ✓ | 72.1 | 69.8 | 65.0 | 58.1 | 50.2
|  $\mathbf{E}$: ✗/ $\mathbf{D}$: ✗ | 9.4 | 6.9 | 5.2 | 4.5 | 2.5
EPIC (V) |  $\mathbf{E}$: ✗/ $\mathbf{D}$: ✓ | 9.9 | 7.5 | 5.5 | 4.9 | 2.7
|  $\mathbf{E}$: ✓/ $\mathbf{D}$: ✗ | 11.4 | 9.0 | 6.9 | 6.3 | 3.4
|  $\mathbf{E}$: ✓/ $\mathbf{D}$: ✓ | 12.0 | 10.3 | 8.2 | 7.1 | 6.1
|  $\mathbf{E}$: ✗/ $\mathbf{D}$: ✗ | 8.9 | 5.4 | 4.9 | 3.6 | 1.7
EPIC (N) |  $\mathbf{E}$: ✗/ $\mathbf{D}$: ✓ | 9.2 | 6.0 | 5.1 | 4.2 | 2.0
|  $\mathbf{E}$: ✓/ $\mathbf{D}$: ✗ | 10.1 | 8.0 | 6.8 | 5.2 | 2.3
|  $\mathbf{E}$: ✓/ $\mathbf{D}$: ✓ | 11.6 | 9.3 | 7.1 | 6.6 | 3.9
|  $\mathbf{E}$: ✗/ $\mathbf{D}$: ✗ | 4.8 | 4.1 | 2.9 | 2.1 | 1.5
EPIC (A) |  $\mathbf{E}$: ✗/ $\mathbf{D}$: ✓ | 5.1 | 4.3 | 3.2 | 2.3 | 1.8
|  $\mathbf{E}$: ✓/ $\mathbf{D}$: ✗ | 7.3 | 6.1 | 5.0 | 3.9 | 3.7
|  $\mathbf{E}$: ✓/ $\mathbf{D}$: ✓ | 7.8 | 6.9 | 5.5 | 4.2 | 3.9

Table T2:  Supplemental Tables: Impact of temporal resolution. Performance of
our AGT for different temporal resolutions of input video. Here, SR indicates
sampling rate of frames for feature extraction, _i.e_., SR=1/$k$ means frames
sampled at 1/$k$ -th factor of the original frame rate. EPIC (A), EPIC (V),
EPIC (N) indicate tasks ‘Action’, ‘Verb’, ‘Noun’ classification on dataset
EPIC-Kitchens100. We report the mean average precision at different
intersection over union thresholds (mAP@tIoU) for
tIoU$\in\\{0.1,0.2,0.3,0.4,0.5\\}$. $\uparrow$ indicates higher is better.

Dataset | Model | mAP@tIoU $\uparrow$
---|---|---
0.1 | 0.2 | 0.3 | 0.4 | 0.5
THUMOS14 | SR=1/8 | 70.4 | 65.8 | 62.3 | 54.1 | 48.6
| SR=1/4 | 72.1 | 69.8 | 65.0 | 58.1 | 50.2
EPIC (V) | SR=1/8 | 10.9 | 8.5 | 6.3 | 5.5 | 4.9
| SR=1/4 | 12.0 | 10.3 | 8.2 | 7.1 | 6.1
EPIC (N) | SR=1/8 | 10.2 | 8.1 | 5.9 | 5.2 | 2.8
| SR=1/4 | 11.6 | 9.3 | 7.1 | 6.6 | 3.9
EPIC (A) | SR=1/8 | 7.0 | 5.0 | 4.1 | 2.5 | 1.9
| SR=1/4 | 7.8 | 6.9 | 5.5 | 4.2 | 3.9

####  A.1.2 Ablation Study: Loss function

Note that for any version of the loss function, the model requires cross
entropy loss to be able to classify the action label pertaining to an
instance. The model also requires some form of regression loss to produce
predictions pertaining to the start and end timestamps of an action instance.
Recall, our overall loss (see Eq. (9) in the main paper) is a combination of
cross-entropy loss and regression loss, _i.e_., segment loss
$\mathcal{L}_{s}$. The segment loss contains two components: $L_{1}$ loss and
IoU loss $\mathcal{L}_{iou}$. Table T3 shows the results of the performance of
our model when trained with ablated versions of the segment loss. The results
indicate that the models trained with only $L_{1}$ loss perform better than
the ones trained with only IoU loss $\mathcal{L}_{iou}$. Additionally, models
trained with both losses are better than the ones trained with only one of the
losses. Nonetheless, all versions of our AGT model perform better than state-
of-the-art methods. We only provide the mAP values averaged over the various
intersection-over-union thresholds (tIoU) for THUMOS14 and EPIC-Kitchens100.

Table T3: Ablation Study: Loss function. We report performance of our AGT
model when trained with ablated versions of the loss function. We report mAP
for evaluation performance (higher is better). We train the model with a
combination of cross-entropy loss and segment loss containing $L_{1}$ loss
and/or IoU loss $\mathcal{L}_{iou}$. ✓ and ✗ indicate whether the specific
component of the segment loss is used or not respectively. EPIC(A), EPIC (V),
EPIC (N) indicate tasks ‘Action’, ‘Verb’, ‘Noun’ classification on EPIC-
Kitchens100.

Dataset |  $L_{1}$: ✓ |  $L_{1}$: ✗ |  $L_{1}$: ✓
---|---|---|---
|  $\mathcal{L}_{iou}$: ✗ |  $\mathcal{L}_{iou}$: ✓ |  $\mathcal{L}_{iou}$: ✓
THUMOS14 | 61.3 | 59.6 | 63.0
Charades | 26.0 | 25.3 | 28.6
EPIC (A) |  04.8 |  03.7 |  05.9
EPIC (V) |  07.1 |  06.4 |  08.7
EPIC (N) |  06.3 |  05.0 |  07.7

####  A.1.3 Impact of number of layers

Table T4 shows the results of the performance of our model with different
number of layers in encoder and decoder component of the transformer. While
increase in number of layers increases the training time, we did not observe
much difference in the performance of the model with increased depth of the
transformer components. We only provide the mAP values averaged over the
various intersection-over-union thresholds (tIoU) for THUMOS14 and EPIC-
Kitchens100.

Table T4: Impact of number of layers. We report performance of our AGT model
with different number of layers in encoder and decoder. We report mAP for
evaluation performance (higher is better). EPIC (A), EPIC (V), EPIC (N)
indicates task ‘Action’, ‘Verb’, ‘Noun’ classification on EPIC-Kitchens100. #E
indicates number of layers in encoder and #D indicates number of layers in
decoder.

Dataset | #E | #D | mAP
---|---|---|---
| 4 | 2 | 62.5
THUMOS14 | 4 | 4 | 63.0
| 2 | 4 | 62.7
| 4 | 2 | 28.0
Charades | 4 | 4 | 28.6
| 2 | 4 | 28.2
| 4 | 2 | 5.5
EPIC (A) | 4 | 4 | 5.9
| 2 | 4 | 5.6
| 4 | 2 | 8.5
EPIC (V) | 4 | 4 | 8.7
| 2 | 4 | 8.6
| 4 | 2 | 7.2
EPIC (N) | 4 | 4 | 7.7
| 2 | 4 | 7.3

####  A.1.4 Impact of number of heads

Table T5 shows the results of the performance of our AGT model with different
number of heads in the attention modules of the transformer. The results
suggest a slight improvement with more number of heads in the transformer
network. We only provide the mAP values averaged over the various
intersection-over-union thresholds (tIoU) for THUMOS14 and EPIC-Kitchens100.

Table T5: Impact of number of heads. We report performance of our AGT model
with different number of heads in the attention modules of the transformer
network. We report mAP for evaluation performance (higher is better). EPIC
(A), EPIC (V), EPIC (N) indicates task ‘Action’, ‘Verb’, ‘Noun’ classification
on EPIC-Kitchens100. #heads indicates number of heads in attention modules of
the transformer.

Dataset | #heads | mAP
---|---|---
THUMOS14 | 8 | 63.0
| 4 | 61.4
Charades | 8 | 28.6
| 4 | 26.4
EPIC (A) | 8 | 5.9
| 4 | 5.2
EPIC (V) | 8 | 8.7
| 4 | 8.4
EPIC (N) | 8 | 7.7
| 4 | 7.1

####  A.1.5 Impact of action query graph size

Table T6 shows the results of the performance of our AGT model with different
number of node encodings in the action query graph. Intuitively, a very large
size of action query graph implies the model will require more time to learn
the non-maximal suppression of the irrelevant predictions. On the other hand,
a very small size of action query graph might limit the ability of model to
learn complex structure in the action instances. Note that, any value used for
our experiments is higher than the maximum number of action instances per
video in the dataset. The results suggest minor improvement with more number
of nodes in the action query graph, however, the models with more number of
nodes require longer training times. Our experiments also suggest that when
the size of the action query graph is reduced, the localization performance of
our model degrades. We only provide the mAP values averaged over the various
intersection-over-union thresholds (tIoU) for THUMOS14 and EPIC-Kitchens100.

Table T6: Impact of action query graph size. We report performance of our AGT
model with different number of nodes in the action query graph. We report mAP
for evaluation performance (higher is better). EPIC (A), EPIC (V), EPIC (N)
indicates task ‘Action’, ‘Verb’, ‘Noun’ classification on EPIC-Kitchens100.
#queries indicates number of nodes in the action query graph.

Dataset | #queries | mAP
---|---|---
| 150 | 59.1
THUMOS14 | 300 | 63.0
| 900 | 63.2
| 30 | 24.0
Charades | 50 | 28.6
| 100 | 28.6
| 900 | 4.3
EPIC (A) | 1500 | 5.9
| 2000 | 7.0
| 900 | 7.0
EPIC (V) | 1500 | 8.7
| 2000 | 8.8
| 900 | 6.1
EPIC (N) | 1500 | 7.7
| 2000 | 7.9

###  A.2 Additional Qualitative Analysis

In this section, we visualize the results of our proposed model AGT to
supplement the qualitative analysis in the main paper.

####  A.2.1 Visualization: Predictions

---
Figure F1: Visualization: Predictions.  Visualization of predictions and
groundtruth action instances

We provide additional visualizations of the predictions of our AGT on several
diverse samples in Figure F1. The visualizations indicate that our model is
able to predict the correct number of action instances as well as most of the
correct action categories with minimal errors in start and end timestamps for
videos containing overlapping instances with varying temporal extents.

####  A.2.2 Visualization: Learned Graphs

|
---|---

Figure F2: Visualization: Learned Graphs.  Visualizations of embeddings
corresponding to the last layer of the decoder and ground truth instances. The
thickness of edges show the strength of interaction between the nodes. For
ease of visibility, the nodes have been numbered based on the order of their
predictions sorted with respect to the start time (_i.e_., node 0 represents
the instance that starts first). These visualizations demonstrate that the
model indeed learns non-linear dependencies between the action instances in a
video. The legend below each figure shows the action labels corresponding to
the color coded elements. For details on the visualization process, please
refer to Section A.2.2

.

We visualize the learned action query graph in Figure F2. by observing the
graph embeddings obtained from the last layer of decoder. For better
visibility, we do not plot the nodes (or their edges) that are classified as
no action (_i.e_. class label $\varnothing$) by the prediction head. Note that
the edge matrix is also learnable in our model. For the purpose of this
visualization, we obtained the edge weights from the attention coefficients in
the self-attention based graph message passing module . We show samples with
reoccurring and/or overlapping action instances. The visualizations
demonstrate that the model indeed learns non-linear dependencies among the
action instances that appear in the video.

####  A.2.3 Analysis: Effect of Action Instance Durations

We conduct further analysis to study the performance of our model in terms of
the durations of the action instances. Figure F3 shows the trend of
segmentation error, _i.e_., $L_{1}$ norm computed between the ground truth and
predicted timestamps of actions instances plotted against the duration of the
ground truth instances (normalized with respect to the video duration). The
error is computed over normalized values of the timestamps. This analysis
indicates that action instances with larger durations (with respect to the
whole video duration) have lower segmentation errors in their predictions as
compared to the instances with smaller durations.

Figure F3: Analysis (THUMOS14). Analysis of segmentation error (L1 loss) with
respect to the duration of corresponding ground truth instances. All the
values are normalized with respect to the overall video duration. We observe
that the action instances of longer durations have lower segmentation errors
in their predictions. Figure F4: Detailed Architecture Architecture of
Activity Graph Transformer. Please see Section A.3.1 for details. ‘Q’,‘K’,‘V’
are query, key and value to the self-attention layer as described in [53].

###  A.3 Technical Details

In this section, we provide additional implementation details to supplement
the model section in the main paper.

####  A.3.1 Additional details

Detailed Architecture. Figure F4 presents the architecture of our AGT in
detail. Activity Graph Transformer (AGT) consists of three components: (1)
backbone network to obtain features corresponding to the input video; (2)
transformer network consisting of an encoder network and a decoder network
that operates over graphs; and (3) prediction heads for the final prediction
of action instances of the form (label, start time, end time). The encoder
network receives the compact video-level representation from the backbone
network and encodes it to a latent graph representation, referred to as
context graph. The decoder network receives graph-structured abstract query
encodings (referred to as action query graph) as input along with the context
graph. The decoder uses the context graph to transform the action query graph
to a graph-structured set of embeddings. Each node embedding of this decoder
output is fed into a prediction head to obtain predictions of action
instances. The whole AGT network is trained end-to-end using a combination of
classification and regression losses for the action labels and timestamps
respectively.

Positional Encoding. Positional encoding layer consists of a layer that
retrieves encodings based on an integer index provided to it. In our case,
given a video feature
$\mathbf{v}=[\mathbf{v}^{(1)},\mathbf{v}^{(2)}\ldots\mathbf{v}^{(N_{v})}]$,
the positional encoding layer receives input $i$ and provides an embedding
$\mathbf{p}_{v}^{(i)}$ corresponding to the $i$-th element of the video
feature $\mathbf{v}^{(i)}$ where $i\in{1,2,\ldots,N_{v}}$. In our
implementation, the embedding size is same as that of the video feature so as
to allow addition of the positional encodings and input video features. Since
the weights of the layer are learnable during training, the positional
encoding layer is learnable. We use torch.nn.Embedding in Pytorch to implement
it. This layer initialization requires maximum possible value of $N_{v}$ in
the features corresponding to the video.

Action Query Graph. Similar to positional encoding layer, the $N_{o}$
encodings in the action query graph $\mathbf{q}$ is obtained using an
embedding layer. Specifically, the layer receives $i$ as input to provide
$i$-th node $\mathbf{q}^{(i)}$ of the query graph where
$i\in{1,2,\ldots,N_{o}}$. In our implementation, we use torch.nn.Embedding in
Pytorch to implement this. The weights of this layer are learnable during
training.

Losses. For completeness, we describe the IoU loss ($\mathcal{L}_{iou}$) which
is used as a component of segment loss $\mathcal{L}_{s}$ to train our model.
The segment loss is described as:

$\mathcal{L}_{s}=\lambda_{iou}\mathcal{L}_{iou}(s^{(i)},\tilde{s}^{(\phi(i))})+\lambda_{L1}||s^{(i)}-\tilde{s}^{(\phi(i))}||_{1},$
(10)

where $\lambda_{iou},\lambda_{L1}\in\mathbbm{R}$ are hyperparameters.

$\mathcal{L}_{iou}(s^{(i)},\tilde{s}^{(\phi(i))})=1-\frac{|s^{(i)}\cap\tilde{s}^{(\phi(i))}|}{|s^{(i)}\cup\tilde{s}^{(\phi(i))}|}$
(11)

where $|.|$ is the duration of the instance, _i.e_., difference between end
and start timestamp.

####  A.3.2 Training Details

Feature Extraction & Data augmentation. To obtain I3D features corresponding
to an input video $V$ containing $T$ frames sampled at a specific sample rate,
we first divide the video into short overlapping segments of 8 frames with an
overlap of 4 frames resulting in $T^{\prime}$ chunks. We use an I3D model
pretrained on the Kinetics [6] dataset to extract features of dimension $C$
($=2048$). In our implementation, we obtain two-stream features (both RGB and
flow streams). We obtain features for these $T^{\prime}$ chunks to obtain a
tensor of size $T^{\prime}\times 2048$. Here, the length of the video $T$
depends on the duration of the video, and, hence the size of the temporal
channel (_i.e_. $T^{\prime}$) of the feature tensor varies based on the input.

To prevent severe overfitting, we perform data augmentation to train our model
on the features directly obtained from I3D model (described above). We use a
hyperparameter $N_{v}^{max}$ as the maximum size of temporal channel used for
training. This helps in stabilizing the training as the video datasets contain
high variance in their duration. If the size of temporal channel of the video
tensor $T^{\prime}$ is less than $N_{v}^{max}$, we repeat each element in the
temporal channel $\gamma$ times ($\gamma=4$) in our implementation to obtain a
modified tensor of size $\gamma T^{\prime}\times 2048$ and then randomly
sample $T^{\prime}$ elements from the modified tensor. If the size of temporal
channel of the video tensor $T^{\prime}$ is more than $N_{v}^{max}$, we just
randomly sample $T^{\prime}$ elements from the modified tensor. Note that,
positional encoding is applied on this feature of size
$N_{v}=\min(T^{\prime},N_{v}^{max})$.

We find such data augmentation during training to be crucial to prevent
overfitting and obtain good performance of our model, especially for smaller
datasets such as THUMOS14. During testing, if the size of temporal channel of
the video tensor $T^{\prime}$ is less than $N_{v}^{max}$, we don’t perform any
augmentation. If the size of temporal channel of the video tensor $T^{\prime}$
is more than $N_{v}^{max}$, we uniformly sample $T^{\prime}$ elements from the
feature in order to match the maximum index of the positional encoding layer.

Furthermore, to perform training in minibatches, we apply $0$-padding to
ensure all elements have the same size as the largest element of the batch.
For training efficiency and minimizing the amount of $0$-padding, we sort all
the dataset based on the duration of the video. We observe that this type of
batch formation leads to improvement in training speed without affecting the
model performance.

Hyperparameters. We provide the hyperparameters used to train our model below.

We train all our models using AdamW optimizer [31] with a learning rate of
1e-5 and a weight decay of 1e-5 for 3000k steps. We reduce the learning rate
by factor of 10 after 2000k steps. The hyperparameters in the loss functions
$\lambda_{L1}$ and $\lambda_{iou}$ are set to 5 and 3 respectively for all our
experiments. All the learnable weights are initialized using Xavier
initialization.

For our experiments, we sample frames at 1/4 of the original frame rate and
obtain the I3D features as decribed earlier. We do not finetune the I3D model.

We mention the dataset specific hyperparameters below.

THUMOS14.  We do not use dropout for this dataset. We use maximum number of
nodes in the context graph $N_{v}^{max}$ equal to 256. The size of the action
query graph is 300 for our experiments (except when conducting ablation on the
size of action query graph). We use base model dimension in the transformer as
512 and set the number of encoder and decoder layers as 4 (except when
conducting ablation on the number of layers).

Charades.  We use dropout with default probability $0.1$. We use maximum
number of nodes in the context graph $N_{v}^{max}$ equal to 64. The size of
the action query graph is 100 for our experiments (except when conducting
ablation on the size of action query graph). We use base model dimension in
the transformer as 512 and set the number of encoder and decoder layers as 4
(except when conducting ablation on the number of layers).

Epic-Kitchens100. We do not use dropout for this dataset. We use maximum
number of nodes in the context graph $N_{v}^{max}$ equal to 1024. The size of
the action query graph is 1200 for our experiments (except when conducting
ablation on the size of action query graph). We use base model dimension in
the transformer as 512 and set the number of encoder and decoder layers as 4
(except when conducting ablation on the number of layers).