# Deep Learning-based Action Detection in Untrimmed Videos: A Survey

Elahe Vahdani and Yingli Tian∗ E. Vahdani is with the Department of Computer
Science, The Graduate Center, The City University of New York, NY, 10016.
E-mail<EMAIL_ADDRESS>
Y. Tian is with the Department of Electrical Engineering, The City College,
and the Department of Computer Science, the Graduate Center, the City
University of New York, NY, 10031<EMAIL_ADDRESS>∗Corresponding
authorThis material is based upon work supported by the National Science
Foundation under award number IIS-2041307.

###### Abstract

Understanding human behavior and activity facilitates advancement of numerous
real-world applications, and is critical for video analysis. Despite the
progress of action recognition algorithms in trimmed videos, the majority of
real-world videos are lengthy and untrimmed with sparse segments of interest.
The task of temporal activity detection in untrimmed videos aims to localize
the temporal boundary of actions and classify the action categories. Temporal
activity detection task has been investigated in full and limited supervision
settings depending on the availability of action annotations. This paper
provides an extensive overview of deep learning-based algorithms to tackle
temporal action detection in untrimmed videos with different supervision
levels including fully-supervised, weakly-supervised, unsupervised, self-
supervised, and semi-supervised. In addition, this paper also reviews advances
in spatio-temporal action detection where actions are localized in both
temporal and spatial dimensions. Moreover, the commonly used action detection
benchmark datasets and evaluation metrics are described, and the performance
of the state-of-the-art methods are compared. Finally, real-world applications
of temporal action detection in untrimmed videos and a set of future
directions are discussed.

###### Index Terms:

Action Understanding, Temporal Action Detection, Untrimmed Videos, Deep
Learning, Full and Limited Supervision.

##  1 Introduction

This paper provides a comprehensive overview of temporal action detection.
This task aims to detect the start and end of action instances in long
untrimmed videos and predict the action categories. Temporal action detection
is crucial for many video analysis applications such as sports analysis,
autonomous driving, anomaly detection in surveillance cameras, understanding
instructional videos, etc. Learning with limited supervision is a scheme where
annotations of actions are unavailable or only partially available during the
training phase. Because annotation of long untrimmed videos is very time-
consuming, designing action detection methods with limited supervision has
been very popular. This survey reviews temporal action detection methods with
full and limited supervision signals.

Figure 1: Temporal action detection aims to localize action instances in time
and recognize their categories. The first row demonstrates an example of
action “long jump” detected in an untrimmed video from THUMOS14 dataset [1].
The second row is an example of an untrimmed video including several action
instances of interest with various lengths.

###  1.1 Motivation

Social networks and digital cameras have led to substantial video and media
content produced by individuals each day. Hence, video understanding and
analysis continues to be one of the essential research subjects in computer
vision. While deep learning has accomplished remarkable performance in many
computer vision tasks, video understanding is still far from ideal. Action
understanding, notably, as a vital element of video analysis, facilitates the
advancement of numerous real-world applications. For instance, collaborative
robots need to recognize how the human partner completes the job to cope with
the variations in the task [2]. Sport analysis systems must comprehend game
actions to report commentaries of live activities [3]. Autonomous driving cars
demand an understanding of operations performed by the surrounding cars and
pedestrians [4].

In this paper, we define trimmed videos as pre-segmented video clips that each
contains only one action instance. In other words, the context of the action,
i.e., moments before or after the action are not included in the video.
Therefore, action detection in trimmed videos only need to classify the action
categories without the need to detect starting and ending timestamps.
Recognizing actions in trimmed videos has many applications in video
surveillance, robotics, medical diagnosis [5], and has achieved excellent
performance in recent years [6, 7, 8]. However, the majority of videos in the
wild, i.e., recorded in unconstrained environments, are naturally untrimmed.
Untrimmed videos are lengthy unsegmented videos that may include several
action instances, the moments before or after each action, and the transition
from one action to another. The action instances in one video can belong to
several action classes and have different duration.

Temporal activity detection in untrimmed videos aims to localize the action
instances in time and recognize their categories. This task is considerably
more complicated than action recognition which merely seeks to classify the
categories of trimmed video clips. Fig. 1 shows an example of temporal
activity detection in an untrimmed video recorded in a stadium. The first row
demonstrates the detection of action ”long jump” in temporal domain where the
start and end time of the action are localized. The goal is to only detect the
actions of interest, i.e., actions that belong to a predefined set of action
classes. The temporal intervals of other activities that do not belong to this
set of actions are called temporal background. For example, the segments right
before or right after action ”long jump” may belong to other diverse
activities such as crowd cheering in the stadium. In some cases, the frames
right before or right after an action are visually very similar to the start
or end of the action which makes the localization of action intervals very
challenging. Another challenge (as shown in the second row of Fig. 1) is that
action instances may occur at any time of the video and have various duration,
lasting from less than a second to several minutes [9].

Temporal action detection mainly targets activities of high-level semantics
and videos with a sparse set of actions (e.g., actions only cover $30\%$ of
the frames in [10]). However, in some cases, the goal is to predict action
labels at every frame of the video. In such cases, the task is referred to as
temporal action segmentation which targets the fine-grained actions and videos
with dense occurrence of actions ($93\%$ of the frames in [11]). One can
convert between a given segmentation and a set of detected instances in the
temporal domain by simply adding or removing temporal background segments
[12]. Temporal action detection similar to object detection belongs to the
family of detection problems. Both of these problems aim to localize the
instances of interest, i.e., action intervals in temporal domain versus object
bounding boxes in spatial domain, Fig. 2 (a and c). When targeting fine-
grained actions, temporal action detection (segmentation) is similar to
semantic segmentation as both aim to classify every single instance, i.e.,
frames in temporal domain versus pixels in spatial domain, Fig. 2 (b and d).
As a result, many techniques for temporal action detection and segmentation
are inspired by the advancements in object detection and semantic segmentation
[13, 14, 15].

Action detection has drawn much attention in recent years and has broad
applications in video analysis tasks. As surveillance cameras are increasingly
deployed in many places, the demand for anomaly detection has also surged.
Anomalous events such as robbery or accidents occur less frequently compared
with normal activities and it can be very time-consuming to detect such events
by humans. Therefore, automatic detection of suspicious events has a great
advantage. By growing popularity of social media many people follow online
tutorials and instructional videos to learn how to perform a task such as
“changing the car tire” properly for the first time. The instructional videos
are usually untrimmed and include several steps of the main task, e.g., “jack
up the car” and “put on the tire” for changing the tire. Automatic
segmentation of these videos to the main action steps can facilitate and
optimize the learning process. Another application is in sport video analysis
to localize the salient actions and highlights of a game and analyze the
strategies of specific teams. Furthermore, action detection has a critical
role in self-driving cars to analyze the behavior of pedestrians, cyclists,
and other surrounding vehicles to make safe autonomous decisions.

Figure 2: Task Relations: (a) Temporal detection of action “Long Jump” on
THUMOS14 [1]. (b) Temporal detection (segmentation) of fine-grained actions
shown by different colors in a “making pancakes” video on Breakfast [11]. (c)
and (d) Results from [16] on PASCAL [17].

###  1.2 Taxonomy

To the best of our knowledge, this is the first comprehensive survey
describing deep learning based algorithms for activity detection in untrimmed
videos with different supervision levels. We describe the fully-supervised
methods in Section 2.3 and methods with limited supervision (weakly-
supervised, unsupervised, self-supervised, and semi-supervised) in Section
2.4. Section 3 summarizes action detection benchmark datasets, evaluation
metrics, and performance comparison between the-state-of-the-art methods.
Finally, Section 4 discusses the most common real-world applications of action
detection and possible future directions. We provide a brief introduction of
the tasks here.

Temporal action detection aims to find the precise temporal boundary and label
of action instances in untrimmed videos. Depending on annotation availability
in train set, temporal action detection can be studied in the following
settings (also listed in Table I).

  * •

Fully supervised action detection: Temporal boundaries and labels of action
instances are available for training.

  * •

Weakly-supervised action detection: Only the video-level labels of action
instances are available. The order of action labels can be provided or not.

  * •

Unsupervised action detection: There are no annotations for the action
instances.

  * •

Semi-supervised action detection: The data is split to a small set $S_{1}$ and
a large set $S_{2}$. The videos in $S_{1}$ are fully annotated (as in fully-
supervised) while the videos in $S_{2}$ are either not annotated
(unsupervised) or only annotated with video-level labels (as in weakly-
supervised).

  * •

Self-supervised action detection: A pretext task is defined to extract
information from the data in an unsupervised setting by leveraging its
structure. Then, this information is used to improve the performance for
temporal action detection (downstream task) which can be supervised,
unsupervised, or semi-supervised.

  * •

Action detection with limited supervision: Limited supervision is the opposite
of full supervision where the annotations are unavailable or partially
available. In this paper, we define limited supervision to include weakly-
supervised, unsupervised, self-supervised, and semi-supervised settings as
they are defined above.

TABLE I: Main categories of temporal action detection task with different
supervision levels in training set. “✓” indicates “available”; “✗” is for
“unavailable”, and $\ast$ is “partially available”.

Supervision Level | Action Temporal Boundaries | Action Labels
---|---|---
Fully-supervised | ✓ | ✓
Weakly-supervised | ✗ | ✓
Unsupervised | ✗ | ✗
Semi-supervised | $\ast$ | $\ast$
Self-supervised | ✓$\ast$ ✗ | ✓$\ast$ ✗

##  2 Temporal Action Detection Methods

We begin this section by introducing important technical terms in Section 2.1.
Given an input video, video feature encoding is necessary to extract
representative visual features of the video (discussed in Section 2.2). Action
detection methods with full supervision are described in Section 2.3 and
action detection methods with limited supervision are reviewed in Section 2.4.

###  2.1 Term Definition

To facilitate reading subsequent sections, we define common terms, scores, and
loss functions here.

######  Definition 1.

Temporal action detection. This task aims to find the precise temporal
boundaries and categories of action instances in untrimmed videos. Annotation
of an input video is denoted by ${\Psi}_{g}$ and includes a set of action
instances as the following:

${\Psi}_{g}=\\{{\varphi}_{n}=(t_{s,n},t_{e,n},l_{n})\\}_{n=1}^{N},$ (1)

where $N$ is the number of action instances, and ${\varphi}_{n}$ is the $n$-th
action instance. The start time, end time, and label of ${\varphi}_{n}$ are
denoted by $t_{s,n}$, $t_{e,n}$, and $l_{n}$, respectively. Label $l_{n}$
belongs to set $\\{1,\cdots,C\\}$, where $C$ is the number of action classes
of interest in the whole dataset. The annotation ${\Psi}_{g}$ can be fully,
partially, or not available for the videos of the training set.

######  Definition 2.

Temporal proposals. The temporal regions of input video that are likely to
contain an action are called temporal proposals. Each temporal proposal
$P_{n}$ is an interval identified with a starting time $t_{s,n}$, an ending
time $t_{e,n}$, and a confidence score $c_{n}$. The confidence score is the
predicted probability that the interval contains an action. Proposal $P_{n}$
can be formulated as:

$P_{n}=(t_{s,n},t_{e,n},c_{n}).$ (2)

######  Definition 3.

Temporal IoU (tIoU). This is the ratio of temporal intersection over union
between two temporal intervals. It is often measured between a predicted
proposal (interval $I_{p}$) and its closest ground-truth action instance
(interval $I_{g}$), formulated as:

$tIoU(I_{p},I_{g})=\frac{I_{p}\cap I_{g}}{I_{p}\cup I_{g}}.$ (3)

######  Definition 4.

Temporal proposal labeling. Label of proposal $I_{p}$ is determined by ground-
truth action instance $I_{g}$ that has the maximum tIoU with $I_{p}$. Let’s
denote the class label of $I_{g}$ with $c$. Then, depending on a predefined
threshold $\sigma$, the proposal is declared as positive (true positive) with
label $c$ if $tIoU\geq\sigma$. Otherwise, it is negative (or false positive).
Also, if a ground-truth action instance is matched with several proposals,
only the proposal with the highest confidence score is accepted as true
positive and the others are declared as false positives.

######  Definition 5.

Precision and recall for proposal generation. Precision is the ratio of true
positive proposals to the total number of predicted proposals. Precision must
be high to avoid producing exhaustively many irrelevant proposals. Recall is
the ratio of true positive proposals to the total number of ground-truth
action instances. Recall must be high to avoid missing ground-truth instances.

######  Definition 6.

Actionness score. Actionness score at a temporal position is the probability
of occurrence of an action instance at that time. This score is often denoted
by $a_{t}\in[0,1]$ for time $t$.

######  Definition 7.

Startness and endness scores. Startness score (endness score) at a temporal
position is the probability of start (end) of an action instance at that time.

######  Definition 8.

Action completeness score. The maximum tIoU between a candidate proposal and
ground truth action instances is called action completeness of that proposal.
It was shown by [18] that incomplete proposals that have low tIoU with ground-
truth intervals could have high classification scores. Therefore, action
completeness must be considered to evaluate and rank the predicted proposals.

######  Definition 9.

Action classification score. Generated temporal proposals are fed to action
classifiers to produce a probability distribution over all action classes.
This can be represented by vector $(p^{1},\cdots,p^{C})$ where $p^{i}$ is the
probability of action class $i$, and $C$ is the number of classes. For a fair
comparison, researchers utilize classifiers from earlier work SCNN-classifier
[19], UntrimmedNet [19], [20], etc. They uniformly sample a constant number of
frames from the video segment and feed it to ConvNets such as C3D [21], two
stream CNNs [22] or temporal segment networks [23]. In some cases, the
recognition scores of sampled frames are aggregated with the Top-k pooling or
weighted sum to yield the final prediction.

###  2.2 Video Feature Encoding

Untrimmed videos are often lengthy and can be as long as several minutes, and
thus it is difficult to directly input the entire video to a visual encoder
for feature extraction due to the limits of computational resources. For
instance, popular video feature extractors such as 3D-CNNs can only operate on
short clips spanning about 4 seconds. A common strategy for video
representation is to partition the video into equally sized temporal intervals
called snippets, and then apply a pre-trained visual encoder over each
snippet. Formally, given input video $X$ with $l$ frames, a sequence $S$ of
snippets with regular duration $\sigma$ is generated where

$S=\\{s_{n}\\}_{n=1}^{l_{s}}\hskip 7.22743pt,\hskip
7.22743ptl_{s}=\frac{l}{\sigma},$ (4)

and $s_{n}$ is the n-th snippet. Then, each snippet is fed to a pre-trained
visual encoder such as two-stream [22], C3D [21], or I3D [24] for feature
extraction. In two-stream network [22], snippet $s_{n}$ which is centered at
$t_{n}$-th frame of the video, has an RGB frame $x_{t_{n}}$, and a stacked
optical flow $o_{t_{n}}$ derived around the center frame. The RGB frame
$x_{t_{n}}$ is fed to spatial network ResNet [25], extracting feature vector
$f_{S,n}$. The optical flow $o_{t_{n}}$ is fed to temporal network BN-
Inception [26], extracting feature $f_{T,n}$. The spatial and temporal
features, $f_{S,n}$ and $f_{T,n}$, are concatenated to represent the visual
feature $f_{n}$ for snippet $s_{n}$.

Similarly, in I3D [24], a stack of RGB and optical flow frames from each
snippet $s_{n}$ are fed to I3D network, extracting spatial and temporal
feature vectors $f_{S,n}$ and $f_{T,n}$ which are concatenated to create
feature $f_{n}$. In C3D [21], the frames of each snippet $s_{n}$ are directly
fed to a 3D-CNN architecture to capture spatio-temporal information, and
extracting feature vector $f_{n}$.

###  2.3 Action Detection with Full Supervision

In fully-supervised action detection, the annotation (${\Psi}_{g}$ in Eq. (1))
of temporal boundaries and labels of action instances are provided for each
video of training set. During inference, the goal is to find the temporal
boundaries of action instances and predict their labels. A main step in action
detection is temporal proposal generation to identify the temporal intervals
of the video that are likely to include action instances. Fully-supervised
temporal proposal generation methods can be categorized to anchor-based and
anchor-free. Anchor-based methods generate action proposals by assigning dense
and multi-scale intervals with pre-defined lengths at each temporal position
of the video (Section 2.3.1). Anchor-free methods often predict action
boundary confidence or actionness scores at temporal positions of the video,
and employ a bottom-up grouping strategy to match pairs of start and end
(Section 2.3.2). There are also several methods that combine the advantages of
anchor-free and anchor-based proposal generation methods (Section 2.3.3).
After generating the proposals, rich features must be extracted from the
proposals to evaluate the quality of proposals. Section 2.3.4 reviews common
loss functions that are used during training for proposal evaluation. Section
2.3.5 discusses modeling long-range dependencies to capture the relation
between video segments in untrimmed videos to improve action localization.
Finally, Section 2.3.6 summarizes spatio-temporal action detection methods.

####  2.3.1 Anchor-based Proposal Generation and Evaluation

Anchor-based methods, also known as top-down methods, generate temporal
proposals by assigning dense and multi-scale intervals with pre-defined
lengths to uniformly distributed temporal locations in the input video.
Formally, given a video with $T$ frames, $\frac{T}{\sigma}$ temporal
positions, known as anchors, are uniformly sampled from every $\sigma$ frames.
Then, several temporal windows with different duration
$\\{d_{1},d_{2},\cdots,d_{n}\\}$ are centered around each anchor as initial
temporal proposals. The proposal lengths ($d_{i}$ s) must have a wide range to
align with action instances of various lengths that can last from less than a
second to several minutes in untrimmed videos [9]. Then visual encoders and
convolution layers are applied on the temporal proposals for feature
extraction and features are used to evaluate the quality of temporal proposals
and adjust their boundaries (Section 2.3.4).

Figure 3: Anchor-based methods assign multi-scale intervals with pre-defined
lengths at uniformly distributed temporal positions.

##### 2.3.1.1 Feature Extraction of Multi-scale Proposals

As mentioned earlier, temporal proposals have very diverse time spans to align
with action instances. However, fixed-size features must be extracted from
each proposal to be fed to fully connected layers for proposal evaluation
(action classification and regression). Here, we review different strategies
to extract fixed-size features from proposals with different lengths.

Sampling and Feature Concatenation: Shou et al. in SCNN [19] uniformly sampled
a fixed number of frames from each proposal and fed them to a visual encoder
for feature extraction. This is not computationally efficient because there
are many overlapping proposals and overlapping segments are processed multiple
times. To address this problem, Gao et al. in Turn-Tap [27] and CBR [28]
decomposed the video into non-overlapping equal-length units and extracted the
features of each unit only once. Different numbers of consecutive units are
grouped together at each anchor unit to generate multi-scale proposals. To
obtain the proposal features, the features of all units are concatenated.
Using this approach, the proposal features are computed from unit-level
features, which are calculated only once. However, concatenation of features
within each proposal or sampling frames do not lead to rich feature
extraction.

3D RoI Pooling: This approach extracts fixed size features from multi-scale
proposals using 3D RoI pooling. Specifically, an input feature volume of size
$l\times h\times w$ ( $l$ for temporal dimension, $h$ for height and $w$ for
width dimensions) is divided into $l_{s}\times h_{s}\times w_{s}$ sub-volumes
(where $l_{s},h_{s},$ and $w_{s}$ are fixed), and max pooling is performed
inside each sub-volume. Therefore, proposals of various lengths generate
output volume features of the same size, which is $d\times l_{s}\times
h_{s}\times w_{s}$, where $d$ is the channel dimension. The idea of 3D RoI
pooling for action detection is an extension of the 2D RoI pooling for object
detection in Faster R-CNN [29]. This idea was first introduced in R-C3D[30]
and used in other frameworks such as AGCN [31] and AFNet [32]. The limitation
of this approach is that the multi-scale proposals at each location share the
same receptive field, which may be too small or too large for some anchor
scales.

##### 2.3.1.2 Receptive Field Alignment with Proposal Span

To address the variation of action duration, multi-scale anchors are assigned
to each temporal location of the video. Before receptive field alignment,
multi-scale anchors at any position share the same receptive field size. This
is problematic because if the receptive field is too small or too large with
respect to the anchor size, the extracted feature may not contain sufficient
information or include too much irrelevant information. Here, we review the
strategies to align the receptive field size with proposal span.

Multi-tower Network: TAL-Net [13] proposed a multi-tower network, compose of
several temporal convNets, each one responsible for a certain anchor-size.
Then, the receptive field of each anchor segment was aligned with its temporal
span using dilated temporal convolutions. This idea was also used in TSA-Net
[33]. However, assigning pre-defined temporal intervals limits the accuracy of
generated proposals.

Temporal Feature Pyramid Network: In a temporal feature pyramid network
(TFPN), the predictions are yielded from multiple resolution feature maps.
This idea was first introduced in Single Shot Detector (SSD) [34] for object
detection, and then extended to temporal domain for action detection in SSAD
[35] and $\text{S}^{3}\text{D}$ [36]. They proposed an end-to-end network
where the lower-level feature maps with higher resolution and smaller
receptive field are responsible to detect short action instances while the top
layers with lower resolution and larger receptive field, detect long action
instances. For each feature map cell, several anchor segments with multiple
scales are considered around the center that are fed to convolutional layers
for evaluation. The limitation of this approach is that lower layers in the
pyramid are unaware of high-level semantic information, and top layers lack
enough details, so they all fail to localize the actions accurately.

U-shaped Temporal Feature Pyramid Network: In order to mitigate the problem
with regular TFPNs, a U-shaped TFPN architecture was designed to connect high-
level and low-level features. This idea was first introduced in Unet [37], FPN
[38], and DSSD [39] for object detection and then was generalized to temporal
domain in MGG [40], PBRNet [41], RapNet [42], C-TCN [43], and MLTPN [44]. The
video representation features are extracted using off-the-shelf feature
extractors. Then temporal convolution and max pooling layers are applied to
reduce the temporal dimension and increase the receptive field size. This is
followed by temporal deconvolution layers for upscaling. Then, high-level
features are combined with corresponding low-level features with lateral
connections between the convolutional and deconvolutional layers. U-shaped
TFPNs have drawn much attention recently and achieved state-of-the art results
for temporal action detection task.

####  2.3.2 Anchor-free Proposal Generation and Evaluation

Anchor-free methods employ a bottom-up grouping strategy for proposal
generation based on predicted boundary probability or actionness scores at
temporal positions of the video. Anchor-free methods are capable to generate
proposals with precise boundary and flexible duration because the proposal
lengths are not predefined.

##### 2.3.2.1 Proposal Generation with Actionness Scores

Zhao et al. in SSN [18] proposed to identify continuous temporal regions with
high actionness scores (def 6) as proposals (known as TAG proposals).
Continuous temporal regions are grouped using a classic watershed algorithm
[45] applied on the 1D signal formed by complemented actionness values. The
proposals are fed to a temporal pyramid for feature extraction and proposal
evaluation. The feature extraction process is too simple to capture rich
features.

##### 2.3.2.2 Proposal Generation with Boundary Scores

These methods predict three probability signals for actionness (def 6),
startness and endness scores (def 7). They generate temporal proposals by
matching the temporal positions that are likely to be the start or end of an
action (peak of startness and endness signals). In BSN [46] proposal features
are constructed by concatenation of a fixed number of points, sampled from the
actionness scores (def 6) by linear interpolation. BSN ignores the global
information for actions with blurred boundaries, causing unreliable confidence
scores. Also, proposal features are too weak to capture enough temporal
context, and the feature construction and confidence evaluation are performed
for each proposal separately, which is inefficient. BMN [47] explores the
global context for simultaneously evaluating all proposals end-to-end. They
construct a feature map by aggregating the features of all proposals together.
The feature map is fed to convolution layers to simultaneously evaluate all
proposals. The advantage of this approach is to extract rich feature and
temporal context for each proposal and exploit the context of adjacent
proposals. Also, proposal evaluation is very fast during inference. However,
they use the same method as BSN [46] to generate boundary probabilities (start
and end) which ignores the global information for actions with blurred
boundaries. DBG [48] simultaneously evaluates all proposals to explore global
context and extract rich features similar to BMN [47]. Moreover, instead of
only exploiting the local information to predict boundary probabilities
(probability of start and end), DBG proposed to employ global proposal-level
features.

Figure 4: Anchor-free proposal generation with boundary matching. These
methods predict action boundary probabilities at uniformly distributed
temporal positions and match the start and end points with high probabilities
as proposals.

In order to model the relations between the boundary and action content of
temporal proposals, BC-GNN [49] proposed a graph neural network where
boundaries and content of proposals are taken as the nodes and edges of the
graph, and their features are updated through graph operations. Then the
updated edges and nodes are used to predict boundary probabilities and content
confidence score to generate proposals. A2Net [50] and AFSD [51] visit the
anchor-free mechanism, where the network predicts the distance to the temporal
boundaries for each temporal location in the feature sequence. AFSD [51] also
proposes a novel boundary refinement strategy for precise temporal
localization.

####  2.3.3 Anchor-based and Anchor-free Combination

Anchor-based methods consider segments of various lengths as initial proposals
at regularly distributed temporal positions of the video. However, because the
segment sizes are designed beforehand, they cannot accurately predict the
temporal boundary of actions. Also, because the duration of action instances
varies from seconds to minutes, covering all ground-truth instances with
anchor-based methods is computationally expensive. Anchor-free methods predict
action boundary confidence or actionness score at all temporal positions of
the video, and employ a bottom-up grouping strategy to match pairs of start
and end. Anchor-free methods are capable to generate proposals with precise
boundaries and flexible duration. However, in some cases they only exploit
local context to extract the boundary information. Therefore, they are
sensitive to noise, likely to produce incomplete proposals, and fail to yield
robust detection results.

Several methods such as CTAP [52], MGG [40], PBRNet [41], RapNet [42] balance
the advantages and disadvantages between anchor-based and anchor-free
approaches for proposal generation. CTAP [52] designed a complementary filter
applied on the initial proposals to generate the probabilities of proposal
detection by anchor-free TAG [18] (defined in 2.3.2). The original use of
complementary filtering is to estimate a signal given two noisy measurements,
where one of them is mostly high-frequency (maybe precise but not stable)
similar to TAG proposals and the other one is mostly low-frequency (stable but
not precise) similar to sliding-window proposals. Also, several temporal
feature pyramid networks (TFPN, defined in 2.3.1) such as MGG [40], PBRNet
[41], and RapNet [42] generate coarse segment proposals of various length with
TFPN (anchor-based), and simultaneously predict fine-level frame actionness
(anchor-free). The advantage of this idea is to adjust the segment boundary of
proposals with frame actionness information during the inference.

####  2.3.4 Common Loss Functions for Proposal Evaluation

After generating the temporal proposals, rich features are extracted from the
proposals to evaluate their quality. Several convolutional layers are applied
on the features to predict actionness score (def 6), completeness score (def
8), classification score (def 9), and to adjust the temporal boundary of the
proposals. Here, we review common loss functions that are used during training
to supervise these predicted scores and evaluate the quality of proposals.

######  Definition 10.

Actionness loss. This is a binary cross-entropy loss that classifies the
temporal proposals as action or background. Given $N$ proposals, this loss is
defined as:

$L_{\text{act}}=-\frac{1}{N}\sum_{i=1}^{N}b_{i}\log(a_{i})+(1-b_{i})\log(1-a_{i}),$
(5)

where $a_{i}$ is the predicted actionness score (def 6), and
$b_{i}\in\\{0,1\\}$ is a binary ground-truth label for the $i$-th proposal. If
the proposal is positive (def 4), then $b_{i}=1$. Otherwise, $b_{i}=0$.

######  Definition 11.

Action completeness loss. Given $N$ proposals, the completeness loss is
defined as:

$L_{\text{com}}=\frac{1}{N_{\text{pos}}}\sum_{i=1}^{N}d(c_{i},g_{i})\cdot[l_{i}>0],$
(6)

where $c_{i}$ is the predicted action completeness score (def 8) for $i$-th
proposal, and $g_{i}$ is the ground-truth action completeness score. $d$ is a
distance metric which is often $L_{2}$ or smooth $L_{1}$ loss. $l_{i}$ is the
label of the $i$-th proposal and condition $[l_{i}>0]$ implies that action
completeness is only considered for positive proposals (def 4).
$N_{\text{pos}}$ is the number of positive proposals during each mini-batch.

######  Definition 12.

Action overlap loss. This is another variation of action completeness loss
which rewards the proposals with higher temporal overlap with ground truths
and is defined as the following:

$\mathcal{L}_{overlap}=\frac{1}{N_{\text{pos}}}\sum_{i}\frac{1}{2}\cdot\Big{(}\frac{(p^{l_{i}}_{i})^{2}}{(g_{i})^{\alpha}}-1\Big{)}\cdot[l_{i}>0],$
(7)

where $p_{i}$ is the classification probability vector over action labels for
the $i$-th proposal, and $p^{l_{i}}_{i}$ is the probability of action class
$l_{i}$. Other notations are the same as in $L_{\text{com}}$ (def 11) and
$\alpha$ is a hyper-parameter.

######  Definition 13.

Action classification loss. This is the classification (cross-entropy) loss
and the probability distribution is over all action classes as well as
temporal background as the following:

$L_{\text{cls}}=-\frac{1}{N}\sum_{i=1}^{N}\log(p^{l_{i}}_{i}),$ (8)

where $l_{i}\in\\{0,1,\cdots,C\\}$ is the label of $i$-th proposal, and
$p^{l_{i}}_{i}$ is the probability of class $l_{i}$.

######  Definition 14.

Action regression loss. To adjust the temporal boundary of proposals, the
start and end offset of proposals are predicted and supervised by a regression
loss as the following:

$L_{\text{reg}}=\frac{1}{N_{\text{pos}}}\sum_{i=1}^{N}|(o_{s,i}-o^{\star}_{s,i})+(o_{e,i}-o^{\star}_{e,i})|\cdot[l_{i}>0],$
(9)

where term $o_{s,i}$ is the difference between the start coordinate of $i$-th
proposal and the start coordinate of the closest ground truth action instance.
The term $o^{\star}_{s,i}$ is the predicted offset. Similarly, $o_{e,i}$ and
$o^{\star}_{e,i}$ are the ground-truth and predicted offset for end coordinate
of the $i$-th proposal. The condition $[l_{i}>0]$ implies that boundary
adjustment is only considered for positive proposals (def 4).

####  2.3.5 Modeling Long-range Dependencies

As mentioned earlier, untrimmed videos are often lengthy and must be
partitioned into shorter clips for feature extraction. Processing these
shorter clips independently can lead to loss of temporal or semantic
dependencies between video segments. Therefore, several tools such as
recurrent neural networks, graph convolutions, attention mechanism and
transformers are used to capture these dependencies. The advantage of modeling
dependencies is to refine the temporal boundary of proposals or predict their
action category or action completeness given the information from other
neighboring proposals.

##### 2.3.5.1 Recurrent Neural Networks

RNNs are used for sequence modeling and are capable of capturing long-term
dependencies in videos. Buch et al. in Sst [53] and SS-TAD [54] used RNNs for
action detection. They partition the video into non-overlapping equal-length
segments and feed each segment to a visual encoder for feature extraction. At
time $t$, visual feature $f_{t}$ and the hidden state of the previous time
step ($h_{t-1}$) are fed to a Gated Recurrent Unit (GRU)-based architecture to
produce hidden state $h_{t}$. This hidden state is then fed to fully connected
layers to evaluate multi-scale proposals at time $t$ by producing actionness
scores (def 6). In an earlier work, Yuan et al. in PSDF [55] captured the
motion information over multiple resolutions and utilized RNNs to improve
inter-frame consistency. Yeung et al. learn decision policies for an RNN-based
agent [56], and later proposed an LSTM model to process multiple input frames
with temporal attention mechanism [57]. LSTMs are also used in other
frameworks such as [58], [59], [60] to evaluate temporal proposals. The
advantage of using RNNs is that hidden state at time $t$ encodes the
information from previous time steps which is useful to capture temporal
dependencies. However, RNNs are not capable to encode very long videos as the
hidden vector gets saturated after some time steps.

Figure 5: Capturing temporal dependencies in untrimmed videos with RNNs.
Hidden state at time $t$, $h_{t}$, encodes the information from previous time
steps. This picture is regenerated from [53].

##### 2.3.5.2 Graph Models

A full action often consists of several sub-actions that may independently be
detected in several overlapping proposals. Based on this observation, Zeng et
al. in PGCN [61] captured proposal-proposal relations by applying graph-
convolution networks (GCNs). They constructed a graph where the nodes are the
proposals. The edges connect highly overlapped proposals as well as disjoint
but nearby proposals to provide contextual information. The edge weights model
the relation between the proposals by measuring cosine similarity of their
features. Through graph convolutions feature of each proposal gets updated by
aggregating the information from other proposals. The updated features are
then used to predict action categories, completeness, and refining the
boundaries.

Figure 6: Modeling proposal-proposal relations with graph convolutional
networks (GCNs), where the nodes are the proposals and the edges model the
relations between proposals. The feature of proposal $p_{3}$ is influenced by
the features of proposals $p_{1},p_{2}$, and $p_{4}$. Image is reproduced from
PGCN [61].

Li et al. in AGCN [31] proposed an attention based GCN to model the inter and
intra dependencies of the proposals. Intra attention learns the long-range
dependencies among pixels inside each action proposal and inter attention
learns the adaptive dependencies among the proposals to adjust the imprecise
boundary. Bai et al. in BC-GNN [49] proposed a graph neural network to model
the relations between the boundary and action content of temporal proposals.

Xu et al. proposed G-TAD [62] to capture the relations between different
snippets of input video. They constructed a graph where the nodes are temporal
segments of the video and the edges are either temporal or semantic. The
temporal edges are pre-defined according to the snippets’ temporal order but
the semantic edges are dynamically updated between the nodes according to
their feature distance. Temporal and semantic context of the snippets are
aggregated using graph convolutions. All possible pairs of start and end with
duration within a specific range are considered to generate the proposals. To
evaluate each proposal, the temporal and semantic features of the
corresponding sub-graph are extracted. Chang et al. in ATAG [63] also designed
an adaptive GCN similar to G-TAD [62] to capture local temporal context where
the graph nodes are the snippets and the edges model the relation between
snippets. The temporal context is then captured through graph convolutions
where the feature of each snippet is influenced and updated by the features of
other snippets. VSGN [64] builds a graph on video snippets similar to G-TAD
[62], but also exploits correlations between cross-scale snippets. They
propose a cross-scale graph pyramid network which aggregates features from
cross scales, and progressively enhances the features of original and
magnified scales at multiple network levels.

##### 2.3.5.3 Transformers

Some action instances in a video have non-sequential dependencies, meaning
that they are related but are separated by other events in the video. Also,
some action instances may have overlaps in their temporal extents. Based on
these observations, Nawhal et al. in AGT [65] proposed an encoder decoder
transformer to capture non-linear temporal structure by reasoning over videos
as nonsequential entities. Their encoder generates a context graph where the
nodes are initially video level features and the interactions among nodes are
modeled as learnable edge weights. Also, positional information for each node
is provided using learnable positional encodings. Their decoder learns the
interactions between context graph (latent representation of the input video)
and graph structured query embeddings (latent representations of the action
queries).

Tan et al. in RTD-Net [66] proposed a relaxed transformer to directly generate
action proposals without the need to human prior knowledge for careful design
of anchor placement or boundary matching mechanisms. The transformer encoder
models long-range temporal context and captures inter-proposal relationships
from a global view to precisely localize action instances. They also argued
that the snippet features in a video change at a very slow speed and direct
employment of self-attention in transformers can lead to over-smoothing.
Therefore, they customized the encoder with a boundary-attentive architecture
to enhance the discrimination capability of action boundary. Chang et al. in
ATAG [63] designed an augmented transformer to mine long-range temporal
context for noisy action instance localization. The snippet-level features
generated by transformer are used to classify the snippets to action or
background under supervision of a binary classification loss. Throughout this
process the transformer learns to capture long-term dependencies at snippet
level.

####  2.3.6 Spatio-temporal Action Detection

Spatio-temporal action detection aims to localize action instances in both
space and time, and recognize the action labels. In the fully-supervised
setting of this task, the temporal boundary of action instances at the video-
level, the spatial bounding box of actions at the frame-level, and action
labels are provided during training and must be detected during inference.
Fig. 7 shows an example of this task. The start and end of action “long jump”
are detected in temporal domain. Also, bounding box of the actor performing
the action is detected in each frame in spatial domain.

Figure 7: Spatio-temporal activity detection task: action ”long jump” is
localized in time and space. Other than temporal interval of the action,
bounding box of the person performing the action is detected in each frame.

##### 2.3.6.1 Frame-level Action Detection

Early methods [67, 68] were based on extensions of the sliding window scheme,
requiring strong assumptions such as a cuboid shape, i.e., a fixed spatial
extent of the actor across frames. Later, advancements in object detection
inspired frame-level action detection methods to recognize human action
classes at the frame level. In the first stage action proposals are produced
by a region proposal algorithm or densely sampled anchors, and in the second
stage the proposals are used for action classification and localization
refinement. Hundreds of action proposals are extracted per video given low-
level cues, such as super-voxels [69, 70] or dense trajectories [71, 72, 73],
and then proposals are classified to localize actions.

After detecting the action regions in the frames, some methods [74, 75, 76,
77, 78, 79, 80, 81] use optical flow to capture motion cues. They employ
linking algorithms to connect the frame-level bounding boxes into spatio-
temporal action tubes. Gkioxari et al. [74] used dynamic programming approach
to link the resulting per-frame detection. The cost function of the dynamic
programming is based on detection scores of the boxes and overlap between
detection of consecutive frames. Weinzaepfel et al. [79] replaced the linking
algorithm by a tracking-by-detection method. Then, two-stream Faster R-CNN was
introduced by [76, 78]. Saha et al. [78] fuse the scores of both streams based
on overlap between the appearance and the motion. Peng et al. [76] combine
proposals extracted from the two streams and then classify and regress them
with fused RGB and multi-frame optical flow features. They also use multiple
regions inside each action proposal and then link the detection across a video
based on spatial overlap and classification score.

Another group [82, 83, 84] rely on an actionness measure, i.e., a pixel-wise
probability of containing any action. To estimate actionness, they use low-
level cues such as optical flow [84], CNNs with a two-stream architecture [83]
or RNNs [83]. They extract action tubes by thresholding the actionness scores
[82] or by using a maximum set coverage [84]. The output is a rough
localization of the action as it is based on noisy pixel-level maps.

The main disadvantage of these methods is that the temporal property of videos
is not fully exploited as the detection is performed on each frame
independently. Effective temporal modeling is crucial as a number of actions
are only identifiable when temporal context information is available.

##### 2.3.6.2 Clip-level Action Detection

As mentioned earlier, temporal modeling is necessary for accurate action
localization. Here, we discuss methods that exploit temporal information by
performing action detection at the clip (i.e., a short video snippet) level.

Kalogeiton et al. [85] proposed action tubelet detector (ACT-detector) that
takes as input a sequence of frames and outputs action categories and
regressed tubelets, i.e., sequences of bounding boxes with associated scores.
The tubelets are then linked to construct action tubes (sequence of bounding
boxes of action). Gu et al. [86] further demonstrate the importance of
temporal information by using longer clips and taking advantage of I3D pre-
trained on the large-scale video dataset [24]. In order to generate action
proposals, they extend 2D region proposals to 3D by replicating them over
time, assuming that the spatial extent is fixed within a clip. However, this
assumption would be violated for the action tubes with large spatial
displacement over time, in particular when the clip is long or involves rapid
movement of actors or camera. Thus, using long cuboids directly as action
proposals is not optimal, since they introduce extra noise for action
classification.

Yang et al. [87] perform action detection at clip level, and then linked them
to build action tubes across the video. They employ multi-step optimization
process to progressively refine the initial proposals. Other methods [6], [88]
exploited human proposals coming from pretrained image detectors and
replicated them in time to build straight spatio-temporal tubes.

##### 2.3.6.3 Modeling Spatio-temporal Dependencies

Understanding human actions usually requires understanding the people and
objects around them. Therefore, state-of-the-art methods model the relation
between actors and the contextual information such as other people and other
objects. Some methods used the graph-structured networks [89, 90] and
attention mechanism [91, 88, 92] to aggregate the contextual information from
other people and objects in the video.

Wu et al. [88] provided long-term supportive information that enables video
models to better understand the present. The designed a long-term feature bank
and a feature bank operator FBO that computes interactions between the short-
term and long-term features. They integrate information over a long temporal
support, lasting minutes or even the whole video. Girdhar et al. [91] proposed
a transformer-style architecture to weight actors with features from the
context around him. Tomei et al. [93] employed self-attention to encode people
and object relationships in a graph structure, and use the spatio-temporal
distance between proposals. Ji et al. proposed Action Genome [94] to model
action-object interaction, by decomposing actions into spatio-temporal scene
graphs. Ulutan et al. [92] suggested combining actor features with every
spatio-temporal region in the scene to produce attention maps between the
actor and the context. Pan et al. [95] proposed a relational reasoning module
to capture the relation between the two actors based on their respective
relations with the context. Tomei et al. [96] proposed a graph-based framework
to learn high-level interactions between people and objects, in both space and
time. Spatio-temporal relationships are learned through self-attention on a
multi-layer graph structure which can connect entities from consecutive clips,
thus considering long-range spatial and temporal dependencies.

###  2.4 Action Detection with Limited Supervision

Fully supervised action detection requires the full annotation of temporal
boundaries and action labels for all action instances in training videos,
which is very time-consuming and costly. To eliminate the need for exhaustive
annotations in the training phase, in recent years, researchers have explored
the design of efficient models that require limited ground-truth annotations.
We discuss weakly-supervised methods in Section 2.4.1, and other learning
methods with limited supervision (unsupervised, semi-supervised, and self-
supervised) are described in Section 2.4.2.

####  2.4.1 Weakly-supervised Action Detection

Weakly-supervised learning scheme requires coarse-grained or noisy labels
during the training phase. Following the work of [97], weakly-supervised
action detection in common settings requires only the video-level labels of
actions during training while the temporal boundaries of action instances are
not needed. During testing both labels and temporal boundaries of actions are
predicted. In the following parts of this section, weakly-supervised action
detection refers to this setting. There are also other weak signals utilized
for action detection such as order of actions [98], [99], [100], [101],
frequency of action labels [102], and total number of events in each video
[103]. A common strategy in weakly-supervised action detection is to use
attention mechanism to focus on discriminative snippets and combine salient
snippet-level features into a video-level feature. The attention scores are
used to localize the action regions and eliminate irrelevant background
frames. There are two main strategies to extract attention signals from
videos. First, class-specific attention approaches where attention scores are
generated from class activation sequences (def 15) for each action class
(Section 2.4.1.2). Second, class-agnostic attention approaches where attention
scores are class-agnostic and are extracted from raw data (Section 2.4.1.3).
We discuss these two attention strategies in this section.

##### 2.4.1.1 Term Definition

To facilitate reading this section, we provide the definition of frequently
used terminologies.

######  Definition 15.

Temporal class activation maps (T-CAM). For a given video, T-CAM is a matrix
denoted by $A$ which represent the possibility of activities at each temporal
position. Matrix $A$ has $n_{c}$ rows which is the total number of action
classes, and $T$ columns which is the number of temporal positions in the
video. Value of cell $A[c,t]$ is the activation of class $c$ at temporal
position $t$. Formally $A$ is calculated by:

$A=WX\oplus b,$ (10)

where $X\in{\rm I\\!R}^{d\times T}$ is a video-level feature matrix, and $d$
is the feature dimension. Also, $W\in{\rm I\\!R}^{n_{c}\times d}$ and
$b\in{\rm I\\!R}^{n_{c}}$, are learnable parameters and $\oplus$ is the
addition with broadcasting operator.

######  Definition 16.

Class-specific attention scores. In a given video, class-specific attention
score is the occurrence probability of action class $c$ at temporal position
$t$, denoted by $a[c,t]$. Formally, $a[c,t]$ is computed by normalizing the
activation of class $c$ over temporal dimension:

$a[c,t]=\frac{\text{exp}(A[c,t])}{\sum_{t=1}^{T}\text{exp}(A[c,t])},$ (11)

where $A$ is the T-CAM (def 15), and $T$ is the number of temporal positions.
Therefore, row $a_{c}$ is the probability distribution of occurrence of class
$c$ over video length.

######  Definition 17.

Class-agnostic attention score. In a given video, class-agnostic attention
score, denoted by $\lambda_{t}$, is the occurrence probability of any action
of interest at temporal position $t$, regardless of the action class. The
attention vector for all temporal positions of the video is denoted by
$\lambda$.

######  Definition 18.

Attention-based aggregated features. The video-level foreground and background
features are generated using temporal pooling of embedded features weighted by
attention scores. Class-specific features are defined based on class-specific
attention scores $a_{c}$ (def 16) for each class $c$ while class-agnostic
features are defined based on class-agnostic attention vector $\lambda$ (def
17). Aggregated foreground feature is most influenced by feature vectors with
high attention that represent actions while background feature is impacted by
features with low attention. $T$ is the video length and $X$ is the video
feature matrix. These features are formulated as the following:

| Foreground: | Background:
---|---|---
Class-specific: | $f_{c}=Xa_{c}$ | $b_{c}=\frac{1}{T-1}X(\mathbb{1}-a_{c}),$
Class-agnostic: | $f=\frac{1}{T}X\lambda$ | $b=\frac{1}{T}X(\mathbb{1}-\lambda).$

##### 2.4.1.2 Class-specific Attention for Action Localization

Class-specific attention module computes the attention weight $a[c,t]$ (def
16) for all action classes $c$ and all temporal positions $t$ in each video.
The attention scores attend to the portions of the video where an activity of
a certain category occurs. Therefore, video segments with attention scores
higher than a threshold are localized as action parts. Class-specific
attention module is used in [104], [105], [102], [106] to localize the
temporal boundary of action instances.

Class-specific attention learning with MIL: In general scheme of MIL (multi-
instance learning), training instances are arranged in sets, called bags, and
a label is provided for the entire bag [107]. In the context of weakly-
supervised temporal action detection, each video is treated as a bag of action
instances and the video-level action labels are provided. In order to compute
the loss for each bag (video in this task), each video should be represented
using a single confidence score per category. The confidence score for each
category is computed as the average of top $k$ activation scores over the
temporal dimension for that category. In a given video, suppose set
$\\{t^{c}_{1},t^{c}_{2},\cdots,t^{c}_{k}\\}$ are $k$ temporal positions with
highest activation scores for class $c$. Then, the video-level class-wise
confidence score $s^{c}$ for class $c$ is defined as:

$s^{c}=\frac{1}{k}\sum_{l=1}^{k}A[c,t^{c}_{l}],$ (12)

where $A[c,t^{c}_{l}]$ is the activation (def 15) of class $c$ at temporal
position $t^{c}_{l}$. Then, probability mass function (PMF) of action classes
is computed by applying softmax function on $s^{c}$ scores over class
dimension:

$p^{c}=\frac{\exp{(s^{c})}}{\sum_{c=1}^{n_{c}}\exp{(s^{c})}},$ (13)

where $n_{c}$ is the number of action classes. MIL loss is a cross-entropy
loss applied over all videos and all action classes. For video $i$ and action
class $c$, $p^{c}_{i}$ is the class-wise probability score, and $y^{c}_{i}$ is
a normalized ground-truth binary label. MIL is defined as:

$L_{MIL}=\frac{1}{n}\sum_{i=1}^{n}\sum_{c=1}^{n_{c}}-y^{c}_{i}\log(p^{c}_{i}),$
(14)

where $n$ is the total number of videos. MIL loss supervises class-wise
probability scores which are computed based on activation scores $A[c,t]$.
Therefore, MIL learns activation scores and T-CAM (def 15) for each video and
is used in W-TALC [105], Action Graphs [108], UNet [104], and Actionbytes
[109].

Class-specific attention learning with CASL: The CASL (co-activity similarity
loss) was initially introduced in W-TALC[105] and then inspired others Deep
Metric [106], Action Graphs [108], WOAD [110], Actionbytes [109]. The main
idea is that for a pair of videos including the same action classes, the
foreground features in both videos should be more similar than the foreground
feature in one video and the background feature in the other video. For a pair
of videos with indices $m$ and $n$ that include action class $c$, the
foreground features are denoted by $f^{m}_{c}$, $f^{n}_{c}$ and the background
features by $b^{m}_{c}$, $b^{n}_{c}$ (def 18). Then CASL is defined based on
ranking hinge loss as the following:

$\begin{split}L^{mn}_{c}&=\frac{1}{2}\\{\max\big{(}0,d(f^{m}_{c},f^{n}_{c})-d(f^{m}_{c},b^{n}_{c})+\delta\big{)}\\\
&+\max\big{(}0,d(f^{m}_{c},f^{n}_{c})-d(b^{m}_{c},f^{n}_{c})+\delta\big{)}\\},\end{split}$
(15)

where $d$ is a metric (e.g., cosine similarity) to measure the degree of
similarity between two feature vectors and $\delta$ is a margin parameter. The
average of $L^{mn}_{c}$ is computed over all video pairs that include action
class $c$. This loss trains class-specific attention scores $a_{c}$ as
foreground and background features $f_{c}$ and $b_{c}$ are defined based on
$a_{c}$ (def 18).

Islam et al. in Deep Metric [106] replaced metric $d$ with a class-specific
metric $D_{c}$ defined for each class $c$. Rashid et al. in Action Graphs
[108] applied a GCN to transform each temporal segment’s feature
representation to a weighted average of its neighbors. Then updated features
are used in CASL for localization. The advantage of this GCN is to model
temporal dependencies, cluster the semantically-similar time segments, and
pushing dissimilar segments apart.

Class-specific attention learning with center loss: The center loss which was
first introduced in [111], learns the class-specific centers and penalizes the
distance between the features and their class centers. Narayan et al. in
3C-Net [102] employed center loss to enhance the feature discriminability and
reduce the intra-class variations. For each video $i$ and each action class
$c$, center loss computes the distance (L2 norm) between class-specific
foreground feature $f^{i}_{c}$ (def 18) and cluster center feature $z_{c}$ as
the following:

$\mathcal{L}_{center}=\frac{1}{N}\sum_{i}\sum_{c:y^{i}(c)=1}\left\lVert
f^{i}_{c}-z_{c}\right\rVert^{2}_{2},$ (16)

where cluster center feature $z_{c}$ is updated during training. Here, $N$ is
the total number of videos, and condition $y^{i}(c)=1$ checks if action class
$c$ occurs in video $i$.

##### 2.4.1.3 Class-agnostic Attention for Action Localization

Class-agnostic attention module computes attention vector $\lambda$ (def 17)
directly from raw data, by applying fully connected and ReLU layers over video
features, followed by a sigmoid function to scale attention weights to
$[0,1]$. Learning class-agnostic attention weights is used in many methods
such as RPN [112], BG modeling [113], AutoLoc[114], CleanNet [115], DGAM
[116], STPN [117] , BaSNet [118] , MAAN [119], and CMCS [120].

Class-agnostic attention learning with cross-entropy: The video-level class-
agnostic foreground and background features $f$ and $b$ (def 18) are fed to a
classification module, and supervised with a cross entropy loss:

$p_{fg}[c]=\frac{\exp{(w_{c}\cdot f)}}{\sum_{i=0}^{C}\exp{(w_{i}\cdot
f)}},\mathcal{L}_{fg}=-\log(p_{fg}[y]),$ (17)

where $w_{c}$ s are the weights of the classification module, $C$ is the
number of action classes in the entire dataset, and $y$ is the label of action
that happens in the video. Also, label $0$ represents the background class.
Similarly, $\mathcal{L}_{bg}$ is defined for $p_{bg}$ which is a softmax
applied over multiplication of background feature $b$ and the classification
module. This loss trains attention vector $\lambda$ through class-agnostic
features $f$ and $b$ (def 18), and is used in STPN [117].

Class-agnostic attention learning with clustering loss: Nguyen et al in
background modeling [113] propose a method to separate foreground and
background using a clustering loss by penalizing the discriminative capacity
of background features. Class-agnostic foreground and background features $f$
and $b$ (def 18) are encouraged to be distinct using a clustering loss:

$z_{f}=\frac{\exp(uf)}{\exp(uf)+\exp(vf)}\ ,\
z_{b}=\frac{\exp(vb)}{\exp(ub)+\exp(vb)},$ (18)

$\mathcal{L}_{cluster}=-\log{z_{f}}-\log{z_{b}},$ (19)

where $u,v\in{\rm I\\!R}^{d}$ are trainable parameters. Attention $\lambda$ is
trained by separating class-agnostic features $f$ and $b$ (def 18).

Class-agnostic attention learning with prototypes: Prototypical network which
was introduced in [121] for classification task, represents each class as a
prototype and matches each instance with a prototype with highest similarity.
During training, the semantically-related prototypes are pushed closer than
unrelated prototypes. Huang et al. in RPN [112] proposed a prototype learning
scheme for action localization. For temporal position $t$ and action class
$c$, the similarity score $s_{t,c}$ between feature $x_{t}$ and prototype
$p_{c}$ is computed and similarity vector $s_{t}$ consists of $s_{t,c}$ for
all classes. Then the similarity vector $s_{t}$ is fused with attention score
$\lambda_{t}$ into a video-level score $\hat{s}$:

$s_{t,c}=-\left\lVert x_{t}-p_{c}\right\rVert^{2}_{2}\ \ ,\ \
\hat{s}=\sum_{t=1}^{T}\lambda_{t}s_{t}.$ (20)

Score $\hat{s}$ is supervised by a classification loss with respect to the
video-level labels, training attention scores $\lambda_{t}$.

Class-agnostic attention learning with CVAE: DGAM [116] aims to separate
actions from context frames by imposing different attentions on different
features using a generative model, conditional VAE (CVAE) [122]. Formally, the
objective of DGAM is:

$\max_{\lambda\in[0,1]}\underbrace{\log p(y|X,\lambda)}_{\text{term
1}}+\underbrace{\log p(X|\lambda)}_{\text{term 2}},$ (21)

where $X$ denotes the features, $y$ is the video-level label, and $\lambda$ is
the attention signal. Term 1 encourages high discriminative capability of the
foreground feature $f$ and punishes any discriminative capability of the
background feature $b$. Term 2 is approximated by a generative model which
forces the feature representation $X$ to be accurately reconstructed from the
attention $\lambda$ using CVAE. By maximizing this conditional probability
with respect to the attention, the frame-wise attention is optimized by
imposing different attentions on different features, leading to separation of
action and context frames.

##### 2.4.1.4 Direct Action Proposal Generation

Many methods [117], [105], [123], [104] localize the actions by applying
thresholds on attention scores. The disadvantage of thresholding is that the
snippets are treated independently and their temporal relations are neglected.
Also, thresholding may not be robust to noises in class activation maps. Shou
et al. [114] in AutoLoc directly predict the temporal boundary of each action
instance. A localization branch is designed to directly predict the action
boundaries (inner boundaries). The outer boundaries are also obtained by
inflating the inner boundaries. Knowing that a video includes action class
$c$, an outer-inner-contrastive (OIC) loss is applied on the activation scores
of action $c$. The OIC loss computes the average activation in the outer area
minus the average activation in the inner area to encourage high activations
inside and penalize high activations outside because a complete action clip
should look different from its neighbours. Liu et al. [115] proposed CleanNet
to exploit temporal contrast for action localization. A contrast score is
generated by summing up action, starting and ending scores for each action
proposal. The action localization is trained by maximizing the average
contrast score of the proposals, which penalizes fragmented short proposals
and promotes completeness and continuity in action proposals.

##### 2.4.1.5 Action Completeness Modeling

Previous methods used random hiding and iterative removal to enforce action
completeness. Singh et al. in Hide-and-seek [123] force the model to see
different parts of the video by randomly masking different regions of the
videos in each training epoch. However, randomly hiding frames does not always
guarantee the discovery of new parts and also disrupts the training process.
Zhong et al. in Step-by-step erasion [124] trained a series of classifiers
iteratively to find complementary parts, by erasing the predictions of
predecessor classifiers from input videos. The major draw-back with this
approach is the extra time cost and computational expense to train multiple
classifiers. Zeng et al. [125] propose an iterative-winners-out strategy that
selects the most discriminative action instances in each training iteration
and hide them in the next iteration. Liu et al. in CMCS [120] proposed to
enforce multiple branches in parallel to discover complementary pieces of an
action. Each branch generates a different class activation map (def 15). A
diversity loss (introduced in [126]) is imposed on class activation maps,
which computes cosine similarities between pairs of branches and all action
categories. Minimizing the diversity loss, encourages the branches to produce
activations on different action parts.

####  2.4.2 Unsupervised, Semi-supervised, and Self-supervised

Although weakly-supervised action detection has been extensively studied in
recent years, there are fewer articles addressing action detection task in
unsupervised, semi-supervised, or self-supervised setting that are briefly
reviewed here.

##### 2.4.2.1 Unsupervised Action Detection

Unsupervised learning does not need any human-annotated labels during
training. Seneret et al. [127] introduced an iterative approach which
alternates between discriminative learning of the appearance of sub-activities
from visual features and generative modeling of the temporal structure of sub-
activities. Kukleva et al. [128] proposed a combination of temporal encoding
(generated using a frame time stamp prediction network) and a Viterbi decoding
for consistent frame-to-cluster assignment. Gong et al. in ACL [129] used only
the total count of unique actions that appear in the video set as supervisory
signal. They propose a two-step clustering and localization iterative
procedure. The clustering step provides noisy pseudo-labels for the
localization step, and the localization step provides temporal co-attention
models to improve the clustering performance.

##### 2.4.2.2 Self-supervised Action Detection

Self-supervised learning refers to training with pseudo labels where pseudo
labels are automatically generated for a pre-defined pretext task without
involving any human annotations. Chen et al. in SSTDA [130] proposed self-
supervised temporal domain adaptation method to address the spatio-temporal
variations (different people performing the tasks in different styles) in
action segmentation. They designed two self-supervised auxiliary tasks, binary
and sequential domain prediction, to jointly align local and global embedded
feature spaces across domains. The binary domain prediction task predicts a
single domain for each frame-level feature, and the sequential domain
prediction task predicts the permutation of domains for an untrimmed video,
both trained by adversarial training with a gradient reversal layer (GRL)
[131, 132]. Jain et al. in Actionbytes [109] only use short trimmed videos
during the training and train an action localization network with cluster
assignments as pseudo-labels to segments a long untrimmed videos into
interpretable fragments (called ActionBytes). They adopt a self-supervised
iterative approach for training boundary-aware models from short videos by
decomposing a trimmed video into ActionBytes and generate pseudo-labels to
train a CNN to localize ActionBytes within videos.

TABLE II: The benchmark datasets for temporal and spatio-temporal action detection. Dataset | Activities Types |  # Videos |  # Action Categories |  Avg Video Length (Sec) |  # Action Instances (avg per video) |  Multi-label (  # labels per frame)
---|---|---|---|---|---|---
THUMOS [1] | Sports | 413 | 20 | 212 | 15.5 | No
MultiTHUMOS [57] | Sports | 413 | 65 | 212 | 97 | Yes
ActivityNet [133] | Human Activities | 19,994 | 200 | 115 | 1.54 | No
HACS Segment [134] | Human Activities | 50K | 200 | 156 | 2.8 | No
Charades [135] | Daily Activities | 9,848 | 157 | 30 | 6.75 | Yes
Breakfast [11] | Cooking | 1712 | 48 | 162 | 6 | No
50Salads [136] | Cooking | 50 | 17 | 384 | 20 | No
MPII cooking 2 [137] | Cooking | 273 | 59 | 356 | 51.6 | No
COIN [138] | Daily Activities | 11,827 | 180 | 142 | 3.9 | No
Ava [86] | Movies | 437 | 80 | 900 | 3361.5 | Yes

##### 2.4.2.3 Semi-supervised Action Detection

In Semi-supervised setting, a small number of videos are fully annotated with
the temporal boundary of actions and class labels while a large number of
videos are either unlabeled or include only video-level labels. Ji et al.
[139] employ a fully supervised framework, known as BSN [46], to exploit the
small set of labeled data. They encode the input video into a feature sequence
and apply sequential perturbations (time warping and time masking [140]) on
it. Then, the student proposal model takes this perturbed sequence as the
input but the teacher model predicts directly on the original feature
sequence. In the end, the student model is jointly optimized with a supervised
loss applied to labeled videos and a consistency loss to all videos.

##  3 Datasets and Evaluation

In this section, we describe the datasets collected for action detection and
the evaluation metrics for this task.

###  3.1 Datasets

Gaidon et al. [141, 142] introduced the problem of temporally localizing the
actions in untrimmed videos, focusing on limited actions such as “drinking and
smoking” [67] and “open door and sitdown” [143]. Later, researchers worked on
building the following datasets that include large number of untrimmed videos
with multiple action categories and complex background information. Some of
these datasets target activities of high-level semantics (such as sports)
while others include fine-grained activities (such as cooking). The details
are summarized in Table II.

$\bullet$ THUMOS14 [1] is the most widely used dataset for temporal action
localization. There are $220$ and $213$ videos for training and testing with
temporal annotations in $20$ classes. Action instances are rather sparsely
distributed through the videos and about $70\%$ of all frames are labeled as
background. The number of action instances per video on average is $15.5$ (and
$1.1$ for distinct action instances). Also, maximum number of distinct actions
per video is 3.

$\bullet$ MultiTHUMOS [57] has the same set of videos as in THUMOS14 [1], but
it extends the latter from $20$ action classes with $0.3$ labels per frame to
$65$ classes with $1.5$ labels per frame. Also, the average number of distinct
action classes in a video is $10.5$ (compared to $1.1$ in THUMOS14), making it
a more challenging multi-label dataset. Also, maximum number of distinct
actions per video is 25.

$\bullet$ ActivityNet [133] has two versions, v1.2 and v1.3. The former
contains $9,682$ videos in $100$ classes, while the latter, which is a
superset of v1.2 and was used in the ActivityNet Challenge 2016, contains
$19,994$ videos in $200$ classes. In each version, the dataset is divided into
three disjoint subsets, training, validation, and testing, by 2:1:1.

$\bullet$ HACS [134] includes $504K$ untrimmed videos retrieved from YouTube
where each one is strictly shorter than $4$ minutes. HACS clips consists of
$1.5M$ annotated clips of 2-second duration and HACS Segments contains $139K$
action segments densely annotated in $50K$ untrimmed videos spanning $200$
action categories.

$\bullet$ CHARADES [135] consists of $9,848$ videos recorded by Amazon
Mechanical Turk users based on provided scripts. This dataset contains videos
with multiple actions and involves daily life activities from $157$ classes of
$267$ people from three continents. Over $15\%$ of the videos have more than
one person.

$\bullet$ Breakfast [11] includes $1712$ videos for breakfast preparation
activities performed by $52$ subjects. The videos were recorded in $18$
different kitchens and belong to $10$ different types of breakfast activities
(such as fried egg or coffee) which consist of $48$ different fine-grained
actions. Each video contains $6$ action instances on average and only $7\%$ of
the frames are background.

$\bullet$ 50Salads [136] contains $50$ videos for salad preparation activities
performed by $25$ subjects and with $17$ distinct action classes. On average,
each video contains $20$ action instances and is $6.4$ minutes long.

$\bullet$ MPII Cooking 2 [137] consists of $273$ videos with about $2.8$
million frames. There are $59$ action classes and about $29\%$ of the frames
are background. The dataset provides a fixed split into a train and test set,
separating $220$ videos for training.

$\bullet$ COIN dataset [138],[144] contains $180$ tasks and $11,827$ videos
and $46,354$ annotated segments. The videos are collected from YouTube in $12$
domains (e.g., vehicles, gadgets, etc.) related to daily activities.

$\bullet$ AVA [86] is designed for spatio-temporal action detection and
consists of $437$ videos where each video is a $15$ minute segment taken from
a movie. Each person appearing in a test video must be detected in each frame
and the multi-label actions of the detected person must be predicted
correctly. The action label space contains $80$ atomic action classes but
often the results are reported on the most frequent $60$ classes.

###  3.2 Evaluation Metrics

Here, we discuss the metrics designed to evaluate the performance of proposal
generation, temporal action detection, and spatio-temporal action detection.

Temporal Action Proposal Generation. For this task, Average Recall (AR) with
multiple IoU thresholds is usually used as evaluation metrics. Most methods
use IoU thresholds set $[0.5$ : $0.05$ : $0.95]$ in ActivityNet-1.3 [133] and
$[0.5:0.05:1.0]$ in THUMOS14 [1]. To evaluate the relation between recall and
proposals number, most methods evaluate AR with Average Number of proposals
(AN) on both datasets, which is denoted as AR@AN. On ActivityNet-1.3, area
under the AR vs. AN curve (AUC) is also used as metrics, where AN varies from
$0$ to $100$.

Temporal Action Detection. For this task, mean Average Precision (mAP) is used
as evaluation metric, where Average Precision (AP) is calculated on each
action class, respectively. On ActivityNet-1.3 [133], mAP with IoU thresholds
$\\{0.5,0.75,0.95\\}$ and average mAP with IoU thresholds set
$[0.5:0.05:0.95]$ are often used. On THUMOS14[1], mAP with IoU thresholds
$\\{0.3,0.4,0.5,0.6,0.7\\}$ is used.

Spatio-temporal Action Detection. Two metrics are frequently used for this
task. First, frame-AP measures the area under the precision-recall curve of
the detections for each frame. A detection is correct if the intersection-
over-union with the ground truth at that frame is greater than a threshold and
the action label is correctly predicted. Second, video-AP measures the area
under the precision-recall curve of the action tubes predictions. A tube is
correct if the mean per frame intersection-over-union with the ground truth
across the frames of the video is greater than a threshold and the action
label is correctly predicted.

###  3.3 Performance Analysis

Action detection results of the state-of-the-art methods on THUMOS14 [1] and
ActivityNet [133] dataset are compared by mAP (%) in Tables III and IV
respectively. The methods are categorized to fully-supervised, weakly-
supervised, semi-supervised, self-supervised and US (unsupervised). We also
summarize the advantageous and limitations of fully-supervised methods and
methods with limited supervision in Tables V and VI.

TABLE III: Action detection results of the-state-of-the-art on testing set of
THUMOS-14, measured by mAP (%) at tIoU thresholds.

Supervision | Method | 0.3 | 0.4 | 0.5 | 0.6 | 0.7
---|---|---|---|---|---|---
Fully supervised | Yeung et al. [56] | 36.0 | 26.4 | 17.1 | - | -
SMS [145] | 36.5 | 27.8 | 17.8 | - | -
SCNN [19] | 36.3 | 28.7 | 19 | - | -
Sst [53] | - | - | 23.0 | - | -
CDC [14] | 40.1 | 29.4 | 23.3 | 13.1 | 7.9
SSAD [35] | 43 | 35 | 24.6 | - | -
TCN [146] | - | 33.3 | 25.6 | 15.9 | 9.0
TURN TAP [27] | 44.1 | 34.9 | 25.6 | - | -
R-C3D [30] | 44.8 | 35.6 | 28.9 | - | -
SS-TAD [54] | 45.7 | - | 29.2 | - | 9.6
SSN [18] | 51.9 | 41.0 | 29.8 | - | -
CTAP [52] | - | - | 29.9 | - | -
CBR [28] | 50.1 | 41.3 | 31.0 | 19.1 | 9.9
S3D[36] | 47.9 | 41.2 | 32.6 | 23.3 | 14.3
DBS [15] | 50.6 | 43.1 | 34.3 | 24.4 | 14.7
BSN [46] | 53.5 | 45.0 | 36.9 | 28.4 | 20.0
MGG [40] | 53.9 | 46.8 | 37.4 | 29.5 | 21.3
AGCN [31] | 57.1 | 51.6 | 38.6 | 28.9 | 17.0
GTAN [147] | 57.8 | 47.2 | 38.8 | - | -
BMN [47] | 56.0 | 47.4 | 38.8 | 29.7 | 20.5
SRG[148] | 54.5 | 46.9 | 39.1 | 31.4 | 22.2
DBG [48] | 57.8 | 49.4 | 39.8 | 30.2 | 21.7
G-TAD [62] | 54.5 | 47.6 | 40.2 | 30.8 | 23.4
BC-GNN [49] | 57.1 | 49.1 | 40.4 | 31.2 | 23.1
BSN++ [149] | 59.9 | 49.5 | 41.3 | 31.9 | 22.8
TAL-Net [13] | 53.2 | 48.5 | 42.8 | 33.8 | 20.8
TSA-Net [33] | 55.8 | 52.0 | 44.1 | 33.0 | 21.8
BU [150] | 53.9 | 50.7 | 45.4 | 38.0 | 28.5
A2Net [50] | 58.6 | 54.1 | 45.5 | 32.5 | 17.2
ATAG [63] | 62.0 | 53.1 | 47.3 | 38.0 | 28.0
Lianli et al. [151] | 66.4 | 58.4 | 48.8 | 36.7 | 25.5
PGCN [61] | 63.6 | 57.8 | 49.1 | - | -
| TadTR [152] | 62.4 | 57.4 | 49.2 | 37.8 | 26.3
| AFNet [32] | 63.4 | 58.5 | 49.5 | 36.9 | 23.5
| AGT [65] | 65.0 | 58.1 | 50.2 |  |
| PBRNet [153] | 58.5 | 54.6 | 51.3 | 41.8 | 29.5
| RTD-Net[66] | 68.3 | 62.3 | 51.9 | 38.8 | 23.7
| C-TCN [43] | 68.0 | 62.3 | 52.1 | - | -
| VSGN [64] | 66.7 | 60.4 | 52.4 | 41.0 | 30.4
| MLTPN [44] | 66.0 | 62.6 | 53.3 | 37.0 | 21.2
| TSP [154] | 69.1 | 63.3 | 53.5 | 40.4 | 26.0
| DaoTAD [155] | 62.8 | 59.5 | 53.8 | 43.6 | 30.1
| AFSD [51] | 67.3 | 62.4 | 55.5 | 43.7 | 31.1
| SP-TAD [156] | 69.2 | 63.3 | 55.9 | 45.7 | 33.4
| Liu et al.[157] | 68.9 | 64.0 | 56.9 | 46.3 | 31.0
Weakly supervised | Hide-Seek [123] | 19.5 | 12.7 | 6.8 | - | -
UNet [104] | 28.2 | 21.1 | 13.7 | - | -
Step-by-step [124] | 31.1 | 22.5 | 15.9 | - | -
STPN [117] | 35.5 | 25.8 | 16.9 | 9.9 | 4.3
MAAN [119] | 41.1 | 30.6 | 20.3 | 12 | 6.9
AutoLoc [114] | 35.8 | 29 | 21.2 | 13.4 | 5.8
W-TALC [105] | 40.1 | 31.1 | 22.8 | - | 7.6
STAR [158] | 48.7 | 34.7 | 23 | - | -
CMCS [120] | 41.2 | 32.1 | 23.1 | 15 | 7
AdapNet [159] | 41.09 | 31.61 | 23.65 | 14.53 | 7.75
Cleannet [115] | 37 | 30.9 | 23.9 | 13.9 | 7.1
TSM [160] | 39.5 | 31.9 | 24.5 | 13.8 | 7.1
3C-Net [102] | 40.9 | 32.3 | 24.6 | - | 7.7
Shen et al [161] | 44 | 34.4 | 25.5 | 15.2 | 7.2
Action Graphs [108] | 47.3 | 36.4 | 26.1 | - | -
BG modeling [113] | 46.6 | 37.5 | 26.8 | 17.6 | 9
BaSNet [118] | 44.6 | 36 | 27 | 18.6 | 10.4
RPN [112] | 48.2 | 37.2 | 27.9 | 16.7 | 8.1
TSCN [162] | 47.8 | 37.7 | 28.7 | 19.4 | 10.2
DGAM [116] | 46.8 | 38.2 | 28.8 | 19.8 | 11.4
ECM [163] | 46.5 | 38.2 | 29.1 | 19.5 | 10.9
Deep Metric [106] | 46.8 | - | 29.6 | - | 9.7
A2CL-PT [164] | 48.1 | 39.0 | 30.1 | 19.2 | 10.6
EM-MIL [165] | 45.5 | 36.8 | 30.5 | 22.7 | 16.4
Lee et al [166] | 46.9 | 39.2 | 30.7 | 20.8 | 12.5
| ASL [167] | 51.8 | - | 31.1 | - | 11.4
| Huang et al [168] | 49.1 | 40.0 | 31.4 | 18.8 | 10.6
| Ding et al [169] | 48.2 | 39.7 | 31.6 | 22.0 | 13.8
| CoLA [170] | 51.5 | 41.9 | 32.2 | 22.0 | 13.1
| Acsnet [171] | 51.4 | 42.7 | 32.4 | 22.0 | 11.7
| Lee et al. [172] | 52.3 | 43.4 | 33.7 | 22.9 | 12.1
| ACM-Net [173] | 55.0 | 44.6 | 34.6 | 21.8 | 10.8
| D2-Net [174] | 52.3 | 43.4 | 36.0 | - | -
Semi supervised | TTC-Loc [175] | 52.8 | 44.4 | 35.9 | 24.7 | 13.8
Ji et al [139] | 53.4 | 45.2 | 37.2 | 29.5 | 20.5
Self supervised | Actionbytes [109] | 43.0 | 35.8 | 29.0 | - | 9.5
Gong et al. [176] | 50.8 | 42.2 | 32.9 | 21.0 | 10.1
US | ACL [129] | 39.6 | 32.9 | 25.0 | 16.7 | 8.9

TABLE IV: Action detection results of the-state-of-the-art on validation set
of ActivityNet (V is the version), measured by mAP (%) at tIoU thresholds.
$\star$ indicates utilization of weaker feature extractor (UNet [104]).

Supervision | Method | V | 0.5 | 0.75 | 0.95 | Average
---|---|---|---|---|---|---
Fully supervised | R-C3D [30] | 1.3 | 26.8 | - | - | 12.7
AFNet [32] | 1.3 | 36.1 | 17.8 | 5.2 | 18.6
TAL-Net [13] | 1.3 | 38.23 | 18.30 | 1.30 | 20.22
TCN [146] | 1.3 | 37.49 | 23.47 | 4.47 | 23.58
CDC [14] | 1.3 | 45.3 | 26.0 | 0.2 | 23.8
SSN [18] | 1.3 | 39.12 | 23.48 | 5.49 | 23.98
DBS [15] | 1.3 | 43.2 | 25.8 | 6.1 | 26.1
A2Net [50] | 1.3 | 43.55 | 28.69 | 3.7 | 27.75
MLTPN [44] | 1.3 | 44.86 | 28.96 | 4.30 | 28.27
SRG[148] | 1.3 | 46.53 | 29.98 | 4.83 | 29.72
BSN [46] | 1.3 | 46.45 | 29.96 | 8.02 | 30.03
BU [150] | 1.3 | 43.47 | 33.91 | 9.21 | 30.12
AGCN [31] | 1.3 | - | - | - | 30.4
RTD-Net[66] | 1.3 | 47.21 | 30.68 | 8.61 | 30.83
Lianli et al. [151] | 1.3 | 47.01 | 30.52 | 8.21 | 30.88
C-TCN [43] | 1.3 | 47.6 | 31.9 | 6.2 | 31.1
PGCN [61] | 1.3 | 48.26 | 33.16 | 3.27 | 31.11
| TadTR [152] | 1.3 | 49.08 | 32.58 | 8.49 | 32.27
| SP-TAD [156] | 1.3 | 50.06 | 32.92 | 8.44 | 32.99
| BMN [47] | 1.3 | 50.07 | 34.78 | 8.29 | 33.85
| Liu et al.[157] | 1.3 | 50.02 | 34.97 | 6.57 | 33.99
| G-TAD [62] | 1.3 | 50.36 | 34.60 | 9.02 | 34.09
| BC-GNN [49] | 1.3 | 50.56 | 34.75 | 9.37 | 34.26
| GTAN [147] | 1.3 | 52.61 | 34.14 | 8.91 | 34.31
| AFSD [51] | 1.3 | 52.4 | 35.3 | 6.5 | 34.4
| ATAG [63] | 1.3 | 50.92 | 35.35 | 9.71 | 34.68
| BSN++ [149] | 1.3 | 51.27 | 35.70 | 8.33 | 34.88
| PBRNet [153] | 1.3 | 53.96 | 34.97 | 8.98 | 35.01
| VSGN [64] | 1.3 | 52.38 | 36.01 | 8.37 | 35.07
| TSP [154] | 1.3 | 51.26 | 37.12 | 9.29 | 35.81
Weakly supervised (V=1.2) | UNet⋆ [104] | 1.2 | 7.4 | 3.2 | 0.7 | 3.6
Step-by-step [124] | 1.2 | 27.3 | 14.7 | 2.9 | 15.6
AutoLoc⋆ [114] | 1.2 | 27.3 | 15.1 | 3.3 | 16.0
TSM [160] | 1.2 | 28.3 | 17.0 | 3.5 | 17.1
Action Graphs [108] | 1.2 | 29.4 | - | - | -
W-TALC [105] | 1.2 | 37.0 |  |  | 18.0
EM-MIL [165] | 1.2 | 37.4 | - | - | 20.3
Cleannet [115] | 1.2 | 37.1 | 20.3 | 5.0 | 21.6
3C-Net [102] | 1.2 | 37.2 | - | - | 21.7
Deep Metric [106] | 1.2 | 35.2 | - | - | -
CMCS [120] | 1.2 | 36.8 | 22.0 | 5.6 | 22.4
Shen et al [161] | 1.2 | 36.9 | 23.1 | 3.4 | 22.8
RPN [112] | 1.2 | 37.6 | 23.9 | 5.4 | 23.3
TSCN [162] | 1.2 | 37.6 | 23.7 | 5.7 | 23.6
BaSNet [118] | 1.2 | 38.5 | 24.2 | 5.6 | 24.3
DGAM [116] | 1.2 | 41.0 | 23.5 | 5.3 | 24.4
Acsnet [171] | 1.2 | 41.0 | 23.5 | 5.3 | 24.4
| ECM [163] | 1.2 | 41.0 | 24.9 | 6.5 | 25.5
| ASL [167] | 1.2 | 40.2 | - | - | 25.8
| Lee et al. [172] | 1.2 | 41.2 | 25.6 | 6.0 | 25.9
| D2-Net [174] | 1.2 | 42.3 | 25.5 | 5.8 | 26.0
| CoLA [170] | 1.2 | 42.7 | 25.7 | 5.8 | 26.1
| Ding et al [169] | 1.2 | 41.7 | 26.7 | 6.3 | 26.4
Weakly supervised (V=1.3) | STPN [117] | 1.3 | 29.3 | 16.9 | 2.6 | -
STAR [158] | 1.3 | 31.1 | 18.8 | 4.7 | -
AdapNet [159] | 1.3 | 33.61 | 18.75 | 3.40 | 21.97
MAAN [119] | 1.3 | 33.7 | 21.9 | 5.5 | -
BG modeling [113] | 1.3 | 36.4 | 19.2 | 2.9 | -
A2CL-PT [164] | 1.3 | 36.8 | 22.0 | 5.2 | 22.5
Huang et al [168] | 1.3 | 36.5 | 22.8 | 6.0 | 22.9
| ACM-Net [173] | 1.3 | 40.1 | 24.2 | 6.2 | 24.6
Semi | TTC-Loc [175] | 1.2 | 40.6 | 3.6 | 5.3 | 24.5
Self supervised | Actionbytes [109] | 1.2 | 39.4 | - | - | -
Gong et al. [176] | 1.2 | 45.5 | 27.3 | 5.4 | 27.6
US | ACL [129] | 1.2 | 35.2 | 21.4 | 3.1 | 21.1

####  3.3.1 Fully-supervised Methods

Proposal Generation. Anchor-free methods such as SSN [18], BSN [46], BMN [47],
DBG [48], BC-GNN [49], BU [150], and BSN++ [149], A2Net [50] and AFSD [51]
achieved superior results compared with anchor-based methods such as Yeung et
al. [56], SMS [145], TCN [146], SCNN [19], TURN TAP [27], CBR [28], and CDC
[14]. This is because anchor-free methods generate temporal action proposals
with more flexibility and precise temporal boundaries. Some methods such as
CTAP[52], MGG[40], PBRNet [41], SRG[148], and RapNet [42] combine advantageous
of anchor-based and anchor-free methods and attained a higher results.

Proposal Feature Extraction. R-C3D [30] and AFNet [32] employ 3D RoI pooling
for feature extraction and obtained low results on ActivityNet due to lack of
receptive field alignment with proposal span. TAL-Net [13], TSA-Net [33]
employ a multi-tower network and achieve a higher performance compared with 3D
RoI pooling methods. The methods of SSAD [35], S3D[36], MGG [40], PBRNet
[153], MLTPN [44], C-TCN [43], RapNet [42], SP-TAD [156], and DaoTAD [155]
employ temporal feature pyramid to extract features from actions with
different duration and achieved superior performance.

Modeling Long-term Dependencies. Sst [53] and SS-TAD [54] which are RNN-based
methods achieve relatively lower results as they can not generate flexible
proposals. PGCN [61], G-TAD [62], BC-GNN [49], AGCN [31], ATAG [63], and VSGN
[64] are graph models that capture dependencies between proposals or video
segments. Among them VSGN [64] achieved the best performance by exploiting
correlations between cross-scale snippets (original and magnified) and
aggregating their features with a graph pyramid network. AGT [65], RTD-Net
[66], ATAG [63], and TadTR [152] use transformers to model long-range
dependencies. Among them RTD-Net [66] achieved the best results (on THUMOS14)
by customizing the encoder with a boundary-attentive architecture to enhance
the discrimination capability of action boundary.

There are also two state-of-the-art (SOTA) methods that do not belong to the
mentioned categories of methods. TSP [154] proposed a novel supervised
pretraining paradigm for clip features, and improved the performance of SOTA
using features trained with the proposed pretraining strategy. Liu et al. in
[157] leverages temporal aggregation to improve the feature discriminative
power of each snippet and enhance the feature coherence within a single
instance.

TABLE V: Summary of fully-supervised methods for temporal action detection.
$(+)$ and $(-)$ denote the advantages and disadvantages.

Objective | Category | Methods | Advantages and Limitations
---|---|---|---
Proposal Generation | Anchor-based |  SCNN [19], CBR[28], Turn-Tap[27], CDC [14] |  \+ Efficiently generate multiple-scales proposals, use global info of all anchors to generate reliable confidence scores.
\- Proposals are not temporally flexible and precise.
Anchor-free |  TAG [18], BSN [46], BMN [47], DBG [48] BC-GNN [49], BU [150] A2Net [50], AFSD [51] BSN++ [149] |  \+ Generate proposals with flexible duration.
\+ Global context for proposal evaluation ( in BMN, DBG).
\+ Global context for proposal generation (in DBG).
\- Proposal evaluation is not efficient in some cases.
\- Distorting the information of short actions due to down-scaling.
|  Anchor-based +Anchor-free |  CTAP[52], MGG[40] PBRNet [41], RapNet [42] |  \+ Combining advantageous of anchor-based and anchor-free.
| \- Not modeling long-range dependencies.
Proposal Feature Extraction | 3D RoI pooling | R-C3D [30], AFNet [32] |  \+ Fast feature extraction from multi-scale proposals.
\- Proposal features may include insufficient or irrelevant info because of
receptive field misalignment.
Multi-tower Network |  TAL-Net [13], TSA-Net [33] |  \+ Alignment of receptive field to proposal span to extract rich features from proposals.
\- Pre-defined temporal intervals limit the accuracy of proposals.
TFPN |  SSAD [35], S3D[36] MGG [40], C-TCN [43] MLTPN [44], PBRNet [41] A2Net [50], AFSD [51] RapNet [42], SP-TAD [156] DaoTAD [155] | \+ Feature pyramids to detect different scales of actions.
\+ Re-fine the proposal boundaries from coarse to fine (in MGG, PBRNet, and
RapNet).
\+ Combination with anchor-free pipeline for flexible and precise proposal
generation (A2Net, AFSD).
\- No modeling of temporal dependencies in most cases.
Modeling Long-term Dependencies | RNNs | Sst [53], SS-TAD [54] | \+ Modeling long-term dependencies for proposal generation.
\- Proposals are not flexible and precise.
Graphs |  PGCN [61], G-TAD [62], BC-GNN [49], AGCN [31] ATAG [63] , VSGN [64] |  \+ Modeling temporal dependencies between proposals or video segments for proposal generation and refinement.
\- Proposal generation is inefficient or temporal dependencies are used only
for proposal refinement.
Transformer |  AGT [65], RTD-Net [66] ATAG [63], TadTR [152] |  \+ Modeling non-linear temporal structure and inter-proposal relationships for proposal generation.
\- High parametric complexity.

####  3.3.2 Methods with Limited Supervision

Action Localization with Class-specific Attention. UNet [104] is supervised
with MIL loss which is not strong enough to predict accurate attention scores.
The methods of W-TALC [105], Action Graphs [108], and Deep Metric [106] all
target action-action separation by employing a co-activity similarity loss.
3C-Net [102] applied center loss on video-level aggregated features to enhance
feature discriminability. Deep Metric [106] outperforms W-TALC [105], Action
Graphs [108], and 3C-Net [102] by defining a class-specific metric for each
action category.

Action Localization with Class-agnostic Attention. STPN [117] proposed to
learn attention through class-agnostic features but has a low performance as
cross entropy loss alone does not train accurate attention signals. BG
modeling [113] used a clustering loss to separate action from background. BG
modeling [113] and BaSNet [118] force all background frames to belong to one
specific class which is not desirable as they do not share any common
semantics. RPN [112], and Huang et al. [168] increase inter-class separateness
by pushing action (or sub-action) features to their prototypes. Huang et al.
[168] outperforms RPN [112] by modeling the relations between sub-actions of
each action. DGAM [116] addressed the action-context confusion through
imposing different attentions on different features with a generative model.
EM-MIL [165] employed Expectation-Maximization to capture complete action
instances and outperformed DGAM [116] on THUMOS14 dataset.

Direct Action Localization. AutoLoc [114], and CleanNet [115] regress the
intervals of action instances for proposal generation, instead of performing
hard thresholding. They obtained a lower performance compared with most recent
methods as they do not model action completeness or address action-context
confusion.

Action Completeness Modeling. The methods of CMCS [120], Hide-and-Seek [123],
and Step-by-step [124] target the action completeness and CMCS [120] achieves
a superior performance. This is because Hide-and-Seek [123] and Step-by-step
[124] do not guarantee the discovery of new parts by randomly hiding or
removing different video regions. In contrary, CMCS [120] employs a diversity
loss to enforce the model to discover complementary action parts.

ACL [129] is an unsupervised method and only uses the total count of unique
actions that appear in the video, but it still achieves a comparable
performance with respect to some of the weakly-supervised methods such as
3C-Net [102]. Gong et al. [176] is a self-supervised method that attained the
state-of-the-art results on ActivityNet-1.2 among methods with limited
supervision, confirming the advantageous of self-supervised learning. The
recent state-of-the-art weakly supervised methods such as D2-Net [174]
achieved comparable performance to the semi-supervised methods of Ji et al
[139] and TTC-Loc [175]. This is interesting specially because D2-Net [174]
does not use temporal annotation of actions at all while Ji et al [139] and
TTC-Loc [175] use temporal annotations at least for a small percentage of
videos in the dataset.

TABLE VI: Summary of methods with limited supervision for temporal action
detection. $(+)$ and $(-)$ denote the advantages and disadvantages.

Objective | Category | Method | Advantages and Limitations
---|---|---|---
Localization with Class-specific Attention | MIL Loss |  UNet [104], W-TALC [105] Action Graphs [108], BaSNet [118] 3C-Net [102], Actionbytes [109] | \+ Learns temporal class activation maps.
\- MIL loss alone does not predict accurate attention scores.
\- Only supervising temporal positions with highest activation scores.
Co-activity Similarity Loss (CASL) |  W-TALC[105], Action Graphs [108] DM[106], Actionbytes [109] |  \+ Action-background separation and reducing intra-class variations.
\- Action-context confusion is not addressed.
\- Not modeling action completeness.
Center Loss |  3C-Net [102] |  \+ Reduce intra-class variations by pushing action features to class centers.
|  \- Imprecise attention signal, supervises video-level aggregated features.
Localization with Class-agnostic Attention | CE Loss |  STPN [117], RPN [112] BG modeling [113] | \+ Learns attention through class-agnostic features.
\- CE loss alone does not train accurate attention signals.
Clustering Loss |  RPN [112] BG modeling [113] | \+ Separating foreground-background features.
\- Force all background frames to belong to one specific class, but they do
not share any common semantics.
Prototype Learning |  RPN [112] Huang et al. [168] |  \+ Inter-class separateness by pushing action (or sub-action) features to their prototypes.
\- Action-context confusion is not addressed.
| Generative Model |  DGAM [116] EM-MIL [165] |  \+ Conditional VAE / Expectation-Maximization to separate actions from context frames and capture complete action instances.
| \- Not modeling temporal dependencies and relation between sub-actions.
Direct Localization |  Action-boundary Contrast |  AutoLoc [114] CleanNet [115] |  \+ Regress the intervals of action instances for proposal generation, instead of performing hard thresholding.
\- Not modeling action completeness.
Action Completeness Modeling |  Masking |  Hide-and-seek [123] Step-by-step [124] |  \+ Randomly hiding or removing different video regions to see different action parts.
\- Does not guarantee the discovery of new parts.
Diversity Loss |  CMCS [120] |  \+ Enforcing the model to discover complementary pieces of an action.
| \- Not modeling the relation between sub-actions.

##  4 Discussions

In this section, we describe the application of temporal action detection in
real-world applications and introduce several directions for future work in
this domain.

###  4.1 Applications

Temporal action detection has numerous real-world applications as most of the
videos in practice are untrimmed with a sparse set of actions. In this
section, we describe several applications such as understanding instructional
videos, anomaly detection in surveillance videos, action spotting in sports,
and detection in self-driving cars.

####  4.1.1 Action Localization in Instructional videos

With the rising popularity of social media and video sharing sites such as
YouTube, people worldwide upload numerous instructional videos in diverse
categories. Millions of people watch these tutorials to learn new tasks such
as ”making pancakes” or ”changing a flat tire.” Analysis of the instructional
videos has drawn more attention in recent years, leading to the proposition of
several tasks including step localization and action segmentation [177]. Based
on the psychological studies, it has been shown that simplifying and
segmenting the video into smaller steps (sub-actions) is a more effective way
to learn a new task [144, 178]. For example, the task of ”making pancakes” can
be segmented to action steps such as ”add the eggs,” ”pour the mixture into
the pan,” ”heat a frying pan,” and such. Many datasets are designed to study
action localization and action anticipation such as EPIC-Kitchen [179] and
INRIA Instructional Videos Dataset [180]. Both of these tasks (step
localization and action segmentation) are directly related to action
detection. Step localization is the task of localizing the start and endpoints
of a series of steps and recognizing their labels while action segmentation is
the frame-level labeling.

####  4.1.2 Anomaly Detection in Surveillance Videos

Surveillance cameras are increasingly deployed in public places, monitoring
the areas of interest to ensure security. With the stream of data from these
video cameras, there has been a rise in video analysis and anomaly detection
research. Anomalies are significant deviations of scene entities from normal
behavior [181, 182]. Fighting, traffic accidents, burglary, and robbery are a
few examples of anomalies. Compared to normal activities, anomalous events
rarely occur. Therefore, intelligent computer vision algorithms are required
to detect anomalous events automatically, to avoid the waste of time and
labor. In some methods, anomaly detection models are trained with normal
behaviors to learn distributions of normal patterns. These models identify
anomalous activities based on dissimilarity to the standard data distributions
[183, 184]. In other cases, normal and anomalous videos are used during
training to automatically predict high anomaly scores [185, 186]. In many
real-time applications, the system must detect anomalous events as soon as
each video frame arrives, only based on history and the current data; for
instance, an intelligent video surveillance application designed to raise the
alarm when suspicious activity is detected. To this end, online action
detection algorithms are developed to accumulate historical observations and
predicted future information to analyze current events [187], [188], [189],
[190],[191].

####  4.1.3 Action Spotting in Sports

Professional analysts utilize sports videos to investigate the strategies in a
game, examine new players, and generate meaningful statistics. In order to
analyze the videos, they watch many broadcasts to spot the highlights within a
game, which is a time-consuming and costly process. Fortunately, automated
sports analytic methods developed in the computer vision field can facilitate
sports broadcasts understanding. In recent years, many automated methods have
been proposed to help localize the salient actions of a game. They produce
statistics of events within a game by either analyzing camera shots or
semantic information. Human activity localization in sports videos is studied
in [192, 193, 194, 195], salient game actions are identified in [196, 197],
automatic game highlights identification and summarization are performed in
[198, 199, 200, 201, 202]. Moreover, action spotting, which is the task of
temporal localization of human-induced events, has been popular in soccer game
broadcasts [3, 203] and some methods aimed to automatically detect goals,
penalties, corner kicks, and card events [204]. Action detection algorithms
can inspire many of the tasks mentioned above.

####  4.1.4 Action Detection in Autonomous Driving

With the rapid development and advancement of cars and other vehicles in urban
transportation, autonomous driving has attracted more attention in the last
decades. The cameras assembled on the self-driving cars capture the real-time
stream of videos that need to be processed with online algorithms. The car
should be aware of the surrounding environment and spot road users, including
pedestrians, cyclists, and other vehicles, to make safe autonomous decisions.
Also, it should be able to detect and anticipate road users activities such as
moving away, moving towards, crossing the road, and anomalous events in real-
time to adjust the speed and handle the situation. Therefore, spatio-temporal
action localization algorithms need to be developed to guarantee the safety of
self-driving cars [205]. Yao et al. [206] proposed a traffic anomaly detection
with a when-where-what pipeline to detect, localize, and recognize anomalous
events from egocentric videos. To improve the detection and prediction of
pedestrian movements, Rasouli et al. [4] studied pedestrian behavior depending
on various factors, such as demographics of the pedestrians, traffic dynamics,
and environmental conditions. Moreover, Mahadevan et al. [207] proposed an
immersive VR-based pedestrian mixed traffic simulator to examine pedestrian
behavior in street crossing tasks.

###  4.2 Future work

Weakly-supervised action localization in untrimmed videos has drawn much
research attention by providing only video-level labels during training
instead of exhaustive annotation of temporal boundaries in the training phase.
Subsequently, knowledge transfer from publicly available trimmed videos is a
promising trend to make up for the coarse-grained video-level annotations in
weakly-supervised settings. Nevertheless, domain-adaptation schemes must
fulfill the domain gap between trimmed and untrimmed videos to transfer robust
and reliable knowledge. Only a few methods have explored knowledge transfer
from trimmed videos [109], [159], [208], [209], but we expect to see more in
the future.

In recent years, zero-shot learning (ZSL) in the visual recognition domain has
been emerging as a rising trend as it is challenging to collect a large number
of samples for each class during training. ZSL works by transferring the
knowledge from the seen classes with sufficiently many instances to generalize
the models on unseen classes with no samples during training. The task of
zero-shot temporal activity detection (ZSTAD) is introduced in [210] to
generalize the applicability of action detection methods to newly emerging or
rare events that are not included in the training set. The task of ZSTAD is
highly challenging because each untrimmed video in the testing set possibly
contains multiple novel action classes that must be localized and detected. It
is worth mentioning that activity detection with few-shot learning has been
recently explored in [109], [211], [212], [213], [214], [215]. The advancement
of both zero-shot and few-shot action detection is anticipated in the near
future.

##  5 Conclusion

Action detection schemes have expedited the progress in many real-world
applications such as instructional video analysis, anomaly detection in
surveillance videos, sports analysis, and autonomous driving. The advancement
of learning methods with limited supervision has facilitated action detection
by detachment from costly need to annotate the temporal boundary of actions in
long videos. This survey has extensively studied recently developed deep
learning-based methods for action detection from different aspects including
fully-supervised schemes, methods with limited supervision, benchmark
datasets, performance analysis, applications, and future directions. The
performance analysis and future directions are summarized to inspire the
design of new and efficient methods for action detection that serves the
computer vision community.

## References

  * [1] Y.-G. Jiang, J. Liu, A. R. Zamir, G. Toderici, I. Laptev, M. Shah, and R. Sukthankar, “Thumos challenge: Action recognition with a large number of classes,” 2014.
  * [2] F. Rea, A. Vignolo, A. Sciutti, and N. Noceti, “Human motion understanding for selecting action timing in collaborative human-robot interaction,” _Front. Robot. AI_ , vol. 6, p. 58, 2019.
  * [3] A. Cioppa, A. Deliege, S. Giancola, B. Ghanem, M. V. Droogenbroeck, R. Gade, and T. B. Moeslund, “A context-aware loss function for action spotting in soccer videos,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 13 126–13 136.
  * [4] A. Rasouli and J. K. Tsotsos, “Autonomous vehicles that interact with pedestrians: A survey of theory and practice,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 21, no. 3, pp. 900–918, 2019.
  * [5] S. Herath, M. Harandi, and F. Porikli, “Going deeper into action recognition: A survey,” _Image and vision computing_ , vol. 60, pp. 4–21, 2017.
  * [6] C. Feichtenhofer, H. Fan, J. Malik, and K. He, “Slowfast networks for video recognition,” in _Proceedings of the IEEE international conference on computer vision_ , 2019, pp. 6202–6211.
  * [7] D. Ghadiyaram, D. Tran, and D. Mahajan, “Large-scale weakly-supervised pre-training for video action recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 12 046–12 055.
  * [8] H. Duan, Y. Zhao, Y. Xiong, W. Liu, and D. Lin, “Omni-sourced webly-supervised learning for video recognition,” _arXiv preprint arXiv:2003.13042_ , 2020\.
  * [9] H. Idrees, A. R. Zamir, Y.-G. Jiang, A. Gorban, I. Laptev, R. Sukthankar, and M. Shah, “The thumos challenge on action recognition for videos “in the wild”,” _Computer Vision and Image Understanding_ , vol. 155, pp. 1–23, 2017.
  * [10] A. Gorban, H. Idrees, Y.-G. Jiang, A. R. Zamir, I. Laptev, M. Shah, and R. Sukthankar, “Thumos challenge: Action recognition with a large number of classes,” 2015.
  * [11] H. Kuehne, A. Arslan, and T. Serre, “The language of actions: Recovering the syntax and semantics of goal-directed human activities,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2014, pp. 780–787.
  * [12] C. Lea, M. D. Flynn, R. Vidal, A. Reiter, and G. D. Hager, “Temporal convolutional networks for action segmentation and detection,” in _proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 156–165.
  * [13] Y.-W. Chao, S. Vijayanarasimhan, B. Seybold, D. A. Ross, J. Deng, and R. Sukthankar, “Rethinking the faster r-cnn architecture for temporal action localization,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 1130–1139.
  * [14] Z. Shou, J. Chan, A. Zareian, K. Miyazawa, and S.-F. Chang, “Cdc: Convolutional-de-convolutional networks for precise temporal action localization in untrimmed videos,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 5734–5743.
  * [15] Z. Gao, L. Wang, Q. Zhang, Z. Niu, N. Zheng, and G. Hua, “Video imprint segmentation for temporal action detection in untrimmed videos,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 8328–8335.
  * [16] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3431–3440.
  * [17] M. Everingham, L. Van Gool, C. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes challenge 2012 (voc2012) results (2012),” in _URL http://www. pascal-network. org/challenges/VOC/voc2011/workshop/index. html_ , 2011.
  * [18] Y. Zhao, Y. Xiong, L. Wang, Z. Wu, X. Tang, and D. Lin, “Temporal action detection with structured segment networks,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 2914–2923.
  * [19] Z. Shou, D. Wang, and S.-F. Chang, “Temporal action localization in untrimmed videos via multi-stage cnns,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 1049–1058.
  * [20] Y. Xiong, L. Wang, Z. Wang, B. Zhang, H. Song, W. Li, D. Lin, Y. Qiao, L. Van Gool, and X. Tang, “Cuhk & ethz & siat submission to activitynet challenge 2016,” _arXiv preprint arXiv:1608.00797_ , 2016.
  * [21] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning spatiotemporal features with 3d convolutional networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 4489–4497.
  * [22] K. Simonyan and A. Zisserman, “Two-stream convolutional networks for action recognition in videos,” in _Advances in neural information processing systems_ , 2014, pp. 568–576.
  * [23] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. Van Gool, “Temporal segment networks: Towards good practices for deep action recognition,” in _European conference on computer vision_. Springer, 2016, pp. 20–36.
  * [24] J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” in _proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 6299–6308.
  * [25] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [26] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” _arXiv preprint arXiv:1502.03167_ , 2015.
  * [27] J. Gao, Z. Yang, K. Chen, C. Sun, and R. Nevatia, “Turn tap: Temporal unit regression network for temporal action proposals,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 3628–3636.
  * [28] J. Gao, Z. Yang, and R. Nevatia, “Cascaded boundary regression for temporal action detection,” _arXiv preprint arXiv:1705.01180_ , 2017.
  * [29] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in _Advances in neural information processing systems_ , 2015, pp. 91–99.
  * [30] H. Xu, A. Das, and K. Saenko, “R-c3d: Region convolutional 3d network for temporal activity detection,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 5783–5792.
  * [31] J. Li, X. Liu, Z. Zong, W. Zhao, M. Zhang, and J. Song, “Graph attention based proposal 3d convnets for action detection.” in _AAAI_ , 2020, pp. 4626–4633.
  * [32] G. Chen, C. Zhang, and Y. Zou, “Afnet: Temporal locality-aware network with dual structure for accurate and fast action detection,” _IEEE Transactions on Multimedia_ , 2020.
  * [33] G. Gong, L. Zheng, and Y. Mu, “Scale matters: Temporal scale aggregation network for precise action localization in untrimmed videos,” in _2020 IEEE International Conference on Multimedia and Expo (ICME)_. IEEE, 2020, pp. 1–6.
  * [34] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “Ssd: Single shot multibox detector,” in _European conference on computer vision_. Springer, 2016, pp. 21–37.
  * [35] T. Lin, X. Zhao, and Z. Shou, “Single shot temporal action detection,” in _Proceedings of the 25th ACM international conference on Multimedia_ , 2017, pp. 988–996.
  * [36] D. Zhang, X. Dai, X. Wang, and Y.-F. Wang, “S3d: single shot multi-span detector via fully 3d convolutional networks,” _arXiv preprint arXiv:1807.08069_ , 2018.
  * [37] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [38] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2117–2125.
  * [39] C.-Y. Fu, W. Liu, A. Ranga, A. Tyagi, and A. C. Berg, “Dssd: Deconvolutional single shot detector,” _arXiv preprint arXiv:1701.06659_ , 2017.
  * [40] Y. Liu, L. Ma, Y. Zhang, W. Liu, and S.-F. Chang, “Multi-granularity generator for temporal action proposal,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3604–3613.
  * [41] Q. Liu and Z. Wang, “Progressive boundary refinement network for temporal action detection.”
  * [42] J. Gao, Z. Shi, G. Wang, J. Li, Y. Yuan, S. Ge, and X. Zhou, “Accurate temporal action proposal generation with relation-aware pyramid network.” in _AAAI_ , 2020, pp. 10 810–10 817.
  * [43] X. Li, T. Lin, X. Liu, C. Gan, W. Zuo, C. Li, X. Long, D. He, F. Li, and S. Wen, “Deep concept-wise temporal convolutional networks for action localization,” _arXiv preprint arXiv:1908.09442_ , 2019.
  * [44] X. Wang, C. Gao, S. Zhang, and N. Sang, “Multi-level temporal pyramid network for action detection,” in _Chinese Conference on Pattern Recognition and Computer Vision (PRCV)_. Springer, 2020, pp. 41–54.
  * [45] J. B. Roerdink and A. Meijster, “The watershed transform: Definitions, algorithms and parallelization strategies,” _Fundamenta informaticae_ , vol. 41, no. 1, 2, pp. 187–228, 2000.
  * [46] T. Lin, X. Zhao, H. Su, C. Wang, and M. Yang, “Bsn: Boundary sensitive network for temporal action proposal generation,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 3–19.
  * [47] T. Lin, X. Liu, X. Li, E. Ding, and S. Wen, “Bmn: Boundary-matching network for temporal action proposal generation,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 3889–3898.
  * [48] C. Lin, J. Li, Y. Wang, Y. Tai, D. Luo, Z. Cui, C. Wang, J. Li, F. Huang, and R. Ji, “Fast learning of temporal action proposal via dense boundary generator.” in _AAAI_ , 2020, pp. 11 499–11 506.
  * [49] Y. Bai, Y. Wang, Y. Tong, Y. Yang, Q. Liu, and J. Liu, “Boundary content graph neural network for temporal action proposal generation,” _arXiv preprint arXiv:2008.01432_ , 2020.
  * [50] L. Yang, H. Peng, D. Zhang, J. Fu, and J. Han, “Revisiting anchor mechanisms for temporal action localization,” _IEEE Transactions on Image Processing_ , vol. 29, pp. 8535–8548, 2020.
  * [51] C. Lin, C. Xu, D. Luo, Y. Wang, Y. Tai, C. Wang, J. Li, F. Huang, and Y. Fu, “Learning salient boundary feature for anchor-free temporal action localization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 3320–3329.
  * [52] J. Gao, K. Chen, and R. Nevatia, “Ctap: Complementary temporal action proposal generation,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 68–83.
  * [53] S. Buch, V. Escorcia, C. Shen, B. Ghanem, and J. Carlos Niebles, “Sst: Single-stream temporal action proposals,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2911–2920.
  * [54] S. Buch, V. Escorcia, B. Ghanem, L. Fei-Fei, and J. C. Niebles, “End-to-end, single-stream temporal action detection in untrimmed videos,” 2019.
  * [55] J. Yuan, B. Ni, X. Yang, and A. A. Kassim, “Temporal action localization with pyramid of score distribution features,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 3093–3102.
  * [56] S. Yeung, O. Russakovsky, G. Mori, and L. Fei-Fei, “End-to-end learning of action detection from frame glimpses in videos,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 2678–2687.
  * [57] S. Yeung, O. Russakovsky, N. Jin, M. Andriluka, G. Mori, and L. Fei-Fei, “Every moment counts: Dense detailed labeling of actions in complex videos,” _International Journal of Computer Vision_ , vol. 126, no. 2-4, pp. 375–389, 2018.
  * [58] V. Escorcia, F. C. Heilbron, J. C. Niebles, and B. Ghanem, “Daps: Deep action proposals for action understanding,” in _European Conference on Computer Vision_. Springer, 2016, pp. 768–784.
  * [59] B. Singh, T. K. Marks, M. Jones, O. Tuzel, and M. Shao, “A multi-stream bi-directional recurrent neural network for fine-grained action detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 1961–1970.
  * [60] S. Ma, L. Sigal, and S. Sclaroff, “Learning activity progression in lstms for activity detection and early detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 1942–1950.
  * [61] R. Zeng, W. Huang, M. Tan, Y. Rong, P. Zhao, J. Huang, and C. Gan, “Graph convolutional networks for temporal action localization,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 7094–7103.
  * [62] M. Xu, C. Zhao, D. S. Rojas, A. Thabet, and B. Ghanem, “G-tad: Sub-graph localization for temporal action detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 10 156–10 165.
  * [63] S. Chang, P. Wang, F. Wang, H. Li, and J. Feng, “Augmented transformer with adaptive graph for temporal action proposal generation,” _arXiv preprint arXiv:2103.16024_ , 2021.
  * [64] C. Zhao, A. Thabet, and B. Ghanem, “Video self-stitching graph network for temporal action localization,” _arXiv preprint arXiv:2011.14598_ , 2020.
  * [65] M. Nawhal and G. Mori, “Activity graph transformer for temporal action localization,” _arXiv preprint arXiv:2101.08540_ , 2021.
  * [66] J. Tan, J. Tang, L. Wang, and G. Wu, “Relaxed transformer decoders for direct action proposal generation,” _arXiv preprint arXiv:2102.01894_ , 2021.
  * [67] I. Laptev and P. Pérez, “Retrieving actions in movies,” in _2007 IEEE 11th International Conference on Computer Vision_. IEEE, 2007, pp. 1–8.
  * [68] L. Cao, Z. Liu, and T. S. Huang, “Cross-dataset action detection,” in _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_. IEEE, 2010, pp. 1998–2005.
  * [69] M. Jain, J. Van Gemert, H. Jégou, P. Bouthemy, and C. G. Snoek, “Action localization with tubelets from motion,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2014, pp. 740–747.
  * [70] D. Oneata, J. Revaud, J. Verbeek, and C. Schmid, “Spatio-temporal object detection proposals,” in _European conference on computer vision_. Springer, 2014, pp. 737–752.
  * [71] W. Chen and J. J. Corso, “Action detection by implicit intentional motion clustering,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 3298–3306.
  * [72] J. C. Van Gemert, M. Jain, E. Gati, C. G. Snoek _et al._ , “Apt: Action localization proposals from dense trajectories.” in _BMVC_ , vol. 2, 2015, p. 4.
  * [73] M. M. Puscas, E. Sangineto, D. Culibrk, and N. Sebe, “Unsupervised tube extraction using transductive learning and dense trajectories,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 1653–1661.
  * [74] G. Gkioxari and J. Malik, “Finding action tubes,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 759–768.
  * [75] R. Hou, C. Chen, and M. Shah, “Tube convolutional neural network (t-cnn) for action detection in videos,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 5822–5831.
  * [76] X. Peng and C. Schmid, “Multi-region two-stream r-cnn for action detection,” in _European conference on computer vision_. Springer, 2016, pp. 744–759.
  * [77] G. Singh, S. Saha, M. Sapienza, P. H. Torr, and F. Cuzzolin, “Online real-time multiple spatiotemporal action localisation and prediction,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 3637–3646.
  * [78] S. Saha, G. Singh, M. Sapienza, P. H. Torr, and F. Cuzzolin, “Deep learning for detecting multiple space-time action tubes in videos,” _arXiv preprint arXiv:1608.01529_ , 2016.
  * [79] P. Weinzaepfel, Z. Harchaoui, and C. Schmid, “Learning to track for spatio-temporal action localization,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 3164–3172.
  * [80] Z. Yang, J. Gao, and R. Nevatia, “Spatio-temporal action detection with cascade proposal and location anticipation,” _arXiv preprint arXiv:1708.00042_ , 2017.
  * [81] Y. Ye, X. Yang, and Y. Tian, “Discovering spatio-temporal action tubes,” _Journal of Visual Communication and Image Representation_ , vol. 58, pp. 515–524, 2019.
  * [82] Z. Li, K. Gavrilyuk, E. Gavves, M. Jain, and C. G. Snoek, “Videolstm convolves, attends and flows for action recognition,” _Computer Vision and Image Understanding_ , vol. 166, pp. 41–50, 2018.
  * [83] L. Wang, Y. Qiao, X. Tang, and L. Van Gool, “Actionness estimation using hybrid fully convolutional networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 2708–2717.
  * [84] G. Yu and J. Yuan, “Fast action proposals for human action detection and search,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1302–1311.
  * [85] V. Kalogeiton, P. Weinzaepfel, V. Ferrari, and C. Schmid, “Action tubelet detector for spatio-temporal action localization,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 4405–4413.
  * [86] C. Gu, C. Sun, D. A. Ross, C. Vondrick, C. Pantofaru, Y. Li, S. Vijayanarasimhan, G. Toderici, S. Ricco, R. Sukthankar _et al._ , “Ava: A video dataset of spatio-temporally localized atomic visual actions,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 6047–6056.
  * [87] X. Yang, X. Yang, M.-Y. Liu, F. Xiao, L. S. Davis, and J. Kautz, “Step: Spatio-temporal progressive learning for video action detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 264–272.
  * [88] C.-Y. Wu, C. Feichtenhofer, H. Fan, K. He, P. Krahenbuhl, and R. Girshick, “Long-term feature banks for detailed video understanding,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 284–293.
  * [89] C. Sun, A. Shrivastava, C. Vondrick, K. Murphy, R. Sukthankar, and C. Schmid, “Actor-centric relation network,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 318–334.
  * [90] Y. Zhang, P. Tokmakov, M. Hebert, and C. Schmid, “A structured model for action detection,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 9975–9984.
  * [91] R. Girdhar, J. Carreira, C. Doersch, and A. Zisserman, “Video action transformer network,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 244–253.
  * [92] O. Ulutan, S. Rallapalli, M. Srivatsa, C. Torres, and B. Manjunath, “Actor conditioned attention maps for video action detection,” in _The IEEE Winter Conference on Applications of Computer Vision_ , 2020, pp. 527–536.
  * [93] M. Tomei, L. Baraldi, S. Calderara, S. Bronzin, and R. Cucchiara, “Stage: Spatio-temporal attention on graph entities for video action detection,” _arXiv preprint arXiv:1912.04316_ , 2019.
  * [94] J. Ji, R. Krishna, L. Fei-Fei, and J. C. Niebles, “Action genome: Actions as compositions of spatio-temporal scene graphs,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 10 236–10 247.
  * [95] J. Pan, S. Chen, Z. Shou, J. Shao, and H. Li, “Actor-context-actor relation network for spatio-temporal action localization,” _arXiv preprint arXiv:2006.07976_ , 2020.
  * [96] M. Tomei, L. Baraldi, S. Calderara, S. Bronzin, and R. Cucchiara, “Video action detection by learning graph-based spatio-temporal interactions,” _Computer Vision and Image Understanding_ , vol. 206, p. 103187, 2021.
  * [97] C. Sun, S. Shetty, R. Sukthankar, and R. Nevatia, “Temporal localization of fine-grained actions in videos by domain transfer from web images,” in _Proceedings of the 23rd ACM international conference on Multimedia_ , 2015, pp. 371–380.
  * [98] P. Bojanowski, R. Lajugie, F. Bach, I. Laptev, J. Ponce, C. Schmid, and J. Sivic, “Weakly supervised action labeling in videos under ordering constraints,” in _European Conference on Computer Vision_. Springer, 2014, pp. 628–643.
  * [99] D.-A. Huang, L. Fei-Fei, and J. C. Niebles, “Connectionist temporal modeling for weakly supervised action labeling,” in _European Conference on Computer Vision_. Springer, 2016, pp. 137–153.
  * [100] A. Richard, H. Kuehne, and J. Gall, “Weakly supervised action learning with rnn based fine-to-coarse modeling,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 754–763.
  * [101] H. Kuehne, A. Richard, and J. Gall, “Weakly supervised learning of actions from transcripts,” _Computer Vision and Image Understanding_ , vol. 163, pp. 78–89, 2017.
  * [102] S. Narayan, H. Cholakkal, F. S. Khan, and L. Shao, “3c-net: Category count and center loss for weakly-supervised action localization,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 8679–8687.
  * [103] J. Schroeter, K. Sidorov, and D. Marshall, “Weakly-supervised temporal localization via occurrence count learning,” _arXiv preprint arXiv:1905.07293_ , 2019.
  * [104] L. Wang, Y. Xiong, D. Lin, and L. Van Gool, “Untrimmednets for weakly supervised action recognition and detection,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2017, pp. 4325–4334.
  * [105] S. Paul, S. Roy, and A. K. Roy-Chowdhury, “W-talc: Weakly-supervised temporal activity localization and classification,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 563–579.
  * [106] A. Islam and R. Radke, “Weakly supervised temporal action localization using deep metric learning,” in _The IEEE Winter Conference on Applications of Computer Vision_ , 2020, pp. 547–556.
  * [107] M.-A. Carbonneau, V. Cheplygina, E. Granger, and G. Gagnon, “Multiple instance learning: A survey of problem characteristics and applications,” _Pattern Recognition_ , vol. 77, pp. 329–353, 2018.
  * [108] M. Rashid, H. Kjellstrom, and Y. J. Lee, “Action graphs: Weakly-supervised action localization with graph convolution networks,” in _The IEEE Winter Conference on Applications of Computer Vision_ , 2020, pp. 615–624.
  * [109] M. Jain, A. Ghodrati, and C. G. Snoek, “Actionbytes: Learning from trimmed videos to localize actions,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 1171–1180.
  * [110] M. Gao, Y. Zhou, R. Xu, R. Socher, and C. Xiong, “Woad: Weakly supervised online action detection in untrimmed videos,” _arXiv preprint arXiv:2006.03732_ , 2020.
  * [111] Y. Wen, K. Zhang, Z. Li, and Y. Qiao, “A discriminative feature learning approach for deep face recognition,” in _European conference on computer vision_. Springer, 2016, pp. 499–515.
  * [112] L. Huang, Y. Huang, W. Ouyang, L. Wang _et al._ , “Relational prototypical network for weakly supervised temporal action localization,” 2020.
  * [113] P. X. Nguyen, D. Ramanan, and C. C. Fowlkes, “Weakly-supervised action localization with background modeling,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 5502–5511.