# When Did It Happen? Duration-informed Temporal Localization of Narrated
Actions in Vlogs

Oana Ignat<EMAIL_ADDRESS>0000-0003-0272-5147 ,  Santiago Castro
<EMAIL_ADDRESS>,  Yuhang Zhou<EMAIL_ADDRESS>,  Jiajun Bao
<EMAIL_ADDRESS>,  Dandan Shan<EMAIL_ADDRESS>and  Rada Mihalcea
<EMAIL_ADDRESS>University of Michigan500 S State StAnn
ArborMichiganUSA48109

###### Abstract.

We consider the task of temporal human action localization in lifestyle vlogs.
We introduce a novel dataset consisting of manual annotations of temporal
localization for 13,000 narrated actions in 1,200 video clips. We present an
extensive analysis of this data, which allows us to better understand how the
language and visual modalities interact throughout the videos. We propose a
simple yet effective method to localize the narrated actions based on their
expected duration. Through several experiments and analyses, we show that our
method brings complementary information with respect to previous methods, and
leads to improvements over previous work for the task of temporal action
localization.

action temporal localization, action duration, vlogs, natural language
processing, video processing, multimodal processing

††copyright: acmcopyright††journal: TOMM††journalyear: 2021††journalvolume:
1††journalnumber: 1††article: 1††publicationmonth: 1††price: 15.00††doi:
10.1145/3495211††copyright: acmcopyright††journal: TOMM††journalyear:
2021††journalvolume: 1††journalnumber: 1††article: 1††publicationmonth:
1††price: 15.00††doi: 10.1145/3495211

##  1\. Introduction

Targetting the long-term goal of video understanding, recent years have
witnessed significant progress in the task of action localization, starting
with the localization of one action at a time in a short clip (Wang et al.,
2011) or in a longer untrimmed video (Lin et al., 2017), all the way to
localizing more complex natural language queries in videos (Gao et al., 2017;
Anne Hendricks et al., 2017a; Hendricks et al., 2018; Ge et al., 2019; Ghosh
et al., 2019), and recently to localizing complex natural language queries
extracted directly from transcripts in online videos (Tang et al., 2019;
Zhukov et al., 2019; Miech et al., 2019a).

Lifestyle vlogs represent a great challenge and opportunity for this task, as
they depict everyday actions in a complex setting. Unlike traditional action
datasets (Caba Heilbron et al., 2015; Sigurdsson et al., 2016; Abu-El-Haija et
al., 2016; Anne Hendricks et al., 2017a) or instructional video datasets
(Zhukov et al., 2019; Tang et al., 2019; Miech et al., 2019b), vlogs contain a
wide variety of actions that are more akin to real-life settings, such as
“grab my Kindle,” “do some reading,” or “chill out.”

Moreover, vlogs typically include transcripts with complex natural language
expressions, which allow us to find an alternative to the costly process of
manual annotations. Given the prevalence of vlogs in online platforms,
automatically extracting action names from their transcripts can lead to a
large-scale inexpensive action dataset. Previous work (Miech et al., 2019b)
relied on this technique to build very large datasets of video-action
mappings. However, previous work also found that the video and transcript are
often misaligned (Ignat et al., 2019; Miech et al., 2019a): in the best case,
there is a gap of a few seconds between the time when a person verbally
expresses the action and when it is visually illustrated.

Figure 1. Overview of the dataset (Ignat et al., 2019): distinguishing between
actions that are narrated by the vlogger but not visible in the video and
actions that are both narrated and visible in the video (underlined), with a
highlight on visible actions that represent the same activity (same color).
The arrows represent the temporal alignment between when the visible action is
narrated as well as the time it occurs in the video. Best viewed in color.

This paper addresses the task of temporal action localization in vlogs, and
makes three main contributions. First, we introduce a dataset of manual
annotations of temporal localization of actions that addresses new challenges
compared to other action localization datasets. Second, we present 2Seal – a
simple yet effective method that leverages both language and vision to
temporally localize actions, while also accounting for the expected duration
of the actions. Through extensive evaluations, we show that our proposed
method can be used along with existing models to improve their performance on
temporal action localization. Finally, we conduct an analysis of the results,
and gain insight into the role played by the different components, which
further suggests avenues for future work.

##  2\. Related Work

Learning connections between vision and language is crucial to many
applications. These applications include visual question answering (Yang et
al., 2015; Anderson et al., 2018; Lei et al., 2019), visual content retrieval
based on textual queries (Miech et al., 2019b; Mithun et al., 2018; Jiang et
al., 2007), image and video captioning (Donahue et al., 2015; You et al.,
2016; Anderson et al., 2018), video summarization with natural language
(Palaskar et al., 2019; Plummer et al., 2017), action detection (Carreira and
Zisserman, 2017; Lin et al., 2017; Girdhar et al., 2019), action temporal
localization in videos (Regneri et al., 2013; Gao et al., 2017; Ghosh et al.,
2019; Escorcia et al., 2019; Ge et al., 2019) and mapping text descriptions to
image or video content (Regneri et al., 2013; Wang et al., 2016; Lee et al.,
2018; Shi et al., 2019; Krishna et al., 2017).

#### Action Localization Datasets.

Action detection and localization algorithms evolve with the building of
complex datasets. From searching YouTube videos, given a set of predefined
actions (Heilbron et al., 2015; Abu-El-Haija et al., 2016; Carreira and
Zisserman, 2017), or filming in people’s homes who act based on a scenario
(Sigurdsson et al., 2016), these datasets capture the complexity of daily life
activities. However, because of the high annotation cost, these methods are
not scalable. Currently, the latest trend in the vision community is to search
for pre-defined tasks on WikiHow and collect their corresponding videos from
YouTube (Miech et al., 2019b; Tang et al., 2019; Zhukov et al., 2019). This
process is more efficient and guarantees that more relevant actions are shown
in the videos. Another technique for collecting human actions is to perform
implicit data gathering (Fouhey et al., 2018): instead of explicitly searching
for a pre-defined task, find routine videos that contain a broad range of
daily actions.

In our work, we use the data introduced in (Ignat et al., 2019) which
identifies if the actions mentioned in the transcripts are present (visible)
in the video. Although we use implicit data gathering as proposed in the past,
unlike Fouhey et al. (Fouhey et al., 2018), who focus on the visual
information (hand and object locations), we focus on routine videos that
contain rich audio descriptions of the actions being performed, and we use
this transcribed audio to extract actions.

#### Action Localization Methods.

Methods that reason over text and visual information do this by first
extracting the textual embeddings (Pennington et al., 2014; Levy and Goldberg,
2014; Devlin et al., 2019) and visual features (Tran et al., 2014; Carreira
and Zisserman, 2017) and then linearly mapping them to the same embedding
space (Anne Hendricks et al., 2017b, a; Gao et al., 2017; Ge et al., 2019).
This is usually computed using self and cross attention over the textual and
visual features. The visual features can be extracted with a convolutional
neural net as in (Anne Hendricks et al., 2017b; Yuan et al., 2018; Liu et al.,
2018) or from object bounding boxes (Lee et al., 2018). Recent work (Tan and
Bansal, 2019; Sun et al., 2019; Lu et al., 2019) builds on this approach by
combining the attention modules in a large scale Transformer architecture
(Vaswani et al., 2017). Their goal is to learn inter-modality and cross-
modality relationships that can be used in downstream tasks that require
complex reasoning about natural language grounded in visual data (Goyal et
al., 2016; Suhr et al., 2018; Hudson and Manning, 2019).

#### Instructional vs. Routine Videos.

Action localization methods are moving from using simple pre-defined action
labels (Carreira and Zisserman, 2017; Girdhar et al., 2019) to more complex
natural language action descriptions (Anne Hendricks et al., 2017b; Sigurdsson
et al., 2018; Miech et al., 2019b). Our goal is also to localize natural
language descriptions of actions in videos. An important difference between
our task and previous work is that the natural language descriptions come from
the people filming the actions.

Research work such as (Alayrac et al., 2016; Miech et al., 2019b) also take
advantage directly of the actions extracted from the transcripts, however
their videos are instructional videos. Instead of looking at instructional
videos, we choose a broader category: routine videos, which can contain
instructions, but are more focused on describing the typical day of a person.

Compared to instructional videos, routine videos contain a more diverse set of
activities, from waking up in the morning and taking a shower, to working out
and making a meal. This diversity of actions in one video translates to many
more diverse filming perspectives in the same video, which presents a novel
challenge for action localization models. Another difference is that routine
videos contain higher-level actions that can be abstract in nature (e.g.,
“wind down,” “go for a walk”) and thus harder to ground than clear
instructions. This is an important difference, as it presents a challenge that
is essential for webly supervised systems, which are expected to learn from a
diverse mix of both concrete actions and high-level abstract actions. In the
realm of web videos, instructional videos account for only a small fraction.

Finally, note that existing action localization methods by and large rely on
simplifying assumptions (e.g., instructional videos, always visible actions,
non-overlapping actions). In contrast, our paper introduces an evaluation that
accounts for the additional challenges encountered in online videos.

##  3\. Data Collection and Annotation

We collect a dataset of routine and do-it-yourself (DIY) videos from YouTube,
consisting of people performing daily activities, such as making breakfast or
cleaning the house. These videos also typically include a detailed verbal
description of the actions being depicted. We choose to focus on these
lifestyle vlogs because they are very popular, with tens of millions having
been uploaded on YouTube; Table 2 shows the approximate number of videos
available for several routine queries. Vlogs also capture a wide range of
everyday activities; on average, we find thirty different visible human
actions in five minutes of video.

By collecting routine videos, instead of searching explicitly for actions, we
do implicit data gathering, a form of data collection introduced by Fouhey et
al (Fouhey et al., 2018). Because everyday actions are common and not unusual,
searching for them directly does not return many results. In contrast, by
collecting routine videos, we find many everyday activities present in these
videos.

Query | Results
---|---
my morning routine | 28M+
my after school routine | 13M+
my workout routine | 23M+
my cleaning routine | 13M+
DIY | 78M+
Table 1. Approximate number of videos found when searching for routine and do-
it-yourself queries on YouTube.

Videos | 171
---|---
Video hours | 20
Transcript words | 302,316
Clips | 1,246
Actions | 13,380
Visible actions | 3,131
Non-visible actions | 10,249
Table 2. Data statistics

Figure 2. Overview of the data gathering pipeline.

###  3.1. Data Gathering

We build a data gathering pipeline (see Figure 2) to automatically extract and
filter videos and their transcripts from YouTube. The input to the pipeline is
manually selected YouTube channels. Ten channels are chosen for their rich
routine videos, where the actor(s) describe their actions in great detail.
From each channel, we manually select two different playlists, and from each
playlist, we randomly download ten videos. The following data processing steps
are applied:

Transcript Filtering. Transcripts are automatically generated by YouTube. We
filter out videos that do not contain any transcripts or that contain
transcripts with an average (over the entire video) of less than 0.5 words per
second.

These videos do not contain detailed action descriptions so we cannot
effectively leverage textual information.

Extract Candidate Actions from Transcript. Starting with the transcript, we
generate a noisy list of potential actions. This is done using the Stanford
parser (Chen and Manning, 2014) to split the transcript into sentences and
identify verb phrases, augmented by a set of hand-crafted rules to eliminate
some parsing errors. The resulting actions are noisy, containing phrases such
as “found it helpful if you” and “created before up the top you.”

Segment Videos into Clips. The length of our collected videos varies from two
minutes to twenty minutes. To ease the annotation process, we split each video
into clips (short video sequences of maximum one minute). Clips are split to
minimize the chance that the same action is shown across multiple clips. This
is done automatically, based on the transcript timestamp of each action.
Because YouTube transcripts have timing information, we are able to line up
each action with its corresponding frames in the video. We sometimes notice a
gap of several seconds between the time an action occurs in the transcript and
the time it is shown in the video. To address this misalignment, we first map
the actions to the clips using the time information from the transcript. We
then expand the clip by 15 seconds before the first action and 15 seconds
after the last action. This increases the chance that all actions will be
captured in the clip.

Motion Filtering. We remove clips that do not contain significant movement. We
sample one out of every one hundred frames of the clip, and compute the 2D
correlation coefficient between these sampled frames. If the median of the
obtained values is greater than a certain threshold (we choose 0.8), we filter
out the clip.

Videos with low movement tend to show people sitting in front of the camera,
describing their routine, but not acting out what they are saying. There can
be many actions in the transcript, but if they are not depicted in the video,
we cannot leverage the video information.

Figure 3. Sample video frames, transcript, and annotations.

###  3.2. Visual Action Annotation

We start by identifying which of the actions extracted from the transcripts
are visually depicted in the videos. We create an annotation task on Amazon
Mechanical Turk (AMT) to identify actions that are visible. We give each AMT
turker a HIT consisting of five clips with up to seven actions generated from
each clip. The turker is asked to assign a label (visible in the video; not
visible in the video; not an action) to each action. Figure 4 shows the AMT
interface used. Because it is difficult to reliably separate not visible and
not an action, we group these labels together. Each clip is annotated by three
different turkers. For the final annotation, we use the label assigned by the
majority of turkers, i.e., visible or not visible / not an action.

To help detect spam, we identify and reject the turkers that assign the same
label for every action in all five clips that they annotate. Additionally,
each HIT contains a ground truth clip that has been pre-labeled by two
reliable annotators. Each ground truth clip has more than four actions with
labels that were agreed upon by both reliable annotators. We compute accuracy
between a turker’s answers and the ground truth annotations; if this accuracy
is less than 20%, we reject the HIT as spam.

After spam removal, we compute the agreement score between the turkers using
Fleiss kappa (Fleiss and Cohen, 1973). Over the entire data set, the Fleiss
agreement score is 0.35, indicating fair agreement. On the ground truth data,
the Fleiss kappa score is 0.46, indicating moderate agreement. This fair to
moderate agreement indicates that the task is difficult, and there are cases
where the visibility of the actions is hard to label. To illustrate, Figure 5
shows examples where the annotators had low agreement. Table 2 shows
statistics for our final dataset of videos labeled with actions, and Figure 3
shows a sample video and transcript, with annotations.

Figure 4. Annotation tool used by Amazon Mechanical Turk workers to annotate
if an action is visible or not in the video. Figure 5. An example of low
agreement. The table shows actions and annotations from workers #1, #2, and
#3, as well as the ground truth (GT). Labels are: visible - ✓, not visible -
x. The bottom row shows screenshots from the video. The Fleiss kappa agreement
score is -0.2. Figure 6. Action temporal localization annotation. Each action
is localized in the video according to its start and end time offsets. The
action is localized according to its visibility in the video, and if it cannot
be seen, it is marked as not visible.

Note that the goal of our dataset is to capture naturally-occurring, routine
actions. Because the same action can be identified in different ways (e.g.,
“pop into the freezer”, “stick into the freezer”), our dataset has a complex
and diverse set of action labels. These labels demonstrate the language used
by humans in everyday scenarios; because of that, we choose not to group our
labels into a pre-defined set of actions. Table 3 shows the number of unique
verbs, which can be considered a lower bound for the number of unique actions
in our dataset. On average, a single verb is used in seven action labels,
demonstrating the richness of our dataset.

The action labels extracted from the transcript are highly dependent on the
performance of the constituency parser. This can introduce noise or ill-
defined action labels. Some actions contain extra words (e.g., “brush my teeth
of course”), or lack words (e.g., “let me just”). Some of this noise is
handled during the annotation process; for example, most actions that lack
words are labeled as “not visible” or “not an action” because they are hard to
interpret.

Dataset | #Actions | #Verbs | #Actors | Implicit | Label types
---|---|---|---|---|---
Ours | 4340 | 580 | 10 | ✓ | ✓
VLOG (Fouhey et al., 2018) | - | - | 10.7k | ✓ | ✓
Kinetics (Kay et al., 2017) | 600 | 270 | - | x | x
ActivityNet (Caba Heilbron et al., 2015) | 203 | - | - | x | x
MIT (Monfort et al., 2019) | 339 | 339 | - | x | x
AVA (Gu et al., 2018) | 80 | 80 | 192 | ✓ | x
Charades (Sigurdsson et al., 2016) | 157 | 30 | 267 | x | x
MPII Cooking (Rohrbach et al., 2012) | 78 | 78 | 12 | ✓ | x
Table 3. Comparison between our dataset and other video human action
recognition datasets. # Actions show either the number of action classes in
that dataset (for the other datasets), or the number of unique visible actions
in that dataset (ours); # Verbs shows the number of unique verbs in the
actions; Implicit is the type of data gathering method (versus explicit);
Label types are either post-defined (first gathering data and then annotating
actions): ✓, or pre-defined (annotating actions before gathering data): x.

###  3.3. Temporal Action Annotation

Each video is associated with a set of human actions, in the form of verb
phrases extracted from the automatically generated video transcripts. The
actions are labeled into two categories: visible or not visible, depending on
whether the actions are explicitly represented in the video. For example, in
the video sequence shown in Figure 1, the action “drink coffee” is not visible
in the video; it is only mentioned as a reason for performing the visible
action of “use a melatonin spray.” Other not visible actions from Figure 1
are: “help,” “hope,” “enjoyed this video,” “thumbs it up” and “subscribe,”
which relate to video feedback but are not visually shown.

Two of the authors of this paper annotated the start and end time of all the
visible actions in the dataset, as illustrated in Figure 6. Each action is
localized according to its start and end time offsets. The timestamp is marked
according to when the action is visible, which does not necessarily correspond
to when it is talked about. If the annotators were not able to localize the
action in the clips, they marked it as not visible, which corresponds to a
correction of the original dataset (Ignat et al., 2019). They performed the
annotations using a simple annotation tool that we built for this purpose,
which is publicly available at https://github.com/OanaIgnat/video_annotations.

We measure the inter-annotator agreement by computing the Krippendorff’s Alpha
score (Krippendorff, 1970) using the interval difference function for each
video. We obtain scores between 0.78 and 0.90, which indicate a high
agreement.

For our experiments, we split the data by vlog channel. Out of ten channels,
six channels are used for training, two channels for validation, and two for
testing. Statistics for this experimental split are shown in Table 4.

| #actions | Vis. (%) | #videos | #clips
---|---|---|---|---
Train | 4,939 | 35.1 | 110 | 680
Val | 1,264 | 35.9 | 26 | 187
Test | 3,456 | 25.7 | 35 | 275
Table 4. Statistics for the experimental data split. “Vis.” is the percentage
of visible actions among the narrated actions.

###  3.4. Data Analysis

We perform two types of analyses to gain a better understanding of our
dataset.

Action Duration.

First, we measure the distribution of action durations in our dataset. As
shown later, this information is important, as the action durations can have
an impact on the performance of different models. Table 5(a) shows the action
duration distribution in the dataset. A summary of long actions found in other
datasets is shown in Table 5(c) (we define an action as long if it exceeds
fifteen seconds). Table 5(b) shows examples of long actions, grouped by verb
and sorted by frequency.

Duration (s) | #actions
---|---
0-5 | 1,136
5-15 | 1,200
15-25 | 475
25-35 | 157
35-45 | 72
45-60 | 99
(a)

Long actions | #actions
---|---
use (a whisk) | 87
make (oatmeal) | 81
clean (my skin) | 60
Short actions | #actions
add (spice) | 362
use (the clamps) | 228
put (a lid on top) | 179
(b)

Dataset | Long actions (%)
---|---
Charades-STA (Gao et al., 2017) | 4.2
CrossTask (Zhukov et al., 2019) | 16.4
COIN (Tang et al., 2019) | 31.6
Ours | 25.5
(c)

Table 5. Action duration analysis: (a) Distribution in our dataset; (b)
Example of long and short actions, each with a sample object, grouped by verbs
and sorted by verb frequency; (c) Percentage of long ($>$15s) actions in other
datasets.

Temporal Relations between Actions.

Second, we analyze the temporal relations between actions mentioned in the
transcripts. These actions can be challenging to model as they capture the
complexities of real life. While there are several actions that follow each
other (as more naturally expected), there are also actions that overlap, are
included in one another, or even happen at the same time. From a total of
2,070 number of overlapping actions, 1,573 are included in each other and 269
occur exactly at the same time. Table 6 shows examples of such actions. While
several action localization datasets have been proposed in the past (Tang et
al., 2019), to the best of our knowledge, this dataset is the only action
localization dataset that contains overlapping actions, making it challenging
and novel. For the purpose of this work, we localize each action independent
of other actions, but future work may leverage the relations that exist
between actions.

Actions that follow each other | Actions that overlap
---|---
“make super quick chicken tacos” ; “do the dishes” |  “toss everything together” $\cap$ “chop it up”
“put them in a bowl” ; “cover in water” |  “add fresh herbs” $\cap$ “add chickpeas to a bowl”
“give a little mix” ; “add half cup of berries” |  “scoop out of the processor” $\cap$ “scoop it into a bowl”
“get a little water on your skin” ; “rinse it off” |  “combine our dry ingredients” $\cap$ “give it a mix”
… | …
Actions that are included in each other | Actions that occur exactly at the same time
“use a plastic scraper” $\subseteq$ “wipe thoroughly” |  “write out” $\equiv$ “make your bucket list”
“throw the cushions around” $\subseteq$ “fix my cushions up” |  “go to bed” $\equiv$ “head to bed”
“do this scrub vigorously” $\subseteq$ “clean some ovens” |  “add good protein” $\equiv$ “use one tablespoon of cashew nut butter”
“do some yoga” $\subseteq$ “wind down” |  “grab my Kindle” $\equiv$ “do some reading”
… | …

Table 6. Examples of different types of action temporal relations: actions
that overlap ($\cap$), actions that are included in each other ($\subseteq$),
actions that occur exactly at the same time ($\equiv$). From a total of 2,070
number of overlapping actions, 1,573 are included in each other and 269 occur
exactly at the same time.

##  4\. Two-Stage Action Localization

For a given action mentioned in a video transcript, our goal is to: (1) decide
if it is visible within the video clip; and (2) if it is visible, identify its
temporal location (i.e., the time interval start and end times).

To achieve this goal, we propose a two-stage method which we call 2Seal
(2-StagE Action Localization).

Figure 7 shows the overall architecture of 2Seal. Following our analysis of
the variation in action duration (see Section 3.4), and empirical observations
made on the development dataset, we hypothesize that shorter actions can be
localized mainly based on the temporal information inferred from the
transcript (i.e., _when_ an action was narrated within the transcript),
whereas longer actions are often temporally shifted with respect to their
mention in the transcript and thus can benefit from a multimodal model. We
thus devise an architecture that first aims to predict whether the action is
short or long, and correspondingly activates a transcript alignment (for short
actions) or a multimodal model (for long actions). We describe below each of
these main components.

Figure 7. 2SEAL method architecture. Note the depicted MPU-based multimodal
model can be replaced with any multimodal model. The MPU model is composed of
vector element-wise addition (’+’), vector element-wise multiplication (’x’)
and vector concatenation followed by a Fully Connected (’FC’) layer to combine
the information from both textual and visual modalities.

Action Duration Classification We use the annotated temporal locations in the
videos to determine the expected duration of each action, and build a binary
classifier to discriminate between short ($\leq$15s) and long ($>$15s)
actions. We choose this threshold based on the validation data. The classifier
uses as input an action text embedding obtained from a text encoder, as
described in Section 5.

Transcript Alignment. Each video contains a transcript automatically generated
by the YouTube API. The transcript contains time information for every
utterance. Given an action mention extracted from an utterance, the Transcript
Alignment method assumes the action is visible, and predicts its temporal
location to be the time interval associated with the corresponding utterance,
as illustrated in Figure 8. The transcript alignment is also illustrated in
Figure 7.

Figure 8. Example of applying the Transcript Alignment method. The transcript
contains time intervals for utterances. Each action contained in an utterance
is assigned the corresponding time interval.

Multimodal Model. We split the video clips into fixed-duration spans and
convert the action temporal localization task into binary classification tasks
based on the output from a scorer model s. We aim to predict if the visual
information from a video clip span corresponds to the linguistic
representation of an action. For a given action mention within the transcript
and a fixed-duration video clip span, we compute a similarity score to decide
if they correspond to each other. The action mention is represented using a
text encoder and the features for the video clip span are obtained from a
video encoder (see Section 5).

The process of pairing action mentions to video clip spans is shown in Figure
7; s can be represented by any multimodal model, and we describe several
models in Section 5.2. At test time, given a video clip and its corresponding
transcript, we input all the pairs of action mentions and fixed-duration video
clip spans. We merge all the spans that surpass a certain threshold and are
separated by less than three seconds into _proposals_. Each proposal is
assigned the maximum similarity score of its spans. We then perform non-
maximum suppression to select the best proposal as the predicted action
location interval. At training time, we focus only on the binary task and
train s with the standard cross-entropy loss. Given that an action mention has
many more negative (not visible) fixed-duration video clip spans in a given
video clip, we balance the classes out via downsampling by taking negative
random samples from the same video clip. The question of how different
negative sampling strategies affect the scorer model performance is left for
future work.

##  5\. Experiments

To evaluate our duration-informed action localization method, we run several
comparative experiments on the dataset described in Section 3. We compare our
method with several strong baselines, and also perform feature ablation and a
breakdown of results by action duration.

In all our experiments, we use a video encoder consisting of the last layer
(mixed_5c) from a Kinetics (Carreira and Zisserman, 2017) pre-trained I3D
model. The video clips are divided into overlapping three-second spans with a
stride of 1s. We freeze both the text and the video encoders and take their
outputs as features. For the Action Duration Classification, we use an SVM
classifier with C=1.0 and an RBF kernel, and weight the samples inversely
proportional to their class frequency. We train the models using an Adam
optimizer with early stopping (tolerance 15 epochs), with a learning rate of
0.001 and a batch size of 64.

###  5.1. Action Duration Classification.

We train the action duration classifier described in the previous section
using only the visible actions. The results are reported in Table 7. For
comparison, we also show the performance of a majority classifier, which
labels every action as “short” by default. As shown in the table, despite the
simplicity of the classifier, the action duration classifier obtains good
improvement over the majority baseline.

Method | A | P | R | F1
---|---|---|---|---
Majority | 74.4 | 74.4 | 100.0 | 85.3
Action Duration Clf. | 80.6 | 81.8 | 97.6 | 89.0
Table 7. Action duration classification results on the validation set. The
classification is binary, where the positives are the short actions
($\leq$15s) and the negatives the long ones ($>$15s). The columns are in
order: accuracy (A), precision (P), recall (R) and F1 score (F1).

###  5.2. Temporal Action Localization

Our 2Seal method includes a scorer that measures the similarity between a
video clip and an action mention (see Figure 7). To implement this scorer, we
experiment with three methods proposed in previous work: multimodal processing
unit, multiple instance learning noise contrastive estimation, and stacked
cross attention.

Multimodal Processing Unit (MPU). We use the MPU model (Gao et al., 2017) to
compute the similarity score between the language representation of a narrated
action and a video clip span. For the text features, we fine-tune a pre-
trained BERT-base-uncased (Devlin et al., 2019) for domain adaption by on 884
vlog transcripts with 80,749 sentences. We take embeddings from this model for
the action mentions in the transcripts by average pooling (the final embedding
size is 768). In Section 6.2 we experiment with variations of this text
encoder. The text and visual features for each pair are linearly mapped to the
same embedding space. Next, the MPU model is applied to compute the
interaction between the two vectors of the same duration. The MPU model is
composed of vector element-wise addition (’+’), vector element-wise
multiplication (’x’) and vector concatenation followed by a Fully Connected
(’FC’) layer to combine the information from both textual and visual
modalities. The outputs from all three operations are concatenated to
construct a multi-modal representation. This process is also illustrated in
the overall architecture in Figure 7. The resulting representation is given as
input to a linear layer and finally to a sigmoid function to obtain a
similarity score.

Multiple Instance Learning Noise Contrastive Estimation (MIL-NCE). We use the
MIL-NCE model from (Miech et al., 2019a) which was trained on HowTo100M (Miech
et al., 2019b). The similarity score is computed as a dot product between the
text and video encoder outputs. The text encoder takes embeddings from a
GoogleNews-pretrained skipgram word2vec (Mikolov et al., 2013) implementation
and further processes and pools the embeddings to obtain a fixed-size
representation. We use the MIL-NCE I3D111https://tfhub.dev/deepmind/mil-
nce/i3d/1 visual features, and not the S3D features, for consistency reasons
and to ensure a fair comparison between the multimodal models. We empirically
find it beneficial to threshold the similarities at mid-range value after
experimenting with linear regression models on the validation data. Note we do
not fine-tune this model but freeze it. Future work can explore how the method
benefits from fine-tuning.

Stacked Cross Attention (SCA). We also experiment with the SCA method (Lee et
al., 2018), and adapt its Text-Image formulation. It first attends to image
frames with respect to each word, and then compares each word to its
corresponding attended frame vector to determine the importance of each word.
The relevance $R$ between the $i$-th word and the image is defined as the
cosine similarity between the $i$-th word vector $v_{i}$ and its attended
frame vector $a_{i}^{t}$. The final similarity score between image $I$ and
sentence $T$ is summarized by average pooling:
$S_{AVG}^{\prime}(I,T)=\frac{1}{n}\sum_{j=1}^{n}R^{\prime}(e_{j},a_{j}^{v})$.
The textual features are represented using a Gated Recurrent Unit (GRU)
(Graves, 2008) as in (Lee et al., 2018). We use the mid-range threshold for
the similarity score.

2D Temporal Adjacent Networks (2D-TAN). We find the 2D-TAN model (Zhang et
al., 2020) suitable for our task as it is built to localize multiple natural
language queries in a video.

The video clips are represented using C3D (Tran et al., 2015) features and the
action queries using GloVe (Pennington et al., 2014) embeddings, as described
in the 2D-TAN paper (Zhang et al., 2020). We take as final proposal the action
localization proposal with the highest score.

We test the pre-trained model and also fine-tune it on our training and
validation data. We run two model configurations, which were trained on TACoS
(Regneri et al., 2013), namely “Pool” and “Conv” in our test set. “Pool” and
“Conv” represent max-pooling and stacked convolution respectively, which
indicates two different ways for moment feature extraction in the 2D-TAN
model. We report the results of fine-tuned “Conv” 2D-TAN model, which is the
best performing 2D-TAN model configuration on our test dataset.

###  5.3. Results

We evaluate the predictions made by the action localization methods using two
evaluation metrics. First, we compute the Visibility Accuracy (VA) to decide
if the method can distinguish between visible and not visible actions. Second,
only for the visible actions, we compute the recall at different Intersection
over Union (IoU) thresholds: 0.1, 0.3, 0.5 and 0.7. A higher threshold means a
stronger constraint on how exact the match between the predicted and the
ground truth location needs to be. If the predicted interval has an IoU score
with the ground truth greater than the threshold, we consider the prediction
as being correct. We also compute the average recall over all IoU values, as
the mIoU. Note that if a method predicted that a visible action is non-
visible, then the recall score is penalized.

Table 8 presents the temporal action localization results on our data. The
Transcript Alignment method performs better than the MPU, MIL-NCE, SCA and
2D-TAN methods if we do not previously apply our proposed 2Seal method before.
However, when using our 2Seal method that combines both the Transcript
Alignment and a method to score long actions (either MPU, MIL-NCE, SCA, or
2D-TAN), the performance improves significantly, with the system integrating
the MPU model leading to the best results. We suspect MIL-NCE may perform
better if fine-tuned, however our intention is not to compare MPU and MIL-NCE
but to show how our method can improve over other existing methods. The
results confirm our initial hypothesis that actions of different duration
benefit from different methods: the transcript alignment excels at short
actions, while the multimodal model performs better for long actions.

|  | Recall
---|---|---
Method | VA | IoU=0.1 | IoU=0.3 | IoU=0.5 | IoU=0.7 | mIoU
All visible | 25.7 | 67.4 | 23.6 | 8.3 | 4.1 | 21.6
All non-visible | 74.3 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
Transcript Alignment (ours) | 25.7 | 73.3 | 47.3 | 22.2 | 7.2 | 30.8
MPU | 75.5 | 57.9 | 27.0 | 12.4 | 6.2 | 21.4
2SEAL (ours) + MPU | 79.0 | 74.6 | 48.7 | 22.8 | 8.6 | 31.9
MIL-NCE | 26.1 | 62.9 | 22.2 | 8.0 | 4.2 | 20.5
2SEAL (ours) + MIL-NCE | 34.4 | 74.4 | 47.8 | 21.7 | 7.9 | 31.4
SCA | 24.2 | 49.9 | 17.0 | 6.0 | 3.4 | 15.9
2SEAL (ours) + SCA | 26.1 | 72.2 | 46.7 | 21.4 | 7.6 | 30.5
2D-TAN | 25.7 | 49.4 | 23.1 | 10.9 | 3.7 | 17.6
2SEAL (ours) + 2D-TAN | 25.7 | 73.4 | 47.0 | 21.6 | 7.7 | 30.8
Human | 85.9 | 83.5 | 71.8 | 52.0 | 35.0 | 50.3
Table 8. Results on the test set. “VA” stands for Visibility Accuracy.

##  6\. Analyses and Discussion

To gain insights into the performance of our proposed model in relation to
action duration, and to understand the role played by different features, we
perform several analyses.

###  6.1. Action Duration Impact

If the action is brief, the IoU metric will be influenced by a few seconds
compared to when the action is longer in duration. This metric penalizes more
the mislocalization of short actions, as compared to the longer ones. This
analysis is often done for the task of object detection, where the IoU scores
are grouped by bounding box size (Redmon et al., 2015). To verify our initial
hypothesis that actions of different duration benefit from different
localization methods, we break down the results of the MPU (the best scorer
from among MPU, MIL-NCE, SCA and 2D-TAN without applying the 2SEAL method) by
action duration in Table 9. As shown in the table, the performance of the
model is connected to the duration of the actions. For long actions, the
multimodal method obtains better results compared to the transcript alignment
method, while the opposite is true for short actions.

| 0–15s | 16–35s | 36–60s
---|---|---|---
Recall | MPU | Align | MPU | Align | MPU | Align
IoU=0.1 | 49.5 | 71.6 | 90.7 | 76.6 | 95.2 | 83.3
IoU=0.3 | 5.4 | 49.0 | 73.4 | 51.4 | 81.0 | 0.0
IoU=0.5 | 2.0 | 25.0 | 22.0 | 17.8 | 78.6 | 0.0
IoU=0.7 | 0.8 | 9.4 | 5.6 | 1.9 | 66.7 | 0.0
mIoU | 12.0 | 32.0 | 38.9 | 29.9 | 71.7 | 16.5
Table 9. Breakdown by action duration (time span) on the validation set. The
MPU model performance increases with the increase of action time span, while
transcript alignment (Align) performance decreases. Figure 9. Randomly sampled
qualitative results for different cases of action overlapping. Best viewed in
color.

###  6.2. Text and Visual Features

In Table 10, we experiment with the MPU model (without applying the 2SEAL
method) and look into how each modality contributes to solving this task, by
removing one modality at a time from our best performing model. We also
analyze other types of text embeddings. Inspired by (Motwani and Mooney, 2012;
Sigurdsson et al., 2017), we focus on verbs and nouns, which we extract from
the actions and compute their BERT embeddings. We observe that the visual
information contributes the most to the task of action localization, as
removing this information drastically lowers the model performance. Another
observation is that processing the entire action is more beneficial to the
model than focusing only on nouns and verbs.

| Recall
---|---
Method | IoU=0.1 | IoU=0.3 | IoU=0.5 | IoU=0.7 | mIoU
MPU | 57.9 | 27.0 | 12.4 | 6.2 | 21.4
MPU verb only | 33.5 | 18.5 | 9.2 | 4.8 | 13.7
MPU verb+noun only | 33.8 | 18.7 | 9.8 | 4.8 | 14.0
MPU BERT w/o DA | 46.9 | 26.4 | 14.1 | 5.4 | 19.0
MPU ELMo | 48.5 | 23.7 | 10.6 | 6.2 | 18.4
MPU GloVe | 41.6 | 22.5 | 11.6 | 6.9 | 17.2
MPU video only | 41.5 | 25.4 | 13.9 | 6.8 | 18.0
MPU text only | 25.3 | 11.6 | 4.3 | 2.2 | 9.1
Table 10. Results on the test set for different variations of the input to the
MPU model. “DA” stands for Domain Adaptation.

###  6.3. Qualitative Results

Randomly sampled results are shown in Figure 9. They are grouped by the
different levels of action overlapping: no overlap, intersection, inclusion
and perfect overlap. From analyzing these results, a future work direction
emerges: detecting which actions are likely to happen at the same time, which
in turn can lead to better algorithms for action localization.

##  7\. Conclusion

In this paper, we introduced a new dataset for action localization in vlogs —
a growing form of online video communication where everyday routine actions
are described in language and also presented visually. Using this dataset, we
addressed the task of temporal action localization in videos. We proposed
2Seal – a simple yet effective method to visually localize the actions
mentioned in a video transcript, which relies on both language and vision, and
specifically accounts for the duration of an action for the purpose of
building a more accurate system.

Through several extensive evaluations, we showed that our method improves and
complements other methods by first computing the expected duration of an
action, and selectively applying a language-based or multimodal model
depending on the action duration. This work contributes to the larger body of
work for multimodal understanding, and at the same time builds a large
repository of vision-language representations covering a wide spectrum of
actions that can be used for downstream tasks such as action recognition
systems, human behavior understanding, event recognition, and others. The
dataset introduced in this paper, the annotation tool, and the system code are
publicly available at https://github.com/MichiganNLP/vlog_action_localization.

##  8\. Acknowledgments

We thank Weiji Li for his help with the human annotations of action
visibility. This research was partially supported by a grant from the
Automotive Research Center (ARC) at the University of Michigan in accordance
with Cooperative Agreement W56HZV-19-2-0001.

## References

  * (1)
  * Abu-El-Haija et al. (2016) Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. 2016\.  Youtube-8m: A large-scale video classification benchmark.  _arXiv preprint arXiv:1609.08675_ (2016).
  * Alayrac et al. (2016) Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, and Simon Lacoste-Julien. 2016.  Unsupervised learning from narrated instruction videos. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 4575–4583.
  * Anderson et al. (2018) Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018.  Bottom-up and top-down attention for image captioning and visual question answering. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 6077–6086.
  * Anne Hendricks et al. (2017a) Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2017a.  Localizing Moments in Video With Natural Language. In _The IEEE International Conference on Computer Vision (ICCV)_.
  * Anne Hendricks et al. (2017b) Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2017b.  Localizing moments in video with natural language. In _Proceedings of the IEEE international conference on computer vision_. 5803–5812.
  * Caba Heilbron et al. (2015) Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015.  Activitynet: A large-scale video benchmark for human activity understanding. In _Proceedings of the ieee conference on computer vision and pattern recognition_. 961–970.
  * Carreira and Zisserman (2017) Joao Carreira and Andrew Zisserman. 2017.  Quo vadis, action recognition? a new model and the kinetics dataset. In _proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 6299–6308.
  * Chen and Manning (2014) Danqi Chen and Christopher Manning. 2014.  A fast and accurate dependency parser using neural networks. In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_. 740–750.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _NAACL-HLT_.
  * Donahue et al. (2015) Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. 2015\.  Long-term recurrent convolutional networks for visual recognition and description. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2625–2634.
  * Escorcia et al. (2019) Victor Escorcia, Mattia Soldan, Josef Sivic, Bernard Ghanem, and Bryan Russell. 2019.  Temporal Localization of Moments in Video Collections with Natural Language.  _arXiv preprint arXiv:1907.12763_ (2019).
  * Fleiss and Cohen (1973) Joseph L Fleiss and Jacob Cohen. 1973.  The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability.  _Educational and psychological measurement_ 33, 3 (1973), 613–619.
  * Fouhey et al. (2018) David F Fouhey, Wei-cheng Kuo, Alexei A Efros, and Jitendra Malik. 2018.  From lifestyle vlogs to everyday interactions. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 4991–5000.
  * Gao et al. (2017) Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. 2017\.  Tall: Temporal activity localization via language query. In _Proceedings of the IEEE International Conference on Computer Vision_. 5267–5275.
  * Ge et al. (2019) Runzhou Ge, Jiyang Gao, Kan Chen, and Ram Nevatia. 2019\.  MAC: Mining Activity Concepts for Language-based Temporal Localization. In _2019 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 245–253.
  * Ghosh et al. (2019) Soham Ghosh, Anuva Agarwal, Zarana Parekh, and Alexander Hauptmann. 2019.  ExCL: Extractive Clip Localization Using Natural Language Descriptions.  _arXiv preprint arXiv:1904.02755_ (2019).
  * Girdhar et al. (2019) Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zisserman. 2019.  Video action transformer network. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 244–253.
  * Goyal et al. (2016) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2016.  Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ (2016), 6325–6334.
  * Graves (2008) Alex Graves. 2008\.  Supervised Sequence Labelling with Recurrent Neural Networks. In _Studies in Computational Intelligence_.
  * Gu et al. (2018) Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al. 2018\.  Ava: A video dataset of spatio-temporally localized atomic visual actions. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 6047–6056.
  * Heilbron et al. (2015) Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015.  ActivityNet: A large-scale video benchmark for human activity understanding.  _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ (2015), 961–970.
  * Hendricks et al. (2018) Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2018.  Localizing moments in video with temporal language.  _arXiv preprint arXiv:1809.01337_ (2018).
  * Hudson and Manning (2019) Drew A Hudson and Christopher D Manning. 2019.  Gqa: a new dataset for compositional question answering over real-world images.  _arXiv preprint arXiv:1902.09506_ (2019).
  * Ignat et al. (2019) Oana Ignat, Laura Burdick, Jia Deng, and Rada Mihalcea. 2019\.  Identifying Visible Actions in Lifestyle Vlogs. In _ACL_.
  * Jiang et al. (2007) Yu-Gang Jiang, Chong-Wah Ngo, and Jun Yang. 2007.  Towards optimal bag-of-features for object categorization and semantic video retrieval. In _CIVR ’07_.
  * Kay et al. (2017) Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. 2017\.  The kinetics human action video dataset.  _arXiv preprint arXiv:1705.06950_ (2017).
  * Krippendorff (1970) Klaus Krippendorff. 1970\.  Estimating the reliability, systematic error and random error of interval data.  _Educational and Psychological Measurement_ 30, 1 (1970), 61–70.
  * Krishna et al. (2017) Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017.  Dense-captioning events in videos. In _Proceedings of the IEEE international conference on computer vision_. 706–715.
  * Lee et al. (2018) Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. 2018.  Stacked cross attention for image-text matching. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 201–216.
  * Lei et al. (2019) Jie Lei, Licheng Yu, Tamara L. Berg, and Mohit Bansal. 2019\.  TVQA+: Spatio-Temporal Grounding for Video Question Answering.  _ArXiv_ abs/1904.11574 (2019).
  * Levy and Goldberg (2014) Omer Levy and Yoav Goldberg. 2014.  Dependency-Based Word Embeddings. In _ACL_.
  * Lin et al. (2017) Tianwei Lin, Xu Zhao, and Zheng Shou. 2017.  Single shot temporal action detection. In _Proceedings of the 25th ACM international conference on Multimedia_. 988–996.
  * Liu et al. (2018) Meng Liu, Xiang Wang, Liqiang Nie, Xiangnan He, Baoquan Chen, and Tat-Seng Chua. 2018\.  Attentive Moment Retrieval in Videos. In _SIGIR ’18_.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019\.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In _Advances in Neural Information Processing Systems_. 13–23.
  * Miech et al. (2019a) Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. 2019a.  End-to-End Learning of Visual Representations from Uncurated Instructional Videos.  _arXiv preprint arXiv:1912.06430_ (2019).
  * Miech et al. (2019b) Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019b.  HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips.  _arXiv preprint arXiv:1906.03327_ (2019).
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, G. S. Corrado, and J. Dean. 2013\.  Efficient Estimation of Word Representations in Vector Space.  _CoRR_ abs/1301.3781 (2013).
  * Mithun et al. (2018) Niluthpol Chowdhury Mithun, Juncheng Li, Florian Metze, and Amit K Roy-Chowdhury. 2018.  Learning joint embedding with multimodal cues for cross-modal video-text retrieval. In _Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval_. ACM, 19–27.
  * Monfort et al. (2019) Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Yan Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et al. 2019\.  Moments in time dataset: one million videos for event understanding.  _IEEE transactions on pattern analysis and machine intelligence_ (2019).
  * Motwani and Mooney (2012) Tanvi S Motwani and Raymond J Mooney. 2012.  Improving Video Activity Recognition using Object Recognition and Text Mining.. In _ECAI_ , Vol. 1. 2.
  * Palaskar et al. (2019) Shruti Palaskar, Jindrich Libovickỳ, Spandana Gella, and Florian Metze. 2019.  Multimodal abstractive summarization for how2 videos.  _arXiv preprint arXiv:1906.07901_ (2019).
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  Glove: Global Vectors for Word Representation. In _EMNLP_.
  * Plummer et al. (2017) Bryan A Plummer, Matthew Brown, and Svetlana Lazebnik. 2017\.  Enhancing video summarization via vision-language embedding. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 5781–5789.
  * Redmon et al. (2015) Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. 2015.  You Only Look Once: Unified, Real-Time Object Detection.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ (2015), 779–788.
  * Regneri et al. (2013) Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal. 2013.  Grounding Action Descriptions in Videos.  _Transactions of the Association for Computational Linguistics_ 1 (2013), 25–36.  https://doi.org/10.1162/tacl_a_00207
  * Rohrbach et al. (2012) Marcus Rohrbach, Sikandar Amin, Mykhaylo Andriluka, and Bernt Schiele. 2012.  A database for fine grained activity detection of cooking activities. In _2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 1194–1201.
  * Shi et al. (2019) Botian Shi, Lei Ji, Yaobo Liang, Nan Duan, Peng Chen, Zhendong Niu, and Ming Zhou. 2019.  Dense Procedure Captioning in Narrated Instructional Videos. In _Proceedings of the 57th Conference of the Association for Computational Linguistics_. 6382–6391.
  * Sigurdsson et al. (2018) Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. 2018\.  Charades-ego: A large-scale dataset of paired third and first person videos.  _arXiv preprint arXiv:1804.09626_ (2018).
  * Sigurdsson et al. (2017) Gunnar A Sigurdsson, Olga Russakovsky, and Abhinav Gupta. 2017\.  What actions are needed for understanding human actions in videos?. In _Proceedings of the IEEE International Conference on Computer Vision_. 2137–2146.
  * Sigurdsson et al. (2016) Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. 2016.  Hollywood in homes: Crowdsourcing data collection for activity understanding. In _European Conference on Computer Vision_. Springer, 510–526.
  * Suhr et al. (2018) Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2018\.  A corpus for reasoning about natural language grounded in photographs.  _arXiv preprint arXiv:1811.00491_ (2018).
  * Sun et al. (2019) Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019.  Videobert: A joint model for video and language representation learning. In _Proceedings of the IEEE International Conference on Computer Vision_. 7464–7473.
  * Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019.  Lxmert: Learning cross-modality encoder representations from transformers.  _arXiv preprint arXiv:1908.07490_ (2019).
  * Tang et al. (2019) Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. 2019\.  Coin: A large-scale dataset for comprehensive instructional video analysis. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 1207–1216.
  * Tran et al. (2015) Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. 2015.  Learning Spatiotemporal Features with 3D Convolutional Networks.  arXiv:1412.0767 [cs.CV]
  * Tran et al. (2014) Du Tran, Lubomir D. Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. 2014.  C3D: Generic Features for Video Analysis.  _ArXiv_ abs/1412.0767 (2014).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is All you Need. In _NIPS_.
  * Wang et al. (2011) Heng Wang, Alexander Kläser, Cordelia Schmid, and Cheng-Lin Liu. 2011.  Action recognition by dense trajectories.  _CVPR 2011_ (2011), 3169–3176.
  * Wang et al. (2016) Mingzhe Wang, Mahmoud Azab, Noriyuki Kojima, Rada Mihalcea, and Jia Deng. 2016.  Structured matching for phrase localization. In _European Conference on Computer Vision_. Springer, 696–711.
  * Yang et al. (2015) Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alexander J. Smola. 2015.  Stacked Attention Networks for Image Question Answering.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ (2015), 21–29.
  * You et al. (2016) Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. 2016.  Image captioning with semantic attention. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4651–4659.
  * Yuan et al. (2018) Yitian Yuan, Tao Mei, and Wenwu Zhu. 2018.  To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regression.  _ArXiv_ abs/1804.07014 (2018).
  * Zhang et al. (2020) Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo. 2020\.  Learning 2D Temporal Adjacent Networks forMoment Localization with Natural Language. In _AAAI_.
  * Zhukov et al. (2019) Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. 2019.  Cross-task weakly supervised learning from instructional videos. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 3537–3545.