#  Exo2EgoDVC: Dense Video Captioning of Egocentric Procedural Activities
Using Web Instructional Videos

Takehiko Ohkawa1,2, Takuma Yagi3, Taichi Nishimura4, Ryosuke Furuta1,
Atsushi Hashimoto2, Yoshitaka Ushiku2, and Yoichi Sato1
1The University of Tokyo 2OMRON SINIC X Corp.
3National Institute of Advanced Industrial Science and Technology (AIST)
4Kyoto University

###### Abstract

We propose a novel benchmark for cross-view knowledge transfer of dense video
captioning, adapting models from web instructional videos with exocentric
views to an egocentric view. While dense video captioning (predicting time
segments and their captions) is primarily studied with exocentric videos
(_e.g_., YouCook2), benchmarks with egocentric videos are restricted due to
data scarcity. To overcome the limited video availability, transferring
knowledge from abundant exocentric web videos is demanded as a practical
approach. However, learning the correspondence between exocentric and
egocentric views is difficult due to their dynamic view changes. The web
videos contain mixed views focusing on either human body actions or close-up
hand-object interactions, while the egocentric view is constantly shifting as
the camera wearer moves. This necessitates the in-depth study of cross-view
transfer under complex view changes. In this work, we first create a real-life
egocentric dataset (EgoYC2) whose captions are shared with YouCook2, enabling
transfer learning between these datasets assuming their ground-truth is
accessible. To bridge the view gaps, we propose a view-invariant learning
method using adversarial training in both the pre-training and fine-tuning
stages. While the pre-training is designed to learn invariant features against
the mixed views in the web videos, the view-invariant fine-tuning further
mitigates the view gaps between both datasets. We validate our proposed method
by studying how effectively it overcomes the view change problem and
efficiently transfers the knowledge to the egocentric domain. Our benchmark
pushes the study of the cross-view transfer into a new task domain of dense
video captioning and will envision methodologies to describe egocentric videos
in natural language.

Figure 1:  Our proposed cross-view knowledge transfer of dense video
captioning. We propose to utilize existing web instructional videos with
exocentric views, YouCook2 (YC2) [75], to improve dense video captioning on
newly recorded egocentric videos (EgoYC2). The EgoYC2’s captions are shared
with the YC2, enabling the study of transfer learning under view gaps in
videos.

##  1 Introduction

Perceiving procedural human activities from an egocentric (first-person) view
has been a long-standing problem. Unlike action recognition focusing on
labeling specific activities [9, 49], video-to-text description extends this
realm, which offers a detailed textual interpretation of ongoing activities.
This not only facilitates a deeper understanding of the procedure of tasks,
but also enriches intuitive and communicative interfaces between humans and
assistive systems, such as augmented reality [33] and human-robot interactions
[15].

One formulation of the video-to-text description is dense video captioning
[21], which densely detects time segments of a video and generates their
captions. This task has been studied with instructional videos (_e.g_.,
YouCook2 [75]), which primarily feature exocentric (third-person) videos
available on the Web. In contrast, egocentric benchmarks for dense video
captioning remain underexplored due to limited dataset scale (_e.g_., 16 hours
of MMAC Captions (ego) [39] _vs_. 176 hours of YouCook2).

Given this data scarcity, finding ways to bridge exocentric (source) and
egocentric (target) views is vital to utilizing numerous web exocentric videos
to enhance the understanding of egocentric activities. Prior works of this
cross-view transfer have been studied in action recognition [25, 61] and
human/hand pose estimation [16, 41].

Unlike short-term modeling in these prior works (_e.g_., per-frame pose
estimation and per-clip action recognition), dense video captioning requires
capturing longer dependencies of actions. This emphasizes the problem of
dynamic camera view changes in learning the correspondence between both views.
The source instructional videos are not purely captured from a single fixed
camera, but composed of multiple alternating views (_e.g_., face and hand
shots [36]). In contrast, egocentric videos inherently include dynamic view
changes due to the camera wearer’s motion, which obstructs learning the
procedure from egocentric views [3]. This necessitates adapting captioning
models from mixed source views to a moving target view.

In this work, we propose a knowledge transfer benchmark for dense video
captioning between source and target datasets with access to their ground-
truth during training ( Exo2EgoDVC: Dense Video Captioning of Egocentric
Procedural Activities Using Web Instructional Videos). To study this cross-
view transfer, we first create a new egocentric dataset, EgoYC2, with caption
annotations shared with a source dataset, YouCook2. We collect 226 egocentric
cooking videos from 44 users, featuring real-life home kitchens (_vs_. a
laboratory kitchen in MMAC Captions). These paired datasets reduce
discrepancies in caption content and granularity, allowing us to pre-train a
model on the source data and fine-tune it on the target data.

To address the view gaps, we encourage the pre-training and fine-tuning to be
less affected by view-dependent bias, using adversarial invariant feature
learning [12, 6]. The pre-training aims to learn features invariant to the two
views in the source data: face shots showing body actions and hand shots
focusing on hand-object interactions. The view-invariant fine-tuning is
performed using both datasets, further adapting to the egocentric domain. To
address the moving impact in the egocentric videos, we stabilize the videos by
a fine-grained and temporally coherent tracking of hand-object interactions,
which consists of hand detection-based tracking [50, 4] and hand-object
segmentation [74].

We evaluate our transfer learning method regarding how effectively the method
overcomes view changes and efficiently transfers knowledge to the egocentric
domain. Our pre-training with the decoupling of mixed source views has shown
further improvement in egocentric video captioning against a naive pre-trained
model. The stabilization of the target view movement with the hand tracking
improves fine-tuning performance, and additional support of hand-object
features is more effective. Our view-invariant fine-tuning further improves
adapting the pre-trained model to the egocentric domain. Our benchmark allows
us to provide a practical solution for transfer learning from exocentric to
egocentric videos under dynamic view changes.

Our contributions are summarized as follows:

  * •

We offer a new real-life egocentric video dataset (EgoYC2) for dense video
captioning, whose captions are shared with an exocentric video dataset
(YouCook2).

  * •

We propose view-invariant learning in the pre-training and fine-tuning with
unified adversarial training and video processing that mitigates the view
change effects.

  * •

We demonstrate how effectively the proposed method overcomes the problem of
view changes and efficiently transfers the knowledge to the egocentric domain.

##  2 Related Work

Dense video captioning consists of two sub-tasks localizing multiple time
segments occurring in a video and describing their captions. While Krishna
_et al_. initially proposes to describe coarse activities [21], subsequent
works focus on more fine-grained activities using instructional videos, such
as cooking [75, 39] (_e.g_., YouCook2 (YC2)), makeup [64], and daily activity
videos [1, 64, 35]. These instructional videos promote the understanding of
the procedure [75], a series of steps to accomplish certain tasks.

Nevertheless, dense video captioning for egocentric (first-person) videos has
been less studied compared to exocentric (third-person) benchmarks [1, 75, 64,
53, 35]. A video captioning dataset for egocentric cooking videos, MMAC
Captions [39], contains x11 smaller amount of videos than YC2. Other related
egocentric datasets (_e.g_., EPIC-KITCHENS [9] and Ego4D [13]) are collected
with spoken narrations. Their transcribed texts help define action labels and
video-language pre-training [28, 46], but they drastically differ from
procedural captions [69, 70]. Also, their supervision is typically weak and
noisy, including incorrect visual grounding, irrelevant caption contents, and
errors in automatic speech recognition [34, 20, 52, 29].

We newly construct an egocentric video dataset (EgoYC2) with procedural
caption annotations. Compared to the MMAC Captions dataset captured in a
laboratory kitchen, we record real-life cooking activities and annotate
captions shared with the exocentric dataset YC2. This allows us to let models
resolve the view gaps only without considering the gaps in caption content and
granularity. In fact, we use the same vocabulary list as the YC2 and have
close average step sizes per video (7.7 in YC2 _vs_. 6.5 in EgoYC2 _vs_. 30.1
in MMAC Captions).

Egocentric perception using exocentric videos. Exocentric view data can inform
the state of humans, actions, and the surrounding environment that are not
always observable from a limited field-of-view of an egocentric camera. Based
on this insight, knowledge transfer from exocentric to egocentric views has
been studied, _e.g_., in action recognition [51, 25, 56, 61, 68] and
human/hand pose estimation [16, 42, 41]. This transfer has been categorized
into two settings: paired _vs_. unpaired scenarios.

The paired setting assumes to capture the same actions from different views
[51] or using synchronized egocentric and exocentric cameras [23, 49].
Sigurdsson  _et al_. propose to learn a shared feature space between both
views using metric learning [51]. With camera calibration and head-camera
tracking, other works [41, 23, 16] project annotated hand-object poses from
exocentric to egocentric views. The unpaired setting relaxes this assumption
of view correspondence; thus, two videos are neither synchronized nor captured
in the same environment. Several ways to learn view-invariant features across
unpaired two videos have been proposed, such as knowledge distillation [25]
and cross-view feature alignment [61, 68] or attention [56]. Other methods
employ domain adaptation techniques, such as adversarial training [8, 38] and
pseudo-labeling [42, 43].

Our work addresses this cross-view transfer problem for a new task domain,
dense video captioning. Unlike action recognition, the transfer of dense video
captioning requires modeling long-term dependencies of actions, which
complicates the problem of view changes considering a longer sequence of a
video. Since our used videos are asynchronous, we propose an adversarial
training method that works in the unpaired setting. Unlike adversarial
adaptation in action recognition [8, 38], we perform view-invariant learning
in the pre-training stage to explicitly handle the mixed views in the source
data.

##  3 Exo-to-Ego Transfer Learning

We propose a solution for a transfer learning task of dense video captioning
from YouCook2 (source) to our collected EgoYC2 (target). We assume that the
ground-truth is accessible for both datasets during training due to its task
complexity, which follows supervised domain adaptation settings [57, 37]. We
first introduce our setup and transfer learning method in Secs. 3.1 and 3.2,
and then describe the details of hand-object-aware video representation in
Sec. 3.3 and a captioning network in Sec. 3.4.

The challenge of this task is addressing the dynamics of view changes in both
datasets. Unlike cross-view transfer in action recognition [51, 25, 56, 61,
68], dense video captioning requires modeling long-term dependencies of
actions, which further complicates the view change problem considering a
longer sequence of a video. We observe that the source web videos can be
decomposed into several shots captured from different views, based on video
composition analysis, _e.g_., instruction classes in how-to videos [71], scene
cut categorization [45, 36], temporal shot segmentation [7, 65]. In contrast,
egocentric videos are typically untrimmed and captured from a single camera,
but involve dynamic scene changes due to the head movement. This dynamic
movement prevents learning the procedure (key steps) from egocentric views
[3]. Therefore, the view change problem stems from different causes between
both datasets, and it is necessary to overcome view-dependent bias to adapt
models to the target egocentric domain.

Figure 2:  Structure analysis of exocentric cooking videos [36]. The shots in
instructional cooking videos can be coarsely categorized into (a) face shots
and (b) hand shots. As shown in the time segments below, these shots are
typically alternating within a single step. This motivates us to partition the
source videos into exo (face shot) and ego-like (hand shot) classes.  Figure
3:  View-invariant learning across exocentric and egocentric views. (i) We
define an intermediate view (ego-like) in the source domain, which represents
the one between exo and ego views. We treat source images where the face is
detected as the exo view and the others as the ego-like view due to its
similarity to the ego view. We generate video features using a fixed encoder
$\phi$ and describe this processing for egocentric videos in Sec. 3.3. (ii) We
design our view-invariant (VI) learning to gradually adapt from exo to ego
views. Our method consists of pre-training (PT) on the source data and fine-
tuning (FT) across the source and target data. Following adversarial domain
adaptation [12], we train a feature converter $F$ and a view classifier $C$
with a gradient reversal layer (GRL) to reduce the view gaps in a feature
space. The former PT takes the source data with the exo and ego-like classes,
and the latter FT aligns all three views.

Preliminary. We have access to both labeled source and target datasets
$\mathcal{D}_{\mathrm{s}}$ and $\mathcal{D}_{\mathrm{t}}$. These datasets
contain $m$ ($n$) labeled videos where the input $\mathbf{X}$ is features for
a video and $\mathbf{y}$ corresponds to its ground-truth of dense video
captions, written as
$\mathcal{D}_{\mathrm{s}}=\left\\{\left(\mathbf{X}_{\mathrm{s}1},\mathbf{y}_{\mathrm{s}1}\right),\left(\mathbf{X}_{\mathrm{s}2},\mathbf{y}_{\mathrm{s}2}\right),\ldots,\left(\mathbf{X}_{\mathrm{s}m},\mathbf{y}_{\mathrm{s}m}\right)\right\\}$
and
$\mathcal{D}_{\mathrm{t}}=\left\\{\left(\mathbf{X}_{\mathrm{t}1},\mathbf{y}_{\mathrm{t}1}\right),\left(\mathbf{X}_{\mathrm{t}2},\mathbf{y}_{\mathrm{t}2}\right),\ldots,\left(\mathbf{X}_{\mathrm{t}n},\mathbf{y}_{\mathrm{t}n}\right)\right\\}$.
The video features $\mathbf{X}$ are encoded by a fixed feature extractor
$\phi$ and represented as a set of frame-wise features:
$\mathbf{X}=\left\\{\mathbf{x}_{1},\mathbf{x}_{2},\ldots,\mathbf{x}_{t}\right\\}$,
where $\mathbf{x}$ is frame features and $t$ is a fixed length of a video. Our
inference model consists of a feature converter network $F$ and a task network
$G$, learning the mapping from $\mathbf{X}$ to $\mathbf{y}$. We also define
the view class for each frame with a view labeling function $V(\cdot)$, which
takes a frame $\mathbf{x}$ and assigns it to one of the predefined view labels
(_i.e_., exo, ego-like, and ego as denoted in Sec. 3.1).

Given an entire model $G\circ F$ and a dataset $\mathcal{D}$, we denote a task
loss function for dense video captioning as
$\mathcal{L}_{\text{task}}\left(F,G,\mathcal{D}\right)$. Using the source and
target datasets, the joint training on the two datasets is defined as

$\mathcal{L}_{\text{task}}\left(F,G,\mathcal{D}_{\mathrm{t}}\right)+\lambda_{\text{src}}\mathcal{L}_{\text{task}}\left(F,G,\mathcal{D}_{\mathrm{s}}\right),$
(1)

where $\lambda_{\text{src}}$ is a weight controlling the training on
$\mathcal{D}_{\mathrm{s}}$.

###  3.1 View labeling and preprocessing

We construct the view labeling function $V(\cdot)$. Based on the composition
analysis of exocentric cooking videos (Fig. 2), the source data can be divided
into face shots showing body actions and hand shots indicating necessary
attention to specific objects. These shots can be interleaved even in a single
step. Due to the visual similarity between the hand shots and the egocentric
images, we define three view classes as exo (face shots) and ego-like (hand
shots) from the source exocentric videos, and ego views from the target
egocentric videos (see the left of Fig. 3).

To process the source videos, we use a face tracking that classifies the face
and hand shots, which consists of a face detector [48] and a simple tracking
method SORT [4]. We assign positive source images where the face is detected
to the exo view and the others to the ego-like view.

To reduce the moving impact of the ego view, we propose to utilize a fine-
grained and temporally coherent tracking of hand-object interactions,
informing actions occurring in moving scenes. We implement this tracking by
the combination of hand detection-based tracking and frame-wise hand-object
segmentation. Specifically, we track the location of bounding boxes covering
two hands using a pre-trained hand detector [50, 54] and the SORT algorithm.
Additionally, we use the pixel-level location of hands and interacting objects
[74], which can provide more precise descriptions of actions. These two
techniques complement each other; the hand tracking is temporally coherent but
coarse localization and the segmentation is fine-grained but frame-wise
localization (see Sec. 3.3 for details).

###  3.2 View-invariant learning

We aim to learn view-invariant features among the mixed source views and the
unique egocentric view. We perform pre-training and fine-tuning separately for
the following reasons. First, from an aspect of gradual domain adaptation [22,
30], it is beneficial to handle the smaller domain gap first and gradually
adapt from the source to an intermediate domain and finally to the target
domain. Unlike standard video domain adaptation across two datasets [6, 17,
38] or two views [8], we resolve the domain gap in camera viewpoints by
introducing an intermediate domain on the source data. This encourages us to
propose a solution of gradual adaptation with the separate training stages.
Second, when developing models with the target data, we may only have access
to a pre-trained model [67, 24], but not to the source web videos due to the
privacy concern [27]. Activity videos usually contain the users’ sensitive
information about their identity and recorded scenes. Indeed, obtaining
consent from the owners of the web videos is cumbersome. Under this
restriction, the decoupled pre-training allows us to study the transferability
of our pre-trained model against a naive pre-training.

We facilitate the learning of invariant features with adversarial training,
while gradually adapting from the exo to the ego-like and finally to the ego
view. Fig. 3 illustrates the overview of the view labeling and our learning
method. The video features are converted to learnable representation by the
converter $F$. These features are fed to the task network $G$ to solve the
captioning task and the classifier $C$ to let the converted features be view-
independent. The classifier $C$ is trained by adversarial adaptation with a
gradient reversal layer [12]. The converter $F$ attempts to produce features
undistinguished by the classifier while the classifier is trained to classify
their views.

View-invariant pre-training in source domain. We pre-train the model on the
source data to let the feature converter $F$ produce view-invariant features.
This learning is facilitated by adversarial loss [12] with the classifier $C$,
which learns the mapping from a frame $\mathbf{x}$ to a view label
$V(\mathbf{x})$:

$\mathcal{L}_{\text{adv}}\left(F,C,\mathcal{D},V\right)=\mathbb{E}_{\mathbf{x}\sim
X\sim\mathcal{D}}\mathcal{L}_{\text{ce}}\left(C\left(F\left(\mathbf{x}\right)\right),V\left(\mathbf{x}\right)\right).$
(2)

We use a cross-entropy loss $\mathcal{L}_{\text{ce}}$ as the objective to
reuse $\mathcal{L}_{\text{adv}}$ in the three-view classification problem in
the next section. The total loss of the pre-training is defined as

$\begin{split}\mathcal{L}_{\text{task}}\left(F,G,\mathcal{D}_{\mathrm{s}}\right)-\lambda_{\text{adv}}\mathcal{L}_{\text{adv}}\left(F,C,\mathcal{D}_{\mathrm{s}},V\right),\end{split}$
(3)

where $\lambda_{\text{adv}}$ is a controlling weight for the adversarial loss.

View-invariant training across three views. We follow similar view-invariant
learning during the training with the source and target data. We first
initialize the entire model $G\circ F$ from a pre-trained checkpoint (Eq. 3)
and reinitialize the classifier $C$. Then, we jointly train the model on the
source and target data (Eq. 1), while the classifier $C$ is trained to
classify a frame $\mathbf{x}$ from both datasets into the three view classes,
formulated as

$\begin{split}\mathcal{L}_{\text{task}}\left(F,G,\mathcal{D}_{\mathrm{t}}\right)&+\lambda_{\text{src}}\mathcal{L}_{\text{task}}\left(F,G,\mathcal{D}_{\mathrm{s}}\right)\\\
&-\lambda_{\text{adv}}\mathcal{L}_{\text{adv}}\left(F,C,\right\\{\mathcal{D}_{\mathrm{s}},\mathcal{D}_{\mathrm{t}}\left\\},V\right).\end{split}$
(4)

Note that the source web data are typically much larger than recorded
egocentric videos ($m\gg n$). To address this, we employ an under-sampling
technique for the task loss of Eq. 1, where the number of the input source
data is balanced to the number of target data per iteration. Similarly, we
address the imbalance of view classes in the adversarial loss of Eq. 2 with
the under-sampling.

Figure 4:  Baseline for egocentric dense video captioning. Our baseline
consists of (i) hand-object encoding and (ii) one-stage captioning with
parallel decoding (PDVC [62]). We first preprocess the egocentric videos with
hand detection (“crop area”) and hand-object segmentation (“hands”, “1st
obj.”, and “2nd obj.”). We extract features for these regions by the fixed
encoder $\phi$ and pass their concatenated features to the feature converter
$F$. The generated video features are fed to a transformer-based captioning
model with two prediction heads of time segment and caption.

###  3.3 Hand-object feature generation

We describe feature encoding for egocentric videos. As discussed in Sec. 3.1,
we crop the videos with the hand tracking. Additionally, we utilize hand-
object masks to localize actions in moving scenes. Not only reducing the
moving effect, these hand and object cues also improve recognition of action
verbs (_e.g_., “pour”) and objects (_e.g_., “olive oil”).

Estimates of hand-object interactions have been considered as a means of
egocentric video representation, such as hand segmentation [26], hand pose
estimation [41], hand-object detection [47], and 3D hand-object features [55,
23]. Owing to the recent advancement of diverse object segmentation [19], our
work attempts to utilize the hand tracking and hand-object segmentation for
the video representation.

Dataset | view |  | paired?
---
text? |  | proc.
---
caption?
domain | source
ActivityNet Captions [21] | exo* |  | ✓ |  | various | YouTube
YouCook2 (YC2) [75] | exo* |  | ✓ | ✓ | cooking | YouTube
EPIC-KITCHENS [9] | ego |  | ✓ |  | cooking | recorded
MMAC Captions [39] | ego |  | ✓ | ✓ | cooking | recorded
YouMakeup [64] | exo* |  | ✓ | ✓ | makeup | YouTube
COIN [53] | exo* |  | ✓ | ✓ | various | YouTube
CrossTask [77] | exo* |  | ✓ | ✓ | various | YouTube
HowTo100M [35] | exo* |  | ✓ |  | various | YouTube
HIREST [73] | exo* |  | ✓ | ✓ | various | YouTube
Ego4D [13] | ego |  | ✓ |  | various | recorded
Charades-Ego [51] | both | P |  |  | various | recorded
H2O [23] | both | P |  |  | various | recorded
Assembly101 [49] | both | P |  |  | assembly | recorded
EgoYC2 (Ours) | ego | WP | ✓ | ✓ | cooking | recorded
Table 1:  The comparison of datasets for human activity understanding. We show
the type of viewpoints (“ego” or “exo”) and whether their views are paired
(“P”: paired, “WP”: weakly paired). We also compare the presence of text
information and procedural captions. The last two columns indicate the domain
and the source of the videos.

Datasets | YC2 | EgoYC2
---|---|---
#video | 2,000 | 226
#user | - | 44
#recipe_class | 89 | 21
total duration | 176 h | 43 h
avg. video dur. | 5.3 min | 11.6 min
avg. segment dur. | 19 sec | 103 sec
avg. step size | 7.7 | 6.5
viewpoint | exo* | ego
user consent | $\times$ | ✓
Table 2:  Statistics of YouCook2 (YC2) and Ego-YouCook2 (EgoYC2). We re-record
11.3% of YouCook2 recipes with a head-mounted camera, resulting in 43 hours of
226 videos.

From the affordance analysis [74, 10, 72], hand-object interactions are
classified into direct and indirect interactions. The direct (first-order)
interaction refers to the case where the hand is in physical contact with an
object. The indirect (second-order) interaction describes the situation where
an object is manipulated using a tool without direct contact. In the scene of
“pouring olive oil to bread” in Fig. 4, the hand directly grasps an olive oil
bottle (_i.e_., first-order object) and pours it on the bread (_i.e_., second-
order object) where the person indirectly interacts with the bread.

To recognize these objects, we propose a practical refinement scheme of hand-
object masks using two segmentation models. We first adopt EgoHOS [74], a pre-
trained segmentation model for hands and interacting objects in egocentric
videos. This model tends to produce inaccurate masks for the objects, as
detecting object boundaries is hard in real-life scenes. To correct this
error, we use a generic segmentation model for any objects (SAM [19]), which
can segment objects in any category and generalize to real-world scenes. Using
this capability, we take an intersection between the two models’ predictions
and use the most overlapped mask from the SAM for the final output.

Given the hand-object masks, we extract features for regions that enclose each
entity using the pre-trained encoder $\phi$ , namely cropped area (blue),
hands (red), first/second-order interacting objects (green/yellow), as shown
in the left of Fig. 4. The feature converter $F$ takes their concatenated
features and produces frame-wise features for an entire video.

###  3.4 Captioning baseline

We employ a one-stage captioning model for the task network $G$, which has a
unified architecture for the two sub-tasks, predicting time segments (_e.g_.,
[033s-125s]) and captions (_e.g_., “apply olive oil and cheesy sauce on the
bread”). The primary reason for this choice is that our proposed transfer
learning is designed to learn view-invariant features in a middle layer of the
encoding pipeline. Thus, using models with separate encoding for the two sub-
tasks would complicate this view-invariant learning. Specifically, we adopt a
strong baseline for dense video captioning with parallel decoding (PDVC) [62]
(see the right of Fig. 4). This model has shown superiority over the two-stage
models with the separate encoding scheme [76, 63]. The task loss
$\mathcal{L}_{\text{task}}$ of PDVC consists of the localization loss of time
segments, the classification for event query, the cross-entropy for predicted
words, and the cross-entropy for event count.

##  4 EgoYC2 Dataset

We describe the details of the target egocentric dataset, EgoYC2. We adopt the
captions from YouCook2 (YC2) [75] to annotate procedural captions, describing
the sequence of necessary steps to complete complex tasks. We then re-record
these cooking videos by participants wearing a head-mounted camera. This
allows us to align the two datasets with unified caption content and
granularity.

The dataset comparison is shown in Tab. 2111“exo*” view indicates that the
video may not be captured from a fixed viewpoint, _e.g_., YouTube videos
contain scene cuts with different views.. Our dataset is the unique egocentric
dataset with procedural captions shared with exocentric videos. Charades-Ego
[51], H2O [23], and Assembly101 [49] are captured from both egocentric and
exocentric views in the paired setting, but these datasets feature action
labels. We denote our dataset as weakly-paired data because captured
environments are different, but the caption definition is aligned to YC2.
Unlike the knowledge transfer of action recognition between exo-to-ego views
in Charades-Ego and ego-to-ego views in EPIC-KITCHENS [38], our transfer
learning benchmark features a cross-dataset setting of the exo-to-ego
transfer, relaxing the strict alignment of video capture setups.

Tab. 2 shows the statistics compared to YC2. Our videos reach 11.3 % of YC2
videos downloaded from YouTube. Our untrimmed videos have longer video and
per-segment duration. Unlike the MMAC Captions [39], we provide caption
annotations aligned to the YC2 to let models address the view gaps without
considering the gaps in captions. We use the same vocabulary list as the YC2
and have close step sizes (7.7 in YC2 _vs_. 6.5 in EgoYC2 _vs_. 30.1 in MMAC
Captions). We present other details in the supplement.

Figure 5:  Time segmentation by detected AR markers. In the transition of
cooking steps, we ask participants to check the next step on their smartphone
or tablet and display an AR marker once they confirm the next step. Given a
recorded video, we postprocess it to detect the marker and segment the video
temporally.

Method | Input | dvc_eval | SODA
---|---|---|---
| B4 | M | C | M | C | tIoU
| Source only | V | 0.00 | 0.77 | 3.6 | 0.89 | 1.47 | 17.9
Baseline | PT+FT | V | 1.54 | 7.03 | 38.1 | 7.03 | 25.2 | 50.5
| VC | 1.97 | 8.20 | 46.3 | 8.04 | 32.3 | 55.0
| VC+HO | 1.68 | 8.91 | 52.5 | 8.91 | 37.3 | 59.0
\+ DANN [12] | VC+HO | 1.59 | 8.90 | 51.4 | 8.83 | 37.5 | 58.1
Ours | VI-PT + FT | VC+HO | 2.06 | 9.44 | 55.2 | 9.02 | 39.5 | 56.0
PT + VI-FT | VC+HO | 1.77 | 8.89 | 53.0 | 8.91 | 37.2 | 59.1
VI-PT + VI-FT | VC+HO | 2.66 | 9.19 | 59.0 | 9.27 | 45.2 | 58.1
Table 3:  Quantitative results in transfer learning from YouCook2 (YC2) to
EgoYC2.  We run pre-training (PT) and fine-tuning (FT) with or without the
view-invariant (VI) learning. We also compare various input feature types: raw
videos (V), cropped videos (VC), and that with hand-object features (VC+HO).

VI? | dvc_eval | SODA
---|---|---
PT | FT | B4 | M | C | M | C
- | - | 0.98 | 10.00 | 50.6 | 12.62 | 40.4
✓ | - | 1.62 | 10.35 | 55.5 | 13.57 | 47.9
- | ✓ | 1.07 | 10.13 | 51.7 | 12.70 | 40.9
✓ | ✓ | 1.68 | 10.20 | 54.1 | 13.36 | 48.7
Table 4:  Analysis of captioning performance with GT proposals. We evaluate
our comparison models in Tab. 4 given ground-truth (GT) time segments. We use
the VC+HO feature as the input. ‘VI” indicates our proposed method of view-
invariant learning introduced in Sec. 3.2.

Transfer learning setup. We use YC2 and EgoYC2 as the source and target data,
respectively. We split the EgoYC2 dataset into train and evaluation sets with
151 (964) and 75 (511) videos (step descriptions), respectively. To align both
datasets, we re-split the YC2 dataset according to the EgoYC2’s split, where
train and evaluation sets have 1,716 (13,324) and 75 (511) videos (step
descriptions), respectively. The evaluation sets correspond to each other, and
all the YC2 data that are not re-recorded in this work are included in the
training set.

Time segment annotation. Besides the annotations of step descriptions, we need
to annotate the time segment of each step description (_i.e_., start and end
time). We propose automatic time stamp annotation using AR markers displayed
on a virtual screen, as shown in Fig. 5. We ask the participants to use their
smartphone or tablet to see a step description in the transition of the steps.
We show an AR marker on the screen once they confirm the next step. We
postprocess a recorded video by detecting the displayed markers and segment it
temporally. This allows us to annotate the time segments without manually
inspecting the entire video.

##  5 Experiments

###  5.1 Experimental setup

Implementation details. We employ PyTorch for implementation and run all
experiments on a single NVIDIA V100 16GB GPU. The video features are generated
by a pre-trained encoder $\phi$, ResNet152 [14]. We present different video
presentations for egocentric videos: raw video features (V), cropped video
features (VC), and the one with hand-object features (VC+HO). We train models
jointly on the source and target data to avoid overfitting the target data. We
set $\lambda_{\text{src}}$ and $\lambda_{\text{adv}}$ as 0.1, and the fixed
video length $t$ as 200. We denote naive pre-training and fine-tuning as PT+FT
and the view-invariant learning as VI-(PT/FT). Additional details are shown in
the supplement.

Evaluation. We evaluate the performance in the target domain with five times
experiments with different random seeds. We report the average scores of these
two metrics: dvc_eval [21] and SODA [11]. The dvc_eval metric computes the
average precision of the matched pairs between the prediction and the ground
truth, namely BLEU4 (B4) [44], METEOR (M) [2], and CIDEr (C) [58]. The SODA
metric considers the storytelling quality for an entire video, w.r.t. the
order of captions and their redundancy. Following [40], we show METEOR (M),
CIDEr (C), and temporal Intersection-over-Union (tIoU) as the SODA scores.

Figure 6:  Qualitative results (recipe: scrambled eggs). We show generated
captions given time segment proposals from prediction or ground-truth. We
compare our ablation models: view-invariant (VI) pre-training (PT) and/or
view-invariant (VI) fine-tuning (FT). The marks $\square$ and $\vartriangle$
indicate failure cases for irrelevant ingredients and duplicate captions.

###  5.2 Results

We report the results of the transfer learning task quantitatively and
qualitatively, and show ablation results. We provide additional results in the
supplement, _e.g_., the choice of hyperparameters and hand-object mask
refinement.

Results of transfer learning. Tab. 4 shows performance comparisons in the
transfer learning setting. The source only shows the zero-shot generalization
ability of a trained model on YC2 to EgoYC2. The significant view gaps prevent
its generalization to egocentric videos. Naive supervised fine-tuning (FT)
after pre-training (PT), PT+FT, boosts the performance and achieves better
results with hand-object cues (see the following section). This shows that,
despite the view gaps, pre-training on the web video benefits the video
captioning task in egocentric videos.

Our proposed view-invariant (VI) pre-training on the source data, VI-PT + FT,
exhibits better captioning results than adaptation baselines from a naively
pre-trained model (PT + VI-FT and PT+FT + DANN [12]). This shows that the
proposed pre-training improves the transferability of the pre-trained model to
unknown perspectives like the ego view. Our full method (VI-PT + VI-FT) has
shown superiority over the VI-PT + FT method by 6.8% and 14.4% in the CIDEr
scores of dvc_eval and SODA, respectively. This proves the effectiveness of
adapting gradually to the ego view with the pre-training and fine-tuning
stages.

Ablation study of egocentric video representations. We validate different
representations for egocentric videos in Tab. 4, namely raw videos (V),
cropped videos (VC), and the VC with hand-object features (VC+HO). We set
PT+FT as a base training setting. First, a simple cropping technique with the
hand tracking (VC) is effective, exhibiting a 21.5 % improvement in the CIDEr
score over the V input. This suggests that tracking the hand region reduces
the effect of complex scene changes and makes the task more tractable. With
the explicit cues of hand-object features (VC+HO), we observe the gain of the
CIDEr score by 13.4 % over the VC input and achieve better results except for
the B4 score.

Analysis of captioning capability. Tab. 4 shows captioning results given GT
time segments, enabling the evaluation of pure captioning ability. Similarly
to Tab. 4, models based on view-invariant pre-training (VI-PT + FT and our
full method) have higher captioning performance. Since the given time segments
will not overlap, the generated captions have fewer duplicate sentences, as
shown in Fig. 6.

Qualitative results. Fig. 6 shows qualitative results of our ablation models
with generated captions and predicted time segments. Regarding time
segmentation, while PT+FT and PT + VI-FT generate overlapped segments and
overlook some segments, VI-PT + FT and our full method correct these
localization errors. In generated captions, we observe several failure
patterns: appearing unrelated ingredients and duplicate captions. The captions
of PT+FT and PT + VI-FT include the unrelated ingredients (_e.g_., “tofu”,
“pork”, and “udon noodles”) with and without GT proposals. We also find
repeated sentences (triangles) in the models without the view-invariant fine-
tuning (_i.e_., PT+FT and VI-PT + FT). These observations suggest that the
view-invariant pre-training reduces the mixing of unrelated ingredients, and
the later view-invariant fine-tuning helps produce fewer repeated sentences.

Figure 7:  Visualization of feature distribution ($\CIRCLE$: exo, $\CIRCLE$:
ego-like, $\blacktriangle$: ego). We visualize the source and target features
encoded in each training stage with t-SNE [32]. Left: initial features
generated by the encoder $\phi$, Middle: after view-invariant pre-training
(VI-PT) on the source data, Right: after view-invariant fine-tuning (VI-PT +
VI-FT) on both datasets.

Visualization of feature distribution. To see the transition of the view
distribution, we visualize feature distribution for the view classes in Fig.
7. This confirms that our adaptation method aligns the exo and ego-like views
on the source data first (after VI-PT) and then aligns all three views (after
VI-PT + VI-FT). Interestingly, our method enables aligning visually similar
domains: the ego-like (green) view indicating attention to objects and the ego
view (red).

##  6 Conclusion

We present a novel benchmark for cross-view knowledge transfer of dense video
captioning from exocentric to egocentric views, along with a new dataset
EgoYC2. We collect 11.3% of YouCook2 videos from an egocentric view with
aligned captions, enabling transfer learning between both datasets. Our
proposed view-invariant learning based on adversarial training succeeds in the
pre-training and fine-tuning stages while resolving the mixed source views and
the moving target view. We validate our proposed method in the cross-view
transfer task with quantitative and qualitative analysis. This benchmark will
promote further studies of transfer learning across the two views and modeling
to describe egocentric activities in natural language.

### Acknowledgments

We thank Kuniaki Saito and Ta Gu for helpful discussions on this manuscript.
This work was supported by JST ACT-X Grant Number JPMJAX2007, JSPS KAKENHI
Grant Number JP22KJ0999, JST ACT-I Grant Number JPMJPR17U5, JSPS KAKENHI Grant
Number 21H04910, and 20H04205, and partially supported by a hardware donation
from Yu Darvish.

## Appendix

##  Appendix A Dataset Details

Video recording. We ask 44 participants to record cooking activities in their
own home kitchens using a head-mounted GoPro camera. The cooking recipes are
adopted from YouCook2 [75] captions with 2,000 recipes consisting of 82 recipe
classes (_e.g_., “BLT” is a class and multiple recipes belong to the class).
Each participant chooses five recipes at will so that selected classes do not
overlap and then prepares the meal by following the step descriptions written
in the recipe. In total, we collect 226 videos totaling 43 hours. We also
received approval for this activity data collection from an Institutional
Review Board and obtained consent from participants who joined this recording.

Fig. 8 indicates our web application used for our video recording, displaying
the instruction of video collection, step descriptions, reference videos from
YouCook2, and necessary ingredients extracted from annotated captions. This
web interface helps participants prepare ingredients and check how to cook
from the reference videos before their recording. During recording, the
highlighted step description is shown to indicate their current step and
changes by manipulating the bottom below. The AR markers are displayed on the
screen in the transition of steps, which are used to annotate temporal
segments.

Figure 8:  Web user interface for our recording. Top left: Instruction of
recording, Top right: Step description with the focus on the current step,
Bottom left: Reference video from YouCook2 [75], Bottom right: Necessary
ingredients extracted from captions.

Recipe class distribution. Figs. 9 and 10 show the distribution of recipe
classes for EgoYC2 and YC2. We collect 21 recipe classes out of 89 classes in
YouCook2. The collected recipe list of EgoYC2 is as follows: BLT, authentic
japanese ramen, bangers and mash, caesar salad, croque monsieur, galbi, hot
dogs, macaroni and cheese, mapo tofu, miso soup, pancake, pasta salad, pork
fried rice, salmon nigiri, salmon sashimi, scrambled eggs, shrimp tempura,
spaghetti carbonara, tuna sashimi, udon noodle soup, yaki udon noodle.

Apart from the view gaps discussed in the paper, jointly addressing the
category shift [5, 31, 66] (the gap in activity (recipe) classes between the
two datasets) will be an important future challenge.

Figure 9: Recipe distribution in EgoYC2 Figure 10: Recipe distribution in
YouCook2 [75]

##  Appendix B Additional Implementation Details

The architectures of the feature converter $F$ and the view classifier $C$
follow a two-layer one-dimensional CNN and a three-layer MLP, respectively.
The video features are represented as 2,048-dimensional feature vectors for an
input image. We use PDVC [59] as a baseline for dense video captioning. The
PDVC uses a two-layer deformable transformer with a hidden size of 512 in
attention layers and 2,048 in feed-forward layers. The number of event queries
is set to 100 and the mini-batch size is set to 1. We use the Adam [18]
optimizer with an initial learning rate of 1e-5 for the feature converter and
PDVC, and 1e-4 for the view classifier. While we validate various input types
for the target egocentric videos, we use the original video features generated
by TSN [60] on YouCook2.

##  Appendix C Additional Results

Analysis of hyperparameter settings. We set the hyperparameter of view-
invariant learning ($\lambda_{\text{adv}}$) by observing the source
performance of the view-invariant pre-training (VI-PT). We use the sum of two
METEOR metrics (sum_METEOR) for the model selection during the pre-training.
We choose the hyperparameter with the highest sum_METEOR value and set
$\lambda_{\text{adv}}$ as 0.1 consistently for the fine-tuning in the target
domain.

We additionally evaluate the performance in the pre-training and fine-tuning
stages, according to different hyperparameters in Tab. 5. The pre-training
with $\lambda_{\text{adv}}=0.01,0.1$ achieves relatively high performance,
while the fine-tuning with $\lambda_{\text{adv}}=0.01,1$ worsens performance
than the PT+FT baseline (top row). When adding the view-invariant technique to
both the pre-training and fine-tuning, we observe an improvement of captioning
ability with $\lambda_{\text{adv}}=0.01,0.1$, as they are adapted from the
pre-training models where the view-invariant learning performs well. Based on
this study, our setting of $\lambda_{\text{adv}}=0.1$ chosen from the source
pre-training performs stably in the target domain with both the pre-training
and fine-tuning stages.

VI? | $\lambda_{\text{adv}}$ | dvc_eval | SODA
---|---|---|---
PT | FT | B4 | M | C | M | C | tIoU
|  | 0 | 1.68 | 8.91 | 52.5 | 8.91 | 37.3 | 59.0
✓ |  | 0.01 | 2.20 | 9.45 | 52.4 | 8.99 | 39.9 | 55.0
✓ |  | 0.1 | 2.06 | 9.44 | 55.2 | 9.02 | 39.5 | 56.0
✓ |  | 1 | 1.70 | 9.29 | 50.5 | 8.75 | 36.4 | 55.1
| ✓ | 0.01 | 1.47 | 8.75 | 49.8 | 8.72 | 35.8 | 58.8
| ✓ | 0.1 | 1.77 | 8.89 | 53.0 | 8.91 | 37.2 | 59.1
| ✓ | 1 | 1.50 | 8.67 | 49.4 | 8.67 | 35.6 | 59.5
✓ | ✓ | 0.01 | 2.46 | 9.60 | 53.1 | 8.99 | 39.3 | 55.4
✓ | ✓ | 0.1 | 2.66 | 9.19 | 59.0 | 9.27 | 45.2 | 58.1
✓ | ✓ | 1 | 1.58 | 9.30 | 49.7 | 8.67 | 35.3 | 54.9
Table 5:  Analysis of hyperparameter settings. We validate different
hyperparameters for the view-invariant learning ($\lambda_{\text{adv}}$) and
show the performance on the target dataset.

Hand-object segmentation results. We propose a segmentation refinement scheme
based on two segmentation models: EgoHOS [74] and SAM [19]. We show
segmentation results for each method in Fig. 11. The EgoHOS inference (left)
often has noisy results (_e.g_., undersegmetation on the top row and incorrect
localization of long and narrow objects on the middle row). EgoHOS suffers
from generalizing to novel real-life environments where diverse object types
and shapes could be present. The SAM inference (middle) can segment any kind
of object with higher generalization. Our refinement (right) computes the
overlap between the two results and outputs the most overlapped segments from
the SAM predictions. This enables us to obtain further refined results even in
crowded cooking environments (_e.g_., middle row).

Figure 11: Our hand-object segmentation refinement. Each panel shows
segmentation results of EgoHOS [74] (left), SAM [19] (middle), and our refined
scheme (right), respectively. Since we don’t use hand identity information
(right/left), we show merged hand masks compared to the results of EgoHOS.

## References

  * [1] J.-B. Alayrac, P. Bojanowski, N. Agrawal, J. Sivic, I. Laptev, and S. Lacoste-Julien.  Unsupervised learning from narrated instruction videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4575–4583, 2016.
  * [2] S. Banerjee and A. Lavie.  METEOR: an automatic metric for mt evaluation with improved correlation with human judgments.  In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, 2005.
  * [3] S. Bansal, C. Arora, and C. V. Jawahar.  My view is the best view: Procedure learning from egocentric videos.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 657–675, 2022.
  * [4] A. Bewley, Z. Ge, L. Ott, F. T. Ramos, and B. Upcroft.  Simple online and realtime tracking.  In Proceedings of the IEEE International Conference on Image Processing (ICIP), pages 3464–3468, 2016.
  * [5] Z. Cao, L. Ma, M. Long, and J. Wang.  Partial adversarial domain adaptation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 139–155, 2018.
  * [6] M.-H. Chen, Z. Kira, G. Alregib, J. Yoo, R. Chen, and J. Zheng.  Temporal attentive alignment for large-scale video domain adaptation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6320–6329, 2019.
  * [7] S. Chen, X. Nie, D. Fan, D. Zhang, V. Bhat, and R. Hamid.  Shot contrastive self-supervised learning for scene boundary detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9796–9805, 2021.
  * [8] J. Choi, G. Sharma, M. Chandraker, and J.-B. Huang.  Unsupervised and semi-supervised domain adaptation for action recognition from drones.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1706–1715, 2020.
  * [9] D. Damen, H. Doughty, G. M. Farinella, A. Furnari, J. Ma, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray.  Rescaling egocentric vision.  International Journal of Computer Vision (IJCV), 130(1):33–55, 2022\.
  * [10] A. Darkhalil, D. Shan, B. Zhu, J. Ma, A. Kar, R. E. L. Higgins, S. Fidler, D. Fouhey, and D. Damen.  EPIC-KITCHENS VISOR benchmark: Video segmentations and object relations.  In Proceedings of the Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, 2022.
  * [11] S. Fujita, T. Hirao, H. Kamigaito, M. Okumura, and M. Nagata.  SODA: story oriented dense video captioning evaluation framework.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 517–531, 2020.
  * [12] Y. Ganin and V. Lempitsky.  Unsupervised domain adaptation by backpropagation.  In Proceedings of the International Conference on Machine Learning (ICML), pages 1180–1189, 2015.
  * [13] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, M. Martin, T. Nagarajan, I. Radosavovic, S. K. Ramakrishnan, F. Ryan, J. Sharma, M. Wray, M.g Xu, E. Zhongcong Xu, C. Zhao, S. Bansal, D. Batra, V. Cartillier, S. Crane, T. Do, M. Doulaty, A. Erapalli, C. Feichtenhofer, A. Fragomeni, Q. Fu, C. Fuegen, A. Gebreselasie, C. Gonzalez, J. Hillis, X. Huang, Y. Huang, W. Jia, W. Khoo, J. Kolar, S. Kottur, A. Kumar, F. Landini, C. Li, Y. Li, Z. Li, K. Mangalam, R. Modhugu, J. Munro, T. Murrell, T. Nishiyasu, W. Price, P. R. Puentes, M. Ramazanova, L. Sari, K. Somasundaram, A. Southerland, Y. Sugano, R. Tao, M. Vo, Y. Wang, X. Wu, T. Yagi, Y. Zhu, P. Arbelaez, D. Crandall, D. Damen, G. M. Farinella, B. Ghanem, V. K. Ithapu, C. V. Jawahar, H. Joo, K. Kitani, H. Li, R. Newcombe, A. Oliva, H. Soo Park, J. M. Rehg, Y. Sato, J. Shi, M. Z. Shou, A. Torralba, Lo Torresani, M.i Yan, and J. Malik.  Ego4D: Around the world in 3, 000 hours of egocentric video.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18973–18990, 2022.
  * [14] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [15] S.-H. Kang and J.-H. Han.  Video captioning based on both egocentric and exocentric views of robot vision for human-robot interaction.  International Journal of Social Robotics, 15(4):631–641, 2023.
  * [16] R. Khirodkar, A. Bansal, L. Ma, R. Newcombe, M. Vo, and K. Kitani.  Ego-humans: An ego-centric 3d multi-human benchmark.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 19807–19819, 2023.
  * [17] D. Kim, Y.-H. Tsai, B. Zhuang, X. Yu, S. Sclaroff, K. Saenko, and M. Chandraker.  Learning cross-modal contrastive features for video domain adaptation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 13598–13607, 2021.
  * [18] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  In Proceedings of the International Conference on Learning Representations (ICLR), 2014.
  * [19] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollar, and R. Girshick.  Segment anything.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4015–4026, 2023.
  * [20] D. Ko, J. Choi, J. Ko, S. Noh, K.-W. On, E.-S. Kim, and H. J. Kim.  Video-text representation learning via differentiable weak temporal alignment.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5006–5015, 2022.
  * [21] R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. C. Niebles.  Dense-captioning events in videos.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 706–715, 2017.
  * [22] A. Kumar, T. Ma, and P. Liang.  Understanding self-training for gradual domain adaptation.  In Proceedings of the International Conference on Machine Learning (ICML), pages 5468–5479, 2020.
  * [23] T. Kwon, B. Tekin, J. Stühmer, F. Bogo, and M. Pollefeys.  H2O: two hands manipulating objects for first person interaction recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10118–10128, 2021.
  * [24] K. Li, D. Patel, E. Kruus, and M. R. Min.  Source-free video domain adaptation with spatial-temporal-historical consistency learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14643–14652, 2023.
  * [25] Y. Li, T. Nagarajan, B. Xiong, and K. Grauman.  Ego-exo: Transferring visual representations from third-person to first-person videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6943–6953, 2021.
  * [26] Y. Li, Z. Ye, and J. M. Rehg.  Delving into egocentric actions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 287–295, 2015.
  * [27] J. Liang, D. Hu, and J. Feng.  Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation.  In Proceedings of the International Conference on Machine Learning (ICML), pages 6028–6039, 2020.
  * [28] K. Q. Lin, J. Wang, M. Soldan, M. Wray, R. Yan, E. Z. Xu, D. Gao, R.-C. Tu, W. Zhao, W. Kong, C. Cai, H. Wang, D. Damen, B. Ghanem, W. Liu, and M. Z. Shou.  Egocentric video-language pretraining.  In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2022.
  * [29] X. Lin, F. Petroni, G. Bertasius, M. Rohrbach, S.-F. Chang, and L. Torresani.  Learning to recognize procedural activities with distant supervision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13843–13853, 2022.
  * [30] D. Liu, Z. Bao, J. Mi, Y. Gan, M. Ye, and J. Zhang.  Cross-domain video action recognition via adaptive gradual learning.  Neurocomputing, 556:126622, 2023.
  * [31] Y. Luo, L. Zheng, T. Guan, J. Yu, and Y. Yang.  Taking a closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2507–2516, 2019.
  * [32] L. V. D. Maaten and G. Hinton.  Visualizing data using t-SNE.  Journal of Machine Learning Research (JMLR), 9:2579–2605, 2008\.
  * [33] I. Majil, M.-T. Yang, and S. Yang.  Augmented reality based interactive cooking guide.  Sensors, 22(21):8290, 2022.
  * [34] A. Miech, J.-B. Alayrac, L. Smaira, I. Laptev, J. Sivic, and A. Zisserman.  End-to-end learning of visual representations from uncurated instructional videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9876–9886, 2020.
  * [35] A. Miech, D. Zhukov, J. B. Alayrac, M. Tapaswi, I. Laptev, and J. Sivic.  Howto100m: Learning a text-video embedding by watching hundred million narrated video clips.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2630–2640. IEEE, 2019.
  * [36] K. Miura, M. Takano, R. Hamada, I. Ide, S. Sakai, and H. Tanaka.  Associating semantically structured cooking videos with their preparation steps.  IEICE Transactions on Information and Systems, 36(2):51–62, 2005\.
  * [37] S. Motiian, M. Piccirilli, D. A. Adjeroh, and G. Doretto.  Unified deep supervised domain adaptation and generalization.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5716–5726, 2017.
  * [38] J. Munro and D. Damen.  Multi-modal domain adaptation for fine-grained action recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 119–129, 2020.
  * [39] K. Nakamura, H. Ohashi, and M. Okada.  Sensor-augmented egocentric-video captioning with dynamic modal attention.  In Proceedings of the ACM International Conference on Multimedia (MM), pages 4220–4229, 2021.
  * [40] T. Nishimura, A. Hashimoto, Y. Ushiku, H. Kameko, and S. Mori.  Recipe generation from unsegmented cooking videos.  CoRR, abs/2209.10134, 2022.
  * [41] T. Ohkawa, K. He, F. Sener, T. Hodan, L. Tran, and C. Keskin.  AssemblyHands: Towards egocentric activity understanding via 3D hand pose estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12999–13008, 2023.
  * [42] T. Ohkawa, Y.-J. Li, Q. Fu, R. Furuta, K. M. Kitani, and Y. Sato.  Domain adaptive hand keypoint and pixel localization in the wild.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 68–87, 2022.
  * [43] T. Ohkawa, T. Yagi, A. Hashimoto, Y. Ushiku, and Y. Sato.  Foreground-aware stylization and consensus pseudo-labeling for domain adaptation of first-person hand segmentation.  IEEE Access, 9:94644–94655, 2021.
  * [44] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 311–318, 2002.
  * [45] A. Pardo, F. C. Heilbron, J. L. AlcÃ¡zar, A. K. Thabet, and B. Ghanem.  MovieCuts: a new dataset and benchmark for cut type recognition.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 668–685, 2022.
  * [46] S. Pramanick, Y. Song, S. Nag, K. Q. Lin, H. Shah, M. Z. Shou, R. Chellappa, and P. Zhang.  Egovlpv2: Egocentric video-language pre-training with fusion in the backbone.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5285–5297, 2023.
  * [47] F. Ragusa, A. Furnari, S. Livatino, and G. M. Farinella.  The MECCANO dataset: Understanding human-object interactions from egocentric videos in an industrial-like domain.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1568–1577, 2021.
  * [48] F. Schroff, D. Kalenichenko, and J. Philbin.  Facenet: A unified embedding for face recognition and clustering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 815–823, 2015.
  * [49] F. Sener, D. Chatterjee, D. Shelepov, K. He, D. Singhania, R. Wang, and A. Yao.  Assembly101: A large-scale multi-view video dataset for understanding procedural activities.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21096–21106, 2022.
  * [50] D. Shan, J. Geng, M. Shu, and D. Fouhey.  Understanding human hands in contact at internet scale.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9866–9875, 2020.
  * [51] G. A. Sigurdsson, A. Gupta, C. Schmid, A. Farhadi, and K. Alahari.  Actor and observer: joint modeling of first and third-person videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7396–7404, 2018.
  * [52] T. Soucek, J.-B. Alayrac, A. Miech, I. Laptev, and J. Sivic.  Look for the change: Learning object states and state-modifying actions from untrimmed web videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13936–13946, 2022.
  * [53] Y. Tang, D. Ding, Y. Rao, Y. Zheng, D. Zhang, L. Zhao, J. Lu, and J. Zhou.  COIN: A large-scale dataset for comprehensive instructional video analysis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1207–1216, 2019.
  * [54] K. Tango, T. Ohkawa, R. Furuta, and Y. Sato.  Background mixup data augmentation for hand and object-in-contact detection.  In Proceedings of the European Conference on Computer Vision Workshops (ECCVW), 2022.
  * [55] B. Tekin, F. Bogo, and M. Pollefeys.  H+O: unified egocentric recognition of 3D hand-object poses and interactions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4511–4520, 2019.
  * [56] T.-D. Truong and K. Luu.  Cross-view action recognition understanding from exocentric to egocentric perspective.  CoRR, abs/2305.15699, 2023.
  * [57] E. Tzeng, J. Hoffman, T. Darrell, and K. Saenko.  Simultaneous deep transfer across domains and tasks.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4068–4076, 2015.
  * [58] R. Vedantam, C. L. Zitnick, and D. Parikh.  CIDEr: consensus-based image description evaluation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4566–4575, 2015.
  * [59] J. Wang, W. Jiang, L. Ma, W. Liu, and Y. Xu.  Bidirectional attentive fusion with context gating for dense video captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7190–7198, 2018.
  * [60] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. Van Gool.  Temporal segment networks for action recognition in videos.  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 41(11):2740–2755, 2019.
  * [61] Q. Wang, L. Zhao, L. Yuan, T. Liu, and X. Peng.  Learning from semantic alignment between unpaired multiviews for egocentric video recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3307–3317, 2023.
  * [62] T. Wang, R. Zhang, Z. Lu, F. Zheng, R. Cheng, and P. Luo.  End-to-end dense video captioning with parallel decoding.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6827–6837, 2021.
  * [63] T. Wang, H. Zheng, M. Yu, Q. Tian, and H. Hu.  Event-centric hierarchical representation for dense video captioning.  IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 31(5):1890–1900, 2021.
  * [64] W. Wang, Y. Wang, S. Chen, and Q. Jin.  Youmakeup: A large-scale domain-specific multimodal dataset for fine-grained semantic comprehension.  In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5132–5142, 2019.
  * [65] H. Wu, K. Chen, H. Liu, M. Zhuge, B. Li, R. Qiao, X. Shu, B. Gan, L. Xu, B. Ren, M. Xu, W. Zhang, R. Ramachandra, C.-W. Lin, and B. Ghanem.  NewsNet: a novel dataset for hierarchical temporal segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10669–10680, 2023.
  * [66] Y. Xu, J. Yang, H. Cao, Z. Chen, Q. Li, and K. Mao.  Partial video domain adaptation with partial adversarial temporal attentive network.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9312–9321, 2021.
  * [67] Y. Xu, J. Yang, H. Cao, K. Wu, M. Wu, and Z. Chen.  Source-free video domain adaptation by learning temporal consistency for action recognition.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 147–164, 2022.
  * [68] Z. Xue and K. Grauman.  Learning fine-grained view-invariant representations from unpaired ego-exo videos via temporal alignment.  CoRR, abs/2306.05526, 2023.
  * [69] A. Yang, A. Nagrani, I. Laptev, J. Sivic, and C. Schmid.  VidChapters-7M: video chapters at scale.  In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS): Datasets and Benchmarks Track, 2023.
  * [70] A. Yang, A. Nagrani, P. H. Seo, A. Miech, J. Pont-Tuset, I. Laptev, J. Sivic, and C. Schmid.  Vid2Seq: large-scale pretraining of a visual language model for dense video captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10714–10726, 2023.
  * [71] S. Yang, S. Kwak, J. Lee, and J. Kim.  Beyond Instructions: a taxonomy of information types in how-to videos.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI), pages 797:1–797:21, 2023.
  * [72] Z. Yu, Y. Huang, R. Furuta, T. Yagi, Y. Goutsu, and Y. Sato.  Fine-grained affordance annotation for egocentric hand-object interaction videos.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2154–2162, 2023.
  * [73] A. Zala, J. Cho, S. Kottur, X. Chen, B. Oguz, Y. Mehdad, and M. Bansal.  Hierarchical video-moment retrieval and step-captioning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23056–23065, 2023.
  * [74] L. Zhang, S. Zhou, S. Stent, and J. Shi.  Fine-grained egocentric hand-object segmentation: Dataset, model, and applications.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 127–145, 2022.
  * [75] L. Zhou, C. Xu, and J. J. Corso.  Towards automatic learning of procedures from web instructional videos.  In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 7590–7598, 2018.
  * [76] L. Zhou, Y. Zhou, J. J. Corso, R. Socher, and C. Xiong.  End-to-end dense video captioning with masked transformer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8739–8748, 2018.
  * [77] D. Zhukov, J.-B. Alayrac, R. G. Cinbis, D. F. Fouhey, I. Laptev, and J. Sivic.  Cross-task weakly supervised learning from instructional videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3537–3545, 2019.