# Show Me What I Like: Detecting User-Specific Video Highlights Using Content-
Based Multi-Head Attention

Uttaran Bhattacharya
University of Maryland
College Park, MD, USA
Work partially done while Uttaran was an intern at Adobe Research Gang Wu
Adobe Research
San Jose, CA, USA
Stefano Petrangeli
Adobe Research
San Jose, CA, USA
Viswanathan Swaminathan
Adobe Research
San Jose, CA, USA
Dinesh Manocha
University of Maryland
College Park, MD, USA

###### Abstract

We propose a method to detect individualized highlights for users on given
target videos based on their preferred highlight clips marked on previous
videos they have watched. Our method explicitly leverages the contents of both
the preferred clips and the target videos using pre-trained features for the
objects and the human activities. We design a multi-head attention mechanism
to adaptively weigh the preferred clips based on their object- and human-
activity-based contents, and fuse them using these weights into a single
feature representation for each user. We compute similarities between these
per-user feature representations and the per-frame features computed from the
desired target videos to estimate the user-specific highlight clips from the
target videos. We test our method on a large-scale highlight detection dataset
containing the annotated highlights of individual users. Compared to current
baselines, we observe an absolute improvement of 2–4% in the mean average
precision of the detected highlights. We also perform extensive ablation
experiments on the number of preferred highlight clips associated with each
user as well as on the object- and human-activity-based feature
representations to validate that our method is indeed both content-based and
user-specific.

Figure 1: User-Specific Highlights for a Variety of Users and Target Videos.
For each user, we consider a set of highlight clips denoting their individual
preferences (left) and detect highlights for them (right) on different target
videos (center top and bottom). Given the users’ overall highlight
preferences, our method employs a multi-head attention (MHA) mechanism to
learn which segments of the target videos are relevant highlights based on
feature similarities between them (center block). For example, our method
learns that user A prefers watching cooking and workout videos. Therefore,
given a target video containing cooking and eating, our method identifies that
only the cooking segments are relevant between the preferred clips and the
target videos, and detects those as highlights. Similarly, our method learns
that user B prefers skating and surfing videos. Therefore, given a video
containing parkour and skating activities, our method detects only the skating
segments as highlights. Overall, our method significantly advances the state-
of-the-art in user-specific highlight detection given a diverse, large-scale
dataset of user preferences and target videos.

##  1 Introduction

The rapid growth in the number of unedited videos due to the commercial
availability of cameras and video sharing platforms requires the development
of automated tools for condensing videos to their most interesting or
highlightable moments. This problem is commonly referred to as highlight
detection [42]. It makes indexing and browsing videos manageable and expedites
large-scale previewing, sharing, and recommending of videos [35]. Ongoing
research efforts have led to the development of efficient content-based
highlight detection methods [42, 11, 37, 2]. More recently, highlight
detection methods have also considered viewers’ preferences to provide
individualized or user-specific highlights [35], making these methods more
practically relevant. However, current user-specific methods either require
the expensive computation of shot boundaries to divide videos into
highlightable and non-highlightable segments [11] or uniformly pool the users’
selected highlights to compute their highlight preferences [35].

In practice, users’ preferences may not be uniformly distributed across their
preferred highlight clips but vary significantly based on the clip contents,
especially in relation to the target videos. For example, in Fig. 1, the
cooking clips are more relevant indicators of user A’s preferences compared to
the workout clips, given the target video containing cooking and eating. Thus,
combining the users’ preferred clips to learn their relevant highlight
preferences for each target video requires learning the relevant features
describing the highlightable content in each preferred clip. Subsequently,
these relevant features need to be mapped to the contents of the target videos
where the highlights are to be detected, requiring an automatic strategy to
estimate the segments of the target videos that are similar to the relevant
features from the preferred clips and, therefore, highlightable. Thus, for
user A, a user-specific highlight detection approach needs to detect that the
cooking segments in the target video are relevant to the user’s preferred
cooking clips, and that both the user’s preferred workout clips and the eating
segments in the target video are irrelevant in the current context.

Detecting highlights based on the similarities between the preferred clips and
the target videos can also be expressed as an attention-based retrieval
problem. An attention mechanism adaptively weighs the keys of different key-
value pairs based on their relative importance to a given query to predict the
most suitable responses to the query [45]. Depending on the data paradigm of
the key, the value, and the query, attention mechanisms are used in a wide
variety of tasks, including tasks in natural language understanding [9], text-
based image and video retrieval [4], object and action recognition in images
and videos [46, 40], and visual question answering [57]. In the case of user-
specific highlight detection, the key, value, and query need to be based on
the video contents, i.e., follow the paradigm of content-based highlight
detection [42, 37, 2] to perform meaningful retrieval of the highlightable
clips per user.

#### Main Contributions.

In this paper, we consider the visual content of the videos for highlight
detection. Specifically, for both the preferred clips and the target videos,
we consider the constituent non-human entities commonly clubbed as “objects”
[20], and the human activities expressed with their pose movements. We follow
the current paradigm of leveraging the presence of and interactions between
human activities and non-human entities, either directly or indirectly, for
content-based highlight detection [42, 11, 2]. We design an attention
mechanism to leverage both the objects and the pose-based activities in the
users’ preferred highlight clips to detect user-specific highlights in
different target videos. We only consider the preferred highlight clips and
not the corresponding full videos, i.e., we do not require information about
clips the users did not select as highlights. Each highlight clip marked by
the users contains partial information about their highlight preferences,
which we store in latent features learned from the objects and the poses
detected in those clips. We pool these object- and pose-based features using
learned weights to obtain combined features representing the users’ overall
highlight preferences. Given target videos for each user, we then query the
object- and the pose-based features of the target videos against the combined
features of the corresponding user to detect highlight clips for them in the
target videos. To summarize, our main contributions are the following:

  * •

User-Specific Highlight Detection. We leverage the video contents, in terms of
the objects and the pose-based human activities, between the users’ preferred
highlight clips and the target videos to detect the user-specific highlights
in the target videos.

  * •

Multi-Head Attention Mechanism. We design an multi-head attention mechanism
that learns content-based features per user from the users’ preferred
highlight clips. We also learn content-based features from the target videos
and query those against the per-user features to detect the user-specific
highlights.

  * •

State-of-the-art performance. We perform extensive experiments on the
benchmark personal highlight detection dataset (PHD2) [11], containing more
than 6,000 testing videos, to show that our method improves the mean average
precision of highlight detection by an absolute 2–4% over the current
baselines, as well as the F-score in the related problem of video
summarization by an absolute 4–12% over the current baselines on the benchmark
SumMe dataset [14]. Further, we show the qualitative performance of our method
as full videos in our supplementary material.

##  2 Related Work

We briefly review the prior work in highlight detection and the related
problem of video summarization. For a more extensive discourse, including the
development of multimedia-based techniques for content-based clustering, scene
understanding, and temporal variance optimization, we refer the readers to the
works of [55, 6, 43, 21]. Since our goal is to learn user-specific highlights
from their past highlight preferences, our problem is also related to
collaborative filtering, and we review prior work there as well.

#### Highlight Detection.

Highlight detection aims to detect the most interesting moments in videos [43,
42]. Since “interesting” is subjective for the viewer, highlight detection
methods rely on the availability of annotated highlights to learn the
interesting moments. Depending on the nature of the annotations, we can
broadly classify these methods into fully supervised, weakly supervised, or
unsupervised approaches. Fully supervised approaches assume that, for each
video, highlight and non-highlight clips are available, and typically optimize
variants of a pairwise ranking loss that ranks the highlight clips higher than
the non-highlight clips [42, 16, 54, 18, 56, 19, 47]. Weakly supervised and
unsupervised approaches do not require highlight annotations but instead
utilize exemplars or video metadata. Exemplar-based methods use open-source
images depicting highlightable moments, which guide the ranking of the
individual shots or frames in the videos [25], e.g., a surfing image used to
detect highlights in a surfing video. Video metadata include information on
the video domain [53], e.g., collecting all edited videos with “surfing” in
the title to learn surfing highlights, using the video duration itself as a
weak signal with the argument that shorter videos are more likely to be edited
and therefore have a higher percentage of highlightable moments [49], and even
focusing on certain subjects in the videos, such as explicitly leveraging
human activities to learn highlights in human-centric videos [2]. While weakly
supervised and unsupervised methods alleviate over-reliance on annotated data,
they are designed to be content-based and cannot adapt to the individual
preferences of different users.

#### Preference-Based Highlight Detection.

These methods solve the more challenging problem of learning individualized
highlights for the users based on their annotated highlight preferences.
Existing methods solve this problem by learning to score highlight frames
higher than other frames conditioned on the users’ preferences [11], learning
parameters based on the users’ preferred clips to guide a content-based
highlight detection network [35], or learning the users’ overall preferences
by averaging over features learned from their preferred clips [5]. Other
methods do not depend on the video contents but rank the clips in a video for
each user based on the durations of their selected clips using a
recommendation algorithm [32]. Our work solves the same problem of predicting
preference-based user-specific highlights and improves on the performances of
these methods.

#### Video Summarization.

Video summarization aims to condense videos into the summaries of their
contents. Summaries are typically presented in the form of storyline graphs
[24, 50], keyframe sequences [28], clips [14, 59], as well as a mixture of
formats [13]. Video summarization can also be broadly classified into fully
supervised, weakly supervised and unsupervised approaches. Fully supervised
approaches leverage annotated summaries, and learn relevant video subsets for
summaries based on their relative importance in the videos [12, 28, 14, 15,
51, 58, 59, 60, 17]. Weakly supervised approaches rely on learning the
relevant contents for a summary based on exemplars such as images and edited
videos [24, 22, 3, 36], or leveraging keywords in the video title to learn the
relevant content-based features for summaries [61, 41, 34, 33]. Unsupervised
approaches learn relevant information directly from the video contents, such
as scenes that commonly co-occur in videos [7], and temporal consistency
across frames and shots to understand summary boundaries [23, 31, 37, 59].
Other approaches for video summarization define and estimate specific
parameters determining summaries, such as coherence [30], diversity, and
representativeness [33, 62]. Yet other approaches consider user preferences in
the form of the text queries they use to search videos [29, 38, 44, 52], their
feedback on individual summaries [10], and their preferences between pairs of
summaries [39], to learn user-specific summaries. Highlight detection methods
learn to leverage video contents using techniques similar to those employed
for video summarization, albeit with different training objectives. Further,
designing highlight detection methods using user-specific feedback requires
the methods to be interactive at inference time and relies on the users’
availability and ability to provide feedback. This is complementary to
learning user-specific highlights based only on the preferred highlight clips,
which streamlines the inference, and the two approaches can be combined
depending on the use cases.

#### Collaborative Filtering.

Collaborative filtering aims to predict novel user preferences based on their
preference history as well as the preferences of other users [27]. It is
widely used in tackling recommendation problems, such as the Netflix challenge
[1] and online video recommendations [8]. However, extending collaborative
filtering to highlight detection requires the availability of multiple user
responses to each clip within each video in a large training set of videos.
Making such an approach practically feasible firstly requires the availability
of multiple user responses for the each clip, which is not the case in current
highlight detection datasets [11]. Secondly, it requires us to define clip
boundaries apriori; otherwise, each video can contain infinitely many clips.
This, in turn, severely limits what the users can choose as their highlight
preferences, as they can only detect highlight clips of fixed lengths. For
these limitations, collaborative filtering is not amenable to user-specific
highlight detection. By contrast, our highlight detection method works at the
frame level and uses variable-length preferred clips marked by the users to
detect their individualized highlights.

##  3 User-Specific Highlight Detection

Figure 2: Our User-Specific Highlight Detection Network. For each preferred
clip $i$, we use the two priming blocks to map the object-based features and
the pose-based features to respective features $z_{i}$ and $y_{i}$. We use
these features to learn the per-frame weights $w_{i}$ and $v_{i}$ using multi-
head attention (MHA), perform per-frame attention pooling, learn the per-clip
weights $\sigma_{i}$ and $\rho_{i}$ using MHA again, and fuse the per-clip
features using weighted summation to get the fused features $\psi_{i}$ and
$\phi_{i}$. For each target video $\tau$, we train a separate set of attention
priming and MHA layers to obtain fused features $\psi_{\tau}$ and
$\phi_{\tau}$. We compute the similarities between the fused features of the
preferred clips and the target video using scaled matrix products and
concatenate and map the resultant features to per-frame highlight scores for
the target video using a fully-connected prediction block. Figure 3: Priming
and MHA for Objects. Priming on the YOLOv5 [20] features using 3D convolutions
(blue blocks) and 3D batch norms (green blocks). We use the attention-primed
features to learn the per-frame attention weights $w_{i}$ and $w_{\tau}$ using
fully-connected layers (orange blocks).

Given preferred highlight clips marked by users and different target videos,
our goal is to detect highlight clips per user from the target videos that are
similar in content to the user’s preferred highlight clips. For each user $u$,
we consider
$P^{\left(u\right)}=\left\\{i\,|\,i=1,\dots,N^{\left(u\right)}\right\\}$
preferred highlight clips, and a target video $\tau^{\left(u\right)}$. For
numerical consistency, we consider each highlight clip to be $T$ frames long
and zero-pad all videos shorter than $T$ frames. In the subsequent text, we
assume all the variables are for user $u$ unless stated otherwise and drop the
superscript $\left(u\right)$ for brevity of notation.

For each user, our objective is to predict the highlight score
$s_{j}\in\left[0,1\right]$ for each frame $j=1,\dots,T_{\tau}$ for the target
video, where $T_{\tau}$ is the number of frames in the target video (we zero-
pad all videos shorter than $T_{\tau}$ frames). The highlight score $s_{j}$
determines how highlightable a frame is, with $1$ being the highest score. We
require our highlight scores to be relative to the contents of both the target
video and the user’s preferred highlight clips. Thus, we aim to learn a
scoring function $\mathcal{S}$ such that

$\left\\{s_{1},\dots,s_{T_{\tau}}\right\\}=\mathcal{S}\left(\tau,P\right).$
(1)

###  3.1 Attention Priming

We consider the contents to be both the non-human entities or “objects” [20]
and the pose-based human activities, and explicitly leverage their presence in
all the clips. For each frame of each clip, we obtain pre-trained features
$x_{\textrm{YOLOv5}}$ from the penultimate layer of the YOLOv5 network [20],
which contains information on the object categories and their spatial
positions. We also obtain pre-trained features $x_{\textrm{Detectron2}}$ from
the penultimate layer of the Detectron2 network [48] containing information on
the human poses. Our attention priming transforms these pre-trained object and
pose features into lower-dimensional features for learning the per-frame
attention weights for multi-head attention fusion. This provides the twin
benefits of leveraging the localized information common to all the pre-trained
features using the same set of convolutional layers, and reducing the
parameter load required to learn the actual attention weights. It also ensures
our network avoids memorization of the attentions learned for the pre-trained
features.

Figure 4: Priming and MHA for Poses. Priming on the Detectron2 [48] features
using spatial temporal graph convolutions with feature pooling (green arrow)
on the five kinematic chains: trunk, two arms and two legs. We use the
attention-primed features to learn the per-frame attention weights $v_{i}$ and
$v_{\tau}$ using fully-connected layers (orange blocks).

To perform attention priming, we pass the YOLOv5 $x_{\textrm{YOLOv5}}$ and the
Detectron2 features ${}_{\textrm{Detectron2}}$ through separate networks to
get $z\in\mathbb{R}^{T\times d_{z}}$ and $y\in\mathbb{R}^{T\times d_{y}}$
respectively, which we use to learn attention weights. For priming the YOLOv5
features, we use a series of 3D convolutions, denoted PrimingConv3D (Fig. 3),
to leverage the spatial and temporal adjacency of the features
$x_{\textrm{YOLOv5}}$, as

$z=\textrm{PrimingConv3D}\left(x_{\textrm{YOLOv5}};\theta_{\textrm{PrimingConv3D}}\right),$
(2)

where $\theta_{\textrm{PrimingConv3D}}$ are trainable parameters. Similarly,
for priming the Detectron2 features, we use a series of spatial temporal graph
convolutions, denoted PrimingSTGCN (Fig. 4), to leverage adjacencies between
the body joints that comprise the features $x_{\textrm{Detectron2}}$, as

$y=\textrm{PrimingSTGCN}\left(x_{\textrm{Detectron2}};\theta_{\textrm{PrimingSTGCN}}\right),$
(3)

where $\theta_{\textrm{PrimingSTGCN}}$ are trainable parameters.

###  3.2 Multi-Head Attention and Fusion

For each preferred highlight clip $i$, we denote the attention-primed features
as $z_{i}\in\mathbb{R}^{T\times d_{z}}$ and $y_{i}\in\mathbb{R}^{T\times
d_{y}}$. Each $\left(z_{i},y_{i}\right)$ contains information on the user’s
highlight preferences in clip $i$, which is different from the user’s
highlight preferences in all the other preferred clips. Thus, to obtain the
user’s overall highlight preferences, we consider a multi-head attention (MHA)
mechanism that learns relative weights at both the frame and the clip levels.

At the frame level, the number of attention heads is the number of frames $T$
for the preferred clips and $T_{\tau}$ for the target video. The parameters
determining the attention weights are shared for all the preferred clips,
implying that the order in which our network processes them is irrelevant.
This is the desired behavior since the preferred clips need not have any
content overlap or any specified ordering. However, we train a separate set of
parameters to learn the attention weights from the target videos since the
target video contains both highlightable and non-highlightable segments.

At the clip level, the number of attention heads is the number of preferred
clips $N$ that we consider for each user. We use the clip level attention
weights to additively fuse the corresponding attention-primed features and
then compute the similarity of the fused features to the attention-primed
features of the target video. During training, the frame level and the clip
level attention weights collectively determine the features describing the
relevant contents of the preferred clips given the target videos per user.
Conversely, during inference, our network uses these per-user relevant
features to detect the highlightable frames from the target videos.

For our MHA mechanism, we train a pair of attention heads MHAConv3D (Fig. 3)
and MHASTGCN (Fig. 4) to produce frame level attention weights
$w_{i}\in\mathbb{R}^{T}$ and $v_{i}\in\mathbb{R}^{T}$ as

$\displaystyle w_{i}$
$\displaystyle=\textrm{MHAConv3D}\left(z_{i};\theta_{\textrm{MHAConv3D}}\right),$
(4) $\displaystyle v_{i}$
$\displaystyle=\textrm{MHASTGCN}\left(y_{i};\theta_{\textrm{MHASTGCN}}\right),$
(5)

where $\theta_{\textrm{MHAConv3D}}$ and $\theta_{\textrm{MHASTGCN}}$ are
trainable parameters. We train another pair of attention heads
$\textrm{MHAFC}_{z}$ and $\textrm{MHAFC}_{y}$ (Fig. 2, rightmost layers in
central block) to get clip level attention weights $\sigma_{i}\in\mathbb{R}$
and $\rho_{i}\in\mathbb{R}$ from the weighted frame level features as

$\displaystyle\sigma_{i}$
$\displaystyle=\textrm{MHAFC}_{z}\left(z_{i}^{\top}w_{i};\theta_{\textrm{MHAFC}_{z}}\right),$
(6) $\displaystyle\rho_{i}$
$\displaystyle=\textrm{MHAFC}_{y}\left(y_{i}^{\top}v_{i};\theta_{\textrm{MHAFC}_{y}}\right),$
(7)

where $\theta_{\textrm{MHAFC}_{z}}$ and $\theta_{\textrm{MHAFC}_{y}}$ are
trainable parameters. We obtain the fused, attention-weighted features
$\psi\in\mathbb{R}^{d_{z}}$ and $\phi\in\mathbb{R}^{d_{y}}$ as

$\displaystyle\psi$ $\displaystyle=\sum_{i=1}^{N}z_{i}^{\top}w_{i}\sigma_{i},$
(8) $\displaystyle\phi$
$\displaystyle=\sum_{i=1}^{N}y_{i}^{\top}v_{i}\rho_{i}.$ (9)

Parallelly, we compute the per-frame attention-weighted features of the target
video as $\psi_{\tau}=z_{\tau}\bigodot w_{\tau}\in\mathbb{R}^{T_{\tau}\times
d_{z}}$ and $\phi_{\tau}=y_{\tau}\bigodot
v_{\tau}\in\mathbb{R}^{T_{\tau}\times d_{y}}$. Given $\psi_{\tau}$,
$\phi_{\tau}$, $\psi$, and $\phi$, we use scaled matrix products to compute
features $h_{z}\in\mathbb{R}^{T_{\tau}}$ and $h_{y}\in\mathbb{R}^{T_{\tau}}$
representing their pairwise similarities, as

$\displaystyle h_{z}$ $\displaystyle=\textrm{softmax}\left(\psi
d_{z}^{\nicefrac{{-1}}{{2}}}\right)\cdot\psi_{\tau},$ (10) $\displaystyle
h_{y}$ $\displaystyle=\textrm{softmax}\left(\phi
d_{y}^{\nicefrac{{-1}}{{2}}}\right)\cdot\phi_{\tau}.$ (11)

We then use a predictor block HltPred (Fig. 2, bottom-right block) that fuses
the features $h_{z}$ and $h_{y}$ using concatenation, and reduces the fused
features to per-frame highlight scores $s\in\left[0,1\right]^{T_{\tau}}$, as

$s=\textrm{HltPred}\left(h_{z}\parallel
h_{y};\theta_{\textrm{HltPred}}\right),$ (12)

where $\theta_{\textrm{HltPred}}$ are trainable parameters.

##  4 Training and Inference

We discuss the loss function we use to train our network, the implementation
details, and how we use our network to detect user-specific highlights on test
videos.

###  4.1 Loss Function

Our network predicts a highlight score between $0$ and $1$ for each frame in
the target video. For training, we consider a frame to be a highlight if and
only if its score is above a predetermined threshold $\zeta$. We train our
network using a combination of a label loss, a margin loss, and a sparsity
loss, assuming a threshold $\zeta=0.5$.

  * •

Label loss $L_{\tau j}$ for each frame $j$ in target video $\tau$ measures the
weighted cross entropy loss between the ground-truth highlight labels $y_{\tau
j}$ ($1$ for highlight, $0$ otherwise) and the predicted highlight scores
$s_{\tau j}$, as

$L_{\tau j}=-wy_{\tau j}\log\left(s_{\tau j}\right)-\left(1-y_{\tau
j}\right)\log\left(1-s_{\tau j}\right),$ (13)

where $w$ denotes the relative weight assigned to the highlight class. The
label loss is the baseline loss that guides the training process as a binary
classification problem.

  * •

Margin loss $M_{\tau j}$ for each frame $j$ in target video $\tau$ measures
the one-sided distance between the highlight score of that frame and the
threshold $\zeta$ as

$M_{\tau j}=\max\left(0,\widetilde{y}_{\tau j}\left(\zeta-s_{\tau
j}\right)\right),$ (14)

where $\widetilde{y}_{\tau j}=1$ if $j$ is a highlight frame, and $-1$
otherwise. The margin loss provides additional constraints that the scores for
the highlight and the non-highlight frames should be on opposite sides of the
threshold $\zeta$.

  * •

Sparsity loss $S_{\tau}$ for each target video $\tau$ enforces that the total
number of highlight frames should be low, as

$S_{\tau}=\sum_{j=1}^{T}\max\left(0,\textrm{sign}\left(s_{\tau
j}-\zeta\right)\right).$ (15)

The sparsity loss incentivizes our network to detect as few highlight frames
as possible, following the intuition that highlight frames make up a small
fraction of total video [42, 49, 2]. Moreover, the sparsity loss improves the
precision of highlight detection by its design of minimizing the number of
frames detected as highlights.

Combining the individual loss terms, we write the overall loss function
$\mathcal{L}_{\tau}$ for the target video $\tau$ as

$\mathcal{L}_{\tau}=\frac{1}{T}\sum_{j=1}^{T}\left(L_{\tau j}+M_{\tau
j}\right)+S_{\tau}.$ (16)

Based on our experiments, we did not observe significant performance
differences when using relative weights for the margin and the sparsity losses
up to five times that of the baseline label loss. Consequently, we propose
using the same weights for all the loss terms for simplicity. We also note
that none of our losses enforce the temporal continuity of labels, i.e.,
adjacent frames should have the same labels (highlight or non-highlight)
except at the highlight clip boundaries. There are two reasons for this.
First, the durations of highlight clips are not fixed. In our experiments, the
longest highlight clip can be about seven times the length of the shortest
highlight clip for the same video. Thus, defining clip boundaries for
enforcing temporal continuity is non-trivial and would, in fact, limit the
variety of highlight clips that we can detect during inference. Second, our
network explicitly leverages information along both the spatial and the
temporal dimensions using 3D convolutions and spatial temporal graph
convolutions. Consequently, temporal continuity is implicitly enforced in the
learned features and highlight scores, and in practice, we observe the
highlight scores varying smoothly across the frames, with only minor variance
due to noise.

Table 1: Mean Average Precision (mAP) and normalized Meaningful Summary
Duration (nMSD) for Highlight Detection. We report the numbers of all methods
on PHD2 [11]. Bold indicates best.

| Method | mAP | nMSD
---|---|---|---
User-Agnostic | Random | 0.112 | 0.536
FCSN [37] | 0.152 | -
HighlightMe [2] | 0.200 | -
User-Specific | Personalized Summ. [32] | 0.216* | 0.288*
Video2GIF [16] | 0.158 | 0.420
PHD-GIF [11] | 0.166 | 0.402
Adaptive-FCSN [35] | 0.168 | -
PR-Net [5] | 0.187 | -
MHA+Fusion (Ours) | 0.228 | 0.271

  * *

Numbers reported for a subset of the test set consisting of at least 5
selected highlights per user. For a similar test subset, our method has mAP of
0.262 and nMSD of 0.223.

###  4.2 Implementation

We first extract the YOLOv5 [20] and the Detectron2 [48] features for each
frame in both the preferred highlight clips and the target videos in the
training dataset. This takes about $30$ seconds per frame on an NVIDIA Tesla
A100 GPU. We then train our network using the Adam optimizer [26] with a batch
size of $36$, an initial learning rate of $1\textsc{e--}4$ that we decay at a
rate of $0.999$ per epoch, and a weight decay of $3\textsc{e--}4$. We train
for $500$ epochs on $8$ NVIDIA Tesla V100 GPUs at a speed of about $1,480$
seconds per epoch. Once the network is fully trained, inference takes about
$9$ seconds for each user using the same GPU configuration. We provide
additional details on hyperparameter tuning in our supplementary material.

###  4.3 Inference

Similar to the training set up, we consider a set of preferred highlight clips
and target videos for each user during inference. We use the preferred
highlight clips to learn the features $h_{z}$ and $h_{y}$ (Eqs. 10 and 11),
and subsequently the per-frame highlight scores $s_{j}$ (Eq. 1). We also
consider the scenario where the users’ preferred highlight clips are not
available, where our network falls back to content-based highlight detection
[2]. Instead of the softmax operations in Eqs. 10 and 11, we perform uniform
weighting along the feature dimension, i.e., we have
$h_{z}=\sum_{k=1}^{d_{z}}\psi_{\tau k}/d_{z}$ and
$h_{y}=\sum_{k=1}^{d_{y}}\phi_{\tau k}/d_{y}$.

Table 2: F-Score for Video Summarization. We report the F-scores of all methods on the SumMe dataset [14]. Bold indicates best. Method | F-Score
---|---
SumMe baseline [14] | 0.394
Submodular mixtures [15] | 0.397
DPP-LSTM [58] | 0.386
Unsup. Adversarial LSTM [31] | 0.417
Sup. Deep RL [62] | 0.421
S2N [47] | 0.433
Adaptive-FCSN [35] | 0.444
HighlightMe [2] | 0.480
MHA+Fusion (Ours) | 0.526

##  5 Experiments and Results

We provide details on the dataset we use for training and testing our method,
the baselines we compare with, and the performance of our method on
quantitative metrics. We also show the improvement in performance as we add
more preferred highlight clips to our input, the benefits of the two
components of object detection and pose detection used in our approach, and
the contribution of each of the loss functions we use in training.

###  5.1 Dataset

We evaluate on the personal highlight detection dataset (PHD2) introduced by
del Molino and Gygli [11]. This dataset consists of URLs of YouTube videos,
IDs of annotators or “users”, and the segments they selected as highlight
clips from those videos as per their preferences. The last video that a user
annotated is designated as the target video for that user. Since PHD2 only
provides the YouTube URLs, we scraped the videos from YouTube for our
experiments, subject to video availability and IP restrictions. Both our
training and testing sets consist of up to 15 highlight clips and a target
video per user. The highlight clips are between 1 and 672 seconds long and
have a mean length of 5.19 seconds. The target videos are between 1 and 37,434
seconds long and have a mean length of 443.29 seconds. The training set
contains a total of 6,596 users and 26,390 videos. The testing set contains
727 users and 6,004 videos that do not overlap with any user or video in the
training set.

###  5.2 Baselines

We compare with the methods Video2GIF by Gygli et al. [16], PHD-GIF by del
Molino and Gygli [11], Personalized Summarization by Panagiotakis et al. [32],
Adaptive-FCSN by Rochan et al. [35], and PR-Net by Chen et al. [5], which,
like our method, consider user-specific history when detecting highlights for
target videos. Video2GIF computes shot boundaries in the videos and uses C3D
features, used for action recognition, to map the shots to highlight labels
and train using a ranking loss. Adaptive-FCSN trains an encoder network to
learn affine parameters based on the users’ preferred highlight clips and uses
those parameters to guide the detection of per-frame highlight labels in the
target videos. We also compare with state-of-the-art user-agnostic approaches
FCSN by Rochan et al. [37], and HighlightMe by Bhattacharya et al. [2], which
detect highlights based only on the video contents. FCSN utilizes the
GoogLeNet backbone to learn image-based features for per-frame highlight
detection. HighlightMe leverages pose-based activities and facial expressions
in human-centric videos and trains an autoencoder-based architecture to detect
per-frame highlights.

Table 3: Ablation 1: Changing the Number of Preferred Highlights. We report
the mAPs of all ablated versions on PHD2 [11]. Bold indicates best.

# preferred Highlight Clips | 0 | 5 | 10 | 15 (max)
---|---|---|---|---
mAP | 0.151 | 0.197 | 0.211 | 0.228

Table 4: Ablation 2: Using Only One Pre-Trained Backbone for Highlight
Detection. We report the mAPs of all ablated versions on PHD2 [11]. Bold
indicates best.

Using backbone | YOLOv5 [20] | Detectron2 [48] | both
---|---|---|---
mAP | 0.164 | 0.193 | 0.228

###  5.3 Quantitative Comparison

We evaluate the performance of each method using the metrics of mean average
precision (mAP) and normalized meaningful summary duration (nMSD). We compute
mAP as the mean of the average precision of matching the highlight labels in
each target video following [37, 32, 35, 2, 5], and nMSD at a recall rate of
0.5 following [16, 11, 32]. As empirical lower bounds on mAP and nMSD, we also
report the performance of randomly choosing the highlight frames in the target
videos. Table 1 shows the mAP and the nMSD of each method on PHD2 [11]. Our
method achieves the best mAP of 0.219 and nMSD of 0.271, outperforming the
current best user-agnostic method HighlightMe [2] by an absolute 2% and the
current best user-specific methods of PR-Net [5] and Personalized
Summarization [32] by absolute 4% and 6% respectively.

Beyond highlight detection, we also evaluate our method on the related problem
of video summarization, following the approach of Rochan et al. [35]. Video
summarization aims to condense videos into summaries of their contents, which
may or may not correspond to the most interesting moments that users annotate
as highlights. We test the performance of our trained network on the SumMe
[14] dataset, which consists of 25 videos totaling about 66 minutes, for video
summarization. We compute the F-score of matching the summary frames and
compare with the summarization methods listed by Rochan et al. [35], their own
highlight detection method, and the highlight detection method of HighlightMe
[2]. We show the results in Table 2, where we observe our method improves the
F-score by an absolute 4% over the next-best method of HighlightMe and 12%
over the SumMe baseline. Our results further corroborate Rochan et al.’s
argument that methods for video summarization can benefit from using networks
with parameters pre-trained using loss functions for highlight detection.

Table 5: Ablation 3: Ablating the Training Loss Functions. We report the mAPs of all ablated versions on PHD2 [11]. Bold indicates best. Excluding loss | label | margin | sparsity | none
---|---|---|---|---
mAP | 0.155 | 0.192 | 0.163 | 0.228
Figure 5: Qualitative Results. We show qualitative results of our method and
the current best baseline of HighlightMe [2] for four users in the testing set
of PHD2[11]. For each user, we show sample frames of (i) their preferred
clips, (ii) ground-truth highlights selected by them from their target videos,
(iii) highlights detected by HighlightMe [2], and (iv) highlights detected by
our method. For each user, we observe that our method matches the ground-truth
more closely than HighlightMe [2].

###  5.4 Ablation Studies

We ablate our method in terms of (i) the number of preferred highlight clips
used, (ii) the two backbones leveraging objects and poses, and (iii) the loss
functions used for training our network.

To evaluate the benefit of using the preferred highlight clips, we train four
versions of our network using at most 0, 5, 10, and 15 preferred highlight
clips per user. We report the mAPs of the corresponding trained networks on
PHD2 in Table 3. We observe a monotonic increase in performance as we increase
the number of preferred highlight clips. This shows that our network learns
more accurate representations of user-specific highlight preferences as we
feed it more data on the preferred highlight clips. At the same time, the rate
of increase drops as we increase the number of preferred highlight clips,
indicating a diminishing marginal benefit of providing additional clips. We
also note that the performance of our method without a substantial number of
preferred highlight clips is worse than the user-agnostic method of
HighlightMe [2]. This is because the highlights in PHD2 are largely human-
centric, which HighlightMe is fine-tuned to detect. However, the performance
of HighlightMe remains fixed, while ours grows and outperforms it as we
increase the number of preferred highlight clips. We expect that combining our
user-specific approach with the human-centric approach of HighlightMe will
lead to the best performance on datasets such as PHD2. For the sake of
completeness, we further test the performance of our method without using any
preferred clips on a domain-specific highlight dataset (DSH) [42] and a title-
based summarization dataset (TVSum) [41], which focus on content domains such
as surfing, cooking, grooming animals, and do not provide user-specific
highlight preferences. We report the results in our supplementary material and
observe that our method is at par with the current methods performing content-
based highlight detection.

We also ablate each of our two network components, the one with the YOLOv5
backbone and the one with the Detectron2 backbone, and train the network using
only the remaining component. For these experiments, there is no concatenation
of the two components. Instead, we adjust the number of parameters of the
first fully-connected layer of the prediction block HltPred (Fig. 2 bottom-
right block) accordingly. We show the mAP of each of these ablated networks on
PHD2 in Table 4. We observe a more significant drop in mAP when ablating the
component with the Detectron2 backbone, which results in the network not
explicitly learning from the human activities in the videos. This is again a
consequence of the annotated highlights in PHD2 being largely human-centric.

Lastly, we ablate each of the three loss functions we describe in Sec. 4.1 and
train our network using the remaining loss functions. We report the mAP of
these ablated versions in Table 5. Without the label loss, the label
assignment becomes random, being only regularized by the margin and the
sparsity losses. This results in a sharp drop in the mAP. Without the margin
loss, all the predicted highlight scores are closer to the threshold $\zeta$,
resulting in more noise in the highlight scores around the threshold $\zeta$
and consequently more confusion between the highlight and the non-highlight
labels. Without the sparsity loss, more frames are incorrectly labeled as
highlights, again resulting in a sharp drop in the mAP.

###  5.5 Qualitative Results

We show sample visual results in Fig. 5. Full video results are available in
our supplementary. We show sample frames from the users’ preferred highlights,
the ground-truth highlights in the users’ target videos, and the highlights
detected by HighlightMe [2] and our method. For our method, we observe content
similarities between the users’ preferred highlights and the target videos,
which also largely match the ground-truth. By contrast, HighlightMe [2] only
looks at the target videos and does not adapt to the users’ preferences. Our
method also fails when the users break their preference patterns in the target
videos, or the target videos did not contain any features similar to the
preferred highlights. Please refer to our video results for details.

##  6 Conclusion

We have presented a method for user-specific highlight detection based on
multi-head attention fusion of features from the users’ preferred highlight
clips. Our method only requires the highlight clips and not the clips users
did not select as highlights, thus significantly reducing the data overhead
for training. Our multi-head attention mechanism leverages both objects and
human activities in videos to learn highlights based on their spatial and
temporal variations and mutual interactions. We also perform experiments to
show that our method advances the state-of-the-art in both highlight detection
in a large-scale dataset with annotated user preferences and the related
problem of video summarization. We also show that, unlike user-agnostic
highlight detection methods, the performance of our method increases as we
increase the maximum number of preferred highlight clips considered for each
user.

##  7 Limitations and Future Work

Our work has some limitations. First, we consider users’ highlight preferences
only at training time and do not consider user feedback at inference. To this
end, we plan to combine our method with complementary methods that rely on
user feedback to gradually learn their preferences. Second, our attention
mechanism relies on only the non-human entities and the human activities.
While this helps us achieve state-of-the-art results currently, we intend to
extend to a broader definition of content, including the associated audio,
semantic segmentation of the video frames, and other human-centric modalities
such as faces. Lastly, we do not fine-tune our method to video categories
based on domains (e.g., surfing videos) or the constituent subjects (e.g.,
human-centric videos). Rather, we design an approach to combine diverse
highlight clips assuming the presence of only generic elements, i.e., objects
and humans. Nevertheless, our method is suitable for fine-tuning as necessary
to improve user-specific highlight detection in various video categories.

## Acknowledgment

Bhattacharya and Manocha were supported in part by ARO Grants W911NF1910069
and W911NF2110026.

## References

  * [1] Robert M. Bell and Yehuda Koren.  Lessons from the netflix prize challenge.  SIGKDD Explor. Newsl., 9(2):75–79, dec 2007.
  * [2] Uttaran Bhattacharya, Gang Wu, Stefano Petrangeli, Viswanathan Swaminathan, and Dinesh Manocha.  Highlightme: Detecting highlights from human-centric videos.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8157–8167, October 2021.
  * [3] Sijia Cai, Wangmeng Zuo, Larry S. Davis, and Lei Zhang.  Weakly-supervised video summarization using variational encoder-decoder and web prior.  In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  * [4] Binghui Chen and Weihong Deng.  Hybrid-attention based decoupled metric learning for zero-shot image retrieval.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [5] Runnan Chen, Penghao Zhou, Wenzhe Wang, Nenglun Chen, Pai Peng, Xing Sun, and Wenping Wang.  Pr-net: Preference reasoning for personalized video highlight detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7980–7989, October 2021.
  * [6] Chong-Wah Ngo, Yu-Fei Ma, and Hong-Jiang Zhang.  Automatic video summarization by graph modeling.  In Proceedings Ninth IEEE International Conference on Computer Vision, pages 104–109 vol.1, 2003.
  * [7] Wen-Sheng Chu, Yale Song, and Alejandro Jaimes.  Video co-summarization: Video summarization by visual co-occurrence.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
  * [8] Paul Covington, Jay Adams, and Emre Sargin.  Deep neural networks for youtube recommendations.  In Proceedings of the 10th ACM Conference on Recommender Systems, RecSys ’16, page 191–198, New York, NY, USA, 2016. Association for Computing Machinery.
  * [9] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [10] Ana Garcia del Molino, Xavier Boix, Joo-Hwee Lim, and Ah-Hwee Tan.  Active video summarization: Customized summaries via on-line interaction with the user.  Proceedings of the AAAI Conference on Artificial Intelligence, 31(1), Feb. 2017.
  * [11] Ana Garcia del Molino and Michael Gygli.  Phd-gifs: Personalized highlight detection for automatic gif creation.  In Proceedings of the 26th ACM International Conference on Multimedia, MM ’18, page 600–608, New York, NY, USA, 2018. Association for Computing Machinery.
  * [12] Boqing Gong, Wei-Lun Chao, Kristen Grauman, and Fei Sha.  Diverse sequential subset selection for supervised video summarization.  In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27, pages 2069–2077. Curran Associates, Inc., 2014.
  * [13] Hongxiang Gu and Viswanathan Swaminathan.  From thumbnails to summaries-a single deep neural network to rule them all.  In 2018 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2018.
  * [14] Michael Gygli, Helmut Grabner, Hayko Riemenschneider, and Luc Van Gool.  Creating summaries from user videos.  In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, pages 505–520, Cham, 2014. Springer International Publishing.
  * [15] Michael Gygli, Helmut Grabner, and Luc Van Gool.  Video summarization by learning submodular mixtures of objectives.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
  * [16] Michael Gygli, Yale Song, and Liangliang Cao.  Video2gif: Automatic generation of animated gifs from video.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [17] Zhong Ji, Kailin Xiong, Yanwei Pang, and Xuelong Li.  Video summarization with attention-based encoder–decoder networks.  IEEE Transactions on Circuits and Systems for Video Technology, 30(6):1709–1717, 2020.
  * [18] Yifan Jiao, Zhetao Li, Shucheng Huang, Xiaoshan Yang, Bin Liu, and Tianzhu Zhang.  Three-dimensional attention-based deep ranking model for video highlight detection.  IEEE Transactions on Multimedia, 20(10):2693–2705, 2018.
  * [19] Yifan Jiao, Tianzhu Zhang, Shucheng Huang, Bin Liu, and Changsheng Xu.  Video highlight detection via region-based deep ranking model.  International Journal of Pattern Recognition and Artificial Intelligence, 33(07):1940001, 2019.
  * [20] Glenn Jocher, Ayush Chaurasia, Alex Stoken, Jirka Borovec, NanoCode012, Yonghye Kwon, TaoXie, Jiacong Fang, imyhxy, Kalen Michael, Lorna, Abhiram V, Diego Montes, Jebastin Nadar, Laughing, tkianai, yxNONG, Piotr Skalski, Zhiqiang Wang, Adam Hogan, Cristi Fati, Lorenzo Mammana, AlexWang1900, Deep Patel, Ding Yiwei, Felix You, Jan Hajek, Laurentiu Diaconu, and Mai Thanh Minh.  ultralytics/yolov5: v6.1 - TensorRT, TensorFlow Edge TPU and OpenVINO Export and Inference, Feb. 2022.
  * [21] Hideo Joho, Jacopo Staiano, Nicu Sebe, and Joemon M Jose.  Looking at the viewer: analysing facial activity to detect personal highlights of multimedia contents.  Multimedia Tools and Applications, 51(2):505–523, 2011.
  * [22] Aditya Khosla, Raffay Hamid, Chih-Jen Lin, and Neel Sundaresan.  Large-scale video summarization using web-image priors.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2013.
  * [23] Gunhee Kim, Leonid Sigal, and Eric P. Xing.  Joint summarization of large-scale collections of web images and videos for storyline reconstruction.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2014.
  * [24] Gunhee Kim and Eric P. Xing.  Reconstructing storyline graphs for image recommendation from web community photos.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2014.
  * [25] Hoseong Kim, Tao Mei, Hyeran Byun, and Ting Yao.  Exploiting web images for video highlight detection with triplet deep ranking.  IEEE Transactions on Multimedia, 20(9):2415–2426, 2018.
  * [26] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [27] Yehuda Koren, Robert Bell, and Chris Volinsky.  Matrix factorization techniques for recommender systems.  Computer, 42(8):30–37, 2009.
  * [28] Y. J. Lee, J. Ghosh, and K. Grauman.  Discovering important people and objects for egocentric video summarization.  In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 1346–1353, 2012.
  * [29] Wu Liu, Tao Mei, Yongdong Zhang, Cherry Che, and Jiebo Luo.  Multi-task deep visual-semantic embedding for video thumbnail selection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
  * [30] Zheng Lu and Kristen Grauman.  Story-driven summarization for egocentric video.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2013.
  * [31] Behrooz Mahasseni, Michael Lam, and Sinisa Todorovic.  Unsupervised video summarization with adversarial lstm networks.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 202–211, 2017.
  * [32] Costas Panagiotakis, Harris Papadakis, and Paraskevi Fragopoulou.  Personalized video summarization based exclusively on user preferences.  In Joemon M. Jose, Emine Yilmaz, João Magalhães, Pablo Castells, Nicola Ferro, Mário J. Silva, and Flávio Martins, editors, Advances in Information Retrieval, pages 305–311, Cham, 2020. Springer International Publishing.
  * [33] Rameswar Panda and Amit K. Roy-Chowdhury.  Collaborative summarization of topic-related videos.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [34] Danila Potapov, Matthijs Douze, Zaid Harchaoui, and Cordelia Schmid.  Category-specific video summarization.  In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, pages 540–555, Cham, 2014. Springer International Publishing.
  * [35] Mrigank Rochan, Mahesh Kumar Krishna Reddy, Linwei Ye, and Yang Wang.  Adaptive video highlight detection by learning from user history.  In Proceedings of the European Conference on Computer Vision (ECCV), August 2020.
  * [36] Mrigank Rochan and Yang Wang.  Video summarization by learning from unpaired data.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [37] Mrigank Rochan, Linwei Ye, and Yang Wang.  Video summarization using fully convolutional sequence networks.  In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  * [38] Aidean Sharghi, Boqing Gong, and Mubarak Shah.  Query-focused extractive video summarization.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 3–19, Cham, 2016. Springer International Publishing.
  * [39] Adish Singla, Sebastian Tschiatschek, and Andreas Krause.  Noisy submodular maximization via adaptive sampling with applications to crowdsourced image collection summarization.  Proceedings of the AAAI Conference on Artificial Intelligence, 30(1), Mar. 2016.
  * [40] Sijie Song, Cuiling Lan, Junliang Xing, Wenjun Zeng, and Jiaying Liu.  Spatio-temporal attention-based lstm networks for 3d action recognition and detection.  IEEE Transactions on Image Processing, 27(7):3459–3471, 2018.
  * [41] Yale Song, Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes.  Tvsum: Summarizing web videos using titles.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
  * [42] Min Sun, Ali Farhadi, and Steve Seitz.  Ranking domain-specific highlights by analyzing edited videos.  In European conference on computer vision, pages 787–802. Springer, 2014.
  * [43] Ba Tu Truong and Svetha Venkatesh.  Video abstraction: A systematic review and classification.  ACM Trans. Multimedia Comput. Commun. Appl., 3(1):3–es, Feb. 2007\.
  * [44] Arun Balajee Vasudevan, Michael Gygli, Anna Volokitin, and Luc Van Gool.  Query-adaptive video summarization via quality-aware relevance estimation.  In Proceedings of the 25th ACM International Conference on Multimedia, MM ’17, page 582–590, New York, NY, USA, 2017. Association for Computing Machinery.
  * [45] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc., 2017.
  * [46] Wenju Wang, Yu Cai, and Tao Wang.  Multi-view dual attention network for 3d object recognition.  Neural Computing and Applications, 34(4):3201–3212, 2022.
  * [47] Zijun Wei, Boyu Wang, Minh Hoai Nguyen, Jianming Zhang, Zhe Lin, Xiaohui Shen, Radomir Mech, and Dimitris Samaras.  Sequence-to-segment networks for segment detection.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31, pages 3507–3516. Curran Associates, Inc., 2018.
  * [48] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * [49] Bo Xiong, Yannis Kalantidis, Deepti Ghadiyaram, and Kristen Grauman.  Less is more: Learning highlight detection from video duration.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1258–1267, 2019.
  * [50] Bo Xiong, Gunhee Kim, and Leonid Sigal.  Storyline representation of egocentric videos with an applications to story-based search.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
  * [51] Jia Xu, Lopamudra Mukherjee, Yin Li, Jamieson Warner, James M. Rehg, and Vikas Singh.  Gaze-enabled egocentric video summarization via constrained submodular maximization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
  * [52] Hui Yang, Lekha Chaisorn, Yunlong Zhao, Shi-Yong Neo, and Tat-Seng Chua.  Videoqa: Question answering on news video.  In Proceedings of the Eleventh ACM International Conference on Multimedia, MULTIMEDIA ’03, page 632–641, New York, NY, USA, 2003. Association for Computing Machinery.
  * [53] Huan Yang, Baoyuan Wang, Stephen Lin, David Wipf, Minyi Guo, and Baining Guo.  Unsupervised extraction of video highlights via robust recurrent auto-encoders.  In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
  * [54] Ting Yao, Tao Mei, and Yong Rui.  Highlight detection with pairwise deep ranking for first-person video summarization.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 982–990, 2016.
  * [55] Minerva Yeung, Boon-Lock Yeo, and Bede Liu.  Segmentation of video by clustering and graph analysis.  Computer Vision and Image Understanding, 71(1):94 – 109, 1998.
  * [56] Youngjae Yu, Sangho Lee, Joonil Na, Jaeyun Kang, and Gunhee Kim.  A deep ranking model for spatio-temporal highlight detection from a 360 video.  pages 7525–7533, 2018.
  * [57] Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian.  Deep modular co-attention networks for visual question answering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [58] Ke Zhang, Wei-Lun Chao, Fei Sha, and Kristen Grauman.  Video summarization with long short-term memory.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 766–782, Cham, 2016. Springer International Publishing.
  * [59] Ke Zhang, Kristen Grauman, and Fei Sha.  Retrospective encoders for video summarization.  In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  * [60] Bin Zhao, Xuelong Li, and Xiaoqiang Lu.  Hierarchical recurrent neural network for video summarization.  In Proceedings of the 25th ACM International Conference on Multimedia, MM ’17, page 863–871, New York, NY, USA, 2017. Association for Computing Machinery.
  * [61] Bin Zhao and Eric P. Xing.  Quasi real-time summarization for consumer videos.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2014.
  * [62] Kaiyang Zhou, Yu Qiao, and Tao Xiang.  Deep reinforcement learning for unsupervised video summarization with diversity-representativeness reward.  pages 7582–7589, 2018.

##  Appendix A Hyperparameter Tuning

We have experimented with batch sizes between 2 (minimum possible) and 36
(maximum allowable given our memory constraints), initial learning rates
between $1\textsc{e--}5$ (very slow convergence) and $1\textsc{e--}3$
(optimization diverges), learning rate decays between $0.9$ (very slow
convergence) and $1$ (optimization oscillates), weight decays between
$1\textsc{e--}4$ (optimization oscillates) and $5\textsc{e--}4$ (very slow
convergence), and training epochs between $100$ and $1000$ (validation loss
saturates around $500$ epochs).

##  Appendix B Additional Highlight Detection Results

We show the quantitative performance of our highlight detection method on two
additional datasets: Domain-Specific Highlights (DSH) [42] for highlight
detection and Title-Based Video Summarization (TVSum) [41] for video
summarization. Both these datasets consist of videos from various domains. DSH
consists of six domains, namely, dog show, gymnastics, parkour, skating,
skiing, and surfing. Each domain contains about 100 videos, and the total
duration over all the six domains is around 1,430 minutes. TVSum consists of
50 videos, totaling around 210 minutes, from ten domains, namely, beekeeping
(BK), bike tricks (BT), dog show (DS), flash mob (FM), grooming animal (GA),
making sandwich (MS), parade (PR), parkour (PK), vehicle tire (VT), and
vehicle unstuck (VU). However, neither of these datasets provide any user
preferences. Therefore, to test our method on these datasets, we use the
testing setup with no preferred highlight clips for any user (Sec. 4.3). We
show the results on the DSH dataset in Table 6, and on the TVSum dataset in
Table 7. The results of all the methods we compare with are taken from
Bhattacharya et al. [2]. We observe that our method has the second-best mean
average precision of all the methods over all the domains. Our method is also
at par with the human-centric approach of HighlightMe [2] for most of the
domains, being only significantly outperformed in domains that have an
abundance of both face and pose modalities and no other detected objects in
the highlighted segments, e.g., flash mob (FM) and parade (PR) in TVSum. The
precision of our method also suffers in the surfing domain in DSH and the bike
tricks (BT) domain in TVSum where our method assigns highlight labels to many
objects and human interactions not related to the ground-truth highlights
(such as prepping surfboards, people interviewing, cars on the street). As
part of our future work, for such scenarios, we can incorporate more human-
centric modalities into our pipeline or provide domain-specific videos as the
preferred clips to improve performance.

Table 6: Mean average precision on the DSH dataset [42]. Bold indicates best, underline indicates second-best. Domain | RRAE [53] | Video2 GIF [16] | LSVM [42] | Less is More [49] | HighlightMe [2] | Ours
---|---|---|---|---|---|---
dog show | 0.49 | 0.31 | 0.60 | 0.58 | 0.63 | 0.60
gymnastics | 0.35 | 0.34 | 0.41 | 0.44 | 0.73 | 0.73
parkour | 0.50 | 0.54 | 0.61 | 0.67 | 0.72 | 0.71
skating | 0.25 | 0.55 | 0.62 | 0.58 | 0.64 | 0.62
skiing | 0.22 | 0.33 | 0.36 | 0.49 | 0.52 | 0.61
surfing | 0.49 | 0.54 | 0.61 | 0.65 | 0.62 | 0.58
Mean | 0.38 | 0.46 | 0.54 | 0.57 | 0.64 | 0.62
Table 7: Mean average precision on the TVSum dataset [41]. Bold indicates best, underline indicates second-best. Domain | MBF [7] | KVS [34] | CVS [33] | Adv-LSTM [31] | Less is More [49] | HighlightMe [2] | Ours
---|---|---|---|---|---|---|---
BK | 0.31 | 0.34 | 0.33 | 0.42 | 0.66 | 0.57 | 0.60
BT | 0.37 | 0.42 | 0.40 | 0.48 | 0.69 | 0.93 | 0.85
DS | 0.36 | 0.39 | 0.38 | 0.47 | 0.63 | 0.60 | 0.62
FM | 0.37 | 0.40 | 0.37 | 0.46 | 0.43 | 0.88 | 0.77
GA | 0.33 | 0.40 | 0.38 | 0.48 | 0.61 | 0.50 | 0.63
MS | 0.41 | 0.42 | 0.40 | 0.49 | 0.54 | 0.50 | 0.55
PR | 0.33 | 0.40 | 0.38 | 0.47 | 0.53 | 0.84 | 0.70
PK | 0.32 | 0.38 | 0.35 | 0.46 | 0.60 | 0.76 | 0.68
VT | 0.30 | 0.35 | 0.33 | 0.42 | 0.56 | 0.65 | 0.61
VU | 0.36 | 0.44 | 0.41 | 0.47 | 0.50 | 0.77 | 0.75
Mean | 0.35 | 0.40 | 0.37 | 0.46 | 0.58 | 0.70 | 0.68