# Towards Automated Movie Trailer Generation

Dawit Mureja Argaw1,2 Mattia Soldan1 Alejandro Pardo1 Chen Zhao1
Fabian Caba Heilbron3 Joon Son Chung2 Bernard Ghanem1

1 King Abdullah University of Science and Technology (KAUST) 2 KAIST 3 Adobe
Research
Corresponding author.

###### Abstract

Movie trailers are an essential tool for promoting films and attracting
audiences. However, the process of creating trailers can be time-consuming and
expensive. To streamline this process, we propose an automatic trailer
generation framework that generates plausible trailers from a full movie by
automating shot selection and composition. Our approach draws inspiration from
machine translation techniques and models the movies and trailers as sequences
of shots, thus formulating the trailer generation problem as a sequence-to-
sequence task. We introduce Trailer Generation Transformer (TGT), a deep-
learning framework utilizing an encoder-decoder architecture. TGT movie
encoder is tasked with contextualizing each movie shot representation via
self-attention, while the autoregressive trailer decoder predicts the feature
representation of the next trailer shot, accounting for the relevance of
shots’ temporal order in trailers. Our TGT significantly outperforms previous
methods on a comprehensive suite of metrics.

##  1 Introduction

Figure 1: Trailer Generation Problem and Solutions. The top row depicts the
movie and expert-created trailer. The process composes shots from the movie in
non-chronological order to create a compelling and intriguing story. Depicted
in the central row are the classification/ranking strategies that
classify/rank each shot in the movie independently (classification) or with
limited relative interaction (ranking). The bottom row represents our approach
which can reason over the entire input movie sequence before producing a
provisional trailer with non-chronological shots order.

Movie trailers are essential marketing tools for the film industry, generating
anticipation by showcasing captivating scenes, storylines, and cast members.
They enable studios to fuel marketing campaigns and build interest ahead of a
movie’s release. For audiences, trailers serve as decision-making tools,
offering a condensed preview of the film’s content. Despite their importance,
creating trailers is costly, time-consuming, and demands expert knowledge.

The process of creating a trailer can be broadly divided into two
stages111https://www.youtube.com/watch?v=zkEXtwCL684. In the first stage,
video editors immerse themselves in the entire movie, viewing all the shots.
They carefully select relevant trailer shots and arrange them in a specific
order to craft an engaging flow and rhythm for the movie trailer (see Fig. 1,
top row). This is a time-consuming and tedious process as the editor has to
sort through an extensive collection of movie shots [38]. The second stage
involves fine-editing which incorporates dialogue and sound modeling into the
trailer. This work delves into the domain of Automatic Trailer Generation
(ATG) with the aim of streamlining the first stage, _i.e._ shot selection and
sequencing, to create a trailer montage from a given full movie (see Fig. 1,
bottom row).

ATG has remained a relatively underexplored problem mainly due to the
complexity of the task and the lack of well-established benchmarks.
Nevertheless, few prior works [6, 32, 23, 38] have made efforts to tackle this
challenging task. These works have approached trailer generation in different
ways. Some works [32, 23] have framed it as a binary classification problem,
where each shot in a movie is classified as either a trailer shot or not,
while others [38, 6] have treated it as a ranking problem, wherein the top-
ranked movie shots are considered as trailer shots.

While the progress has been promising, previous works on trailer generation
exhibit several limitations, mainly due to their problem formulations. First,
both classification and ranking are likely vulnerable to a long-tail
distribution problem since only a few percent of movie shots are trailer
shots, and hence, there will always be a class imbalance in classification
[32] or a significant amount of hard negatives in ranking [38]. Second, in
classification-based works [32, 23], all trailer shots are classified in
parallel, and the decision on whether a given shot is a trailer is not
conditioned by shots that are already selected as trailer shots. This
eventually leads to numerous repetitive shots being categorized as trailer
shots. Similarly, ranking-based works [38, 6] only consider two shots at a
time. Third, and most importantly, existing works do not consider shot
composition (ordering), as their formulation does not allow it. Consequently,
the shot sequence in the generated trailer mirrors the chronological order of
the shots in the original movie. See Fig. 1, central row, for a graphical
illustration of the predicted trailer generated by classification and ranking
methods.

Our work addresses these limitations by introducing a new trailer generation
framework. First, we formulate trailer generation as a regression problem
where we predict continuous feature representations of trailer shots instead
of a discrete binary class. This makes our approach less vulnerable to a long-
tail distribution problem [43]. Second, we employ an autoregressive approach
in which the prediction of a trailer shot embedding at a specific time step is
conditioned not only on the input movie sequence but also on the generated
trailer sequence up to the current step. Third, we design a model that takes
shot composition into account, and we reinforce this concept through training
on a large set of movie-trailer pairs with carefully designed losses.

In pursuit of these specifications, we introduce Trailer Generation
Transformer (TGT), a model designed to automatically generate plausible
trailers using full movies as input. Our approach uses an encoder-decoder
architecture that models ATG as a sequence-to-sequence (seq2seq) problem [36,
37], where the input sequence corresponds to the movie sequence, and the
target sequence represents the desired trailer. To make this process
computationally feasible and follow the natural structure of movies, we
utilize a shot boundary detector [35] to tokenize movies and trailers and use
a state-of-the-art visual encoder (_i.e._ CLIP [28]) to obtain sequences of
visual embeddings.

TGT’s encoder comprises two components: trailerness encoder and context
encoder. The trailerness encoder ingests the entire movie sequence and
estimates the likelihood of each shot being relevant for trailer creation.
This step identifies the visually captivating shots often used for trailer
generation. The scores are fused with the input sequence, a process we term
trailerness encoding drawing a parallel to positional encoding in [37]. The
context encoder consists of a stack of transformer encoder layers, leveraging
self-attention mechanisms [37] to contextualize each shot representation based
on the entire movie sequence, effectively achieving temporal modeling across
multiple shots.

The decoder of TGT is an autoregressive model that predicts the feature
representation of the next trailer shot. This design choice is instrumental in
enabling the model to learn shot composition, specifically, the ordering of
shots, differentiating our approach from previous works [6, 32, 23, 38]. See
the comparison between the central and bottom rows in Fig. 1. At training
time, the decoder takes the encoder output and the target trailer sequence and
employs a causal mask to learn to predict the next shot representation. At
test time, the model generates outputs in an autoregressive manner. Finally, a
greedy algorithm based on nearest neighbor retrieval matches the generated
trailer shot representations with all shots in the movie, thereby selecting
the most suitable shots for assembling the output trailer.

We curate an extensive dataset of movie-trailer pairs and employ tailored loss
functions to train TGT for automatic trailer generation effectively.
Furthermore, to foster further research on ATG, our work presents new
benchmarks based on two widely-used movie datasets: MAD [33] and MovieNet
[10]. We construct these benchmarks by pairing corresponding trailers and
segmenting both movies and trailers into shot sequences, resulting in two
datasets with movie-trailer pairs. We perform extensive experiments to assess
the performance of our approach and compare it to previous methods using
metrics that redefine how we measure trailer generation across various
aspects.

Our contributions are: (1) A novel trailer generation formulation that
effectively overcomes the inherent limitations of previous approaches. (2)
Trailer Generation Transformer (TGT), an autoregressive encoder-decoder
architecture that generates provisional trailers from full movies, which can
be refined further by experts. (3) New ATG benchmarks built atop two movie
datasets: MAD [33] and MovieNet [10]. We evaluate trailer generation
performance using a comprehensive suite of metrics that consider all relevant
aspects of the process.

##  2 Related Works

##### Trailer Generation

Several papers have attempted to automate the process of trailer generation.
Chen  _et al._ [4] analyze movie composition based on sets of rules and
grammar to generate a trailer by concatenating shots with a specific “movie
tempo” above a threshold. Smeaton  _et al._ [31] employ audiovisual features
and a support vector machine to automatically select shots for trailers,
highlighting the importance of shot sequence but leaving it to artists. Some
works [6, 9, 12] utilize text features from TV show descriptions, subtitles,
or metadata, to find similar moments in the video transcript or subtitles and
select corresponding shots for trailer creation. Others [14, 17, 32] use
audio-visual features to identify potential trailer moments from long videos.

Some works [11, 22] tackle the trailer generation task using predefined
templates and automatically fill them with clips from the initial sequence. In
contrast, our approach solely relies on the movie sequence and does not use
templates, additional metadata, or audio features to generate the trailer.
Finally, Wang  _et al._ [38] recently proposed CCANet, a deep learning model
that utilizes co-attention and contrastive attention modules to match and
distinguish trailer moments from non-trailer moments. However, CCANet is
limited to a single genre and requires genre-specific training. Our work, on
the other hand, is genre-agnostic and operates effectively across the diverse
range of genres present in MovieNet [10] and MAD [33].

##### Video Summarization

Video summarization aims to select the most important clips from a given video
to create a concise video summary. Similar to trailer generation, this task
involves a shot selection aspect that models must address. Earlier works
attempted video summarization across different video domains without explicit
supervision [2, 13, 20, 21, 25, 27, 44]. Other works approached this task
using supervised learning, coming from web video summaries [8] or TV series
summaries [34]. Representative works employed various techniques, including
fully convolutional sequence networks that learn from paired data [30] and
unpaired data [29], graph modeling [26], and determinantal point process (DPP)
for structured predictions of video sequences [16, 40, 41].

Recent attempts include using attention-based encoding to score the importance
of each frame using regression [5] or modeling the interaction between video
and text to compute a saliency score [39]. Additionally, Zhang  _et al._ [42]
proposed a sequence-to-sequence modeling approach using an encoder-decoder
LSTM-based architecture to preserve the video semantics in the output
sequence. Gan  _et al._ [7] propose leveraging movie trailers as supervision
for video summarization, while our work uses pairs of movies and trailers for
trailer generation rather than summarization. Lastly, CLIP-It [23] guides
video summarization with text through dense video captions, while TL:DW? [24]
focuses on exploring cross-modal saliency between video and text (transcript)
signals to summarize instructional videos.

##  3 Methodology

##### Problem Formulation

Given a movie sequence ${\mathcal{M}}$, the problem of automatic trailer
generation (ATG) aims at generating its corresponding trailer sequence
${{\mathcal{T}}}$. Here, ${\mathcal{M}}$ denotes a sequence of movie shots
$\\{U_{1},U_{2},\ldots,U_{n}\\}$ and ${{\mathcal{T}}}$ denotes a sequence of
trailer shots $\\{V_{1},V_{2},\ldots,V_{m}\\}$, where $n$ and $m$ represent
the number of shots in ${\mathcal{M}}$ and ${\mathcal{T}}$, respectively.
During training, we use paired movie and trailer sequences, denoted as
$\\{{\mathcal{M}}_{i},{\mathcal{T}}_{i}\\}$, where $i$ refers to the index of
the movie-trailer pair. These pairs serve as a basis for the model to learn
the essential features and patterns necessary for generating trailers from
movie sequences. In contrast, during testing, the model is given only the
movie sequences as input, without any corresponding trailer sequences. The
purpose of this setup is to evaluate the model’s ability to generate trailer
sequences $\mathcal{T^{\prime}}$ that are coherent and relevant to the given
movie sequences $\mathcal{M}$. The performance of the model is assessed by
comparing the generated trailer sequences $\mathcal{T^{\prime}}$ to the ground
truth trailer sequences $\mathcal{T}$, using suitable evaluation metrics.

Figure 2:  Architecture Overview. Subfigure (a) illustrates our TGT model’s
training pipeline. Movies are segmented into shots and transformed into visual
embeddings via a pre-trained CLIP model [28]. Enhanced with positional
embeddings and trailerness scores, these tokens undergo context encoding. The
trailer decoder, during training, uses ground-truth trailer shots as queries
for cross-attention with encoder output, then parallelly regresses the next
shot feature using a causal mask. Subfigure (b) shows the inference pipeline
where the trailer decoder sequentially generates trailer shots in an
autoregressive manner while the movie encoder process remains unchanged.

###  3.1 Proposed Trailer Generation Transformer

Camera shots in movies and trailers provide a convenient video structure that
we leverage. Our initial step involves segmenting the movie and trailer into
shot sequences ($\\{U_{i}\\}_{i=1}^{n}$ and $\\{V_{j}\\}_{j=1}^{m}$,
respectively) using a state-of-the-art shot-boundary detector [35]. Learning
directly from the pixel space of long-form videos in an end-to-end manner
poses demanding computational burdens. Therefore, to efficiently represent the
input sequences, we leverage a pretrained CLIP model [28] as a base encoder to
extract the feature representations of each shot, thereby transforming the
movie and trailer into a sequence of visual tokens, _i.e._
$\\{u_{i}\\}_{i=1}^{n}$ and $\\{v_{j}\\}_{j=1}^{m}$. To indicate the beginning
and end of a movie/trailer sequence, we use learnable start-of-sequence (SOS)
and end-of-sequence (EOS) tokens, respectively, as shown in Fig. 2.

We draw inspiration from machine translation techniques [3, 36, 37].
Specifically, we approach the trailer generation problem as a sequence-to-
sequence (seq2seq) task [36, 37]. In seq2seq, an encoder captures semantic and
syntactic information from an input token sequence and converts it into a
contextualized sequence. Then, a decoder generates the translated sequence by
producing tokens one at a time, conditioned on the contextualized sequence and
previously generated words. We apply this concept to model the automatic
trailer generation (ATG) problem, essentially “translating” a movie into a
trailer. To do this, we use an attention-based encoder-decoder architecture
[37] and introduce our Trailer Generation Transformer (TGT) model. TGT
consists of two main components: the Movie Encoder (detailed in Sec. 3.1.1)
and an auto-regressive Trailer Decoder (described in Sec. 3.1.2).

####  3.1.1 Movie Encoder

The movie encoder aims at encoding a given movie sequence into a
contextualized feature sequence by using a trailerness encoder followed by a
context encoder, as shown in the Movie Encoder block of Fig. 2.

##### Trailerness Encoder

Given an input sequence of visual tokens representing a movie,
$\\{\mathrm{\texttt{SOS}},u_{1},\ldots,u_{n},\mathrm{\texttt{EOS}}\\}$, a
positional encoding layer is first used to embed information about the
relative position of the input tokens. We then feed the sequence into a
trailerness encoder ${\mathcal{E}}_{\mathcal{T}}$. The purpose of the
trailerness encoder is to observe the full movie and reason over the
likeliness of each movie shot to be in a trailer. In
${\mathcal{E}}_{\mathcal{T}}$, a Transformer [37] self-attention layer
${\mathcal{A}}$, followed by a linear layer $f$ and a Sigmoid layer $s$, is
used to predict a trailerness score for each visual token in the input
sequence, as formulated below:

$\small\big{\\{}t^{\mathrm{p}}_{i}\big{\\}}_{i=0}^{n+1}=s(f({\mathcal{A}}(\texttt{SOS}+\sigma_{0},u_{1}+\sigma_{1},\ldots,\texttt{EOS}+\sigma_{n+1}))),$
(1)

where $\sigma_{i}$ denotes the positional encoding value at position $i$ and
$t^{\mathrm{p}}_{i}\in[0,1]$ represents the predicted score of a movie shot
$U_{i}$ as a noteworthy shot to be included in a trailer. During training, we
compute the ground truth trailerness score $t^{\mathrm{GT}}_{i}$ for each
movie shot by taking the maximum value of the cosine similarity between the
movie shot and all the trailer shots (see Eq. (2)). A high value of
$t^{\mathrm{GT}}_{i}$ means that a movie shot $U_{i}$ is in a trailer, whereas
a low value indicates that the movie shot is not similar to any shots in the
trailer. We optimize the mean-squared error between the predicted and ground
truth trailerness scores during training to guide the trailerness encoder
${\mathcal{E}}_{\mathcal{T}}$ as shown below in Eq. (3)

$t^{\mathrm{GT}}_{i}=\max_{j}~{}s_{i,j},~{}~{}\mathrm{for}~{}1\leq i\leq
n,\mathrm{where}~{}s_{i,j}=\frac{u_{i}\cdot v_{j}}{\|u_{i}\|\|v_{j}\|},$ (2)
${\mathcal{L}}_{t}=\sum_{i=0}^{n+1}{|t_{i}^{\mathrm{p}}-t_{i}^{\mathrm{GT}}|}^{2},$
(3)

where ${\mathcal{L}}_{t}$ denotes trailerness loss and the ground truth
trailerness score for SOS $(t^{\mathrm{GT}}_{0})$ and EOS
$(t^{\mathrm{GT}}_{n+1})$ tokens is set to 0. After positional encoding, the
predicted trailerness scores are added to inject information about the
relative degree of trailerness of the movie shots in the sequence (see Fig. 2
Movie Encoder).

##### Context Encoder

Trailerness encoding adjusts each positionally encoded visual token by a
unique float number, representing the trailerness score. Yet, to understand
the relationships between shots in the movie sequence, the trailerness encoded
sequence is passed through a context encoder ${\mathcal{E}}_{\mathcal{C}}$
(see Eq. (4)). The purpose of the context encoder is to obtain a
contextualized representation of the input sequence, essential for decoding a
trailer. We use a stack of Transformer [37] encoder layers for
${\mathcal{E}}_{\mathcal{C}}$. Through the multi-head self-attention
mechanism, ${\mathcal{E}}_{\mathcal{C}}$ considers the context in which each
movie shot appears, generating a representation that encapsulates the meaning
of the entire movie sequence.

$C={\mathcal{E}}_{\mathcal{C}}(\texttt{SOS}+\sigma_{0}+t_{0}^{\mathrm{p}},u_{1}+\sigma_{1}+t_{1}^{\mathrm{p}},\ldots,\texttt{EOS}+\sigma_{n+1}+t_{n+1}^{\mathrm{p}}),$
(4)

where $C$ denotes a contextualized movie sequence representation from
${\mathcal{E}}_{C}$.

####  3.1.2 Trailer Decoder

Given the output of the context encoder, we utilize a trailer decoder
${\mathcal{D}}_{\mathcal{T}}$ to autoregressively generate shot embeddings,
which are subsequently matched back to the movie sequence to compose the
trailer. At training time, the trailer decoder inputs the learned context $C$
and a positionally encoded but right-shifted trailer sequence as shown in Fig.
2 (a). The decoding is done in a sequential manner that
${\mathcal{D}}_{\mathcal{T}}$ attends to the context $C$ and a trailer
sequence up to the current position
$\\{\mathrm{\texttt{SOS}},v_{1},\ldots,v_{j-1}\\}$ in order to output the next
trailer embedding $\hat{v}_{j}$ (Eq. (5)). This enables the network to
efficiently capture sequential dependencies and positional information of
shots, facilitating the learning of shot composition for generating a trailer
sequence. The next trailer shot generation scheme in our work is equivalent to
next-word prediction in language modeling [37]. We use a stack of Transformer
[37] decoder layers for ${\mathcal{D}}_{\mathcal{T}}$. At inference time, as
ground truth trailer sequence is not available, the trailer decoder starts
with $(C,\\{\texttt{SOS}\\})$ and autoregressively decodes a trailer sequence
using the previously generated trailer shot embeddings until decoding the EOS
token. The decoding process is formulated as follows:

$\hat{v}_{j}={\mathcal{D}}_{\mathcal{T}}(C,\\{\texttt{SOS},v_{1},\ldots,v_{j-1}\\}),~{}~{}~{}~{}~{}1\leq
j\leq m+1.$ (5)

Lastly, we adopt a Greedy Search strategy utilizing Nearest Neighbor
retrieval. This method involves comparing each embedding of the decoded shots
against all shot representations from the movie. The shot with the highest
feature similarity is then selected for assembling the output trailer.

####  3.1.3 Training Losses

We train our network by optimizing the embedding distance between the
predicted trailer sequence
$\\{\hat{v}_{1},\ldots,\hat{v}_{m},\hat{v}_{m+1}\\}$ and the ground truth
sequence $\\{v_{1},\ldots,v_{m},\texttt{EOS}\\}$. To ensure that the trailer
decoder outputs the correct shot at each step, we minimize the reconstruction
loss between each predicted trailer embedding and the corresponding ground
truth embedding as shown in Eq. (6). As mentioned earlier, the order in which
the shots are decoded is crucial for trailer generation. To integrate this
concept into the proposed model, we introduce a sequence-based loss,
minimizing the Kullback-Leibler (KL) divergence loss between the predicted
trailer sequence distribution and the ground truth distribution, as shown in
Eq. (7).

${\mathcal{L}}_{\mathrm{rec}}=\sum_{j=1}^{m+1}|\hat{v}_{j}-v_{j}|^{2},$ (6)
${\mathcal{L}}_{\mathrm{KL}}=\sum_{j}\texttt{softmax}(v_{j})\cdot\log\Big{(}\frac{\texttt{softmax}(v_{j})}{\texttt{softmax}(\hat{v}_{j})}\Big{)}.$
(7)

The total training loss is defined as the sum of the trailerness encoding loss
in Eq. (3), the feature reconstruction loss in Eq. (6), and the sequence-based
loss Eq. (7).

${\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{t}+{\mathcal{L}}_{\mathrm{rec}}+{\mathcal{L}}_{\mathrm{KL}}.$
(8)

###  3.2 Datasets

Movies exhibit a large diversity encompassing genre, style, narrative
structure, theme, artistic choices, and many more. We argue that exposing our
model to a large and comprehensive collection of movies is crucial for
building a generalizable trailer generation pipeline. Following this
motivation, we collect a training set of 23,304 movies and trailers spanning
28 genres. These movies cover over 120 years of cinematic history, albeit with
a natural bias towards more recent films, reflecting the industry’s continuous
growth and increased production rates. The training set is paired with a
validation split containing 300 movie-trailer pairs used for hyperparameter
selection.

Furthermore, we introduce new benchmarks for trailer generation, building upon
two mainstream movie datasets: MAD [33] and MovieNet [10]. To align these
datasets with our task, we enhance them by including their respective trailers
obtained from IMDb [1]. Notably, not all movies in these datasets have
trailers available on their IMDb pages. Consequently, we obtained 602 out of
650 movie-trailer pairs for MAD and 989 out of 1100 for MovieNet. We also
ensure no overlap between the test sets and the training/validation splits.

###  3.3 Evaluation Metrics

We evaluate the trailer generation performance by taking both shot selection
and sequencing into account. We follow previous works [9, 38] and employ
Precision, Recall, and F1-score metrics to measure the accuracy of a model in
predicting trailer shots from an input movie sequence. We also measure the
correctness of the predicted trailer sequence using Levenshtein distance (LD)
and Sequence length difference (SLD) metrics. Levenshtein distance (also known
a edit distance) [15] is defined as the minimum number of edits (insertions,
deletions, or substitutions) required to transform the decoded trailer
sequence into the ground truth sequence. Sequence length difference measures
the absolute difference in the number of shots between the predicted trailer
sequence and the ground truth sequence.

##  4 Experiment

Table 1: Experimental comparison with different baselines.

| MAD [33] | MovieNet [10]
---|---|---
| Precision $\uparrow$ | Recall $\uparrow$ | F1-score $\uparrow$ | LD $\downarrow$ | SLD $\downarrow$ | Precision $\uparrow$ | Recall $\uparrow$ | F1-score $\uparrow$ | LD $\downarrow$ | SLD $\downarrow$
Random | 5.32 | 6.05 | 5.65 | - | - | 4.96 | 5.67 | 5.28 | - | -
CCANet [38] | 32.15 | 30.76 | 31.63 | 81.25 | - | 30.46 | 29.29 | 29.58 | 90.18 | -
CLIP-It [23] | 40.29 | 43.05 | 41.73 | 95.58 | 47.10 | 38.19 | 40.28 | 39.34 | 103.64 | 51.30
TGT (ours) | 55.30 | 49.92 | 52.38 | 21.18 | 10.78 | 49.74 | 44.32 | 46.77 | 24.66 | 13.28

|  |  |  |  |  |  Ours
---|---|---|---|---|---|---
|  |  |  |  |  |  GT

Figure 3: Subjective quality of our trailer generation. We compare a movie
trailer against the one produced by our TGT method. We highlight in green the
correctly selected shots, in orange shots that are visually similar, and in
red mismatched shots.

##### Implementation Details

We use the TransNet-V2 [35] model for preprocessing movies and trailers into
sequences of shots. Additionally, we employ the pretrained CLIP ViT H/14 [28]
for shot feature extraction. The trailerness encoder
${\mathcal{E}}_{\mathcal{T}}$ contains a single Transformer [37] encoder
layer. The context encoder ${\mathcal{E}}_{\mathcal{C}}$ consists of four
encoder layers. Five decoder layers are used for the trailer decoder
${\mathcal{D}}_{\mathcal{T}}$. Each encoder and decoder layer has a hidden
dimension of 1024, 8 attention heads, and a feed-forward dimension of 2048.
SOS and EOS tokens are initialized by creating an embedding vector of size
1024, sampled from a Normal distribution. AdamW [19] optimizer with an initial
learning rate of $1e-4$ and a cosine warm-up scheduler is used during
training. Our model is trained for 200 epochs with a mini-batch size of 8, on
a single NVIDIA A100 GPU 222Training experiments done at KAUST..

###  4.1 Trailer Generation Performance

In this section, we evaluate the performance of our approach by comparing it
with three baseline methods: Random, CCANet [38] and CLIP-It [23], on the MAD
[33] and MovieNet [10] benchmarks. To establish a Random baseline, we randomly
select $m$ shots from each movie, where $m$ represents the total number of
shots present in the corresponding ground truth trailer. As shown in Table 1,
a baseline that randomly selects movie shots for trailer generation exhibits
significantly poor performance across all metrics.

##### Comparison to Trailer Generation SoTA

CCANet [38] employs a co-attention module to rank a set of movie shots in a
contrastive manner, where shots with higher rankings are considered to be
trailer moments. To assess the performance of CCANet for trailer generation,
we select the top $m$ ranked shots from a given movie and utilize the
resulting sequence for evaluation purposes. As evident from Table 1, our
approach outperforms CCANet by a large margin. For instance, on the F1-score
metric, TGT achieves an average accuracy of 52.38% and 46.77% on MAD [33] and
MovieNet [10] datasets, respectively. In comparison, CCANet could only achieve
31.63% and 29.58%. This is mainly because CCANet has a limited attention span,
focusing on two shots at a time, which hinders accurate rankings due to the
abundance of negative samples in full-length movies. In contrast, our approach
generates trailers by simultaneously considering the entire movie sequence. It
can also be inferred from Table 1 that the trailer sequence decoded by TGT
shows much greater similarity to the ground truth sequence ($\sim$21 edits on
LD metric on MAD dataset) compared to CCANet ($\sim$81 edits) even though the
LD metric favors CCANet due to the assumption of an equal sequence length to
the ground truth. This is an expected observation since CCANet [38] mainly
targets trailer moment detection and does not take shot sequencing into
account.

##### Comparison to Video Summarization SoTA

We conduct further comparison by adapting a state-of-the-art video
summarization model, CLIP-It [23], for trailer generation. CLIP-It utilizes a
vanilla Transformer [37] architecture, where the input sequence is passed
through both the encoder and decoder. Additionally, a binary classifier is
employed to identify summary-worthy inputs in the sequence. In a parallel
fashion, we classify each shot in a movie as either belonging to the trailer
or not. The shots classified as trailer shots are subsequently merged to
construct the final trailer. The results presented in Table 1 suggest that the
CLIP-It [23] based method generally achieves better performance compared to
CCANet [38]. However, it is worth highlighting that our approach notably
outperforms CLIP-It. For example, on the Recall metric, TGT outperforms CLIP-
It by 15.95% and 10.03% on MAD [33] and MovieNet [10] benchmarks,
respectively. This is likely because CLIP-It [23] primarily focuses on shot-
level classification and may not fully capture the temporal correlations
between shots necessary for creating a compelling trailer. In contrast, our
autoregressive approach enables TGT to attend to the contextual dependencies
between shots, aligning with the movie’s narrative and dramatic flow to
generate a coherent and structured trailer. In terms of the predicted trailer
length, it is evident from Table 1 that CLIP-It deviates by $\sim$51 shots on
the MovieNet dataset, whereas TGT only deviates by $\sim$11 shots on average.

##### Qualitative Results

In Fig. 3, we qualitatively compare a trailer decoded by our approach with its
corresponding ground truth sequence. For visualization purposes, we select a
subsequence from the original trailer. The shots enclosed in green boxes
represent accurate predictions, while those in orange boxes indicate
resembling predictions. The shots in red boxes correspond to failed
predictions. The results in Fig. 3 indicate that the proposed method outputs a
reasonable trailer sequence from a given movie unseen during training. For
instance, TGT generates a third shot that closely resembles the corresponding
insert shot (black frame with text) in the ground truth, even though the movie
sequence itself does not contain that specific shot.

###  4.2 Ablation Studies

##### Network Components

In Table 2, we present ablation experiments on the different network
components used in the proposed framework. First, we study the importance of
trailerness encoding by training our network without
${\mathcal{E}}_{\mathcal{T}}$, _i.e._ we directly feed the positionally
encoded movie sequence into the context encoder. As can be inferred from Table
2, TGT trained without ${\mathcal{E}}_{\mathcal{T}}$ still gives a competitive
performance across all metrics. This is intuitive since the context encoder
has to implicitly capture the trailerness of each shot as it is trained to
learn a contextualized representation for the trailer decoder. However,
explicitly embedding relative trailerness scores (predicted from
${\mathcal{E}}_{\mathcal{T}}$) into the movie sequence has resulted in a
better performance. For instance, on the F1-score metric, trailerness encoding
results in a 4.35% and 4.53% performance improvement on MAD [33] and MovieNet
[10].

Table 2: Ablation experiment on network components.
${\mathcal{E}}_{\mathcal{T}}$: trailerness encoder;
${\mathcal{E}}_{\mathcal{C}}$: context encoder. It shows both encoders
contribute to TGT performance, and ${\mathcal{E}}_{\mathcal{C}}$ is especially
important.

|  | Precision $\uparrow$ | Recall $\uparrow$ | F1-score $\uparrow$ | LD $\downarrow$
---|---|---|---|---|---
MAD | w/o ${\mathcal{E}}_{\mathcal{T}}$ | 53.23 | 47.64 | 50.20 | 22.72
w/o ${\mathcal{E}}_{\mathcal{C}}$ | 43.36 | 36.07 | 39.23 | 25.70
w/o ${\mathcal{E}}_{\mathcal{T}}+{\mathcal{E}}_{\mathcal{C}}$ | 42.36 | 36.15 | 38.89 | 25.75
w/ ${\mathcal{E}}_{\mathcal{T}}+{\mathcal{E}}_{\mathcal{C}}$ (TGT) | 55.30 | 49.92 | 52.38 | 21.18
MovieNet | w/o ${\mathcal{E}}_{\mathcal{T}}$ | 48.38 | 41.80 | 44.75 | 25.62
w/o ${\mathcal{E}}_{\mathcal{C}}$ | 40.08 | 33.41 | 36.32 | 27.09
w/o ${\mathcal{E}}_{\mathcal{T}}+{\mathcal{E}}_{\mathcal{C}}$ | 39.01 | 33.01 | 35.64 | 27.86
w/ ${\mathcal{E}}_{\mathcal{T}}+{\mathcal{E}}_{\mathcal{C}}$ (TGT) | 49.74 | 44.32 | 46.77 | 24.66

To investigate the benefit of learning contextualized movie representation for
trailer generation, we train our network without the context encoder
${\mathcal{E}}_{\mathcal{C}}$. To do so, we use the trailerness encoded movie
sequence as a context and feed it into the trailer decoder. As shown in Table
2, training TGT without ${\mathcal{E}}_{\mathcal{C}}$ results in a notably
worse performance. For example, on the Recall metric, TGT without
${\mathcal{E}}_{\mathcal{C}}$ could only achieve a top-1 accuracy of 36.07%
and 33.41% on MAD [33] and MovieNet [10] datasets, respectively. In
comparison, TGT with ${\mathcal{E}}_{\mathcal{C}}$ gives an accuracy of 49.92%
and 44.32%. It can be observed from Table 2 that a similar performance drop
happens when using a decoder-only architecture, _i.e._ the positionally
encoded movie sequence is fed into the trailer decoder
${\mathcal{D}}_{\mathcal{T}}$ (without
${\mathcal{E}}_{\mathcal{T}}+{\mathcal{E}}_{\mathcal{C}}$). These results
demonstrate the challenge of directly generating a trailer sequence from a
movie sequence and highlight the importance of obtaining a contextualized
representation of the movie for effective trailer decoding.

Table 3: Ablation experiment on loss functions.
${\mathcal{L}}_{\mathrm{rec}}$: reconstruction loss of trailer embeddings;
${\mathcal{L}}_{t}$: trailerness loss; ${\mathcal{L}}_{\mathrm{KL}}$: KL
divergence loss of trailer sequence distribution. It shows that both
${\mathcal{L}}_{t}$ and ${\mathcal{L}}_{\mathrm{KL}}$ are beneficial, and
using them together with ${\mathcal{L}}_{\mathrm{rec}}$ leads to the highest
performance.

|  | Precision $\uparrow$ | Recall $\uparrow$ | F1-score $\uparrow$ | LD $\downarrow$
---|---|---|---|---|---
MAD | ${\mathcal{L}}_{\mathrm{rec}}$ | 49.74 | 43.61 | 46.38 | 26.81
${\mathcal{L}}_{\mathrm{rec}}+{\mathcal{L}}_{t}$ | 52.35 | 46.06 | 48.90 | 25.84
${\mathcal{L}}_{\mathrm{rec}}+{\mathcal{L}}_{\mathrm{KL}}$ | 53.86 | 48.63 | 51.03 | 21.99
${\mathcal{L}}_{\mathrm{rec}}+{\mathcal{L}}_{t}+{\mathcal{L}}_{\mathrm{KL}}$ (TGT) | 55.30 | 49.92 | 52.38 | 21.18
MovieNet | ${\mathcal{L}}_{\mathrm{rec}}$ | 45.40 | 39.32 | 42.04 | 31.84
${\mathcal{L}}_{\mathrm{rec}}+{\mathcal{L}}_{t}$ | 46.52 | 40.15 | 42.99 | 30.91
${\mathcal{L}}_{\mathrm{rec}}+{\mathcal{L}}_{\mathrm{KL}}$ | 48.48 | 43.19 | 45.60 | 25.19
${\mathcal{L}}_{\mathrm{rec}}+{\mathcal{L}}_{t}+{\mathcal{L}}_{\mathrm{KL}}$ (TGT) | 49.74 | 44.32 | 46.77 | 24.66

##### Loss Functions

Here, we examine the contribution of various loss functions used to train our
network. First, we train TGT using the reconstruction loss
${\mathcal{L}}_{\mathrm{rec}}$ in Eq. (6) as the only loss function. As can be
seen from Table 3, ${\mathcal{L}}_{\mathrm{rec}}$ is a strong enough
constraint to train a competitive baseline model. We also investigate the
benefit of optimizing the predicted trailerness scores with ground truth
scores. It can be inferred from Table 3 that TGT trained with trailerness loss
${\mathcal{L}}_{t}$ in Eq. (3) performs consistently better compared to TGT
trained without ${\mathcal{L}}_{t}$. For instance, on the Precision metric, a
network trained with ${\mathcal{L}}_{\mathrm{rec}}+{\mathcal{L}}_{t}$
outperforms a network trained with ${\mathcal{L}}_{\mathrm{rec}}$ by 5.25% and
2.46% on MAD [33] and MovieNet [10] datasets, respectively.

We investigate the impact of minimizing the KL divergence between the
predicted trailer sequence and the ground truth sequence as defined in Eq.
(7). The results in Table 3 show that training with
${\mathcal{L}}_{\mathrm{KL}}$ results in a considerable performance gain.
Particularly, on the Levenshtein distance (LD) metric, a trailer decoded by
TGT trained with ${\mathcal{L}}_{\mathrm{KL}}$ requires an average of 4.83 and
6.65 fewer edit steps (to change it to the ground truth sequence) on MAD [33]
and MovieNet [10] datasets, respectively, compared to TGT trained without
${\mathcal{L}}_{\mathrm{KL}}$. These results emphasize the significance of the
KL-divergence loss ${\mathcal{L}}_{\mathrm{KL}}$ in ensuring that the trailer
sequence generated by TGT closely matches the ground truth sequence. As can be
inferred from Table 3, the combination of all training losses
(${\mathcal{L}}_{\mathrm{rec}}+{\mathcal{L}}_{t}+{\mathcal{L}}_{\mathrm{KL}}$)
yields the best network performance.

###  4.3 Experimental Analyses

##### Text-Controlled Trailer Generation

Thus far, we have formulated movie trailer generation as a video-to-video
problem. Here, we experiment with incorporating language into TGT to guide
trailer decoding. For this purpose, we obtain the plot summary of each movie
from IMDb [1] and encode the text using a pretrained RoBERTa [18] model. We
explore two types of network variants for text-conditioned trailer generation.
First, we simply concatenate the encoded text with the output of the context
encoder as a control to enrich the context pool of TGT, and the resulting
representation will be fed into the trailer decoder (referred to as “encoded
text” in Table 4). Second, we experiment with reasoning over the encoded text
before feeding it to the trailer decoder. We use a single Transformer [37]
encoder layer to obtain a contextualized text representation (referred to as
“contextualized text” in Table 4). As evident from Table 4, “encoded text”
leads to a significant improvement in trailer decoding performance across all
metrics. Using “contextualized text” on top of pretrained RoBERTa [18] further
improves trailer generation performance. This finding is consistent with our
expectation that additional context enhances generation quality. More
importantly, as a plot summary typically includes major plot points such as
the setting, characters, and conflicts, it enables the trailer decoder to
establish a more structured trailer decoding rule by attending to the relevant
movie shots.

Table 4: Analysis on text-controlled trailer generation. “Encoded text” leads
to a significant improvement across all metrics. Using “contextualized text”
further improves the performance.

|  | Precision $\uparrow$ | Recall $\uparrow$ | F1-score $\uparrow$ | LD $\downarrow$
---|---|---|---|---|---
MAD | TGT | 55.30 | 49.92 | 52.38 | 21.18
\+ Encoded text | 59.27 | 54.68 | 56.81 | 19.34
\+ Contextualized text | 60.93 | 56.44 | 58.52 | 18.40
MovieNet | TGT | 49.74 | 44.32 | 46.77 | 24.66
\+ Encoded text | 53.69 | 49.05 | 51.19 | 23.59
\+ Contextualized text | 56.13 | 51.65 | 53.72 | 21.81

Table 5: Analysis on trailer shot selection. Relaxing the metric enhances
performance, indicating the potential of TGT to be used in collaboration with
a human editor.

|  | Precision $\uparrow$ | Recall $\uparrow$ | F1-score $\uparrow$ | LD $\downarrow$
---|---|---|---|---|---
MAD | Top-1 | 55.30 | 49.92 | 52.38 | 21.18
Top-5 | 65.31 | 60.57 | 62.76 | 15.51
Top-10 | 69.02 | 64.85 | 66.79 | 13.05
MovieNet | Top-1 | 49.74 | 44.32 | 46.77 | 24.66
Top-5 | 58.58 | 54.05 | 56.12 | 18.86
Top-10 | 61.84 | 57.74 | 59.63 | 16.23

##### Shot Selection

One of the key challenges of generating a trailer from a movie is shot
selection. At each step of decoding, there could be multiple movie shot
candidates that are semantically similar to the target trailer shot. In Table
5, we evaluate our model’s performance by analyzing the quality of the shots
decoded at each step, considering the top-5 and top-10 matching shots. As
anticipated, expanding the pool of shot candidates leads to enhanced model
performance across all evaluated metrics. For instance, on MAD [33] benchmark,
TGT achieves a top-10 precision of 69.02% and the resulting trailer is only
$\sim$13 edit steps away from the ground truth trailer. This is particularly
noteworthy as it showcases the potential of our model to serve as a shot
recommender during the editing process, where a human editor collaborates with
the model to make informed decisions.

##### Scaling Effect

In Table 6, we present the results of a scaling experiment conducted to
evaluate the performance of our approach using different training data sizes,
specifically 10%, 50%, and 100% of the available data. As can be inferred from
the table, model performance directly correlates with the size of the training
data. This observation is intuitive, as a larger dataset enables the model to
encompass a broader range of patterns and variations found in movie and
trailer sequences.

Table 6: Analysis on the training data size. Performance drops significantly
when using only 10% of the data, while using 50% of the data yields decent
performance compared to the full dataset.

|  | Precision $\uparrow$ | Recall $\uparrow$ | F1-score $\uparrow$ | LD $\downarrow$
---|---|---|---|---|---
MAD | 10 % | 35.23 | 33.25 | 34.11 | 27.34
50 % | 50.30 | 47.71 | 48.80 | 24.32
100 % | 55.30 | 49.92 | 52.38 | 21.18
MovieNet | 10 % | 28.38 | 26.81 | 27.46 | 30.84
50 % | 44.71 | 41.61 | 42.96 | 27.43
100 % | 49.74 | 44.32 | 46.77 | 24.66

###  4.4 Discussion and Limitations

The proposed TGT method streamlines the trailer creation process by enabling
efficient selection and ordering of shots. However, the current method does
not incorporate dialogue and sound modeling, which are crucial for fine
editing. This limitation could be addressed in future work by incorporating
these additional factors into the TGT model. Despite this limitation, the TGT
method can arguably provide significant time-saving to editors by automating
the initial steps of shot selection and ordering. This shift in workload
empowers editors to focus on the artistic aspects of trailer creation, such as
refining cuts, durations, and injecting subtle audio elements to enhance the
trailer further. Therefore, the TGT method, while limited in scope, has the
potential to significantly improve the efficiency and creativity of trailer
creation.

##  5 Conclusion

This work presents a novel approach to automatic trailer generation. Our TGT
models the task as a machine translation problem and uses an effective encode-
decoder architecture to generate plausible trailers. We present two newly
constructed benchmarks and show that TGT outperforms state-of-the-art
approaches. We believe this study has the potential to advance video
summarization and promotional content creation across various domains.

##### Acknowledgement

This work was supported by the King Abdullah University of Science and
Technology (KAUST) Office of Sponsored Research through the Visual Computing
Center (VCC) funding, as well as the SDAIA-KAUST Center of Excellence in Data
Science and Artificial Intelligence (SDAIA-KAUST AI).

## References

  * [1] Imdb.  https://www.imdb.com/.  Accessed: March 15, 2023.
  * Badamdorj et al. [2022] Taivanbat Badamdorj, Mrigank Rochan, Yang Wang, and Li Cheng.  Contrastive learning for unsupervised video highlight detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14042–14052, 2022.
  * Bahdanau et al. [2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_ , 2014.
  * Chen et al. [2004] Hsuan-Wei Chen, Jin-Hau Kuo, Wei-Ta Chu, and Ja-Ling Wu.  Action movies segmentation and summarization based on tempo analysis.  In _Proceedings of the 6th ACM SIGMM international workshop on Multimedia information retrieval_ , pages 251–258, 2004.
  * Fajtl et al. [2019] Jiri Fajtl, Hajar Sadeghi Sokeh, Vasileios Argyriou, Dorothy Monekosso, and Paolo Remagnino.  Summarizing videos with attention.  In _Computer Vision–ACCV 2018 Workshops: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers 14_ , pages 39–54. Springer, 2019.
  * Gaikwad et al. [2021] Bhagyashree Gaikwad, Ankita Sontakke, Manasi Patwardhan, Niranjan Pedanekar, and Shirish Karande.  Plots to previews: Towards automatic movie preview retrieval using publicly available meta-data.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops_ , pages 3205–3214, 2021.
  * Gan et al. [2023] Bei Gan, Xiujun Shu, Ruizhi Qiao, Haoqian Wu, Keyu Chen, Hanjun Li, and Bo Ren.  Collaborative noisy label cleaner: Learning scene-aware trailers for multi-modal highlight detection in movies.  _arXiv preprint arXiv:2303.14768_ , 2023.
  * Gygli et al. [2014] Michael Gygli, Helmut Grabner, Hayko Riemenschneider, and Luc Van Gool.  Creating summaries from user videos.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VII 13_ , pages 505–520. Springer, 2014.
  * Hesham et al. [2018] Mohammad Hesham, Bishoy Hani, Nour Fouad, and Eslam Amer.  Smart trailer: Automatic generation of movie trailer using only subtitles.  In _2018 First International Workshop on Deep and Representation Learning (IWDRL)_ , pages 26–30. IEEE, 2018.
  * Huang et al. [2020] Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin.  Movienet: A holistic dataset for movie understanding.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16_ , pages 709–727. Springer, 2020.
  * Irie et al. [2010] Go Irie, Takashi Satou, Akira Kojima, Toshihiko Yamasaki, and Kiyoharu Aizawa.  Automatic trailer generation.  In _Proceedings of the 18th ACM international conference on Multimedia_ , pages 839–842, 2010.
  * Kawai et al. [2007] Yoshihiko Kawai, Hideki Sumiyoshi, and Nobuyuki Yagi.  Automated production of tv program trailer using electronic program guide.  In _Proceedings of the 6th ACM international conference on Image and video retrieval_ , pages 49–56, 2007.
  * Khosla et al. [2013] Aditya Khosla, Raffay Hamid, Chih-Jen Lin, and Neel Sundaresan.  Large-scale video summarization using web-image priors.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2698–2705, 2013.
  * King et al. [2021] Allyson King, Eric Zavesky, and Michael J Gonzales.  User preferences for automated curation of snackable content.  In _26th International Conference on Intelligent User Interfaces_ , pages 270–274, 2021.
  * Levenshtein et al. [1966] Vladimir I Levenshtein et al.  Binary codes capable of correcting deletions, insertions, and reversals.  In _Soviet physics doklady_ , pages 707–710. Soviet Union, 1966.
  * Li et al. [2018] Yandong Li, Liqiang Wang, Tianbao Yang, and Boqing Gong.  How local is the local diversity? reinforcing sequential determinantal point processes with dynamic ground sets for supervised video summarization.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 151–167, 2018.
  * Liu and Jiang [2015] Xingchen Liu and Jianming Jiang.  Semi-supervised learning towards computerized generation of movie trailers.  In _2015 IEEE International Conference on Systems, Man, and Cybernetics_ , pages 2990–2995. IEEE, 2015.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Lu and Grauman [2013] Zheng Lu and Kristen Grauman.  Story-driven summarization for egocentric video.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2714–2721, 2013.
  * Mahasseni et al. [2017] Behrooz Mahasseni, Michael Lam, and Sinisa Todorovic.  Unsupervised video summarization with adversarial lstm networks.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , pages 202–211, 2017.
  * Mishra et al. [2022] Prakhar Mishra, Chaitali Diwan, Srinath Srinivasa, and G Srinivasaraghavan.  A semi-automatic approach for generating video trailers for learning pathways.  In _Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners’ and Doctoral Consortium: 23rd International Conference, AIED 2022, Durham, UK, July 27–31, 2022, Proceedings, Part II_ , pages 302–305. Springer, 2022.
  * Narasimhan et al. [2021] Medhini Narasimhan, Anna Rohrbach, and Trevor Darrell.  Clip-it! language-guided video summarization.  _Advances in Neural Information Processing Systems_ , 34:13988–14000, 2021.
  * Narasimhan et al. [2022] Medhini Narasimhan, Arsha Nagrani, Chen Sun, Michael Rubinstein, Trevor Darrell, Anna Rohrbach, and Cordelia Schmid.  Tl; dw? summarizing instructional videos with task relevance and cross-modal saliency.  In _European Conference on Computer Vision_ , pages 540–557. Springer, 2022.
  * Panda and Roy-Chowdhury [2017] Rameswar Panda and Amit K Roy-Chowdhury.  Collaborative summarization of topic-related videos.  In _Proceedings of the IEEE Conference on computer vision and pattern recognition_ , pages 7083–7092, 2017.
  * Park et al. [2020] Jungin Park, Jiyoung Lee, Ig-Jae Kim, and Kwanghoon Sohn.  Sumgraph: Video summarization via recursive graph modeling.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXV 16_ , pages 647–663. Springer, 2020.
  * Potapov et al. [2014] Danila Potapov, Matthijs Douze, Zaid Harchaoui, and Cordelia Schmid.  Category-specific video summarization.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13_ , pages 540–555. Springer, 2014.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Rochan and Wang [2019] Mrigank Rochan and Yang Wang.  Video summarization by learning from unpaired data.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7902–7911, 2019.
  * Rochan et al. [2018] Mrigank Rochan, Linwei Ye, and Yang Wang.  Video summarization using fully convolutional sequence networks.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 347–363, 2018.
  * Smeaton et al. [2006] Alan F Smeaton, Bart Lehane, Noel E O’Connor, Conor Brady, and Gary Craig.  Automatically selecting shots for action movie trailers.  In _Proceedings of the 8th ACM international workshop on Multimedia information retrieval_ , pages 231–238, 2006.
  * Smith et al. [2017] John R Smith, Dhiraj Joshi, Benoit Huet, Winston Hsu, and Jozef Cota.  Harnessing ai for augmenting creativity: Application to movie trailer creation.  In _Proceedings of the 25th ACM international conference on Multimedia_ , pages 1799–1808, 2017.
  * Soldan et al. [2022] Mattia Soldan, Alejandro Pardo, Juan León Alcázar, Fabian Caba, Chen Zhao, Silvio Giancola, and Bernard Ghanem.  Mad: A scalable dataset for language grounding in videos from movie audio descriptions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5026–5035, 2022.
  * Song et al. [2015] Yale Song, Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes.  Tvsum: Summarizing web videos using titles.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5179–5187, 2015.
  * Souček and Lokoč [2020] Tomáš Souček and Jakub Lokoč.  Transnet v2: an effective deep network architecture for fast shot transition detection.  _arXiv preprint arXiv:2008.04838_ , 2020.
  * Sutskever et al. [2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le.  Sequence to sequence learning with neural networks.  _Advances in neural information processing systems_ , 27, 2014.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. [2020] Lezi Wang, Dong Liu, Rohit Puri, and Dimitris N Metaxas.  Learning trailer moments in full-length movies with co-contrastive attention.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVIII 16_ , pages 300–316. Springer, 2020.
  * Xu et al. [2023] Yifang Xu, Yunzhuo Sun, Yang Li, Yilei Shi, Xiaoxiang Zhu, and Sidan Du.  Mh-detr: Video moment and highlight detection with cross-modal transformer.  _arXiv preprint arXiv:2305.00355_ , 2023.
  * Zhang et al. [2016a] Ke Zhang, Wei-Lun Chao, Fei Sha, and Kristen Grauman.  Summary transfer: Exemplar-based subset selection for video summarization.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1059–1067, 2016a.
  * Zhang et al. [2016b] Ke Zhang, Wei-Lun Chao, Fei Sha, and Kristen Grauman.  Video summarization with long short-term memory.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part VII 14_ , pages 766–782. Springer, 2016b.
  * Zhang et al. [2018] Ke Zhang, Kristen Grauman, and Fei Sha.  Retrospective encoders for video summarization.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 383–399, 2018.
  * Zhang et al. [2023] Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng.  Deep long-tailed learning: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * Zhou et al. [2018] Kaiyang Zhou, Yu Qiao, and Tao Xiang.  Deep reinforcement learning for unsupervised video summarization with diversity-representativeness reward.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2018.


# Dependence of the electronic structure of the EuS/InAs interface on the
bonding configuration

Maituo Yu  Department of Materials Science and Engineering, Carnegie Mellon
University, Pittsburgh, PA 15213, USA  Saeed Moayedpour  Department of
Chemistry, Carnegie Mellon University, Pittsburgh, PA 15213, USA  Shuyang Yang
Derek Dardzinski  Department of Materials Science and Engineering, Carnegie
Mellon University, Pittsburgh, PA 15213, USA  Chunzhi Wu  Department of Civil
and Environmental Engineering, Carnegie Mellon University, Pittsburgh, PA
15213, USA  Vlad S. Pribiag  School of Physics and Astronomy, University of
Minnesota, Minneapolis, MN 55455, USA  Noa Marom  Electronic mail:
<EMAIL_ADDRESS>Department of Materials Science and Engineering,
Carnegie Mellon University, Pittsburgh, PA 15213, USA  Department of
Chemistry, Carnegie Mellon University, Pittsburgh, PA 15213, USA  Department
of Physics, Carnegie Mellon University, Pittsburgh, PA 15213, USA

###### Abstract

Recently, the EuS/InAs interface has attracted attention for the possibility
of inducing magnetic exchange correlations in a strong spin-orbit
semiconductor, which could be useful for topological quantum devices. We use
density functional theory (DFT) with a machine-learned Hubbard $U$ correction
[npj Comput. Mater. 6, 180 (2020)] to elucidate the effect of the bonding
configuration at the interface on the electronic structure. For all interface
configurations considered here, we find that the EuS valence band maximum
(VBM) lies below the InAs VBM. In addition, dispersed states emerge at the top
of the InAs VBM at the interface, which do not exist in either material
separately. These states are contributed mainly by the InAs layer adjacent to
the interface. They are localized at the interface and may be attributed to
charge transfer from the EuS to the InAs. The interface configuration affects
the position of the EuS VBM with respect to the InAs VBM, as well as the
dispersion of the interface state. For all interface configurations studied
here, the induced magnetic moment in the InAs is small. This suggests that
this interface, in its coherent form studied here, is not promising for
inducing equilibrium magnetic properties in InAs.

††preprint: APS/123-QED

##  I  Introduction

Majorana zero modes may provide a pathway to the realization of fault tolerant
quantum computing [1, 2, 3, 4, 5, 6]. Several types of hybrid materials
systems have been proposed to produce Majorana zero modes. These include the
surface of a 3D topological insulator interfaced with an s-wave
superconductor; a 2D semiconductor with strong orbit-coupling in proximity to
an s-wave superconductor; and a 1D semiconductor with the Majorana quasi-
particles appearing at the two ends of the 1D superconductor wire [7, 8, 9,
10, 4, 11]. To induce superconductivity at the interface, the semiconductor is
in contact with a superconductor, such as Al, Pb, and Nb[12, 13, 14, 15, 16].
One challenge that may be encountered in such platforms is that a large
magnetic field is required to drive the system into the topological regime,
which may be detrimental to the superconductivity [17, 18, 19, 10]. A possible
solution is to add a ferromagnet (FM) in contact with the semiconductor to
induce magnetism in the semiconductor internally via non-equilibrium spin
injection [20], stray fields [21] or the magnetic proximity effect, thus
obviating the need for an external magnetic field [22].

FM/semiconductor heterostructures have been widely used for spintronics [23,
24, 25]. Typically, the FM is a metal, such as Fe, Ni, and Co [26, 27, 28].
However, a metallic FM may induce undesirable states in the gap of a
semiconductor and can have detrimental effects by shunting away the current in
a hybrid device. A ferromagnetic insulator (FMI) may be advantageous for
Majorana devices if it provides proximity induced magnetism in the
semiconductor without introducing such deleterious effects [29]. Europium
containing FMIs have a particularly high magnetic moment per Eu atom and a
large exchange coupling [30]. Several studies have reported proximity induced
magnetism in various materials in contact with europium oxide and europium
chalcogenides, including the EuS/Bi2Se3 interface [31], phosphorene on
EuO[32], MoS2-EuS heterojunctions [33], and EuS/Al heterostructures [34, 35,
36, 37]. A strong interfacial exchange field has been detected at the EuS/Pt
interface [38]. Moreover, proximity to EuS may break the time-reversal
symmetry and open a gap in the interface state of topological insulators [39,
40, 41].

EuS has a rock salt structure with less than 1% lattice mismatch to InAs,
which makes it favorable for epitaxial growth. Therefore, the EuS/InAs
interface could be an ideal option for a hybrid FMI-semiconductor-
superconductor structure if the proximity-induced magnetism in the InAs is
sufficiently strong. Recently, an epitaxial EuS/InAs interface with a (001)
orientation has been grown and characterized [42]. Angle-resolved
photoemission spectroscopy (ARPES) has shown that the Eu 4$f$ states, which
form the top of the valence manifold of EuS, lie below the valence band
maximum of InAs and that an interfacial quantum well state forms in the InAs.
However, proximity-induced magnetism in the InAs could not be detected in
neutron and X‐ray reflectivity measurements. Based on this, it was concluded
in Ref. [42] that the magnetic proximity effect at the EuS/InAs interface is
weak and has a small influence on the overall magnetism in InAs. Subsequent
work on a hybrid structure of InAs nanowires, Al, and EuS, in which the EuS
was in contact with both the InAs and the Al has argued for the presence of
magnetic exchange effects [43]. Very recently zero-bias peaks, which are a
signature of topological superconductivity, have been reported in such a
hybrid structure [44]. However, these observations could be attributed to
distinct mechanisms, such as the EuS inducing magnetism in the Al [36, 34],
rather than the InAs or changing stray magnetic fields during the complex
magnetic reversal [20].

To elucidate the electronic structure of the EuS/InAs interface and help
resolve the question of whether and to what extent proximity induced magnetism
is achieved in the InAs, we conduct first principles simulations within
density functional theory (DFT). A challenge for DFT simulations of the
EuS/InAs interface is that commonly used semi-local exchange-correlation
functionals, such as the generalized gradient approximation of Perdew, Burke,
and Erzerhof (PBE) [45] provide a poor description of EuS and InAs, producing
no band gap for both materials [46, 47, 48, 49]. This may be corrected by
using more accurate hybrid functionals, which contain a fraction of exact
exchange [50, 51]. However, hybrid functionals are impractical for simulations
of large interface models, owing to their high computational cost.
Alternatively, within the DFT+$U$ approach [52, 53], a Hubbard $U$ correction
may be added to a semi-local DFT functional, such as the PBE functional.
Recently, we have developed a method of machine learning the optimal value of
the effective Hubbard $U$ parameter by Bayesian optimization (BO) [46]. The
PBE+U(BO) method has been demonstrated to provide a reliable description of
bulk EuS and InAs [46], as well as InAs surfaces [54] and interfaces [55].
Here, we use PBE+U(BO) to study the electronic structure of the EuS/InAs (001)
interface. The bonding configuration at the interface may significantly affect
the electronic and magnetic properties [56], therefore we consider different
interface configurations.

We find that the interface configurations studied here share qualitatively
similar electronic and magnetic properties. However, the quantitative details
differ between configurations. In all interface configurations the valance
band maximum (VBM) of the EuS lies below the VBM of the InAs and a dispersed
interface state emerges in agreement with the ARPES experiments reported in
Ref. [42]. The interface configuration affects the extent of charge transfer
from EuS to InAs, and as a result, the band alignment between the EuS and
InAs, as well as the interface state dispersion. Greater charge transfer
pushes the EuS VBM lower with respect to the InAs VBM and leads to more
dispersed interface states. In all configurations studied here, the induced
magnetic moment in the InAs is found to be small and short-ranged, which is
also in agreement with Ref. [42]. This indicates that the EuS/InAs interface,
in its coherent form, is not particularly well suited for inducing magnetic
exchange correlations in InAs.

Figure 1:  The band gap of (a) InAs and (b) EuS slabs as a function of the
number of atomic layers. Dashed lines denote the bulk limit of 0.41 eV for
InAs and 1.58 eV for EuS.  Figure 2:  (a) Bulk unfolded band structure of an
InAs slab with 75 layers compared to (b) the band structure of bulk InAs
oriented in the (001) direction. (c) Bulk unfolded band structure of a EuS
slab with 21 layers compared to (d) the band structure of bulk EuS oriented in
the (001) direction.

##  II  Methods

DFT calculations were performed using the Vienna ab initio simulation package
(VASP) with the projector augmented wave method (PAW) [57, 58, 59]. PBE+U was
used to describe the exchange-correlation interactions among electrons. The
Hubbard $U$ correction was applied within the Dudarev approach [53] to the $p$
orbitals of In and As and the $f$ orbitals of Eu. The effective U parameters,
$U_{eff}=U-J$, which correspond to the difference between the on-site Coulomb
interaction, $U$, and the exchange interaction, $J$, were machine learned by
Bayesian optimization (BO), as described in Ref. [46]. The $U_{eff}$ values
obtained therein were $U_{eff}^{In,p}$= -0.5 eV, $U_{eff}^{As,p}$= -7.5 eV,
and $U_{eff}^{Eu,f}$= 8.4 eV. The PBE+U(BO) method with these parameters has
been used successfully for InAs surfaces [54] and the InAs/GaSb interface
[55]. A plane-wave basis set was used with a kinetic energy cutoff of 450 eV.
The Brillouin zone was sampled using the Monkhorst-Pack scheme with an
$8\times 8\times 1$ k-point mesh. Spin-orbit coupling (SOC) [60] was included
in all calculations with the $z$ spin quantization axis, as well as the dipole
correction [61]. To avoid spurious interactions between periodic replicas in
slab models, a sufficiently large vacuum region of $\sim$40 Å was added in the
$z$ direction. To eliminate surface states due to dangling bonds, the InAs
slab was passivated by pseudo-hydrogen atoms with 1.25 electrons. In the
structural relaxation and the interfacial distance optimization, the
Tkatchenko-Scheffler (TS) pairwise dispersion method [62] was used to describe
the van der Waals interactions between the EuS film and the InAs substrate.
The convergence criterion used in the structural relaxation was for the
Hellman-Feynman forces acting on ions to be below 0.01 eV/Å. Bulk band
unfolding [54] was applied to all band structures to project the slab band
structure onto the primitive cell and facilitate comparison with ARPES
experiments. The spin-polarized density of states (DOS) of calculations with
SOC was plotted by extracting the DOS of the majority spin channels and
minority spin channels along the easy axis.

##  III  Results and discussion

We begin by studying the band structures of InAs(001) and EuS(001) slabs
separately. Owing to the effect of quantum confinement, the band gap of a slab
model decreases as the number of layers increases, eventually approaching the
bulk band gap, as shown in Fig 1. The band gap of InAs converges with 75
atomic layers and the band gap of EuS converges with 21 atomic layers. The
bulk-unfolded band structures of these slab models of InAs and EuS are
compared to bulk band structures with a (001) orientation in Fig. 2. With this
number of layers the band gap of InAs is 0.02 eV larger than the bulk limit
and the band gap of EuS is 0.22 eV larger than the bulk limit. The main
features of the bulk band structure are reproduced well by the slab models.

To model the EuS/InAs interface, we assumed that an epitaxially matched EuS
film would grow on top of In-terminated InAs(001), based on the experiments
reported in Ref. [42]. Therefore, the lattice constant of the InAs substrate
was fixed at 6.06 Å and the EuS film was strained to match it. Four unique
interface configurations are possible with the EuS atoms positioned directly
above the InAs lattice sites, as shown in Fig. 3a-d. In Ref. [42]
configuration C1 (shown in Fig. 3a) was proposed based on high-angle annular
dark-field (HAADF) imaging using an aberration-corrected scanning transmission
electron microscope (AC-STEM). It was also noted therein that single layer
intermixing caused a displacement from ideal atomic rows and that atomic steps
were likely present at the interface. Here, we only consider ideal interfaces.
Intermixing, atomic steps, and the possible co-existence of different
interface configurations are not taken into account.

Figure 3:  Side view and top view of four configurations of an ideal EuS/InAs
interface. From top to bottom, the configurations are labeled as (a)
configuration 1 (C1), (b) configuration 2 (C2), (c) configuration 3 (C3), and
(d) configuration 4 (C4).

To determine the interfacial distance to use as a starting point for
relaxation for each configuration, the EuS film was moved along $z$ direction
with respect to the InAs substrate with a step size of 0.1 Å. The resulting
energy curves are shown in Fig. 4. We find that the interface configuration
significantly affects the interfacial distance. The largest interfacial
distance of 3.2 Å is found for configuration C2 and the smallest interfacial
distance of 2.4 Å is found for configuration C3.

Figure 4:  Total energy change as a function of the interfacial distance for
the four configurations. The lowest minimum is referenced to zero.

A potential energy surface (PES) scan was performed by shifting the EuS film
in the $xy$ plane at a fixed distance of 2.4 Å from the InAs substrate along
$z$. The PES scan was performed with 11 layers of InAs and 3 layers of EuS. We
performed a convergence test with a varying number of layers of InAs and EuS
to verify that the energy ranking exhibited in the PES is not affected by the
number of layers. The resulting PES is shown in Fig. 5. Configuration C3 is
the global minimum. Configuration C4 is a local minimum. Configuration C2 is
the global maximum. Based on our calculations, configuration C1, reported in
Ref. [42] is, in fact, a local maximum.

Figure 5:  PES scan at the interfacial distance of 2.4 $\AA$

Starting from the optimal interfacial distance obtained for each
configuration, geometry relaxation was performed for the first 6 InAs layers
and 2 EuS layers around the interface. The atomic positions were constrained
in the $xy$ plane and allowed to vary in the $z$ direction. Fig. 6 shows the
relative energy ranking of the four configurations before and after
relaxation. Relaxation resulted in reduction of the energies of all
structures, however it did not affect the stability ranking. After relaxation,
configuration C3 is still the most stable, followed by configurations C4, C1,
and C2. Configuration C2 is the least stable owing to unfavorable cation-
cation bonding at the interface. The higher stability of configurations C3 and
C4 may be attributed to the tetrahedral site being favored over the edge-
center site because it matches better with the zincblende packing of InAs. The
HAADF-STEM micrograph of the EuS/InAs interface shown in Ref. [42] could also
be compatible with the C3 model in terms of atomic positions. However, a more
complex 3D structure including, for example, surface atomic steps, would be
needed to explain the HAADF intensity reduction and atomic column elongation
in the positions nearby the interface for a full matching of the experimental
image with the C3 epitaxial arrangement.

Figure 6:  Energy ranking of the four interface configurations before and
after relaxation. The lowest energy structure after relaxation is referenced
to zero. Figure 7:  Unfolded band structure of different interface
configurations:(a) C1, (b) C3, and (c) C4. The red, green, and purple markers
denote the contributions from InAs, EuS, and the interface states,
respectively.

We now proceed to examine the effect of the atomistic configuration of the
interface on its electronic structure. Because configuration C2 is unstable we
do not consider it further. Configuration C1 is considered despite being a
local maximum because it was proposed in Ref. [42]. Fig. 7 shows bulk-unfolded
band structures of configurations C1, C3, and C4. For all three configurations
the the top of the EuS valence band (green) is found below the top of the InAs
valence band (red) at the $\Gamma$ point in agreement with the ARPES results
from Ref. [42]. However, the interface configuration significantly affects the
band alignment: for configuration C1, the EuS VBM is found 0.72 eV below the
InAs VBM; for configuration C3, the EuS VBM is found 0.36 eV below the InAs
VBM; and for configuration C4 it is found 0.09 eV below the InAs VBM. The band
alignment produced by configuration C3 appears to be in better agreement with
the ARPES results from Ref. [42] than configuration C1.

Interestingly, dispersed states, colored in purple in Fig. 7, appear at the
top of the valence band at the EuS/InAs interface, which are not present in
either InAs or EuS alone (see Fig. 2). By resolving the atomic contributions
to the band structure, we find that these interface states are contributed
predominantly by the first InAs layer at the interface. The interface states
found here are in agreement with the quantum well state observed in ARPES in
Ref. [42]. The dispersion of the interface states depends on the interface
configuration with configuration C1 producing the greatest band dispersion and
configuration C4 yielding the smallest band dispersion. The local density of
states (LDOS), shown in Fig. 8, shows that the interface states (purple) are
present around the Fermi level only at the interface. Away from the interface,
the bulk DOS of InAs (red) and EuS (green) is recovered. Fig. 9 shows the
partial charges associated with the interface states at the $\Gamma$ point.
The interface configuration also affects the wave-function of the interface
state: For configuration C1 the interface state wave-function is predominantly
localized in the EuS side of the interface; for configuration C3 the interface
state wave-function is predominantly localized in the InAs side of the
interface; and for configuration C4 the interface state wave-function is
evenly distributed across the InAs and EuS.

Figure 8:  Local density of states (LDOS) of (a) configuration C1, (b)
configuration C3, and (c) configuration C4. The interface layer refers to the
first InAs and EuS layers at the interface; The middle layer of InAs refers to
the 38th InAs layer from the interface; The middle layer of EuS refers to the
11th EuS layer from the interface. The Fermi levels are shifted to 0 eV.
Figure 9:  The partial charges associated with the interface states of (a)
configuration C1, (b) configuration C3, and (c) configuration C4 at $\Gamma$
point. The electron charge isosurface is colored in yellow and its
intersections with the unit cell boundary are colored in blue. Hollow spaces
in the isosurface that intersect with the unit cell boundary are colored in
red. Figure 10:  Charge transfer as a function of distance from the interface
for InAs/EuS with different configurations. The center of the interface is
referenced to 0. The InAs is on the left side and the EuS is on the right
side.

The effect of the interface configuration on the band alignment and the
dispersion of the interface state may be explained by the extent of charge
transfer at the interface. The net charge transfer was calculated by:

$C_{net}(z)=C[interface]-C[substrate]-C[film]$ (1)

where $C[\cdot]$ is the charge averaged over the $xy$ plane along the $z$-axis
[41]. The center of the interface is defined as $z=0$. To evaluate the charge
transfer, DFT calculations were performed for the interface slab and for
separate slabs containing only the substrate and only the film with the same
geometry as the interface slab. Fig. 10 shows the resulting charge transfer
for different interface configurations. $C_{net}$ is positive at the
interfacial layer of InAs and negative at the first layer of EuS, meaning that
charge is transferred from the EuS to the InAs. Configuration C1 exhibits the
greatest charge transfer, followed by configuration C3 and configuration C4.
The larger the charge transfer, the lower the EuS VBM lies below the InAs VBM
and the greater the band dispersion of the interface state.

Figure 11:  Spin polarization (the difference between the majority DOS and the
minority DOS) as a function of the layer index for configuration C3. Up and
down refer to the spin-majority DOS and spin-minority DOS. The position of the
interface is indicated by the black dashed line.

The computed magnetic moment of the Eu atoms in bulk EuS is found to be 7.0
$\mu$B, as expected. However, the proximity-induced magnetic moment in the
InAs is small for all three interface configurations. The magnetic moment in
the first layer of InAs (the In atom) adjacent to the EuS is 0.013 $\mu$B for
configuration C1, 0.087 $\mu$B for configuration C3, and 0.166 $\mu$B for
configuration C4. The magnetic moment decays rapidly with the distance from
the interface and vanishes completely by the 8th atomic layer of InAs from the
interface. The induced magnetic moment is inversely correlated with the extent
of charge transfer at the interface. In Ref. [42], a DFT calculation was
conducted for a small interface model with two unit cells of each material in
configuration C1 and 10 $\AA$ of vacuum without surface passivation. PBE+U was
used with a $U$ value of 6 eV for the $f$ orbital of Eu. They found a magnetic
moment of 0.07 $\mu$B on the In atom adjacent to the EuS. Despite the small
induced magnetic moment, the DOS at the interface is spin-polarized, as shown
in Fig. 11. About 0.25 eV below the Fermi level the DOS is dominated by the
majority spin channel, whereas the minority spin channel dominates about 0.2
eV above the Fermi level. This spin-polarization is highly localized to the
first few InAs layers near the interface - the same region where the dispersed
interface states exist. Based on our results, it is unlikely that the
proximity-induced magnetic moment in InAs is sufficiently strong to account
for large effects such as the recent observations of magnetic hysteresis and
of zero-bias conductance peaks at zero magnetic field in EuS/InAs/Al hybrid
quantum devices [43, 44]. However, weakly spin-polarized in-plane transport
may be possible via the interface state.

##  IV  Conclusion

In summary, we have conducted a first-principles investigation of the effect
of the bonding configuration on the electronic properties of the EuS/InAs
(001) interface, using DFT with a machine-learned Hubbard U correction. The
DFT+U(BO) method enables unprecedented simulations of large, well-converged
interface models. For all interface configurations studied here, the valence
band of the EuS lies below the valence band of the InAs. In addition, a
dispersed interface state appears at the top of the InAs valence band, which
is contributed primarily by the InAs layer closest to the interface. These
findings are in agreement with ARPES experiments reported in Ref. [42]. The
different interface configurations differ in the band alignment between the
EuS and InAs and in the dispersion of the interface state. Both are correlated
with the extent of charge transfer from EuS to InAs at the interface. The
greater the charge transfer the lower the EuS VBM lies below the InAs VBM and
the more dispersed the interface state becomes. For all interface
configurations studied here, the induced magnetic moment in the InAs is found
to be modest and short-ranged, which is also in agreement with Ref. [42]. This
suggests that EuS/InAs coherent interfaces are not ideally suited for
applications which seek to leverage equilibrium magnetic moments induced in a
III-V semiconductor by proximity. This work demonstrates that first principles
simulations may help interpret experimental findings and elucidate the
electronic properties as they relate to the interface structure.

##  V  Acknowledgments

We thank Peter Krogstrup and Yu Liu from University of Copenhagen and
Microsoft Quantum Materials Lab Copenhagen and Sara Martí-Sanchez from the
Catalan Institute of Nanoscience and Nanotechnology for helpful discussions of
the details of their HAADF-STEM experiments and DFT calculations. We thank
Sergey Frolov from the University of Pittsburgh, Paul Crowell from the
University of Minnesota, and Chris Palmstrøm from the University of California
Santa Barbara for helpful discussions. This research was funded by the
Department of Energy through grant DE-SC0019274. This research used resources
of the National Energy Research Scientific Computing Center (NERSC), a DOE
Office of Science User Facility supported by the Office of Science of the U.S.
Department of Energy under contract no. DE-AC02-05CH11231.

## References

  * [1] A. Y. Kitaev.  Unpaired majorana fermions in quantum wires.  Phys.-Uspekhi, 44(10S):131, 2001.
  * [2] A. Y. Kitaev.  Fault-tolerant quantum computation by anyons.  Ann. Phys. (N.Y.), 303(1):2–30, 2003.
  * [3] S. D. Sarma, M. Freedman, and C. Nayak.  Majorana zero modes and topological quantum computation.  NPJ Quantum Inf., 1(1):1–13, 2015.
  * [4] R. M. Lutchyn, E. P. A. M. Bakkers, L. P. Kouwenhoven, P. Krogstrup, C. M. Marcus, and Y. Oreg.  Majorana zero modes in superconductor–semiconductor heterostructures.  Nat. Rev. Mater., 3(5):52–68, 2018.
  * [5] C. Nayak, S. H. Simon, A. Stern, M. Freedman, and S. D. Sarma.  Non-abelian anyons and topological quantum computation.  Rev. Mod. Phys., 80(3):1083, 2008.
  * [6] M. Freedman, A. Kitaev, M. Larsen, and Z. H. Wang.  Topological quantum computation.  Bull. Am. Math. Soc., 40(1):31–38, 2003.
  * [7] A. Das, Y. Ronen, Y. Most, Y. Oreg, M. Heiblum, and H. Shtrikman.  Zero-bias peaks and splitting in an Al–InAs nanowire topological superconductor as a signature of Majorana fermions.  Nat. Phys., 8(12):887–895, 2012.
  * [8] J. S. Lee, B. Shojaei, M. Pendharkar, A. P. McFadden, Y. Kim, H. J. Suominen, M. Kjaergaard, F. Nichele, H. Zhang, and C. M. Marcus.  Transport Studies of Epi-Al/InAs Two-Dimensional Electron Gas Systems for Required Building-Blocks in Topological Superconductor Networks.  Nano Lett., 19(5):3083–3090, 2019.
  * [9] L. Fu and C. L. Kane.  Superconducting proximity effect and majorana fermions at the surface of a topological insulator.  Phys. Rev. Lett., 100(9):096407, 2008.
  * [10] J. Alicea.  Majorana fermions in a tunable semiconductor device.  Phys. Rev. B, 81(12):125318, 2010.
  * [11] Y. Oreg, G. Refael, and F. Von Oppen.  Helical liquids and majorana bound states in quantum wires.  Phys. Rev. Lett., 105(17):177002, 2010.
  * [12] W. Mayer, J. Yuan, K. S. Wickramasinghe, T. Nguyen, M. C. Dartiailh, and J. Shabani.  Superconducting proximity effect in epitaxial Al-InAs heterostructures.  Appl. Phys. Lett., 114(10):103104, 2019.
  * [13] R. M. Lutchyn, J. D. Sau, and S. D. Sarma.  Majorana fermions and a topological phase transition in semiconductor-superconductor heterostructures.  Phys. Rev. Lett., 105(7):077001, 2010.
  * [14] H. Y. Günel, I. E. Batov, H. Hardtdegen, K. Sladek, A. Winden, K. Weis, G. Panaitov, D. Grützmacher, and T. Schäpers.  Supercurrent in Nb/inas-nanowire/Nb josephson junctions.  J. Appl. Phys., 112(3):034316, 2012.
  * [15] N. A. Güsken, T. Rieger, P. Zellekens, B. Bennemann, E. Neumann, M. I. Lepsa, T. Schäpers, and D. Grützmacher.  MBE growth of Al/InAs and Nb/InAs superconducting hybrid nanowire structures.  Nanoscale, 9(43):16735–16741, 2017.
  * [16] J. Paajaste, M. Amado, S. Roddaro, F. S. Bergeret, D. Ercolani, L. Sorba, and F. Giazotto.  Pb/InAs nanowire Josephson junction with high critical current and magnetic flux focusing.  Nano Lett., 15(3):1803–1808, 2015.
  * [17] M. T. Deng, C. L. Yu, G. Y. Huang, M. Larsson, P. Caroff, and H. Q. Xu.  Anomalous zero-bias conductance peak in a Nb–InSb nanowire–Nb hybrid device.  Nano Lett., 12(12):6414–6419, 2012.
  * [18] W. Y. He, B. T. Zhou, J. J. He, N. F. Q. Yuan, T. Zhang, and K. T. Law.  Magnetic field driven nodal topological superconductivity in monolayer transition metal dichalcogenides.  Commun. Phys., 1(1):1–7, 2018.
  * [19] M. Leijnse and K. Flensberg.  Introduction to topological superconductivity and majorana fermions.  Semicond. Sci. Technol., 27(12):124003, 2012.
  * [20] Z. D. Yang, B. Heischmidt, S. Gazibegovic, G. Badawy, D. Car, P. A. Crowell, E. P. A. M. Bakkers, and V. S. Pribiag.  Spin Transport in Ferromagnet-InSb Nanowire Quantum Devices.  Nano Lett., 20(5):3232–3239, 2020.  PMID: 32338518.
  * [21] Y. Jiang, E. J. De Jong, V. van de Sande, S. Gazibegovic, G. Badawy, E. P. A. M. Bakkers, and S. M. Frolov.  Hysteretic magnetoresistance in nanowire devices due to stray fields induced by micromagnets.  Nanotechnology, 32(9):095001, dec 2020.
  * [22] J. D. Sau, S. Tewari, R. M. Lutchyn, T. D. Stanescu, and S. D. Sarma.  Non-abelian quantum order in spin-orbit-coupled semiconductors: Search for topological majorana particles in solid-state systems.  Phys. Rev. B, 82(21):214509, 2010.
  * [23] C. Adelmann, X. Lou, J. Strand, C. J. Palmstrøm, and P. A. Crowell.  Spin injection and relaxation in ferromagnet-semiconductor heterostructures.  Phys. Rev. B, 71(12):121301, 2005.
  * [24] X. H. Lou, C. Adelmann, S. A. Crooker, E. S. Garlid, J. J. Zhang, K. S. M. Reddy, S. D. Flexner, C. J. Palmstrøm, and P. A. Crowell.  Electrical detection of spin transport in lateral ferromagnet–semiconductor devices.  Nat. Phys., 3(3):197–202, 2007.
  * [25] H. J. Zhu, M. Ramsteiner, H. Kostial, M. Wassermeier, H. P. Schönherr, and K. H. Ploog.  Room-temperature spin injection from Fe into GaAs.  Phys. Rev. Lett., 87(1):016601, 2001.
  * [26] S. F. Alvarado and P. Renaud.  Observation of spin-polarized-electron tunneling from a ferromagnet into GaAs.  Phys. Rev. Lett., 68(9):1387, 1992.
  * [27] A. T. Hanbicki, B. T. Jonker, G. Itskos, G. Kioseoglou, and A. Petrou.  Efficient electrical spin injection from a magnetic metal/tunnel barrier contact into a semiconductor.  Appl. Phys. Lett., 80(7):1240–1242, 2002.
  * [28] A. Fert and H. Jaffres.  Conditions for efficient spin injection from a ferromagnetic metal into a semiconductor.  Phys. Rev. B, 64(18):184420, 2001.
  * [29] J. D. Sau, R. M. Lutchyn, S. Tewari, and S. D. Sarma.  Generic new platform for topological quantum computation using semiconductor heterostructures.  Phys. Rev. Lett., 104(4):040502, 2010.
  * [30] P. Wei, S. Lee, F. Lemaitre, L. Pinel, D. Cutaia, W. Cha, F. Katmis, Y. Zhu, D. Heiman, and J. Hone.  Strong interfacial exchange field in the graphene/EuS heterostructure.  Nat. Mater., 15(7):711–716, 2016.
  * [31] F. Katmis, V. Lauter, F. S. Nogueira, B. A. Assaf, M. E. Jamer, P. Wei, B. Satpati, J. W. Freeland, I. Eremin, and D. Heiman.  A high-temperature ferromagnetic topological insulating phase by proximity coupling.  Nature, 533(7604):513–516, 2016.
  * [32] H. Q. Chen, B. Li, and J. L. Yang.  Proximity effect induced spin injection in phosphorene on magnetic insulator.  ACS Appl. Mater. Interfaces, 9(44):38999–39010, 2017.
  * [33] X. Liang, L. J. Deng, F. Huang, T. T. Tang, C. T. Wang, Y. P. Zhu, J. Qin, Y. Zhang, B. Peng, and L. Bi.  The magnetic proximity effect and electrical field tunable valley degeneracy in MoS2/EuS van der Waals heterojunctions.  Nanoscale, 9(27):9502–9509, 2017.
  * [34] J. S. Moodera, X. Hao, G. A. Gibson, and R. Meservey.  Electron-Spin Polarization in Tunnel Junctions in Zero Applied Field with Ferromagnetic EuS Barriers.  Phys. Rev. Lett., 61:637–640, Aug 1988.
  * [35] B. Li, N. Roschewsky, B. A. Assaf, M. Eich, M. Epstein-Martin, D. Heiman, M. Münzenberg, and J. S. Moodera.  Superconducting spin switch with infinite magnetoresistance induced by an internal exchange field.  Phys. Rev. Lett., 110:097001, Feb 2013.
  * [36] Y. M. Xiong, S. Stadler, P. W. Adams, and G. Catelani.  Spin-resolved tunneling studies of the exchange field in $\mathrm{EuS}/\mathrm{Al}$ bilayers.  Phys. Rev. Lett., 106:247001, Jun 2011.
  * [37] S. Diesch, P. Machon, M. Wolz, C. Sürgers, D. Beckmann, W. Belzig, and E. Scheer.  Creation of equal-spin triplet superconductivity at the Al/EuS interface.  Nat. Commun., 9(1):1–8, 2018.
  * [38] J. M. Gomez-Perez, X. P. Zhang, F. Calavalle, M. Ilyn, C. González-Orellana, M. Gobbi, C. Rogero, A. Chuvilin, V. N. Golovach, L. E. Hueso, F. S. Bergeret, and F. Casanova.  Strong interfacial exchange field in a heavy metal/ferromagnetic insulator system determined by spin hall magnetoresistance.  Nano Lett., 20(9):6815–6823, 2020.  PMID: 32786952.
  * [39] B. A. Assaf, F. Katmis, P. Wei, C. Z. Chang, B. Satpati, J. S. Moodera, and D. Heiman.  Inducing magnetism onto the surface of a topological crystalline insulator.  Phys. Rev. B, 91:195310, May 2015.
  * [40] Q. I. Yang and A. Kapitulnik.  Two-stage proximity-induced gap opening in topological-insulator–insulating-ferromagnet ${({\mathrm{Bi}}_{x}{\mathrm{Sb}}_{1-x})}_{2}{\mathrm{te}}_{3}$–$\mathrm{EuS}$ bilayers.  Phys. Rev. B, 98:081403, Aug 2018.
  * [41] S. Y. Yang, C. Z. Wu, and N. Marom.  Topological properties of SnSe/EuS and SnTe/CaTe interfaces.  Phys. Rev. Materials, 4:034203, Mar 2020.
  * [42] Y. Liu, A. Luchini, S. Martí-Sánchez, C. Koch, S. Schuwalow, S. A. Khan, T. Stankevic, S. Francoual, J. R. L. Mardegan, and J. A. Krieger.  Coherent Epitaxial Semiconductor–Ferromagnetic Insulator InAs/EuS Interfaces: Band Alignment and Magnetic Structure.  ACS Appl. Mater. Interfaces, 12(7):8780–8787, 2020.
  * [43] Y. Liu, S. Vaitiekėnas, S. Martí-Sánchez, C. Koch, S. Hart, Z. Cui, T. Kanne, S. A. Khan, R. Tanta, S. Upadhyay, M. E. Cachaza, C. M. Marcus, J. Arbiol, K. A. Moler, and P. Krogstrup.  Semiconductor–ferromagnetic insulator–superconductor nanowires: Stray field and exchange field.  Nano Lett., 20(1):456–462, 2020.  PMID: 31769993.
  * [44] S. Vaitiekėnas, Y. Liu, P. Krogstrup, and C. M. Marcus.  Zero-bias peaks at zero magnetic field in ferromagnetic hybrid nanowires.  Nat. Phys., 17(1):43–47, 2021.
  * [45] J. P. Perdew, K. Burke, and M. Ernzerhof.  Generalized gradient approximation made simple.  Phys. Rev. Lett., 77(18):3865, 1996.
  * [46] M. T. Yu, S. Y. Yang, C. Z. Wu, and N. Marom.  Machine learning the Hubbard U parameter in DFT+ U using Bayesian optimization.  NPJ Comput. Mater., 6(1):1–6, 2020.
  * [47] D. B. Ghosh, M. De, and S. K. De.  Electronic structure and magneto-optical properties of magnetic semiconductors: Europium monochalcogenides.  Phys. Rev. B, 70(11):115211, 2004.
  * [48] P. Wachter.  The optical electrical and magnetic properties of the europium chalcogenides and the rare earth pnictides.  Crit. Rev. Solid State Mater. Sci., 3(2):189–241, 1972.
  * [49] S. Massidda, A. Continenza, A. J. Freeman, T. M. De Pascale, F. Meloni, and M. Serra.  Structural and electronic properties of narrow-band-gap semiconductors: InP, InAs, and InSb.  Phys. Rev. B, 41(17):12079, 1990.
  * [50] Y. S. Kim, K. Hummer, and G. Kresse.  Accurate band structures and effective masses for InP, InAs, and InSb using hybrid functionals.  Phys. Rev. B, 80(3):035203, 2009.
  * [51] M. Schlipf, M. Betzinger, M. Ležaić, C. Friedrich, and S. Blügel.  Structural, electronic, and magnetic properties of the europium chalcogenides: A hybrid-functional DFT study.  Phys. Rev. B, 88(9):094433, 2013.
  * [52] V. I. Anisimov, J. Zaanen, and O. K. Andersen.  Band theory and Mott insulators: Hubbard U instead of Stoner I.  Phys. Rev. B, 44(3):943, 1991.
  * [53] S. L. Dudarev, G. A. Botton, S. Y. Savrasov, C. J. Humphreys, and A. P. Sutton.  Electron-energy-loss spectra and the structural stability of nickel oxide: An LSDA+ U study.  Phys. Rev. B, 57(3):1505, 1998.
  * [54] S. Y. Yang, N. Schröter, S. Schuwalow, M. Rajpalk, K. Ohtani, P. KrogstrupGeorg, W. Winkler, J. Gukelberger, D. Gresch, and G. Aeppli.  Electronic structure of InAs and InSb surfaces: density functional theory and angle-resolved photoemission spectroscopy.  arXiv preprint arXiv:2012.14935, 2020.
  * [55] S. Y. Yang, D. Dardzinski, A. Hwang, D. I. Pikulin, G. W. Winkler, and N. Marom.  First principles feasibility assessment of a topological insulator at the InAs/GaSb interface.  arXiv preprint arXiv:2101.07873, 2021.
  * [56] R. T. Tung.  The physics and chemistry of the schottky barrier height.  Appl. Phys. Rev., 1(1):011304, 2014.
  * [57] G. Kresse and J. Hafner.  Ab initio molecular dynamics for liquid metals.  Phys. Rev. B, 47(1):558, 1993.
  * [58] G. Kresse and D. Joubert.  From ultrasoft pseudopotentials to the projector augmented-wave method.  Phys. Rev. B, 59(3):1758, 1999.
  * [59] P. E. Blöchl.  Projector augmented-wave method.  Phys. Rev. B, 50(24):17953, 1994.
  * [60] S. Steiner, S. Khmelevskyi, M. Marsmann, and G. Kresse.  Calculation of the magnetic anisotropy with projected-augmented-wave methodology and the case study of disordered $Fe_{1-x}Co_{x}$ alloys.  Phys. Rev. B, 93(22):224425, 2016.
  * [61] J. Neugebauer and M. Scheffler.  Adsorbate-substrate and adsorbate-adsorbate interactions of Na and K adlayers on Al (111).  Phys. Rev. B, 46(24):16067, 1992.
  * [62] A. Tkatchenko and M. Scheffler.  Accurate molecular van der waals interactions from ground-state electron density and free-atom reference data.  Phys. Rev. Lett., 102(7):073005, 2009.