# Deep Multimodal Feature Encoding for Video Ordering

Vivek Sharma1,3, Makarand Tapaswi2,4 and Rainer Stiefelhagen1
1Karlsruhe Institute of Technology, 2University of Toronto, 3Massachusetts
Institute of Technology, 4Inria
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

True understanding of videos comes from a joint analysis of all its
modalities: the video frames, the audio track, and any accompanying text such
as closed captions. We present a way to learn a compact multimodal feature
representation that encodes all these modalities. Our model parameters are
learned through a proxy task of inferring the temporal ordering of a set of
unordered videos in a timeline. To this end, we create a new multimodal
dataset for temporal ordering that consists of approximately 30K scenes (2-6
clips per scene) based on the “Large Scale Movie Description Challenge”. We
analyze and evaluate the individual and joint modalities on three challenging
tasks: (i) inferring the temporal ordering of a set of videos; and (ii) action
recognition. We demonstrate empirically that multimodal representations are
indeed complementary, and can play a key role in improving the performance of
many applications. The datasets and code are available at
https://github.com/vivoutlaw/tcbp.

##  1 Introduction

As humans, watching a movie entails looking at the video frames, listening to
the audio track (both music and speech), and optionally, reading the closed
captions to obtain a better understanding. All these modalities contribute by
bringing together different pieces of information. Similarly, a joint analysis
of these modalities is crucial for many practical applications, such as video
retrieval, surveillance video analysis, automatic description or search within
home movies and online uploaded video clips, describing video content for
visually impaired, and more. Inspired by this, multimodal feature
representation learning [36, 14, 64, 65, 20, 22, 41, 66, 28] has attracted
quite a lot of attention. These multimodal sources often consist of
complementary information for objects, scenes, events, dialogs, and activities
of interest. Therefore, learning a joint representation from multiple sources
is useful for robust inference.

Figure 1: Given an unordered collection of video clips, we infer their correct
temporal ordering by utilizing our compact multimodal feature encoding that
exploits high-level semantic concepts such as objects, scenes, and sounds in
each clip.

Joint representations of images and text are quite popular in captioning and
retrieval and are commonly achieved through a combination of convolutional
neural networks (CNN) [19] and recurrent neural networks (RNN) [24]. Typical
fusion strategies to bring the two domains together include element-wise
product, sum, avg, max pooling or concatenation, or self-attention. Recently,
_bilinear pooling_ [66, 14] has gained interest as it efficiently captures
pairwise interactions between two representations. Typical implementations use
either the Tensor Sketch projection algorithm [43, 15] or matrix factorization
[66]. Beyond images and text, a few other works have investigated the
combination of modalities [36, 65, 20, 22]: video and audio, or video and
text.

We are interested in capturing such pairwise interactions between modalities
for video clips, and propose a new deep multimodal feature encoding inspired
by [56, 15, 43]. The design of the multimodal feature aims to aggregate audio,
visual, video, and text sources into a compact feature encoding. To this end,
we propose Temporal Compact Bilinear Pooling (TCBP) as an extension of the
Tensor Sketch projection algorithm [43] to incorporate a temporal dimension.
TCBP is a temporal encoding of several segments of multimodal features into a
compact representation, backed by deep pre-trained models. Specifically, TCBP
captures the interaction of each modality with one-another at all spatial
locations over a temporal context (typically 2-5 seconds).

We are also motivated by self-supervised visual representation learning
methods [13, 32, 44, 49, 62], where millions of positive/negative training
image pairs are generated from video constraints to learn discriminative
features. Common proxy tasks include predicting the next frame, or picking the
“odd” one among a set of frames. Generating such training data at the frame-
level is made possible due to the stationary context and objects in the short-
term temporal neighborhood. However, we are unaware of any prior work that
obtains strong supervision of constraints for obtaining clip-level training
data. In practice, these self-supervised constraints do not scale to learn
clip-level representations often due to the lack of millions of annotated
videos, and also owing to limited computational resources. In this work we
present a simple yet surprisingly powerful proxy task of temporal ordering of
several clips to learn a self-supervised clip representation. We utilize the
compact temporal multimodal feature representation for the problem of
inferring the temporal ordering of video-clips.

We highlight our key contributions below:
1) We propose a new self-supervised learning paradigm that produces a compact
multimodal video clip representation combining audio, visual, text and video
modalities. In particular, we extend the Tensor Sketch projection algorithm to
incorporate the temporal dimension, and name it Temporal Compact Bilinear
Pooling (TCBP).
2) We introduce temporal ordering of video clips as a novel proxy task for
learning clip representations. To this end, we create a new dataset of almost
30K annotated ordered scenes from movies each consisting of 2-6 clips per
scene.
3) We present an in-depth analysis of individual and joint modalities and
their effects on performance through the temporal ordering. We also
demonstrate that TCBP can lead to performance gains in action recognition by
effectively combining temporal cues.

##  2 Related Work

Self-supervised representation learning is a line of research growing in
popularity in recent years. This paradigm includes obtaining supervision from
within the structure of the data, and thus removes the need for an often
costly labeling effort. A strong cue to learn good image representations is
the spatial consistency of images [8]. Another approach uses a host of spatial
and color transforms to train a model to recognize instances of transformed
images [10]. In particular, face image representations are also fine-tuned for
specific applications such as video face clustering by considering
positive/negative pairs within video sub-structures [55] or mining labels
using a distance matrix [49].

Videos are often used to learn good image representations. Object tracking in
a video can be used to generate several instances of positive/negative pairs
[62]. At the frame level, using temporal consistency within neighboring frames
[32]; or finding an odd frame in a collection of many frames [13] are examples
of methods used to automatically create positive/negative training data. Note
that many of these methods train CNN models from scratch to learn an effective
representation.

In contrast to these, our work differs substantially in scope and technical
approach. We are interested in learning multimodal video clip representations
(and not single images) and propose to use temporal ordering of a variable set
of clips to learn our joint representation.

Time as a supervisory signal. The idea of using temporal continuity or
ordering as a signal for supervision is quite popular. For example, Misra  _et
al_. [32] shuffle a sequence of 3 frames from a video to generate
positive/negative samples and train a model to predict whether a given
sequence is ordered correctly. However, the use of short 3-frame sequences
only allows learning image representations, while we wish to learn multimodal
clip representations.

Similarly, Wei  _et al_. [63] predict whether a video flows forwards or
backwards given a stack of 10 optical flow frames from a short clip. However,
this formulation prevents the method from learning how to encode object
information, and requires computationally intensive optical flow maps. Most
importantly, only a binary forward/backward ordering is learned, and other
shuffled permutations are not considered. In our task of temporal ordering, we
_order video clips_ and not frames, thus allowing to learn clip-level
representations. We order up to 6 clips each with a duration ranging from 2-5
seconds.

It is also worth noting that temporal ordering has been used for other
applications such as predicting what happens next in egocentric videos [70],
using the steadiness of visual change in videos to penalize higher order
differences [21], or to recognize complex, long-term activities [26].

Similar in spirit to our temporal ordering of video clips, arranging photos by
time has also received some attention. In particular, automating the process
of creating temporally ordered photo albums from an unordered image collection
[46]; sorting photo collections spanning many years [30, 48]; or ordering a
set of photos taken from uncalibrated cameras [4, 7, 33] are all related
works. In contrast, we work with an unordered collection of video clips.

Audio-visual learning. Starting with the work by Yuhas  _et al_. [67] which
employs neural networks for audio-visual speech recognition, audio-visual
learning has come a long way. Recently, there has been interest in predicting
the audio modality given a visual input [38], or predicting the ambient sound
that an image conveys [39]. There is also work on learning representations in
a joint space. For example, Aytar  _et al_. [3] trained a student-teacher
network to transfer the knowledge between the outputs of visual and audio
networks. Subsequently, these ideas were extended to learning inputs either
with image+text and image+sound pairs [68]. Audio-visual representation
learning has seen applications in several tasks such as event classification
[40], audio-visual localization [40, 2], biometric matching [35], sound
localization [37, 2], person identification [34], action recognition [37],
on/off-screen audio separation [12, 37], and video captioning/description [64,
36, 20].

##  3 Learning a Clip Representation

Figure 2: Deep Multimodal Feature Encoding. Illustration of the multimodal
feature encoding applied for task of temporal ordering problem. See Sec. 3 for
a detailed explanation of the feature learning scheme shown.

We present our model to learn a multimodal video clip representation that
encodes images, video, audio, and text jointly. Our goal is to create a
representation that is compact and efficiently encodes all modalities into a
single descriptor. We train this model in a self-supervised paradigm, with
supervision from the proxy task of video clips ordering .

###  3.1 Deep Multimodal Feature Encoding

For a video clip with $N$ modalities, we compute one output feature map per
modality produced by a pre-trained (convolutional) neural network. We denote
these feature maps as matrices $\\{S_{1},S_{2},\ldots,S_{N}\\}$, with size
$S_{i}\in\mathbb{R}^{c_{i}\times t}$, $c_{i}$ denotes the number of channels
for the $i$th modality, and $t$ is the temporal length or number of segments
in which the video is divided for feature extraction. Note that as our clips
can some times be quite long (about 5 seconds) we divide them into segments of
a fixed duration to compute features.

We aggregate all feature maps by concatenating them across channels, to
compute
$x=S_{1}\operatorname{+\kern-3.99994pt+}S_{2}\operatorname{+\kern-3.99994pt+}\ldots\operatorname{+\kern-3.99994pt+}S_{N}$,
where $\operatorname{+\kern-3.99994pt+}$ represents the concatenation operator
and obtain $x\in\mathbb{R}^{c\times t}$, where $c=\sum_{i=1}^{N}c_{i}$.
Concatenation along the channels allows us to combine a variable number of
modalities. Additionally it ensures that information from all modalities is
preserved at this stage of feature encoding. The aggregated feature $x$ is
then passed through some encoding method $E:x\rightarrow v$, to compute the
multimodal encoded feature $v\in\mathbb{R}^{d}$, where $d$ is the
dimensionality of the multimodal embedding space.

###  3.2 Temporal Compact Bilinear Pooling

In this work, we utilize bilinear models (or bilinear pooling) for learning
multimodal representations. We choose to use bilinear models as they possess
the ability to capture interactions between each element of the feature with
one-another (much like a polynomial kernel), thus leading to a powerful
feature representation. Additionally, they have been shown to work well for
related tasks of image classification [15, 27, 5], video classification [6]
and visual-question answering [14]. We will first introduce bilinear models,
and then propose our extension for temporal video data.

Preliminaries. A bilinear pooling function is an operator on the outer product
of a vector $x\in\mathbb{R}^{c}$ to obtain $v\in\mathbb{R}^{d}$ such that:

$v=W[x\otimes x^{T}]\,,$ (1)

where $\otimes$ denotes the outer product, $[\cdot]$ linearizes the matrix,
and $W\in\mathbb{R}^{c^{2}\times d}$ are model parameters to be learned.
However, as feature dimensions are often high (e.g., $c=1024$), the number of
parameters for $W$ can easily be in the range of billions (assuming $d=1024$
as well), which makes learning challenging.

To alleviate this problem, Gao  _et al_. [15] show how the high-dimensional
second-order function can be approximated by a low dimensional projection
function using the Tensor Sketch algorithm [43]. This removes the need for
computation of the expensive outer product and obtains the final vector
directly.

Tensor Sketch (TS) [43] also called the Count Sketch algorithm, projects a
vector $x\in\mathbb{R}^{c}$ to $v\in\mathbb{R}^{d}$. Two sets of auxiliary
vectors $h$ and $s$ are randomly initialized from a uniform distribution (and
held fixed thereafter) to perform this projection $u_{1}=\psi(x,h_{1},s_{1})$
and $u_{2}=\psi(x,h_{2},s_{2})$. The sign vector $s\in\\{-1,1\\}^{c}$
indicates whether the element in $x$ will be added or subtracted from the
final value at a random location determined by $h\in\\{1,\ldots,d\\}^{c}$. In
particular, for every element $x[i]$, its destination in $v$ is given by
$j=h[i]$, and the value in $v$ is accumulated as $u[j]=u[j]+s[i]\cdot x[i]$.
Finally, $u_{1}$ and $u_{2}$ (corresponding to $h_{1},s_{1}$ and
$h_{2},s_{2}$) are convolved with each other to compute $v$ by performing
element-wise multiplication in the Fourier domain.

Compact Bilinear Pooling (CBP) [15]. When working with images, a CNN feature
map $x$ often preserves the encoding of spatial locations to yield
$x\in\mathbb{R}^{c\times s}$, where $c$ is number of channels (as in the above
discussion) and $s$ refers to spatial locations. To work with such features,
[15] first performs TS projection on the vector at each spatial element $s$,
followed by sum pooling. In summary,

$x\in\,\mathbb{R}^{c\times s}\,\xrightarrow{\text{TS
projection}}\,\mathbb{R}^{d\times s}\,\xrightarrow{\text{sum pool over
}s}\,v\in\mathbb{R}^{d}\,.$ (2)

We will refer to this process of projecting individual vectors followed by sum
pooling as CBP. For more details please refer to [15].

Temporal Compact Bilinear Pooling (TCBP). We now propose an extension to the
Tensor Sketch projection algorithm to incorporate a temporal dimension.
Recall, $x\in\mathbb{R}^{c\times t}$ is the aggregated feature after the
concatentation, which we will encode with TCBP. The key difference between CBP
and TCBP is the process in which the TS algorithm’s projection matrices $h$
and $s$ are created. In particular, we initialize $h\in\\{1,\ldots,d\\}^{c}$
as a $c$ dimensional vector, while $s\in\\{-1,1\\}^{c\times t}$ as a matrix.
Having $h$ to be independent of time ensures that a feature index $i$ in
$x,x[i,t]$ is always encoded to the same destination $j=h[i]$, albeit with a
different sign.

Similar to the TS paragraph from above we can write $j=h[i]$ (and is
independent of $t$), while, $u[j]=u[j]+\sum_{t}s[i,t]\cdot x[i,t]$. Our TCBP
approach is summarized in Algorithm 1.

Note that the above procedure is different from applying CBP directly on a
flattened vector $x\in\mathbb{R}^{ct}$, as this would require creation of $h$
which is also initialized to be $h\in\\{1,\ldots,d\\}^{ct}$. We compare CBP
and TCBP with regards to number of parameters and computational efficiency in
Table 1.

We wish to assert that the use of CBP as well as TCBP for multimodal feature
encoding is novel. We learn a projection function to model multimodal data
that includes text, images, audio and video.

Algorithm 1 Temporal Compact Bilinear Pooling

Input: Aggregated feature maps of multimodal data $x\in\mathbb{R}^{c\times
t}$, where $c$ and $t$ are the number of channels and temporal segments.

Output: Multimodal encoded feature map $v\in\mathbb{R}^{d}$, where $d$ is the
encoded feature dimension.

Procedure:

1. Initialize random vector $h_{k}\in\\{1,\ldots,d\\}^{c}$, and $s_{k}\in\\{-1,1\\}^{c\times t}$ from a uniform distribution, $k=1,2$.

2. Compute the count sketch projection function $u=\Psi(x,h,s)=\\{(Px)_{1},\ldots(Px)_{d}\\}$, where $(Px)_{j}=\sum_{i:h[i]=j}\sum_{t}s[i,t]\cdot x[i,t]$.

3. Finally, compute the output encoded vector as $v=FFT^{-1}(FFT(\Psi(x,h_{1},s_{1}))\circ FFT(\Psi(x,h_{2},s_{2})))$, where $\circ$ denotes element-wise multiplication operator.

| CBP | TCBP
---|---|---
Input dimensions | $\mathbb{R}^{c}$ | $\mathbb{R}^{c\times t}$
Output dimensions | $d$ | $d$
Parameters ($h,s$) | $2\cdot 2c$ | $2\cdot(c+ct)$
Computation | $O(c+d\log d)$ | $O(ct+d\log d)$

Table 1: Dimension, memory, and computation comparison of bilinear pooling
with tensor sketch (CBP), and the proposed temporal compact bilinear pooling
(TCBP) algorithm. Parameters $c,t,d$ denote the number of channels, temporal
segments, and the projected dimension.

###  3.3 Learning via Temporal Ordering

We wish to learn a multimodal embedding for video clips using temporal
ordering as a proxy task. For example, consider a scene from movie that
consists of $M$ video clips, $(V_{1},\ldots,V_{M})$. These clips are ordered
by the video creators to tell a story in the most natural way, and often
encode general principles of causality (e.g., pushing a car gets it rolling)
or temporal progression (e.g., enter the house before removing coat).

Training. We sample an ordered pair of clips $(V_{i},V_{j})$ randomly from the
entire scene and train our model parameters to learn that $V_{j}$ should apear
_after_ $V_{i}$, denoted as $V_{i}\succ V_{j}$. To ascertain this order rule,
we are inspired by the order violation loss [59] used to encode visual
semantic hierarchy. In our case, we encode the loss between a pair of clips
as:

$L(V_{i},V_{j})=\|\max\left(0,\phi_{t}(V_{i})-\phi_{t}(V_{j})\right)\|^{2}\,,$
(3)

where $\phi_{t}(V)\in\mathbb{R}_{+}$ is the clip representation used for
temporal ordering, and the loss expects the feature $\phi_{t}(V_{i})$ to
appear “before” (below and to the left of in a 2D space) $\phi_{t}(V_{j})$.
All parameters are updated by back-propagating this loss through the network.
Fig. 2 illustrates this learning procedure for a pair of video clips.

Inference. We are also interested in understanding how well our model is able
to order a set of $M$ clips. To do this, we first encode all $M$ clips using
the temporal order feature encoding $\phi_{t}(V)$, and compute the ordering
loss between all $M\cdot(M-1)$ pairs of clips in the scene. Note that, there
are $M!$ possible sequences in which $M$ unordered clips can be sorted. We use
a simple brute-force approach and pick the sequence that results in the
smallest overall pairwise ordering loss. The final ordered list of clips is
said to be correct, iff it fully matches the ground-truth ordered list. While
this approach does not scale well, it is a feasible and quick solution when
the maximum number of clips in a scene is limited to $M_{\max}=6$. We will
explore other sorting and ranking techniques such as [11] in the future.

###  3.4 Implementation details

We present details of our approach for representing individual clips, their
modalities, and other feature extraction procedures below.

Sampling temporal segments from a clip. We observe that most clips in our
dataset have 4 to 6 temporal segments (see Table 2), with a maximum of 11. As
our projection function depends on the number of temporal segments (recall
that $s$ is randomly initialized to be $\\{-1,1\\}^{c\times t}$), we choose to
use $t=3$ segments to represent all clips. We consider three sampling
strategies to pick a consecutive sequence of three temporal segments ($t=3$)
from each clip. (i) Random Sampling involves randomly choosing three
consecutive segments from a clip; (ii) Constrained First (C_First) picks the
first three segments; and (iii) Constrained Last (C_Last) picks the last three
segments in the clip. We observe that C_Last tends to achieve slightly higher
performance and choose this as our default sampling strategy, unless otherwise
stated.

Base representations from pre-trained models. Given a video clip, we first
break it into non-overlapping 16-frame video segments. We extract a variety of
feature descriptors for different modalities and discuss their details below:
(1) Object features (I) in the frame are extracted using a ResNet50 pre-
trained on ImageNet dataset [18]; (2) Place/Scene related features (P) are
obtained using a ResNet50 pre-trained on Places365 dataset [69]; (3) Video
features encoding motion (R) are obtained using 3D ResNet50 [17] pre-trained
on Kinectics 400 dataset; (4) Text features (S) are computed using Glove6B
[42] pre-trained word vectors; and finally (5) Audio features (A) are
extracted using the SoundNet [3] model pre-trained on millions of Flickr
videos.

The input to the 3D ResNet50 model is a complete segment of 16 frames with
size $112\times 112\times 16$. From this, we extract avgpool features
resulting in $\mathbb{R}^{2048}$. For frame-level features (ImageNet and
Places), we first resize the frame to $224\times 224$ and extract avgpool
features resulting in $\mathbb{R}^{2048}$ per frame. We then mean pool across
16 frames within the segment to obtain a single feature map of
$\mathbb{R}^{2048}$.

To process audio, we first extract the part corresponding to the 16 visual
frames (estimated using timestamps) from the entire audio clip. This is passed
to the SoundNet model, and we extract $pool5$ features for the audio-duration
corresponding to the 16 frames. After an averaging across channels, we obtain
a feature map of $\mathbb{R}^{256}$.

The text modality consists of closed captions and subtitles. Note that we do
_not_ use manually curated video captions or any other descriptions that may
not available for most videos. We use the 300-dimensional word embeddings from
the Glove6B [42] model, and mean pool all words that correspond to the clip to
obtain a $\mathbb{R}^{300}$ feature map. As splitting words across temporal
segments is non-trivial, we replicate this same feature to all temporal
segments.

In summary, a video segment has dimensions ($c$) as follows: I, P, R are 2048,
A is 256, and S is 300. The joint concatenated features are PI: 4096, API:
4352, APIS: 4652, and APIR: 6400.

Clip representations. For both CBP and TCBP encoding methods, we use the same
network architecture. The input is a feature map $x\in\mathbb{R}^{c\times t}$
consisting of a combination of modalities discussed above. We first apply a
$1\times 1$ convolution on the channels to obtain a $\mathbb{R}^{2048\times
t}$ matrix. This is fed to the encoding method resulting in
$v_{\text{enc}}\in\mathbb{R}^{d}$, where $d=8192$. We then perform feature
normalization via a signed square-root operation followed by $\ell_{2}$
normalization. We stack a linear layer to obtain a generic clip representation
(that can be used for other tasks) as $v_{\text{clip}}=W_{1}\cdot
v_{\text{enc}}\in\mathbb{R}^{4096}$. Finally, we compute the temporal ordering
feature as $\phi_{t}(V)=|W_{2}(\text{ReLU}(v_{\text{clip}}))|$, where
$W_{2}\in\mathbb{R}^{4096\times 2048}$. Note that the absolute value
$\phi_{t}(V)\in\mathbb{R}^{2048}_{+}$ is used to compute the ordering loss.
Biases are present for all linear layers, but are omitted for brevity.

Training and optimization. Our model is implemented in PyTorch v0.4. We use
mini-batch SGD to learn the model parameters with a fixed weight decay of
$5\times 10^{-4}$, momentum of 0.9, a learning rate of $10^{-3}$ and a batch
size of 32 for training our network. Depending on the variant, we train our
model for 5000-8000 iterations. For efficiency reasons, we do not finetune
networks used to extract features.

##  4 Evaluation

In this section, we first introduce the dataset used to train temporal
ordering models. We then show the applicability of the proposed methods on
three challenging tasks: 1) inferring the temporal order for a set of
unordered video-clips; and 2) using TCBP for action recognition task.

###  4.1 Dataset

To the best of our knowledge, temporal ordering of videos is a new task. Thus,
we build a new dataset to train and evaluate our approach. We choose to use
clips from the Large Scale Movie Description Challenge (LSMDC) [45, 57, 29]
dataset as they are publicly available, contain several modalities, and are
sourced from movies that typically care about temporal order. The LSMDC
dataset consists of 202 movies with 118,081 video clips. These clips are
associated with transcribed captions from the Audio Descriptions that help
visually impaired people understand the visual content on screen, however, we
do not use these captions as our text modality as they may not be available
for all videos. Having clips corresponding to audio descriptions also implies
that most clips contain none to little dialog. Additionally, the LSMDC clips
are not contiguous, and there are often small (few second) and large (few
minutes) gaps between clips. We extend this dataset by first grouping clips
into scenes.

Scene boundary detection. In this task, our goal is to assign clips that are
temporally continuous, happen in the same place, or contain a group of
coherent events, to the same scene. We will learn to order a set of clips
within a scene.

Scene boundary detection for full movies is usually performed after shot
boundary detection. However, as the LSMDC dataset already comes with pre-
segmented clips, we redefine the problem as grouping clips into unique scenes.
We first use a dynamic programming algorithm based on [54] to efficiently
assign clips to scenes. However, scenes that are created using this method are
often quite long and contain between 15-30 clips. In order to segment such
scenes into smaller sub-sequences, we use the FINCH clustering algorithm [47]
that automatically determines the number of clusters. We manually verify these
clip clusters for consistency, and only scenes that contain between 2-6 clips
are retained.

We use the same training, validation, and test splits, as used in the LSMDC
set of captioning and retrieval tasks. In total, there are 25,269 scenes in
the training set, 1,784 scenes in the validation set, and 2,443 scenes in the
test set. The dataset statistics are given in Table 2. The top half of the
Table presents the total number of clips that appear in scenes of size 2, 3,
etc. In the bottom half we indicate the number of temporal segments (or number
of frames) for different videos, also in the range of 2-6 segments. We divide
each clip into $t$ 16-frame segments without overlap, _i.e._ a clip with $F$
frames has $t=\lceil F/16\rceil$ segments. Approximately 65% of scenes have 4
segments, and there are no clips with less than 48 frames (i.e. with t=3). At
a video framerate of 24 fps, this puts most clips in the range of 2-4 seconds.

We will make our annotations publicly available for further research into the
problem of temporal ordering.

#Clips in a scene
---
Scene size | 2 | 3 | 4 | 5 | 6 | 2-6
Training | 13455 | 6711 | 3097 | 1382 | 624 | 25269
Validation | 958 | 472 | 203 | 100 | 51 | 1784
Test | 1333 | 588 | 325 | 135 | 62 | 2443
#Temporal segments
t | 2 | 3 | 4 | 5 | 6 | 2-6
Frames | 32 | 48 | 64 | 80 | 96 | 32-96
Training | 0 | 0 | 17992 | 3102 | 2025 | 25269
Validation | 0 | 0 | 1257 | 233 | 143 | 1784
Test | 0 | 0 | 1542 | 382 | 252 | 2443

Table 2: Number of clips in a scene of variable size and number of temporal
segments in each such clip for our dataset.

| Scenes with 2-6 clips and t = 3
---|---
Split | Modality | C_First | C_Last | Random Sampling | Chance
Validation | A | 22.90 | 57.97 | 57.21 | 31.78
P | 26.40 | 76.35 | 73.93 | 31.78
I | 22.42 | 77.32 | 74.57 | 31.78
R | 39.13 | 28.08 | 26.90 | 31.78
S | 23.79 | 22.74 | 21.70 | 31.78
Validation | PI | 30.16 | 79.65 | 71.86 | 31.78
API | 24.56 | 78.70 | 73.15 | 31.78
APIS | 21.47 | 77.39 | 71.33 | 31.78
APIR | 17.04 | 77.24 | 46.92 | 31.78
Test | PI | 34.10 | 81.13 | 79.39 | 31.89
API | 39.13 | 85.06 | 84.57 | 31.89
APIS | 38.41 | 83.24 | 82.11 | 31.89
APIR | 25.01 | 82.48 | 79.38 | 31.89

Table 3: Feature importance for CBP with different sampling strategies. A, P,
I, R, and S denote the Audio, Places, ImageNet, Video and Text modality
respectively.

###  4.2 Temporal Ordering

In this section, we present results on the temporal ordering task. We first
present an ablation study for design choices of our proposed method, and then
compare our method against other baselines. We use the stringent ordering
accuracy metric that counts a sample as correct only when all video clips are
sorted in the correct sequence.

Study of individual and combined modalities. In this evaluation, we explore
the contribution of individual and combined feature representations for
temporal ordering. For the individual modalities, we do not perform channel
reduction with the $1\times 1$ convolution, as the dimensions are often
already small (e.g., $c=256$ for audio). In Table 3, we report the performance
of using individual or joint modalities. Audio (A), Places (P), and ImageNet
(I) modalities seem important for this task, while the joint PI and API
feature representations perform best. It is interesting to see that the video
features (APIR) and word embeddings (APIS) reduce performance as compared to
API. We believe that motion features captured by R are not very strong to
predict what may happen far into the future, while the sparsity of subtitles
hurts the text modality.

Study on ordering a variable number of clips in a scene. In our dataset, we
have scenes with variable number of clips ranging from 2-6. In Table 4, we
show the performance of different modalities on subsets of scenes that contain
a fixed number of clips. We observe similar performance from PI and API for
scenes with 2-4 clips. However, API is able to correctly order scenes with 5
and 6 clips, while PI fails. The failure cases show that in absence of varying
objects, context and motion, the model fails to predict the future due to
intrinsic ambiguity of stationary scenes, while audio may provide some help.

We choose API as the default modality for all further experiments.

# Clips in scene | 2 | 3 | 4 | 5 | 6 | 2-6
---|---|---|---|---|---|---
Samples | 958 | 472 | 203 | 100 | 51 | 1784
Random | 50.00 | 16.67 | 4.17 | 0.83 | 0.14 | 31.78
PI | 96.86 | 60.38 | 91.13 | 22.00 | 1.96 | 79.65
API | 95.51 | 56.14 | 87.19 | 31.00 | 31.37 | 78.70
APIS | 98.28 | 54.01 | 76.34 | 16.45 | 25.13 | 77.39
APIR | 100 | 42.58 | 88.18 | 40.00 | 0.0 | 77.24

Table 4: Impact of the number of clips per scene on ordering accuracy on the
validation split. Encoding method used is CBP with C_Last sampling strategy.

Number of segments | $t=1$ | $t=2$ | $t=3$
---|---|---|---
PI | 71.14 | 77.30 | 79.65
API | 54.88 | 77.92 | 78.70
APIS | 53.48 | 76.19 | 77.39
APIR | 71.08 | 77.19 | 77.24

Table 5: Impact of the number of temporal segments used in the encoding of
video clips on ordering accuracy on the validation split. Encoding with CBP
with C_Last sampling.

Study of variable temporal segments encoding. In this experiment, we vary the
number of temporal segments that are encoded for each clip. Recall each
segment $t$ represents a chunk of 16 frames, and clips in our dataset range
from 4 to 11 segments. We use $t=3$ for most experiments as it acts as a kind
of data augmentation when using random sampling.Table 5 confirms our
expectation that using more segments leads to better clip representations
improving performance for all modalities.

Comparison of CBP vs. TCBP. Table 6 shows the performance comparison for
variable number of temporal segments. Note that when $t=1$, CBP and TCBP are
identical. However, for $t=2,3$, TCBP shows slight improvements over CBP as it
is better able to encode longer clips, while CBP relies on sum pooling.

Split | Method | $t=1$ | $t=2$ | $t=3$
---|---|---|---|---
Validation | CBP | 54.87 | 77.92 | 78.70
TCBP | 54.87 | 78.13 | 79.43
Test | CBP | 68.65 | 84.69 | 85.06
TCBP | 68.65 | 85.26 | 86.23

Table 6: Comparison between CBP and TCBP with C_Last sampling. Modalities API
are used in this experiment.

API | Validation | Test
---|---|---
Chance | 31.78 | 31.89
ConcatT_MLP | 19.78 | 37.17
Mean_Pool | 66.59 | 69.58
NetVLAD | 68.11 | 83.79
CBP | 78.70 | 85.06
TCBP | 79.43 | 86.23

Table 7: Comparison of TCBP with other commonly used encoding strategies using
C_Last sampling. Modalities API are used here.

API | Validation | Test
---|---|---
NetVLAD | 68.11 | 83.79
NetVLAD w Neg. | 71.21 | 84.99
CBP | 78.70 | 85.06
CBP w Neg. | 84.59 | 88.95
TCBP | 79.43 | 86.23
TCBP w Neg. | 83.18 | 89.19

Table 8: Role of negative mining in CBP and TCBP using C_Last sampling.
Modalities API are used here.

Comparison between TCBP and other encoding methods. We compare TCBP with
several popular pooling methods. As before, we start with an input feature map
$x\in\mathbb{R}^{c\times t}$. ConcatT_MLP: first applies $1\times 1$
convolutions with 2048 kernels, resulting in $\mathbb{R}^{2048\times t}$. We
then simply flatten the temporal segments and concatenate them to form a
vector of size $\mathbb{R}^{2048\cdot t}$. Mean_Pool: We average pool across
the temporal segments $t$, resulting in a vector of $\mathbb{R}^{c}$. NetVLAD:
we first apply $1\times 1$ convolutions with 512 kernels, resulting in a
feature map of $\mathbb{R}^{512\times t}$. This is fed to NetVLAD [1, 16, 31]
with 32 clusters111We optimized the hyperparameters of NetVLAD by evaluating
#cluster={16,32,64}, feat_dim={256,512,1024}, and found that for #cluster=32
and feat_dim=512 resulted in the best performance. resulting in a feature
vector of $\mathbb{R}^{16384}$.

All these methods (except Mean_Pool) are followed by two fully-connected
layers yielding hidden states of $\mathbb{R}^{4096}$ and $\mathbb{R}^{2048}$.
For Mean_Pool we use $\mathbb{R}^{2048}$ and $\mathbb{R}^{1024}$ as the
dimensions are already small. The output of the last linear layer is used in
the ordering loss.

In Table 7, we see that TCBP outperforms all methods, including strong
baselines of CBP and NetVLAD. In fact, surprisingly ConcatT_MLP performs very
poorly, while a simple Mean_Pool shows limited performance. Note that we
expect small improvements for TCBP over CBP as the main difference lies in the
projection of temporal segments directly instead of sum pooling over time
after projection as in CBP. However, we believe that TCBP is a principled way
to handle temporal segments within the bilinear pooling framework.

Role of negative mining. So far, we have only considered training our model
with positive examples ($Pos$) of ordered video clips. Now, we analyze the
impact of also including negative examples ($Neg$). We adopt a popular
negative sampling strategy [51], and corrupt each positive pair by replacing
one of the clips with a randomly chosen clip from the same batch. The overall
loss is:

$\pazocal{L}=\sum_{(V_{i},V_{j})\in
Pos}L(V_{i},V_{j})+\sum_{(V_{i},V_{j}^{\prime})\in
Neg}\max(0,\alpha-L(V_{i},V_{j}^{\prime}))\,,$ (4)

where $L(\cdot,\cdot)$ is the ordering loss (see Eq. 3), $\alpha$ is a margin
(set to 0.2), and $(V_{i},V_{j}^{\prime})$ is the corrupted version of
$(V_{i},V_{j})$ without loss of generality by only replacing $V_{j}$. We use
an equal ratio of positive to negative examples to train our model.

In Table 8, we observe that adding negatives improves the performance for all
methods as compared to training the model with positive examples only. Note
that we do not need to necessarily rely on negative examples, and positive
examples are sufficient to learn a good multimodal representation as seen in
Table 7. Nevertheless, it is good to see that further improvements can be
achieved by including negatives.

###  4.3 Action Recognition

In our last experiment, we evaluate the effectiveness of TCBP for video action
recognition task.

Dataset. We evaluate TCBP on two human action and activities datasets; HMDB51
[25] and UCF101 [52]. The HMDB51 dataset consists of 51 action categories with
6,766 video clips. The UCF101 dataset consists of 101 action classes with
13,320 video clips. We report mean average accuracy over the three pre-defined
splits for both datasets.

Method. We use TLE [6] as the base architecture, and employ TCBP as the
encoding method. In particular, we use the C3D ConvNet [58] pre-trained on the
Sport-1M dataset [23]. The network operates on a stack of 16 RGB frames of
size $112\times 112$. Following the setup of Diba  _et al_. [6], we employ TSN
[61] architecture with 3 segments pushed through the C3D ConvNet. Same as [6],
we extract the convolutional feature maps from the last convolutional layers
and feed them as input to TCBP. This is followed by a classification layer
with a $K$-way softmax, where $K$ is the number of action categories. Note
that convolutional feature maps from each segment of the last layer of C3D
produce an output of size $\mathbb{R}^{512\times 2\times 7\times 7}$. Three
such segments are first aggregated via element-wise multiplication (similar to
TLE), and followed by TCBP projection with $t=2$ resulting in a vector of
$\mathbb{R}^{8192}$ dimensions. We use the same training and evaluation setup
as [6] for a fair comparison.

Results. Table 9 shows the performance of TCBP with C3D ConvNets compared
against several other methods. The goal of this experiment is to show that
TCBP can improve the performance of the base C3D [58]. It is also interesting
to see that, TCBP improves the C3D ConvNet performance over the two-stream
ConvNets [50] on both datasets. Further, TCBP outperforms Bilinear pooling,
CBP [6], and iDT+FV [60] by significant margin. We believe that the reason for
TCBP outperforming other methods is its ability to effectively encode temporal
cues in the video data, which are not considered in other methods such as [6,
50, 58, 60].

Furthermore, in practice, NetVLAD has been shown to perform worse than
Bilinear pooling (e.g., CBP) in [6, 16]. This is most likely due to CBP
capturing interactions between features at each channel, thus leading to a
strong representation.

Method | UCF101 | HMDB51
---|---|---
SpatioTemporal ConvNet [23] | 65.4 | $-$
LRCN [9] | 82.9 | $-$
Composite LSTM Model [53] | 84.3 | 44.0
iDT+FV [60] | 85.9 | 57.2
Two Stream [50] | 88.0 | 59.4
C3D [58] | 82.3 | 56.8
TLE: Bilinear+TS (CBP) [6] | 85.6 | 59.7
TLE: Bilinear [6] | 86.3 | 60.3
TLE:TCBP (ours) | 88.03 | 61.58

Table 9: C3D ConvNets. Comparison of accuracy (%) of TCBP with C3D ConvNet
against state-of-the-art methods over all three splits of UCF101 and HMDB51.

##  5 Conclusion

We proposed a unified network architecture to learn an effective and compact
multimodal video clip representation that jointly encodes images, audio,
video, and text. In particular, we proposed TCBP, an extension to Compact
Bilinear Pooling that effectively encodes multiple temporal segments of a
video clip. Our model is trained in a self-supervised learning approach where
ordering a set of video clips provides necessary supervision. We evaluated our
feature representation for temporal ordering of video clips, where, multiple
modalities proved to be helpful. We also demonstrated that TCBP is a strong
encoding method that performs well on other video tasks such as action
recognition. We strongly believe that complete understanding of video clips
can only be achieved by analyzing all modalities jointly, and hope that such
multimodal representations will inspire the community in the future.

Acknowledgments. This work is supported by the DFG (German Research
Foundation) funded PLUMCOT project.

## References

  * [1] R. Arandjelović, P. Gronat, A. Torii, T. Pajdla, and J. Sivic.  Netvlad: Cnn architecture for weakly supervised place recognition.  In CVPR, 2016.
  * [2] Relja Arandjelović and Andrew Zisserman.  Objects that sound.  In ECCV, 2018.
  * [3] Yusuf Aytar, Carl Vondrick, and Antonio Torralba.  Soundnet: Learning sound representations from unlabeled video.  In NIPS, 2016.
  * [4] Tali Basha, Yael Moses, and Shai Avidan.  Photo sequencing.  In ECCV, 2012.
  * [5] Aruni Roy Chowdhury, Tsung-Yu Lin, Subhransu Maji, and Erik Learned-Miller.  One-to-many face recognition with bilinear cnns.  In WACV, 2016.
  * [6] Ali Diba, Vivek Sharma, and Luc Van Gool.  Deep temporal linear encoding networks.  In CVPR, 2017.
  * [7] Caglayan Dicle, Burak Yilmaz, Octavia Camps, and Mario Sznaier.  Solving temporal puzzles.  In CVPR, 2016.
  * [8] Carl Doersch, Abhinav Gupta, and Alexei A Efros.  Unsupervised visual representation learning by context prediction.  In ICCV, 2015.
  * [9] Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell.  Long-term recurrent convolutional networks for visual recognition and description.  In CVPR, 2015.
  * [10] Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox.  Discriminative unsupervised feature learning with convolutional neural networks.  In NIPS, 2014.
  * [11] Hazel Doughty, Walterio Mayol-Cuevas, and Dima Damen.  The pros and cons: Rank-aware temporal attention for skill determination in long videos.  arXiv:1812.05538, 2018.
  * [12] Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein.  Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation.  In SIGGRAPH, 2018.
  * [13] Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould.  Self-supervised video representation learning with odd-one-out networks.  In CVPR, 2017.
  * [14] Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach.  Multimodal compact bilinear pooling for visual question answering and visual grounding.  arXiv preprint arXiv:1606.01847, 2016.
  * [15] Yang Gao, Oscar Beijbom, Ning Zhang, and Trevor Darrell.  Compact bilinear pooling.  In CVPR, 2016.
  * [16] Rohit Girdhar, Deva Ramanan, Abhinav Gupta, Josef Sivic, and Bryan Russell.  Actionvlad: Learning spatio-temporal aggregation for action classification.  In CVPR, 2017.
  * [17] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh.  Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?  In CVPR, 2018.
  * [18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  arXiv:1512.03385, 2015.
  * [19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [20] Chiori Hori, Takaaki Hori, Teng-Yok Lee, Ziming Zhang, Bret Harsham, John R Hershey, Tim K Marks, and Kazuhiko Sumi.  Attention-based multimodal fusion for video description.  In ICCV, 2017.
  * [21] Dinesh Jayaraman and Kristen Grauman.  Slow and steady feature analysis: higher order temporal coherence in video.  In CVPR, 2016.
  * [22] Qin Jin, Jia Chen, Shizhe Chen, Yifan Xiong, and Alexander Hauptmann.  Describing videos using multi-modal fusion.  In ACM MM, 2016.
  * [23] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei.  Large-scale video classification with convolutional neural networks.  In CVPR, 2014.
  * [24] Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler.  Skip-thought vectors.  In NIPS, 2015.
  * [25] Hilde Kuehne, Hueihan Jhuang, Rainer Stiefelhagen, and Thomas Serre.  Hmdb51: A large video database for human motion recognition.  In High Performance Computing in Science and Engineering. 2013.
  * [26] Benjamin Laxton, Jongwoo Lim, and David Kriegman.  Leveraging temporal, contextual and ordering constraints for recognizing complex activities in video.  In CVPR, 2007.
  * [27] Tsung-Yu Lin, Aruni RoyChowdhury, and Subhransu Maji.  Bilinear cnn models for fine-grained visual recognition.  In ICCV, 2015.
  * [28] Kuan Liu, Yanen Li, Ning Xu, and Prem Natarajan.  Learn to combine modalities in multimodal deep learning.  arXiv preprint arXiv:1805.11730, 2018.
  * [29] Tegan Maharaj, Nicolas Ballas, Anna Rohrbach, Aaron C Courville, and Christopher Joseph Pal.  A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering.  In CVPR, 2017.
  * [30] Kevin Matzen and Noah Snavely.  Scene chronology.  In ECCV, 2014.
  * [31] Antoine Miech, Ivan Laptev, and Josef Sivic.  Learnable pooling with context gating for video classification.  arXiv preprint arXiv:1706.06905, 2017.
  * [32] Ishan Misra, C Lawrence Zitnick, and Martial Hebert.  Shuffle and learn: unsupervised learning using temporal order verification.  In ECCV, 2016.
  * [33] Yael Moses, Shai Avidan, et al.  Space-time tradeoffs in photo sequencing.  In ICCV, 2013.
  * [34] Arsha Nagrani, Samuel Albanie, and Andrew Zisserman.  Learnable pins: Cross-modal embeddings for person identity.  arXiv:1805.00833, 2018.
  * [35] Arsha Nagrani, Samuel Albanie, and Andrew Zisserman.  Seeing voices and hearing faces: Cross-modal biometric matching.  In CVPR, pages 8427–8436, 2018.
  * [36] Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng.  Multimodal deep learning.  In ICML, 2011.
  * [37] Andrew Owens and Alexei A Efros.  Audio-visual scene analysis with self-supervised multisensory features.  In ECCV, 2018.
  * [38] Andrew Owens, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H Adelson, and William T Freeman.  Visually indicated sounds.  In CVPR, 2016.
  * [39] Andrew Owens, Jiajun Wu, Josh H McDermott, William T Freeman, and Antonio Torralba.  Ambient sound provides supervision for visual learning.  In ECCV, 2016.
  * [40] Sanjeel Parekh, Slim Essid, Alexey Ozerov, Ngoc QK Duong, Patrick Pérez, and Gaël Richard.  Weakly supervised representation learning for unsynchronized audio-visual events.  Journal of Multimedia, page 1, 2018.
  * [41] Yuxin Peng, Jinwei Qi, Xin Huang, and Yuxin Yuan.  Ccl: Cross-modal correlation learning with multigrained fusion by hierarchical network.  IEEE Transactions on Multimedia, 2018.
  * [42] Jeffrey Pennington, Richard Socher, and Christopher Manning.  Glove: Global vectors for word representation.  In EMNLP, 2014.
  * [43] Ninh Pham and Rasmus Pagh.  Fast and scalable polynomial kernels via explicit feature maps.  In ACM SIGKDD, 2013.
  * [44] Veith Roethlingshoefer, Vivek Sharma, and Rainer Stiefelhagen.  Self-supervised face-grouping on graph.  In ACM MM, 2019.
  * [45] Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Chris Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele.  Movie description.  IJCV, 2017.
  * [46] Fereshteh Sadeghi, J Rafael Tena, Ali Farhadi, and Leonid Sigal.  Learning to select and order vacation photographs.  In WACV, 2015.
  * [47] M Saquib Sarfraz, Vivek Sharma, and Rainer Stiefelhagen.  Efficient parameter-free clustering using first neighbor relations.  In CVPR, 2019.
  * [48] Grant Schindler, Frank Dellaert, and Sing Bing Kang.  Inferring temporal order of images from 3d structure.  In CVPR, 2007.
  * [49] Vivek Sharma, Makarand Tapaswi, M Saquib Sarfraz, and Rainer Stiefelhagen.  Self-supervised learning of face representations for video face clustering.  In International Conference on Automatic Face and Gesture Recognition, 2019.
  * [50] Karen Simonyan and Andrew Zisserman.  Two-stream convolutional networks for action recognition in videos.  In NIPS, 2014.
  * [51] Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng.  Reasoning with neural tensor networks for knowledge base completion.  In NIPS, 2013.
  * [52] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  arXiv:1212.0402, 2012.
  * [53] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov.  Unsupervised learning of video representations using lstms.  In ICML, 2015.
  * [54] Makarand Tapaswi, Martin Bäuml, and Rainer Stiefelhagen.  StoryGraphs: Visualizing Character Interactions as a Timeline.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
  * [55] Makarand Tapaswi, Omkar M Parkhi, Esa Rahtu, Eric Sommerlade, Rainer Stiefelhagen, and Andrew Zisserman.  Total cluster: A person agnostic clustering method for broadcast videos.  In ACM ICVGIP, 2014.
  * [56] Joshua B Tenenbaum and William T Freeman.  Separating style and content with bilinear models.  Neural computation, 2000.
  * [57] Atousa Torabi, Niket Tandon, and Leon Sigal.  Learning language-visual embedding for movie understanding with natural-language.  arXiv:1609.08124, 2016.
  * [58] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri.  Learning spatiotemporal features with 3d convolutional networks.  In ICCV, 2015.
  * [59] Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun.  Order-embeddings of images and language.  arXiv preprint arXiv:1511.06361, 2015.
  * [60] Heng Wang and Cordelia Schmid.  Action recognition with improved trajectories.  In ICCV, 2013.
  * [61] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool.  Temporal segment networks: towards good practices for deep action recognition.  In ECCV, 2016.
  * [62] Xiaolong Wang and Abhinav Gupta.  Unsupervised learning of visual representations using videos.  In ICCV, 2015.
  * [63] Donglai Wei, Joseph Lim, Andrew Zisserman, and William T Freeman.  Learning and using the arrow of time.  In CVPR, 2018.
  * [64] Aming Wu and Yahong Han.  Multi-modal circulant fusion for video-to-language and backward.  In IJCAI, 2018.
  * [65] Ziwei Yang, Youjiang Xu, Huiyun Wang, Bo Wang, and Yahong Han.  Multirate multimodal video captioning.  In ACM MM, 2017.
  * [66] Zhou Yu, Jun Yu, Jianping Fan, and Dacheng Tao.  Multi-modal factorized bilinear pooling with co-attention learning for visual question answering.  In ICCV, 2017.
  * [67] Ben P Yuhas, Moise H Goldstein, and Terrence J Sejnowski.  Integration of acoustic and visual speech signals using neural networks.  IEEE Communications Magazine, 1989.
  * [68] Antonio Torralba Yusuf Aytar, Carl Vondrick.  See, hear, and read: Deep aligned representations.  In arXiv:1706.00932, 2017.
  * [69] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Places: A 10 million image database for scene recognition.  PAMI, 2017.
  * [70] Yipin Zhou and Tamara L Berg.  Temporal perception and prediction in ego-centric video.  In ICCV, 2015.


# A late times approximation for the SYK spectral form factor

Matteo A<EMAIL_ADDRESS>
Dipartimento di Fisica, Università degli Studi di Milano and INFN,
via Celoria 16, 20133 Milan, Italy

###### Abstract

We find a late times approximation for the SYK spectral form factor from a
large $N$ steepest descent version of the path integral over two replica
collective fields. Main ingredients are a suitable uv regularization of the
two replica kinetic operator, the property of its Fourier transform and some
spectral analysis of the four point function two replica ladder kernel.

There is a tension between the boundary holographic quantum mechanical
description of a region containing a black hole and its bulk semiclassical
gravity description. While the former has a finite dimensional Hilbert space,
whose discreteness manifests at times related to coarse grain scales in the
Hamiltonian spectrum, the bulk has a continuous nature, in terms of classical
fields. In particular, this tension manifests in the behavior of correlation
functions at late times. A boundary thermal correlator cannot decay to zero at
late times, because of the finiteness of the Hilbert space [1],[2],[3]. On the
other hand, this same correlator computed by the bulk dictionary does go to
zero at large separation times [4]. A bulk correlator describes a certain
number of particles, scattered by the black hole horizon. The larger is the
separation time between in and out states, the larger is the boost for
particles in reaching the horizon. The larger is their gravitational boost,
the longer is the tail of their wave functions beyond the horizon. As a
consequence, the scattering reflection coefficient dies off in the infinite
separation time limit. In other words, in a semiclassical description, the
scattering amplitude for particles by the black hole decays to zero in the
large separation time limit, in contradiction to a finite entropy boundary
quantum mechanics. This is a manifestation of the black hole information
paradox [5]. It is therefore of interest to consider systems where this
tension can be quantitatively analyzed and hopefully resolved to some extent,
by comparing quantities in both sides of the correspondence.

A quantity that suits for this analysis [6] is the spectral form factor (SFF)
$Z(\beta-iT)Z(\beta+iT)$, an analytic continuation to time $T$ of the square
of the imaginary time thermal partition function at inverse temperature
$\beta$. SFF is related to the two point correlation function of pairs of
energy eigenvalues of the quantum system. It gives information in the time
variable $T$ on correlations among pairs of eigenvalues at the energy scale
related to $T$. For example, for a certain coarse graining energy scale, a
correlation dependence
$\langle\rho(E)\rho(E^{\prime})\rangle\propto(E-E^{\prime})^{-2}$ corresponds
to a linearly growing in time $T$ form factor, at the related times scale. On
the other hand, a $\langle\rho(E)\rho(E^{\prime})\rangle\rightarrow 0$
behavior at an higher level of energy coarse graining (eigenvalues level
repulsion), corresponds to a constant value for the SFF at later times scales.

In the context of the Sachdev-Ye-Kitaev model (SYK)
[7],[8],[9],[10],[11],[12],[13], numerical analysis show [14] a late times
behavior of the SYK spectral form factor222see also [15] for earlier related
work. Recent works related to the SYK SFF include [17],[18],[19]. typical of
certain ensembles of Random Matrix Theory. In particular, [14] have shown the
existence of a linear ramp in $T$ rising behavior that saturates at late times
to a constant plateau. Such a behavior is observed in various quantum chaotic
systems. In mathematics, there is the so called Montgomery phenomenon, that
involves the positions of the Riemann zeta function zeros on the critical
line. A repulsion among too closed zeros of the zeta function on the critical
line is observed, that is accurately reproduced by a correlator for pairs of
zeroes involving a Sinc function. It accounts for the ramp regime and the
plateau saturation shown by the Riemann zeta spectral form factor.

In [16] the SYK spectral form factor is obtained as a path integral over two
replica of collective continuous bilocal fields with a complex coupling.
Various properties observed numerically [14], were deduced [16] by a replica
non-diagonal conformal complex saddle. Since SYK is not exactly conformal even
in the strongly coupled regime, the ramp part of the SYK SFF was reproduced
[16] by employing an effective action in the two time reparametrization soft
modes. This effective action is given by a complex linear combination of two
Schwarzians local actions. The two Schwarzians action was introduced in the
paper [16] a bit out of the blue. The authors were based on numerical evidence
and some general indirect argument [20], which were not reported on their
article. In [21], the two Schwarzian effective action was derived by direct
computations on the two replica collective field path integral. The method
employed there, relies on a uv regularization of the two replica kinetic
operator, inspired by a similar technique used in regular SYK [22].

In this work, I develop further some of the techniques and computations
presented in [21]. In particular, a late times $T$ approximation for the SYK
spectral form factor is obtained, by working on a large $N$ approximation of
the two replica path integral [21]. The large $T$ approximation is obtained
here by spectral analysis of the four point function two replica ladder
kernel. In particular, the explicit form of the ladder kernel invariant
eigenfunctions is obtained. Then, an argument is provided, based on the
Fourier transform of a uv regularization of the two replica kinetic operator,
that the ladder kernel invariant eigenfunction is the relevant quantity that
governs the spectral form factor at large times. Finally, by extracting the uv
behavior of this eigenfunction, the form of the regularized source, needed for
computing the large $T$ effective action is derived. The argument presented at
the end of this paper provides an alternative route w.r.t. one developed in
[21] for obtaining the correct form of the source that enters in the
computation of the late times SFF effective action.

In [21], we obtained the following large $N$ approximation for the SYK
spectral form factor

$Z(\beta+iT)Z(\beta-iT)\sim e^{\frac{N}{12J^{2}}\langle
s_{\alpha}|\left(\delta_{\alpha\gamma}-\tilde{K}_{\alpha\gamma}^{c}\right)^{-1}\tilde{K}^{c}_{\gamma\beta}|s_{\beta}\rangle+\frac{N}{2}J_{\alpha}^{1/2}\langle
s_{\alpha}||G_{\alpha}^{c}|^{2}\rangle},$ (1)

which is valid in the strongly coupled nearly conformal regime. Eq. (1) was
derived [21] by steepest descent method on the path integral for the spectral
form factor, along contours that go through the replica non diagonal conformal
saddle. The following replica indexing [21] is used, $\alpha=1,\dots,4$,
$\alpha=1=LL$, $\alpha=2=RR$, $\alpha=3=LR$, $\alpha=4=RL$. The bilocal field
$s_{\alpha}(t,t^{\prime})$ is a regularized version of the two replica kinetic
operator $\delta_{ij}\delta(t-t^{\prime})\partial_{t}$, $i,j=L,R$. The uv
regularization replaces the singular Dirac delta kernel $\delta(t-t^{\prime})$
by a smearing function $U(\xi(t,t^{\prime}))$. It implements the impossibility
to resolve times intervals shorter then the SYK time scale $1/J$ [22].
Moreover, the derivative operator $\partial_{t}$ is substituted by a sum of
contributions with different scalings [21] that are analyzed in terms of the
spectrum of the four point function two replica symmetrized ladder kernel at
the conformal point

$\tilde{K}_{\alpha\beta}^{c}=\tilde{K}^{c}_{ij,kl}(t_{1},t_{2};t_{3},t_{4})\equiv
3J^{2}J_{ij}^{1/2}J_{kl}^{1/2}|G_{ij}^{c}(t_{12})|G_{ik}^{c}(t_{13})G_{jl}^{c}(t_{24})|G_{kl}^{c}(t_{34})|,\qquad
i,j=L,R.$ (2)

In the following, we will extract a late times $T$ approximation for (1). To
this aim,

Let us expand the source

$s_{\alpha}(t,t^{\prime})=\sum_{h}c_{h}\psi_{\alpha}^{h}(t,t^{\prime})U(\xi_{\alpha}(t,t^{\prime})).$
(3)

in terms of eigenfunctions of $\tilde{K}^{c}_{\alpha\beta}$

$\tilde{K}^{c}_{\alpha\beta}(t_{1},t_{2};t,t^{\prime})\psi_{\beta}^{h}(t,t^{\prime})=\tilde{k}_{SFF}(h)\psi_{\alpha}^{h}(t_{1},t_{2}).$
(4)

In (4) $h$ labels the eigenvalues of the $SL_{diag}(2)$ Casimir, the diagonal
subgroup of $SL_{L}(2)\times S_{R}(2)$ that survives the spontaneous breaking
of the twofold time diffeomorphisms by the conformal replica non-diagonal
saddle [16]. Details and properties of the smearing function
$U(\xi_{\alpha}(t,t^{\prime}))$ are given in [21]. For our purposes, we find
convenient to introduce the Fourier transform $\tilde{U}(\eta)$

$U(\xi)=\int\frac{d\eta}{2\pi}e^{-i\xi\eta}\tilde{U}(\eta).$ (5)

Since the smearing function $U(\xi(t,t^{\prime}))$ has support over
$\Delta\xi\sim O(T)$, its Fourier transform $\tilde{U}(\eta)$ is non vanishing
over an interval of length $\Delta\eta\sim O\left(\frac{1}{T}\right)$,
centered in $\eta=0$. Therefore, $\tilde{U}(\eta)$ becomes very narrow in the
large $T$ limit. We use (3) and (5) in the first term at the exponent of the
spectral form factor (1)

$\displaystyle\langle
s_{\alpha}|\left(\delta_{\alpha\gamma}-\tilde{K}_{\alpha\gamma}^{c}\right)^{-1}\tilde{K}^{c}_{\gamma\beta}|s_{\beta}\rangle$
(6) $\displaystyle=$ $\displaystyle\sum_{h,h^{\prime}}c_{h}c_{h^{\prime}}\int
dtdt^{\prime}(\psi_{\alpha}^{h^{\prime}}(t,t^{\prime}))^{*}U(\xi_{\alpha}(t,t^{\prime}))\int\frac{d\eta}{2\pi}\frac{\tilde{k}_{SFF}(h+i\eta)}{1-\tilde{k}_{SFF}(h+i\eta)}\psi_{\alpha}^{h+i\eta}(t,t^{\prime})\tilde{U}(\eta),$

where [21]

$e^{-i\eta_{\alpha}\xi_{\alpha}}=\frac{1}{\left(\zeta_{\alpha}(|t-t^{\prime}|)\right)^{i\eta_{\alpha}}}.$
(7)

In the large $T$ limit, one expands $\tilde{k}_{SFF}(h+i\eta)$ up to the first
order in eta,
$\tilde{k}_{SFF}(h+i\eta)=\tilde{k}_{SFF}(h)+i\eta\,\tilde{k}_{SFF}^{\prime}(h)$.
Then, by evaluating the $\eta$ integral in (6) by residues theorem, one finds

$\langle
s_{\alpha}|\left(\delta_{\alpha\gamma}-\tilde{K}_{\alpha\gamma}^{c}\right)^{-1}\tilde{K}^{c}_{\gamma\beta}|s_{\beta}\rangle=\sum_{h,h^{\prime}}c_{h}c_{h^{\prime}}\int
dtdt^{\prime}(\psi_{\alpha}^{h^{\prime}}(t,t^{\prime}))^{*}U(\xi_{\alpha}(t,t^{\prime}))\psi_{\alpha}^{h+i\eta_{h}}(t,t^{\prime})\tilde{U}(\eta_{h}),$
(8)

where

$\eta_{h}=i\frac{\tilde{k}_{SFF}(h)-1}{\tilde{k}_{SFF}^{\prime}(h)}.$ (9)

Since $\tilde{U}(\eta)$ has support around $\eta=0$ over an interval of length
$O\left(\frac{1}{T}\right)$, the only non vanishing $\tilde{U}(\eta_{h})$ in
(8) are those for

$|\eta_{h}|=\left|\frac{\tilde{k}_{SFF}(h)-1}{\tilde{k}_{SFF}^{\prime}(h)}\right|<\frac{\Delta\eta}{2}=O\left(\frac{1}{T}\right)$
(10)

For large enough time $T$, the only non vanishing term in (8) reduces to the
$\eta_{h}=0$ one, which corresponds by eq. (9) to $\tilde{k}_{SFF}(h_{*})=1$.
Therefore, we have the following large times $T$ approximation

$\langle
s_{\alpha}|\left(\delta_{\alpha\gamma}-\tilde{K}_{\alpha\gamma}^{c}\right)^{-1}\tilde{K}^{c}_{\gamma\beta}|s_{\beta}\rangle\sim\int_{0}^{T}dt\int_{0}^{T}dt^{\prime}(\psi_{\alpha}^{h_{*}}(t,t^{\prime}))^{*}U(\xi_{\alpha}(t,t^{\prime}))\psi_{\alpha}^{h_{*}}(t,t^{\prime}),\qquad
T\rightarrow\infty.$ (11)

Therefore, we have shown that in the large $T$ limit, the only rescaled
smeared source which is effective at the exponent for the spectral form factor
(1) has the following form

$s_{\alpha}(t,t^{\prime})=\psi_{\alpha}^{h_{*}}(t,t^{\prime})U(\xi_{\alpha}(t,t^{\prime})),$
(12)

where $\psi_{\alpha}^{h_{*}}(t,t^{\prime})$ is the invariant eigenfunction of
the four points ladder symmetrized kernel (2)

$\tilde{K}^{c}_{\alpha\beta}(t_{1},t_{2};t,t^{\prime})\psi_{\beta}^{h_{*}}(t,t^{\prime})=\psi_{\alpha}^{h_{*}}(t_{1},t_{2}).$
(13)

Let us recall, (see [21] for more details), that the collective fields action
in the spectral form factor path integral splits as follows

$Z(\beta+iT)Z(\beta-
iT)=\int\mathcal{D}G_{\alpha}\mathcal{D}\Sigma_{\alpha}e^{NI_{c}(G_{\alpha},\Sigma_{\alpha})+\int
G_{\alpha}\sigma_{\alpha}},$ (14)

where $I_{c}$ is the time reparametrization invariant two replica critical
action, _without_ the conformal symmetry breaking replica diagonal kinetic
term $\delta_{ij}\delta(t-t^{\prime})\partial_{t}$.

On the other hand,

$I_{s}=\sum_{\alpha=1,2}\int
dtdt^{\prime}G_{\alpha}(t,t^{\prime})\sigma_{\alpha}(t,t^{\prime})$ (15)

is the conformal symmetric breaking source term, where
$\sigma_{\alpha}(t,t^{\prime})$ is a regularized version of the replica
diagonal kinetic kernel $\delta_{ij}\delta(t-t^{\prime})\partial_{t}$, which
is related to the previously introduced $s_{\alpha}(t,t^{\prime})$ by

$\sigma_{\alpha}(t,t^{\prime})=|G_{\alpha}^{c}(t,t^{\prime})|s_{\alpha}(t,t^{\prime}),\qquad\alpha=1,2.$
(16)

The effective action in the time reparametrization soft modes $f_{L}(t)$,
$f_{R}(t)$ is computed from the non conformal source action (15). In
particular, it depends only on the uv $|t-t^{\prime}|\rightarrow 0$ behavior
of the source $\sigma_{\alpha}(t,t^{\prime})$ . The reason for that is that
the failure of time reparametrization is a uv effect, as for large times
intervals, in the deep ir, the conformal saddle is a very accurate
approximation for the actual Green function. As we have shown above, in the
large $T$ regime, the spectral form factor is dominated by a source (12)
proportional to the invariant eigenfunction
$\psi_{\alpha}^{h_{*}}(t,t^{\prime})$ (13) of the symmetrized four point
ladder kernel $\tilde{K}^{c}_{\alpha\beta}$. In the following we compute
explicitly the diagonal components $\alpha=1,2$ of
$\psi_{\alpha}^{h_{*}}(t,t^{\prime})$ and then extract their
$|t-t^{\prime}|\rightarrow 0$ behavior, in order to obtain the form of the
source that governs (15) in the large $T$ limit. By the same method presented
here, one can also compute the off diagonals components $\alpha=3,4$ of
$\psi_{\alpha}^{h_{*}}(t,t^{\prime})$, we will not do it here, since only the
diagonal components of the source matter $\sigma_{\alpha}(t,t^{\prime})$ in
the source action (15).

Let us consider the conformal Schwinger-Dyson equations for the spectral form
factor [23],[21]. An infinitesimal variation of the bilocal fields by a time
reparametrization is still a solution of the conformal saddle point equations,
due to conformal symmetry. After some manipulation, one gets to the following
condition

$(1-K^{c}_{\alpha\beta})\delta_{\epsilon}G_{\beta}^{c}=0,$ (17)

where $K^{c}_{\alpha\beta}$ is the following _non symmetrized_ version of the
four points ladder kernel

$K^{c}_{ij,kl}(t_{1},t_{2};t_{3},t_{4})=3J^{2}J_{kl}G_{ik}^{c}(t_{13})G_{jl}^{c}(t_{24})(G_{kl}^{c}(t_{34}))^{2}$
(18)

and $\delta_{\epsilon}G_{\alpha}^{c}$ is the first order variation of
$G^{c}_{\alpha}(t,t^{\prime})$ by a linear diffeomorphism $t\rightarrow
t+\epsilon(t)$. Eq. (17) provides also a way to compute the invariant
eigenfunction of the symmetrized kernel (2), since one finds that

$\psi_{\alpha}^{h_{*}}(t,t^{\prime})=J_{\alpha}^{1/2}|G_{\alpha}^{c}|\delta_{\epsilon}G^{c}_{\alpha}$
(19)

by (17) is the invariant eigenfunction of $\tilde{K}^{c}_{\alpha\beta}$.

We are now in the position of being able to compute the diagonal components
$\alpha=1,2$ of $\psi_{\alpha}^{h_{*}}(t,t^{\prime})$ by using eq. (19). A
linear time reparametrization $t\rightarrow t+\epsilon(t)$ on
$G_{\alpha}(t,t^{\prime})$, $\alpha=1,2$ gives to the first order in
$\epsilon(t)$

$\delta_{\epsilon}G_{\alpha}(t,t^{\prime})=\frac{1}{4}(\epsilon^{\prime}(t)+\epsilon^{\prime}(t^{\prime}))G_{\alpha}(t,t^{\prime})+(\epsilon(t)-\epsilon(t^{\prime}))\partial_{t}G_{\alpha}(t,t^{\prime})$
(20)

We use the diagonal components of the spectral form factor conformal saddle
[16],[21]

$G_{\alpha}^{c}(t,t^{\prime})\propto\frac{sgn(t-t^{\prime})}{\left|\frac{\tilde{\beta}_{aux}}{\pi}\sinh\left(\frac{\pi}{\tilde{\beta}_{aux}}(t-t^{\prime})\right)\right|^{1/2}},\qquad\alpha=1,2,$
(21)

and its derivative

$\partial_{t}G^{c}_{\alpha}(t,t^{\prime})\propto-\frac{1}{2}\frac{sgn(t-t^{\prime})}{\left|\frac{\tilde{\beta}_{aux}}{\pi}\sinh\left(\frac{\pi}{\tilde{\beta}_{aux}}(t-t^{\prime})\right)\right|^{3/2}}\cosh\left(\frac{\pi}{\tilde{\beta}_{aux}}(t-t^{\prime})\right)\qquad\alpha=1,2.$
(22)

We express the linear time reparmetrization as a Fourier integral

$\epsilon(t)=\int\frac{d\omega}{2\pi}\,\,e^{-i\omega
t}\,\tilde{\epsilon}(\omega).$ (23)

A variation of the Green function by a phase $\epsilon(t)=e^{-i\omega t}$, by
eq. (20) gives

$\delta_{\epsilon}G^{c}_{\alpha}\propto-i\frac{\omega}{4}(e^{-i\omega
t}+e^{-i\omega t^{\prime}})G^{c}_{\alpha}(t,t^{\prime})+(e^{-i\omega
t}-e^{-i\omega t^{\prime}})\partial_{t}G^{c}_{\alpha}(t,t^{\prime}).$ (24)

After a little manipulation, we get to the following fixed $\omega$ form for
the diagonal components of the invariant eigenfunction of the symmetrized
ladder kernel

$\displaystyle\psi_{\alpha}^{h_{*},\omega}(t,t^{\prime})=J_{\alpha}^{1/2}|G_{\alpha}^{c}(t,t^{\prime})|\delta_{\epsilon}G^{c}_{\alpha}$
(25) $\displaystyle=$ $\displaystyle
ie^{-i\omega\frac{t+t^{\prime}}{2}}\frac{sgn(t-t^{\prime})}{\left|\frac{\tilde{\beta}_{aux}}{\pi}\sinh\left(\frac{\pi}{\tilde{\beta}_{aux}}(t-t^{\prime})\right)\right|}\left(\frac{\pi}{\tilde{\beta}_{aux}}\frac{\sin\left(\omega\frac{t-t^{\prime}}{2}\right)}{\tanh\left(\frac{\pi}{\tilde{\beta}_{aux}}(t-t^{\prime})\right)}-\frac{\omega}{2}\cos\left(\omega\frac{t-t^{\prime}}{2}\right)\right),$
$\displaystyle\alpha=1,2.$

On the other hand, by linearity (23), the more general invariant eigenfunction
of $\tilde{K}^{c}_{\alpha\beta}$ has the following diagonal components

$\psi_{\alpha}^{h_{*}}(t,t^{\prime})=\int\frac{d\omega}{2\pi}\,\tilde{\epsilon}(\omega)\,\psi_{\alpha}^{h_{*},\omega}(t,t^{\prime}),\qquad\alpha=1,2,$
(26)

with $\tilde{\epsilon}(\omega)$ being the Fourier transform (23) of
$\epsilon(t)$. Therfore, we have gotten a continuous family of eigenfunctions
$\\{\psi_{\alpha}^{h_{*},\omega}(t,t^{\prime})\\}_{\omega\in I_{\omega}}$ of
the ladder kernel with eigenvalue one, labeled by $\omega\in
I_{\omega}=(-\frac{\Delta_{\omega}}{2},\frac{\Delta_{\omega}}{2})$, with
$\Delta_{\omega}=O\left(\frac{1}{T}\right)$, and (26) is a generic linear
superposition.

We want to extract the uv behavior $|t-t^{\prime}|\rightarrow 0$ in (26),
since by (12) we know that it enters in the computation of the large $T$
effective action for the spectral form factor through the action term (15).
Let us keep in mind that the source (12) is uv regularized by the smeared
function $U(\xi(t,t^{\prime}))$ that vanishes for $|t-t^{\prime}|<1/J$. On the
other hand, $\tilde{\epsilon}(\omega)$ has support over an interval centered
in $\omega=0$ of length $|I_{\omega}|=O\left(\frac{1}{T}\right)$. In the limit
$T\rightarrow\infty$ one can take the following approximation

$\psi_{\alpha}^{h_{*}}(t,t^{\prime})=\int\frac{d\omega}{2\pi}\,\tilde{\epsilon}(\omega)\,\psi_{\alpha}^{h_{*},\omega}(t,t^{\prime}),\
\sim|I_{\omega}|\omega_{*}\tilde{\epsilon}(\omega_{*})\hat{\psi}_{\alpha}^{h_{*}}(t,t^{\prime})=-i\epsilon^{\prime}(t_{\epsilon})\hat{\psi}_{\alpha}^{h_{*}}(t,t^{\prime}),$
(27)

where $\omega_{*}\in I_{\omega}$, $t_{\epsilon}>0$, and

$\hat{\psi}_{\alpha}^{h_{*}}(t,t^{\prime})=i\frac{sgn(t-t^{\prime})}{\left|\frac{\tilde{\beta}_{aux}}{\pi}\sinh\left(\frac{\pi}{\tilde{\beta}_{aux}}(t-t^{\prime})\right)\right|}\left(\frac{\pi}{\tilde{\beta}_{aux}}\frac{1}{\tanh\left(\frac{\pi}{\tilde{\beta}_{aux}}(t-t^{\prime})\right)}-\frac{1}{2}\right).$
(28)

Eq. (28) is a valid approximation as far as $\tilde{\beta}_{aux}<<T$. On the
other hand, $\epsilon^{\prime}(t_{\epsilon})$ is regarded to be a renormalized
quantity. Its meaning follows from eq. (23) by the integral averaged theorem
that was used in (27).

By taking the uv $|t-t^{\prime}|\rightarrow 0$ in eq. (28) we get to the
following form for the replica diagonal components of the effective source in
the large $T\rightarrow\infty$ limit

$s_{\alpha}(t,t^{\prime})\propto\frac{sgn(t-t^{\prime})}{\left|\frac{\tilde{\beta}_{aux}}{\pi}\sinh\left(\frac{\pi}{\tilde{\beta}_{aux}}(t-t^{\prime})\right)\right|^{2}}U(\xi(t,t^{\prime}))\qquad\alpha=1,2.$
(29)

This is the quantity that enters in the computation of the effective action
for the spectral form factor in the $T\rightarrow\infty$ limit through eq.
(16)
$\sigma_{\alpha}(t,t^{\prime})=|G_{\alpha}^{c}(t,t^{\prime})|s_{\alpha}(t,t^{\prime})$
in the conformal symmetry breaking source term (15). Moreover, this same
quantity governs the large $T$ approximation for the spectral form factor path
integral (1). In [21] the uv form of the smeared source displayed in eq. (29),
was obtained by a different argument. and used there for computing the two
Schwarzians action in the time reparametrization soft modes $f_{L}(t)$,
$f_{R}(t)$.

## References

  * [1] L. Dyson, M. Kleban and L. Susskind, _Disturbing implications of a cosmological constant_ , _JHEP_ 10 (2002) 011, [hep-th/0208013].
  * [2] N. Goheer, M. Kleban and L. Susskind, _The Trouble with de Sitter space_ , _JHEP_ 07 (2003) 056, [hep-th/0212209].
  * [3] J. L. F. Barbon and E. Rabinovici, _Very long time scales and black hole thermal equilibrium_ , _JHEP_ 11 (2003) 047, [hep-th/0308063].
  * [4] G. T. Horowitz and V. E. Hubeny, _Quasinormal modes of AdS black holes and the approach to thermal equilibrium_ , _Phys. Rev._ D62 (2000) 024027, [hep-th/9909056].
  * [5] J. M. Maldacena, _Eternal black holes in anti-de Sitter_ , _JHEP_ 04 (2003) 021, [hep-th/0106112].
  * [6] K. Papadodimas and S. Raju, _Local Operators in the Eternal Black Hole_ , _Phys. Rev. Lett._ 115 (2015) 211601, [1502.06692].
  * [7] A. Kitaev, _A simple model of quantum holography (part 1)_ , in _KITP Program: Entanglement in Strongly-Correlated Quantum Matter_ , 2015, http://online.kitp.ucsb.edu/online/entangled15/kitaev/.
  * [8] A. Kitaev, _A simple model of quantum holography (part 2)_ , in _KITP Program: Entanglement in Strongly-Correlated Quantum Matter_ , 2015, http://online.kitp.ucsb.edu/online/entangled15/kitaev2/.
  * [9] S. Sachdev and J. Ye, _Gapless spin fluid ground state in a random, quantum Heisenberg magnet_ , _Phys. Rev. Lett._ 70 (1993) 3339, [arXiv:cond-mat/9212030].
  * [10] A. Georges, O. Parcollet and S. Sachdev, _Mean Field Theory of a Quantum Heisenberg Spin Glass_ , _Phys. Rev. Lett._ 85 (2000) 840, [arXiv:cond-mat/9909239].
  * [11] S. Sachdev, _Holographic metals and the fractionalized Fermi liquid_ , _Phys. Rev. Lett._ 105 (2010) 151602, [arXiv:1006.3794].
  * [12] S. Sachdev, _Strange metals and the AdS/CFT correspondence_ , _J. Stat. Mech._ (2010) 1011, [arXiv:1010.0682].
  * [13] S. Sachdev, _Bekenstein-Hawking Entropy and Strange Metals_ , _Phys. Rev. X_ 5 (2015) 041025, [arXiv:1506.05111].
  * [14] J. S. Cotler, G. Gur-Ari, M. Hanada, J. Polchinski, P. Saad, S. H. Shenker et al., _Black holes and random matrices_ , _JHEP_ 2017 (May, 2017) 118, [arXiv:1611.04650].
  * [15] A. M. García-García and J. J. M. Verbaarschot, _Spectral and thermodynamic properties of the Sachdev-Ye-Kitaev model_ , _Phys. Rev._ D94 (2016) 126010, [1610.03816].
  * [16] P. Saad, S. H. Shenker and D. Stanford, _A semiclassical ramp in SYK and in gravity_ , arXiv:1806.06840.
  * [17] M. Winer and B. Swingle, _Hydrodynamic Theory of the Connected Spectral Form Factor_ , 2012.01436.
  * [18] M. Khramtsov and E. Lanina, _Spectral form factor in the double-scaled SYK model_ , 2011.01906.
  * [19] A. del Campo, J. Molina-Vilaplana and J. Sonner, _Scrambling the spectral form factor: unitarity constraints and exact results_ , _Phys. Rev. D_ 95 (2017) 126008, [1702.04350].
  * [20] D. Stanford, _Private communication_ , .
  * [21] M. A. Cardella, _Derivation of the two Schwarzians effective action for the Sachdev-Ye-Kitaev spectral form factor_ , 1907.09570.
  * [22] A. Kitaev and S. J. Suh, _The soft mode in the Sachdev-Ye-Kitaev model and its gravity dual_ , _JHEP_ 05 (2018) 183, [arXiv:1711.08467].
  * [23] P. Saad, S. H. Shenker and D. Stanford, _JT gravity as a matrix integral_ , 1903.11115.