# Leaping Into Memories: Space-Time Deep Feature Synthesis

Alexandros Stergiou1,2 Nikos Deligiannis1,2
1Vrije Universiteit Brussel 2imec
<EMAIL_ADDRESS>
###### Abstract

The success of deep learning models has led to their adaptation and adoption
by prominent video understanding methods. The majority of these approaches
encode features in a joint space-time modality for which the inner workings
and learned representations are difficult to visually interpret. We propose
LEArned Preconscious Synthesis (LEAPS), an architecture-agnostic method for
synthesizing videos from the internal spatiotemporal representations of
models. Using a stimulus video and a target class, we prime a fixed space-time
model and iteratively optimize a video initialized with random noise. We
incorporate additional regularizers to improve the feature diversity of the
synthesized videos as well as the cross-frame temporal coherence of motions.
We quantitatively and qualitatively evaluate the applicability of LEAPS by
inverting a range of spatiotemporal convolutional and attention-based
architectures trained on Kinetics-400, which to the best of our knowledge has
not been previously accomplished111Our code is available at:
http://tinyurl.com/VisPre..

Figure 1: Illustration of our proposed LEAPS model inversion. Given a video
input initialized with noise $\mathbf{x}^{*}$, we use a stimulus video
$\mathbf{v}$ of a target class $y$, to prime video model $\Phi(\cdot)$ and
iteratively optimize $\mathbf{x}^{*}$ to synthesize and visually represent the
internal spatiotemporal features of $\Phi(\cdot)$. To impose diversity over
the synthesized videos, feature statistics from a domain-specific verifier
network $S(\cdot)$ are distilled to regularize $\mathbf{x}^{*}$. To preserve
the continuity of motions across frames, $\mathbf{x}^{*}$ is temporally
regularized at each training iteration. Although the illustration visualizes
internal representations as $C\\!\times\\!T\\!\times\\!H\\!\times\\!W$
volumes, they can also be flattened patches $(P^{3}C)\times\frac{THW}{P^{3}}$
of $P^{3}$ resolution as in Vision Transformers.

##  1 Introduction

Inverting deep networks’ learned internal representations has been a difficult
task to achieve. The adaptation and deployment of CNNs and more recently
Transformers, to a variety of fields and tasks has led to significant
breakthroughs. The field of video action recognition has experienced drastic
growth in recent years through the convergence of models with increased
complexities and capacities [2, 4, 32, 64] as well as large accuracy
improvements [11, 12, 27, 26, 29]. Despite great progress in their
applicability, there is still a large gap in the interpretability of video
models. As these models compositionally encode space and time modalities of
videos over large feature spaces and require a lot of parameters,
conceptualizing their internal representations remains challenging. In this
paper, we propose a method to invert the learned features of video models
associated with specific actions by optimizing a parameterized input to
synthesize conceptually and visually coherent representations.

In cognitive science, stages of consciousness include the conscious, the
unconscious, and the preconscious. In contrast to the conscious and
unconscious, the preconscious is responsible for learned information currently
outside the conscious awareness to remain readily available [18]. One way of
accessing learned preconscious information and making it part of conscious
awareness is through priming [39]. Priming uses a stimulus to activate learned
related concepts in memory and thus make them more easily and readily
accessible [33, 34]; e.g., one can remember their bedroom if primed with a
picture of a bed.

Motivated by visual priming in cognitive science, we demonstrate that learned
representations of video models can become accessible through _model priming_.
By using a video stimulus and a target action class, we synthesize the
perceived features corresponding to actions. In turn, the visual features of
the synthesized videos provide a conceptual view of the models’ learned
internal representations. We term these features as the _learned preconscious_
of the video model associated with a specific action.

We introduce LEArned Preconscious Synthesis (LEAPS), illustrated in Figure 1,
a spatiotemporal model inversion method that synthesizes interpretable videos
in order to maximize a classification score without prior knowledge of the
training data. LEAPS uses a video of a target action class as stimulus, to
prime a fixed spatiotemporal model. We additionally include two regularization
terms. The first term enforces coherence of motions across frames by
constraining their representations at each update. The second enhances the
synthesized video diversity by using a domain-specific verifier network that
exploits the disagreements between feature statistics similar to [65]. Through
the same architecture-agnostic approach, we show that LEAPS can invert the
spatiotemporal features of 3D-CNNs and spatiotemporal Transformers.

Our main contributions are as follows. First, we introduce LEAPS, a general
approach for inverting space-time models, which to the best of our knowledge,
is the first attempt to create videos of conceptual representations directly
from jointly encoded space-time features. Second, we use LEAPS on multiple
convolution and attention-based video architectures and compare it to prior
image-based model inversion methods that we extend to video. Finally, we show
that LEAPS can invert both video CNNs and transformer models, by using the
same architecture-agnostic method without any modifications.

##  2 Related Work

Approaches for visualizing and interpreting deep models can be divided into
three groups which we detail below.

Attribution-based visualizations. These methods have been used to visualize
feature contributions over a given input. Attribution methods for images have
primarily been based on back-propagating activations of classes [8, 49, 62],
or individual neurons [3, 5, 50, 53] to localize the regions in the input that
are most informative for a given class or feature. Such approaches include
Integrated gradients (IG) [57] which produce pixel-wise attributions from
integrating computed backprop gradients. Subsequent works have also included
gradient smoothing [52], gradient accumulation in saturated regions [36], and
adaptation of the gradient path [23]. Another set of approaches that rely on
attribution-based visualizations use perturbations of the input [17, 16, 45].
Given their straightforward applicability, image attribution methods have also
been extended to videos. The majority of these works (e.g. Saliency Tubes [56,
55], STEP [28], video OSA [60], and BOREx [24]) have focused on the
localization of spatiotemporal salient regions by extending Grad-CAM [49],
Extremal Pertibations (EP) [16], Occlusion Sensitivity Analysis (OSA) [67],
and Gaussian processes regression (GPR) [6, 38] respectively. In contrast to
these methods that localize salient class features in videos, we offer a
visual feature synthesis method to conceptualize the learned representations
of video models through priming.

Input synthesis. Network-centric approaches invert models to either visualize
particular classes [42, 66, 65] or features [44, 54]. One of the main methods
employed for visualizing internal network features is Gradient Ascent (GA)
[10], which optimizes the input by increasing the activation of a specific
network neuron. Activation Maximization (AM) [51] later adapted GA to
visualize CNN features. Following works have further built on top of AM by
including additional regularizers (e.g. total variation [31], blurring [61],
and gradient masking [42]). One of the most popular extensions of AM has been
DeepDream [1] which optimizes the input to yield high responses for a chosen
class while keeping the internal representations constraint-free. The produced
images include repetitions of recognizable features without representing a
coherent whole. In addition to AM, model inversion [9, 19, 30, 65, 67] has
introduced the task of maximizing the classification score of the synthesized
image instead of maximizing a class activation. The only extension of feature
visualization approaches to videos has been introduced by Feichtenhofer _et
al_. [13] in which they use AM to create visual representations of class
features from two-steam models [14, 63], trained on individual RGB frames
(spatial stream) and optical flow (temporal stream). In this paper, we instead
propose a model inversion method for inverting video architectures that
concurrently model space and time modalities.

Visual feature generation. These methods utilize activation maximization by
including an additional generator network [40, 41, 43] to generate class
features, with the cost of requiring access to the training data. Huang _et
al_. [21] adapted feature generation on video data by modeling the temporal
signal with a temporal generator network. Despite the high fidelity of the
produced results, these approaches are less suitable for interpreting internal
model behaviors, as they do not solely depend on the model under inspection.
Instead, they are primarily influenced by the training data used as well as
the capacity and complexity of the generator model that is trained.

##  3 Learned Preconscious Synthesis

In this section, we overview our LEAPS method (shown in Figure 1). We start by
formally introducing model priming for video architectures in Section 3.1. A
stimulus video of a target class is used to initially prime the network. The
training process uses the primed representations to update a randomly
initialized input alongside two regularizers. The first regularizer is used to
enforce temporal coherence, explained in Section 3.2. The second is used to
improve synthesized feature diversity, overviewed in Section 3.3. We present
the final aggregated function in Section 3.4.

###  3.1 Model Priming

Priming deep models is influenced by cognitive science [39] in which a
stimulus is used to recall prior knowledge. We extend this approach to
visualizing the learned preconscious of deep video models associated with a
specific action class $y$. Given a video v of size
$C\\!\times\\!T\\!\times\\!H\\!\times\\!W$, with $C$ channels, $T$ frames, $H$
height, and $W$ width, as a visual cue for action class $y$, and a randomly
initialized input $\mathbf{x}^{*}$ of
$C\\!\times\\!T\\!\times\\!H\\!\times\\!W$ size to optimize. We define a
priming loss $\mathcal{L}_{p}(\mathbf{x}^{*},\mathbf{v})$ between the internal
representations $\textbf{z}^{l}(\cdot)$ of the optimized input
$\mathbf{x}^{*}$ and stimulus $\mathbf{v}$ across
$l\\!\in\mathbf{\Lambda}=\\{1,...,L\\}$ layers:

$\underset{prim}{\mathcal{L}}(\mathbf{x}^{*},\mathbf{v})\\!=\\!\\!\;\frac{1}{L}\sum_{l\in\mathbf{\Lambda}}\lambda_{l}\;JVS\left(\mu\\!\left(\textbf{z}^{l}(\mathbf{x}^{*})\right)\\!,\mu\\!\left(\textbf{z}^{l}(\mathbf{v})\right)\right)$
(1)

where $\mu\\!\left(\textbf{z}^{l}(\cdot)\right)$ is the $C$-length
spatiotemporal mean vector of representations $\textbf{z}^{l}(\cdot)$.
$JVS(\cdot)$ denotes the Jaccard vector similarity [15]. To integrate a degree
of freedom and avoid hard constraints on the internal representations of
$\mathbf{x}^{*}$, we define $0<\lambda_{l}\leq 1$ as priming weight for layer
$l$. An overview of updating $\mathbf{x}^{*}$ by priming appears in Figure 1.

Due to the vastness of the feature space when optimizing (1), we include two
additional regularization terms to constrain the input. Specifically, we apply
a temporal coherence regularization $\underset{coh}{\mathcal{R}}$ and a
feature diversity regularization $\underset{feat}{\mathcal{R}}$, which we
detail below.

Figure 2: Temporal Coherence regularization of _zumba_ target action label.
Applying temporal coherence regularization on representations from all layers
$l\in\mathbf{\Lambda}$ synthesizes static videos (bottom row). Instead,
regularizing only the final network layer $L$ synthesizes videos with
consistent frame transitions and motions (top row). We note that the video
stimulus is shown as a reference as the regularizations are only applied to
synthesized video representations.

###  3.2 Temporal Coherence Regularization

For the first regularizer, we aim to enforce similarity between
representations of consecutive frames in order to enable consistent feature
transitions in the synthesized video. Therefore, we include a coherence
regularizer $\underset{coh}{\mathcal{R}}$, formulated based on the temporal
coherence loss from [37].

Given two spatiotemporal representations
$\mathbf{z}^{L}(\mathbf{x}^{*})_{t_{1}}$ and
$\mathbf{z}^{L}(\mathbf{x}^{*})_{t_{2}}$ at layer $L$, for temporal locations
$t_{1}$ and $t_{2}$, we use their $l_{1}$ norm to enforce similarity if
$t_{1}$ and $t_{2}$ are consecutive in the video. In non-consecutive cases,
the divergence between $\mathbf{z}^{L}(\mathbf{x}^{*})_{t_{1}}$ and
$\mathbf{z}^{L}(\mathbf{x}^{*})_{t_{2}}$ increases. The coherence regularizer
is formulated as:

$\
\underset{coh}{\mathcal{R}}(\mathbf{x}^{*})\\!=\\!\begin{cases}||\mathbf{z}^{L}(\mathbf{x}^{*})_{t_{1}}\\!-\\!\mathbf{z}^{L}(\mathbf{x}^{*})_{t_{2}}||_{1},\text{if
consecutive}\\\
\text{max}\\!\left(0,\delta\\!-\\!||\mathbf{z}^{L}(\mathbf{x}^{*})_{t_{1}}\\!-\\!\mathbf{z}^{L}(\mathbf{x}^{*})_{t_{2}}||_{1}\right)\\!,\text{elsewise}\end{cases}$
(2)

where $\delta$ is a margin hyperparameter. Temporal coherence is enforced at
layer $L$ as in [37]. Although $\underset{coh}{\mathcal{R}}$ can be applied to
any layer $l\in\mathbf{\mathbf{\Lambda}}$, in practice, minimizing (2) for all
layers enforces a very strong regularization, producing synthesized videos
with minimal to no cross-frame variations as shown in Figure 2. We note that
for Transformers using patches of $P^{3}$ resolution, we first reshape
$\textbf{z}^{L}(\mathbf{x}^{*})$ from
$C^{\prime}P^{3}\times\frac{T^{\prime}H^{\prime}W^{\prime}}{P^{3}}$ to
$C^{\prime}\\!\\!\times\\!T^{\prime}\\!\times\\!H^{\prime}W^{\prime}$ before
calculating (2).

###  3.3 Feature Diversity Regularization

Our second regularization term $\underset{feat}{\mathcal{R}}$ is responsible
for improving the diversity of features generated by model priming. Although
priming provides a strong signal based on which the input can be updated,
there is no intuition as to the degree of diversity that the synthesized
features can have. This can cause class features varying from those of the
stimulus, or features not present in the stimulus, to not be explored during
optimization. In order to enhance the search space we introduce an additional
domain-specific verifier network $\mathcal{S}(\cdot)$. Our goal is to use
high- and low-level feature distribution statistics as proposed in [65] to
incorporate the verifier’s prior knowledge during optimization.

Based on input $\mathbf{x}^{*}$, we run inference on the verifier
$\mathcal{S}(\mathbf{x}^{*})$ to obtain representations
$\mathbf{a}^{k}(\mathbf{x}^{*})$ across each verifier layer $k\in\textbf{K}$.
The feature statistics are then obtained by the $C$-length space-time mean
$\mu\left(\textbf{a}^{k}(\mathbf{x}^{*})\right)$ and variance
$\sigma^{2}\left(\textbf{a}^{k}(\mathbf{x}^{*})\right)$ vectors. The feature
diversity regularizer is defined:

$\begin{split}\underset{feat}{\mathcal{R}}(\mathbf{x}^{*})\\!=\\!\underset{l\in\mathbf{L}}{\sum}||\mu\left(\textbf{a}^{k}(\mathbf{x}^{*})\right)-\mathbb{E}\bigl{(}\mu\left(\textbf{a}^{k}(\mathbf{x})\right)|\mathcal{X}\bigr{)}||_{2}+\\\
\underset{l\in\mathbf{L}}{\sum}||\sigma^{2}\left(\textbf{a}^{k}(\mathbf{x}^{*})\right)-\mathbb{E}\bigl{(}\sigma^{2}\left(\textbf{a}^{k}(\mathbf{x})\right)|\mathcal{X}\bigr{)}||_{2}\end{split}$
(3)

where $\mathbb{E}(\cdot)$ corresponds to the expected value of representation
$\mathbf{a}(\cdot)$ for video input $\mathbf{x}$ part of video dataset
$\mathcal{X}$. Instead of requiring access to the dataset to train over
$\mathbf{x}\in\mathcal{X}$ videos, we use the Batch Normalization [22] running
mean and variance to approximate the expected mean and variance as in [65].
The mean and variance estimates are then reformulated as
$\mathbb{E}\bigl{(}\mu\left(\textbf{a}^{k}(\mathbf{x})\right)|\mathcal{X}\bigr{)}\simeq
BN^{k}(\text{running\\_mean})$ and
$\mathbb{E}\bigl{(}\sigma^{2}\left(\textbf{a}^{k}(\mathbf{x})\right)|\mathcal{X}\bigr{)}\simeq
BN^{k}(\text{running\\_variance})$. An illustration of the improved search
space achieved with the inclusion of feature diversity is shown in Figure 3.

Figure 3: Feature diversity regularization. Given the video stimulus in blue,
synthesized videos in orange are optimized only from the stimulus. Synthesized
videos with feature diversity regularization are shown in red. The regularized
videos can also include general class features different from or not existing
in the stimulus.

###  3.4 Aggregation for Model Inversion

We have introduced model priming as a method to obtain a strong signal from a
stimulus video based on which noise-initialized videos can be optimized. To
implicitly enforce feature transition continuity across frames we include a
temporal coherence regularizer. In addition, as the stimulus does not provide
any prior intuition about the diversity of features, we use a feature
diversity regularizer to effectively enhance the search space. Given their
complementary properties, we formulate the final LEAPS objective as the
combination of model priming, alongside the temporal coherence and feature
diversity regularizers. In line with spatial feature visualization losses that
optimize noise into images representing distinguishable features [1, 37, 48,
65], we include a cross-entropy loss $\underset{CE}{\mathcal{L}}$ based on the
network predictions given the synthesized input $\mathbf{x}^{*}$:

$\mathcal{L}(\mathbf{x}^{*},\mathbf{v},y)\\!=\\!\underset{CE}{\mathcal{L}}(\mathbf{x}^{*},y)+\underset{prim}{\mathcal{L}}(\mathbf{x}^{*},\mathbf{v})+r\mathcal{R}(\mathbf{x}^{*})$
(4)

where the regularizer term combines (2) and (3):
$\mathcal{R}(\mathbf{x}^{*})=\underset{coh}{\mathcal{R}}+\underset{feat}{\mathcal{R}}$.
Respectively, $r$ is a regularizer scaling factor. The final LEAPS objective
enables the synthesis of high-fidelity features without being constrained to
data availability or architecture types as shown in the results section.

##  4 Main Results

We first detail the train scheme and implementation settings used in Section
4.1. We then compare our proposed LEAPS method to image-based feature
visualization methods that we extend to video and use as baselines in Section
4.2. We also qualitatively and quantitatively investigate the synthesized
videos across a variety of video models in Section 4.3. Finally, in Section
4.4 we perform ablation studies for the LEAPS objective as well as different
updatable input spatiotemporal resolutions.

###  4.1 Experimental details

Model details. We invert 3D [20]/(2+1)D [59]/CSN [58] ResNet-50 (R50), X3D
[11], TimeSformer [4], Video Swin [29], MViTv2 [27], rev-MViT [32], and
UniFormerv2 [26] networks. For all our experiments, we use the official
networks available from their respective repositories pretrained on
Kinetics-400 [7]. Due to its limited computational overhead and requirement of
BN layers by the feature diversity regularizer in (3), we use
X3D${}_{\text{S}}$ as the verifier network $\mathcal{S}(\cdot)$. We note that
only the internal representations and predictions from the inverted and
verifier models are used. Both models only run inference and remain fixed
throughout the feature synthesis.

Feature synthesis optimization details. Video input $\mathbf{x}^{*}$ is
initialized with size of $8\times 224^{2}$. For models [11, 4, 29, 27, 32, 26]
which use inputs of fixed size, we first interpolate $\mathbf{x}^{*}$ to match
the required size222Due to space limitations in Figures 5,6,7,and 8 the last 2
frames of $\mathbf{x}^{*}$ are not shown.. Priming stimuli are selected from
the Kinetics-400 validation set randomly. For transformer models,
$\mathbf{x}^{*}$ and $\mathbf{v}$ are first tokenized. We use Adam [25] with a
learning rate of 0.2 and a cosine decrease policy as in [65]. We use a total
of 2K gradient updates. As in [37], we set $\delta=1$. To discover the optimal
$\lambda$ and $r$ hyperparameters for each network we use Mango [47] to
perform a simple grid search for 1K gradient updates. A full table of the
hyperparameters used by each model is available in the supplementary material.

###  4.2 Baseline results

We first assess the quality of the synthesized videos. We invert 3D R50,
X3D${}_{\text{M}}$, and Video Swin-B models and report the averaged top-1
classification accuracies and Inception Scores [46] on synthesized videos
using each video from the validation set of Kinetics-400 as a stimulus. We
extend two prominent image-based feature visualization methods making them
applicable to spatiotemporal models:

DeepDream [1] optimizes the input by a cross-entropy loss. It uses two
regularizers including the total variance and $\mathit{l}_{2}$ norm on the
input to improve convergence.

Activation Maximization (AM) [31] optimizes a random noise image by gradient
ascent to maximize the activation of a specific class. We specifically adapt
[13] for visualizing concurrent spatiotemporal representations, as it is the
only prior method for visualizing features over space and time.

Visualization method | top-1 (%) | Inception Score (IS)
---|---|---
model | ver. | model | verifier
3D R50
3D Deep Dream [1] | 34.5 | 2.6 | 1.1 $\pm$ 0.1 | 1.0
3D AM [13] | 41.4 | 5.8 | 1.4 $\pm$ 0.3 | 1.2 $\pm$ 0.2
LEAPS ours ($\underset{prim}{\mathcal{L}}$) | 67.9 | 53.1 | 3.9 $\pm$ 0.9 | 3.2 $\pm$ 0.6
LEAPS ours ($\underset{prim}{\mathcal{L}}$+$\underset{feat}{\mathcal{R}}$) | 74.3 | 60.2 | 5.1 $\pm$ 0.8 | 3.9 $\pm$ 1.4
LEAPS ours (full) | 86.7 | 68.5 | 9.0 $\pm$ 1.0 | 5.7 $\pm$ 0.7
X3D${}_{\text{M}}$
3D Deep Dream [1] | 18.1 | 1.9 | 1.1 $\pm$ 0.1 | 1.1 $\pm$ 0.1
3D AM [13] | 33.2 | 5.6 | 1.3 $\pm$ 0.3 | 1.2 $\pm$ 0.2
LEAPS ours ($\underset{prim}{\mathcal{L}}$) | 73.4 | 59.8 | 5.1 $\pm$ 1.1 | 3.9 $\pm$ 0.4
LEAPS ours ($\underset{prim}{\mathcal{L}}$+$\underset{feat}{\mathcal{R}}$) | 81.1 | 69.3 | 8.8 $\pm$ 1.5 | 6.3 $\pm$ 0.8
LEAPS ours (full) | 90.3 | 82.5 | 11.4 $\pm$ 0.9 | 8.0 $\pm$ 1.4
Video Swin-B
3D Deep Dream [1] | 15.9 | 1.4 | 1.4 $\pm$ 0.2 | 1.1 $\pm$ 0.1
3D AM [13] | 25.6 | 2.2 | 1.6 $\pm$ 0.4 | 1.1 $\pm$ 0.1
LEAPS ours ($\underset{prim}{\mathcal{L}}$) | 71.2 | 58.6 | 4.4 $\pm$ 0.8 | 3.5 $\pm$ 0.6
LEAPS ours ($\underset{prim}{\mathcal{L}}$+$\underset{feat}{\mathcal{R}}$) | 76.0 | 65.4 | 5.3 $\pm$ 1.5 | 4.1 $\pm$ 1.1
LEAPS ours (full) | 87.4 | 74.3 | 9.8 $\pm$ 1.3 | 6.5 $\pm$ 0.9

Table 1: Quantitative results for mean top-1 accuracy and Inception Score
(IS). The best results per metric are in bold and per architecture are
underlined.

Quantitative evaluation. Table 1 shows the average top-1 accuracies and
Inception Scores (IS) obtained by both the inverted models and verifier when
inferring synthesized videos. For DeepDream and AM that do not use priming,
the statistics are averaged across 10 runs per class. In LEAPS the statistics
are calculated using each video in the Kinetics validation set as stimulus.
For both measures, LEAPS yields consistently higher accuracies and IS compared
to DeepDream and AM. Notably, it significantly improves the verifier accuracy
across all architectures. This demonstrates the merits of model priming as
both DeepDream and AM optimize updatable inputs based solely on maximizing
feature or class activations, without using the information-rich internal
representations provided by the stimulus video. This trend is also visible in
the Inception Scores obtained, as LEAPS additionally regularizes the
synthesized features in order to better represent learned temporally coherent
motions.

(a) 3D DeepDream [1]

(b) 3D AM [13]

Time

(c) stimulus video

(d) LEAPS

Figure 4: Different feature synthesis methods for visualizing
X3D${}_{\text{M}}$ features corresponding to class _salsa dancing_. Stimulus
video is only used for LEAPS.
\begin{overpic}[width=433.62pt]{feature_diversity.pdf}
\put(1.4,44.0){\footnotesize{3D
AM~{}\cite[cite]{[\@@bibref{}{feichtenhofer2020deep}{}{}]}}}
\put(1.4,34.0){\footnotesize{LEAPS}} \put(2.0,8.0){\footnotesize{stimulus}}
\end{overpic} Figure 5: Feature synthesis over different runs for action class
_bartending_. MViTv2 features are used based over different runs based on the
same priming stimulus.

\begin{overpic}[trim=0.0pt 42.67912pt 0.0pt
0.0pt,clip,width=496.85625pt]{leaps_examples/leaps_examples_1.pdf}
\put(1.0,21.0){\rotatebox[origin={c}]{90.0}{\small{3D
R50~{}\cite[cite]{[\@@bibref{}{hara2018can}{}{}]}}}}
\put(51.0,21.0){\rotatebox[origin={c}]{90.0}{\small{(2+1)D
R50~{}\cite[cite]{[\@@bibref{}{tran2018closer}{}{}]}}}}
\put(1.0,8.5){\rotatebox[origin={c}]{90.0}{\small{X3D${}_{\text{M}}$~{}\cite[cite]{[\@@bibref{}{feichtenhofer2020x3d}{}{}]}}}}
\put(51.0,6.7){\rotatebox[origin={c}]{90.0}{\footnotesize{Video
Swin-B~{}\cite[cite]{[\@@bibref{}{liu2022video}{}{}]}}}}
\end{overpic}\begin{overpic}[trim=0.0pt 17.07182pt 0.0pt
31.2982pt,clip,width=496.85625pt]{leaps_examples/leaps_examples_2.pdf}
\put(1.0,21.0){\rotatebox[origin={c}]{90.0}{\small{TimesFormer~{}\cite[cite]{[\@@bibref{}{bertasius2021space}{}{}]}}}}
\put(51.0,21.0){\rotatebox[origin={c}]{90.0}{\small{MViTv2-B~{}\cite[cite]{[\@@bibref{}{li2022mvitv2}{}{}]}}}}
\put(1.0,6.7){\rotatebox[origin={c}]{90.0}{\small{rev-
MViT-B~{}\cite[cite]{[\@@bibref{}{mangalam2022reversible}{}{}]}}}}
\put(51.0,6.7){\rotatebox[origin={c}]{90.0}{\small{UniFormerv2~{}\cite[cite]{[\@@bibref{}{li2022uniformerv2}{}{}]}}}}
\end{overpic}

Figure 6: Qualitative examples of synthesized spatiotemporal features with
LEAPS. Models are primed with a stimulus video, shown on the left of each row
of synthesized videos.

(a) stimulus video for doing aerobics

(b) LEAPS synthesized video

Figure 7: SSIM and PSNR statistics over real and synthesized videos using
Video Swin-B features. Lower values correspond to larger cross-frame
differences.

Quantitative examples. Resulting synthesized videos by different methods are
shown in Figure 4 for the class _salsa spin_. Features synthesized with LEAPS
are significantly more visually distinct compared to those of baseline
methods. As shown, videos produced by image-based visualization methods
extended to videos fail to represent the learned spatiotemporal deep features
corresponding to a specific action class. The quality of the synthesized
features also correlates with the accuracies and Inception Scores in Table 1.

Figure 5 illustrates the synthesized videos from different runs given the same
stimulus for class _bartending_. A substantial difference in the quality and
representability of the visualizations between LEAPS and the extended 3D AM
can be seen across runs. Despite the use of a video stimulus to prime the
network, LEAPS visualizations are not constrained by the representations of
the stimulus video. Each run of our LEAPS method shows a distinct visual style
as feature diversity is encouraged during optimization through the homonym
regularizer term. Effectively, LEAPS can be used as a tool for investigating
the different class-specific features learned by each model.

###  4.3 Analysis of LEAPS synthesized video

The generalizability of feature visualization methods across multiple
architectures is largely neglected with features from only a small subset of
models being visualized. To evaluate the architecture-agnostic nature of LEAPS
and better understand the visual fidelity of the synthesized videos to learned
concepts, we investigate the applicability of LEAPS over a range of
spatiotemporal convolutional and attention-based models.

Generalizability. The quality of the produced feature visualizations may
depend on the architecture used, as models vary in terms of their complexities
and feature spaces that they employ. We compare our proposed LEAPS
visualization method by inverting common video models shown in Figure 6 for a
few arbitrary Kinetics classes. A common theme that arises for all models is
the association of objects with specific actions. For example, the feature
visualizations for the throwing ball, catching/throwing frisbee, eating
spaghetti, and snatch weight lifting actions from inverted features of 3D R50,
(2+1)D R50, Video Swin-B, and rev-MViT-B respectively, all optimize the video
to primarily focus on the objects associated with the specific actions.
Instead, for actions that are primarily perceived by motions performed, e.g.
running on treadmill and dancing ballet, the actors/performers of the target
actions are shown to be more conceptually influential to the model’s learned
preconscious of an action, with their associated motions and movements
captured by the produced visualizations. In addition, the visualizations for
cases eating spaghetti, blowing out candles, and sword fighting reveal that
networks also learn to temporally bound distinct spatiotemporal features of
certain actions. This is an important ability to be learned by video models as
it effectively demonstrates their capacity to filter temporal information
alongside the spatial signal. Notably, the LEAPS visualizations for both
convolutional and attention-based models show that learned features have a
good correspondence to their classes across architectures.

Temporal coherence. As we show in Figure 7, LEAPS can visualize motions
performed over different speeds (slow/fast). We report the Peak Signal Noise
Ratio (PSNR) and Structural Similarity Index Measure (SSIM) for consecutive
frame pairs to analyze the variations observed by both fast and slow motions
within a video. We observe that the speeds with which motions are performed
within the same video can be fundamentally different with both large and small
changes occurring between frames. In contrast to image-based methods extended
to video, LEAPS shows the ability to represent such variations in the produced
feature visualizations. As shown, the PSNR and SSIM statistics from the
stimulus video of class doing aerobics in Figure 7(a), follow the same trends
as those produced by the synthesized LEAPS video in Figure 7(b).

\begin{overpic}[width=433.62pt]{iterations.pdf} \end{overpic} Figure 8: Video
synthesis at different optimization stages for class riding on a bike.
X3D${}_{\text{M}}$ features are used.

Video synthesis optimization. We visualize the resulting synthesized video
$\mathbf{x}^{*}$ at different iteration steps during optimization in Figure 8.
General features such as the outlines of objects and actors as well as their
movements are synthesized first by our proposed method. Interestingly, later
iterations show to refine the visualized features by further synthesizing
visual details. This demonstrates a level of learned hierarchy by video models
as to the types of spatiotemporal learned features that are associated with a
specific action. Our proposed use of model priming in tandem with temporal
coherence and feature diversity regularization terms shows to visualize
models’ spatiotemporal representations of actions, at a finer quality and
detail.

(a) 3D R50.

Metric | Distance function
---|---
$l_{2}$ | $l_{1}$ | $cos$ | JVS
top-1 (m) | 84.3 | 82.4 | 72.1 |  86.7
top-1 (v) | 65.8 | 63.7 | 46.1 | 68.5

(b) X3D${}_{\text{M}}$.

Metric | Distance function
---|---
$l_{2}$ | $l_{1}$ | $cos$ | JVS
top-1 (m) | 88.1 | 86.8 | 78.9 |  90.3
top-1 (v) | 79.7 | 78.4 | 70.2 | 82.5

(c) Video Swin-B.

Metric | Distance function
---|---
$l_{2}$ | $l_{1}$ | $cos$ | JVS
top-1 (m) | 85.5 | 83.0 | 69.2 |  87.4
top-1 (v) | 72.8 | 71.6 | 41.4 | 74.3

(d) MViTv2-B.

Metric | Distance function
---|---
$l_{2}$ | $l_{1}$ | $cos$ | JVS
top-1 (m) | 83.5 | 81.9 | 70.7 |  85.9
top-1 (v) | 72.4 | 69.7 | 45.6 | 73.1

Table 2: Top-1 accuracies of the inverted model (m) and verifier (v) on
synthesized videos over different priming distance functions. Best results are
in bold.

Metric | Video model architectures and variants
---|---
R50 | X3D [11] | TS [4] | Video Swin [29] | MViTv2 [27] | rev-MViT [32] | UniFormerv2 [26]
3D | (2+1)D [59] | CSN [58] | XS | S | M | L | T | S | B | S | B | B | L
LEAPS $\underset{prim}{\mathcal{L}}$
top-1 (m) | 67.9 | 63.8 | 72.4 | 67.4 | 68.5 | 73.4 | 73.8 | 64.9 | 69.1 | 69.5 | 71.2 | 70.6 | 71.5 | 64.5 | 72.3 | 73.0
top-1 (v) | 53.1 | 49.5 | 55.8 | 51.9 | 68.5 | 59.8 | 60.4 | 52.6 | 52.3 | 53.5 | 57.6 | 54.6 | 56.7 | 50.7 | 56.4 | 56.9
IS | $3.9\\!\pm\\!0.9$ | $2.4\\!\pm\\!0.5$ | $4.2\\!\pm\\!0.6$ | $4.0\\!\pm\\!1.0$ | $4.3\\!\pm\\!0.6$ | $5.1\\!\pm\\!1.1$ | $5.5\\!\pm\\!1.3$ | $2.7\\!\pm\\!0.7$ | $4.1\\!\pm\\!1.6$ | $4.2\\!\pm\\!1.1$ | $4.4\\!\pm\\!0.8$ | $4.3\\!\pm\\!1.2$ | $4.6\\!\pm\\!0.7$ | $3.1\\!\pm\\!1.7$ | $4.3\\!\pm\\!0.8$ |  $4.5\\!\pm\\!1.2$
LEAPS $\underset{prim}{\mathcal{L}}+\underset{coh}{\mathcal{R}}$
top-1 (m) | 70.4 | 65.7 | 76.1 | 74.8 | 75.6 | 78.0 | 78.5 | 73.1 | 72.8 | 73.2 | 74.5 | 73.7 | 74.2 | 69.2 | 74.9 | 75.3
top-1 (v) | 55.8 | 54.3 | 60.1 | 61.0 | 75.6 | 65.6 | 70.0 | 63.4 | 62.9 | 63.3 | 63.8 | 63.5 | 63.9 | 52.9 | 63.7 | 64.5
IS | $4.6\\!\pm\\!1.2$ | $3.5\\!\pm\\!0.8$ | $4.8\\!\pm\\!1.6$ | $4.3\\!\pm\\!0.9$ | $5.0\\!\pm\\!1.5$ | $7.2\\!\pm\\!1.3$ | $7.5\\!\pm\\!1.0$ | $3.9\\!\pm\\!0.9$ | $4.5\\!\pm\\!0.7$ | $4.8\\!\pm\\!1.3$ | $5.4\\!\pm\\!0.6$ | $5.0\\!\pm\\!0.8$ | $5.3\\!\pm\\!1.0$ | $3.3\\!\pm\\!2.1$ | $5.5\\!\pm\\!0.6$ |  $5.9\\!\pm\\!0.4$
LEAPS $\underset{prim}{\mathcal{L}}+\underset{feat}{\mathcal{R}}$
top-1 (m) | 74.3 | 67.4 | 76.2 | 78.9 | 79.4 | 81.1 | 81.5 | 71.9 | 74.1 | 74.7 | 76.0 | 75.6 | 76.3 | 70.4 | 75.8 | 76.6
top-1 (v) | 60.2 | 56.9 | 64.3 | 65.8 | 79.4 | 69.3 | 70.8 | 67.2 | 63.2 | 64.5 | 65.4 | 66.7 | 68.0 | 58.2 | 69.0 | 69.8
IS | $5.1\\!\pm\\!0.8$ | $4.2\\!\pm\\!1.4$ | $5.6\\!\pm\\!1.2$ | $7.3\\!\pm\\!1.1$ | $7.6\\!\pm\\!0.8$ | $8.8\\!\pm\\!1.5$ | $9.1\\!\pm\\!1.2$ | $3.8\\!\pm\\!1.2$ | $4.7\\!\pm\\!1.1$ | $4.9\\!\pm\\!0.5$ | $5.3\\!\pm\\!1.5$ | $5.1\\!\pm\\!0.6$ | $5.5\\!\pm\\!1.2$ | $4.0\\!\pm\\!1.8$ | $5.7\\!\pm\\!1.0$ |  $6.2\\!\pm\\!0.9$
LEAPS (full)
top-1 (m) | 86.7 | 78.0 | 88.3 | 86.2 | 87.0 | 90.3 | 90.8 | 83.6 | 85.7 | 86.2 | 87.4 | 85.1 | 85.9 | 82.5 | 87.1 | 88.3
top-1 (v) | 68.5 | 65.2 | 71.6 | 76.4 | 87.0 | 82.5 | 83.7 | 69.7 | 71.9 | 73.5 | 74.3 | 72.4 | 73.1 | 67.3 | 75.4 | 76.2
IS | $\textbf{9.0}\\!\pm\\!\textbf{1.0}$ | $\textbf{6.4}\\!\pm\\!\textbf{1.3}$ | $\textbf{9.7}\\!\pm\\!\textbf{0.7}$ | $\textbf{9.6}\\!\pm\\!\textbf{0.4}$ | $\textbf{10.4}\\!\pm\\!\textbf{1.2}$ | $\textbf{11.4}\\!\pm\\!\textbf{0.9}$ | $\textbf{11.9}\\!\pm\\!\textbf{1.5}$ | $\textbf{7.5}\\!\pm\\!\textbf{1.6}$ | $\textbf{8.5}\\!\pm\\!\textbf{0.7}$ | $\textbf{9.4}\\!\pm\\!\textbf{1.5}$ | $\textbf{9.8}\\!\pm\\!\textbf{1.3}$ | $\textbf{8.7}\\!\pm\\!\textbf{1.3}$ | $\textbf{9.3}\\!\pm\\!\textbf{0.8}$ | $\textbf{7.1}\\!\pm\\!\textbf{2.6}$ | $\textbf{9.1}\\!\pm\\!\textbf{1.3}$ | $\textbf{9.6}\\!\pm\\!\textbf{1.2}$

Table 3: Top-1 accuracies and Inceprion Scores over different objectives
across different architectures and model variants. X3D${}_{\text{S}}$ is used
as the verifier (v) for all experiments. The best results per variant are in
bold. The verifier accuracy is denoted in gray for the case of using
X3D${}_{\text{S}}$ for both model inversion and as the verifier.

Model | temp.$\times$spatial2 | top-1 | IS | Latency (secs)
---|---|---|---|---
m | v | ($\downarrow$I / $\uparrow$ B)
3D R50 | $8\times 182^{2}$ | 84.3 | 67.2 | $8.1\\!\pm\\!1.3$ |  0.591 / 0.913
$8\times 224^{2}$ | 86.7 | 68.5 | $9.7\\!\pm\\!1.0$ | 0.832 / 1.205
$16\times 224^{2}$ | 87.0 | 68.7 | $\textbf{9.8}\\!\pm\\!\textbf{1.6}$ | 1.140 / 1.681
(2+1)D R50 | $8\times 182^{2}$ | 75.4 | 62.9 | $4.2\\!\pm\\!1.8$ |  1.684/2.325
$8\times 224^{2}$ | 78.0 | 65.2 | $6.4\\!\pm\\!1.3$ | 1.858/2.793
$16\times 224^{2}$ | 78.5 | 65.4 | $\textbf{6.6}\\!\pm\\!\textbf{1.5}$ | 2.343/3.176

Table 4: Synthesized video size comparisons based on the top-1 accuracies of
the inverted model, and verifier, Inception Scores (IS), and latency times for
inference ($\downarrow$I) and backprop ($\uparrow$B). Best settings per
architecture are in bold.

###  4.4 Ablation studies

In this section, we conduct ablation studies reporting model statistics over
synthesized videos. We initially consider the effect of the distance function
used during model priming. Additionally, we report statistics over different
combinations of priming and introduced regularizers across the tested
architectures. Finally, we present quantitative results when using inputs of
different spatiotemporal sizes alongside the resulting averaged latency times.

Priming distance methods. In Table 2, we evaluate the impact of the distance
functions used during model priming at (1). We test three different magnitude-
based methods including $l_{2}$, $l_{1}$, and $JVS$, as well the cosine
similarity $cos$. Across all four spatiotemporal models, 3D R50,
X3D${}_{\text{M}}$, Video Swin-B, and MViTv2-B, the magnitude-based methods
perform favorably over the cosine similarity. This is due to $cos$ not taking
into account the magnitude of the feature activation vectors from the
synthesized and stimulus videos, which in turn limits the ability to
synthesize representations relevant to the stimulus. Across magnitude-based
methods, an average improvement of $+2.2\%$ and $+4.1\%$ to the inverted
model’s (m) accuracy is observed with JVS compared to $l_{2}$ and $l_{1}$
respectively. The same trend also holds true for the verifier (v), with
$+1.9\%$ and $+3.7\%$ accuracy improvements from $l_{2}$ and $l_{1}$ when
using JVS. Comparatively to $l_{2}$ and $l_{1}$, JVS uses a combination of
vector magnitudes and angles [15], providing a more balanced approach than
magnitude-only or angle-only metrics. Therefore, we adopt $JVS$ as the priming
distance method used between stimulus and synthesized feature vectors.

Regularizers. In Table 3 we provide comparisons over different priming and
regularizer combinations for our LEAPS objective. We report the top-1
accuracies of the inverted model (m), and verifier (v), as well as the
Inception Score (IS) of the inverted model across a range of spatiotemporal
convolutional and attention-based architectures. We note that in the case of
inverting X3D${}_{\text{S}}$ the same model is used for both model inversion
and as the verifier, evidently resulting in matching verifier/inverted model
accuracies. Overall, the priming objective $\underset{prim}{\mathcal{L}}$
combined with either regularizer term yields clear improvements compared to
the sole use of model priming. Modest improvements are observed with the
combination of priming and the feature diversity regularizer over priming with
temporal coherence. We believe that this is due to the enhanced search space
of $\underset{prim}{\mathcal{L}}+\underset{feat}{\mathcal{R}}$ as more diverse
class features not in the stimulus are also explored. The combination of the
priming objective with both regularizer terms for our proposed LEAPS
consistently achieves the best results by a large margin compared to all other
settings. LEAPS improves accuracy, for both the inverted model and verifier,
as well as the quality of the generated videos based on the IS. These results
further demonstrate that our proposed LEAPS optimization can be used across a
range of architectures as a general spatiotemporal feature visualization
method. Further qualitative examples for each network over different Kinetics
classes can be found in the supplementary material.

Synthesized video resolution. Finally, we compare the accuracies, Inception
Scores, and latency times when optimizing video inputs of different
spatiotemporal sizes. Due to the majority of architectures requiring fixed-
size inputs, we use 3D R50 and (2+1)D R50 as they are input size independent.
From the top-1 accuracies and IS summarized in Table 4, the best-performing
setting across metrics is obtained with $16\\!\times\\!224^{2}$ inputs. We
observe comparable performance on the temporally-reduced
$8\\!\times\\!224^{2}$ setting with a significantly more balanced performance-
to-latency. Notable decreases in accuracies and IS are observed with the
potentially limited spatial resolution $8\\!\times\\!182^{2}$ setting, making
it less suitable. Due to the significant improvements in the per-iteration
latency of the $8\\!\times\\!224^{2}$ inputs, we use this setting throughout
our experiments.

##  5 Conclusions

We have introduced LEAPS, a novel method for visualizing the learned internal
representations of spatiotemporal deep networks through video synthesis. LEAPS
uses a stimulus video to prime a video model and iteratively optimizes a video
initialized with random noise. The resulting synthesized video inverts the
learned representations of the model to visually conceivable features. During
optimization, LEAPS regularizes the updated video by enforcing temporal
coherence between frames to represent feature transitions. Additionally, it
uses a feature diversity regularizer to improve the diversity of the
synthesized features through an enhanced search space. We have shown both
qualitatively and quantitatively that our proposed method can produce high-
quality and visually coherent synthesized videos over a wide range of
spatiotemporal convolutional and attention-based video models. The high
classification scores and synthesized video quality metrics make LEAPS a
generalizable and effective spatiotemporal feature visualization method. We
believe that this first step towards learned spatiotemporal representations
synthesis is a promising direction in understanding video models.

## References

  * [1] Mordvintsev Alexander, Olah Christopher, and Tyka Mike.  Inceptionism: Going deeper into neural networks.  Google Research Blog, 2015.
  * [2] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid.  Vivit: A video vision transformer.  In International Conference on Computer Vision (ICCV), pages 6836–6846. IEEE, 2021.
  * [3] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  PloS one, 10(7):e0130140, 2015.
  * [4] Gedas Bertasius, Heng Wang, and Lorenzo Torresani.  Is space-time attention all you need for video understanding?  In International Conference on Machine Learning (ICML), pages 813–824. PMLR, 2021.
  * [5] Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojciech Samek.  Layer-wise relevance propagation for neural networks with local renormalization layers.  In International Conference on Artificial Neural Networks (ICANN), pages 63–71. Springer, 2016.
  * [6] Michael Burke.  Leveraging gaussian process approximations for rapid image overlay production.  In ACM Multimedia Workshop (ACMMW), pages 21–26. ACM, 2017.
  * [7] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 6299–6308. IEEE, 2017.
  * [8] Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian.  Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks.  In Winter Conference on Applications of Computer Vision (WACV), pages 839–847. IEEE, 2018.
  * [9] Alexey Dosovitskiy and Thomas Brox.  Inverting visual representations with convolutional networks.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 4829–4837. IEEE, 2016.
  * [10] Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Visualizing higher-layer features of a deep network.  Technical Report 1341-3, University of Montreal, 2009.
  * [11] Christoph Feichtenhofer.  X3D: Expanding architectures for efficient video recognition.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 203–213. IEEE, 2020.
  * [12] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In International Conference on Computer Vision (ICCV), pages 6202–6211. IEEE, 2019.
  * [13] Christoph Feichtenhofer, Axel Pinz, Richard P Wildes, and Andrew Zisserman.  Deep insights into convolutional networks for video recognition.  International Journal of Computer Vision, 128:420–437, 2020.
  * [14] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman.  Convolutional two-stream network fusion for video action recognition.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 1933–1941. IEEE, 2016.
  * [15] Basura Fernando and Samitha Herath.  Anticipating human actions by correlating past with the future with jaccard similarity measures.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 13224–13233. IEEE, 2021.
  * [16] Ruth Fong, Mandela Patrick, and Andrea Vedaldi.  Understanding deep networks via extremal perturbations and smooth masks.  In International Conference on Computer Vision (ICCV), pages 2950–2958. IEEE, 2019.
  * [17] Ruth C Fong and Andrea Vedaldi.  Interpretable explanations of black boxes by meaningful perturbation.  In International Conference on Computer Vision (ICCV), pages 3429–3437. IEEE, 2017.
  * [18] Sigmund Freud and James Ed Strachey.  Vol. XIX The Ego and the Id and other works (1923-1925).  In The Standard Edition of the Complete Psychological Works of Sigmund Freud. the Hogarth Press, 1959.
  * [19] Amin Ghiasi, Hamid Kazemi, Steven Reich, Chen Zhu, Micah Goldblum, and Tom Goldstein.  Plug-in inversion: Model-agnostic inversion for vision with data augmentations.  In International Conference on Machine Learning (ICML), pages 7484–7512. PMLR, 2022.
  * [20] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh.  Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 6546–6555. IEEE, 2018.
  * [21] De-An Huang, Vignesh Ramanathan, Dhruv Mahajan, Lorenzo Torresani, Manohar Paluri, Li Fei-Fei, and Juan Carlos Niebles.  What makes a video a video: Analyzing temporal information in video understanding models and datasets.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 7366–7375. IEEE, 2018.
  * [22] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International Conference on Machine Learning (ICML), pages 448–456. PMLR, 2015.
  * [23] Andrei Kapishnikov, Subhashini Venugopalan, Besim Avci, Ben Wedin, Michael Terry, and Tolga Bolukbasi.  Guided integrated gradients: An adaptive path method for removing noise.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 5050–5058. IEEE, 2021.
  * [24] Atsushi Kikuchi, Kotaro Uchida, Masaki Waga, and Kohei Suenaga.  Borex: Bayesian-optimization–based refinement of saliency map for image-and video-classification models.  In Asian Conference on Computer Vision (ACCV), pages 2092–2108, 2022.
  * [25] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In International Conference on Learning Representations (ICLR), 2015\.
  * [26] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Limin Wang, and Yu Qiao.  Uniformerv2: Spatiotemporal learning by arming image vits with video uniformer.  arXiv preprint arXiv:2211.09552, 2022.
  * [27] Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer.  Mvitv2: Improved multiscale vision transformers for classification and detection.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 4804–4814. IEEE, 2022.
  * [28] Zhenqiang Li, Weimin Wang, Zuoyue Li, Yifei Huang, and Yoichi Sato.  Towards visually explaining video understanding networks with perturbation.  In Winter Conference on Applications of Computer Vision (WACV), pages 1120–1129. IEEE, 2021.
  * [29] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu.  Video swin transformer.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 3202–3211. IEEE, 2022.
  * [30] Aravindh Mahendran and Andrea Vedaldi.  Understanding deep image representations by inverting them.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 5188–5196. IEEE, 2015.
  * [31] Aravindh Mahendran and Andrea Vedaldi.  Visualizing deep convolutional neural networks using natural pre-images.  International Journal of Computer Vision, 120:233–255, 2016.
  * [32] Karttikeya Mangalam, Haoqi Fan, Yanghao Li, Chao-Yuan Wu, Bo Xiong, Christoph Feichtenhofer, and Jitendra Malik.  Reversible vision transformers.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 10830–10840. IEEE, 2022.
  * [33] Anthony J Marcel.  Conscious and unconscious perception: An approach to the relations between phenomenal experience and perceptual processes.  Cognitive psychology, 15(2):238–300, 1983.
  * [34] Anthony J Marcel.  Conscious and unconscious perception: Experiments on visual masking and word recognition.  Cognitive psychology, 15(2):197–237, 1983.
  * [35] Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger.  Umap: Uniform manifold approximation and projection.  Journal of Open Source Software, 3(29):861, 2018.
  * [36] Vivek Miglani, Narine Kokhlikyan, Bilal Alsallakh, Miguel Martin, and Orion Reblitz-Richardson.  Investigating saturation effects in integrated gradients.  In International Conference on Machine Learning Workshops (ICMLW). PMLR, 2020.
  * [37] Hossein Mobahi, Ronan Collobert, and Jason Weston.  Deep learning from temporal coherence in video.  In International Conference on Machine Learning (ICML), pages 737–744. PMLR, 2009.
  * [38] Mamuku Mokuwe, Michael Burke, and Anna Sergeevna Bosman.  Black-box saliency map generation using bayesian optimisation.  In International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2020.
  * [39] James H. Neely.  Priming.  In Encyclopedia of Cognitive Science. Nature Publishing Group, 2003\.
  * [40] Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski.  Plug & play generative networks: Conditional iterative generation of images in latent space.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 4467–4477. IEEE, 2017.
  * [41] Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune.  Synthesizing the preferred inputs for neurons in neural networks via deep generator networks.  Advances in Neural Information Processing Systems (NeurIPs), 2016\.
  * [42] Anh Nguyen, Jason Yosinski, and Jeff Clune.  Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks.  In International Conference of Machine Learning Workshops (ICML). PMLR, 2016.
  * [43] Augustus Odena, Christopher Olah, and Jonathon Shlens.  Conditional image synthesis with auxiliary classifier gans.  In International Conference on Machine Learning (ICML), pages 2642–2651. PMLR, 2017.
  * [44] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert.  Feature visualization.  Distill, 2(11):e7, 2017.
  * [45] Vitali Petsiuk, Abir Das, and Kate Saenko.  Rise: Randomized input sampling for explanation of black-box models.  In British Machine Vision Conference (BMVC). BMVA, 2018.
  * [46] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  In Advances in Neural Information Processing Systems (NeurIPS). PMLR, 2016.
  * [47] Sandeep Singh Sandha, Mohit Aggarwal, Igor Fedorov, and Mani Srivastava.  Mango: A python library for parallel hyperparameter tuning.  In International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3987–3991. IEEE, 2020.
  * [48] Shibani Santurkar, Andrew Ilyas, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry.  Image synthesis with a single (robust) classifier.  Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
  * [49] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In International Conference on Computer Vision (ICCV), pages 618–626. IEEE, 2017.
  * [50] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje.  Learning important features through propagating activation differences.  In International Conference on Machine Learning (ICML), pages 3145–3153. PMLR, 2017.
  * [51] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  arXiv preprint arXiv:1312.6034, 2013.
  * [52] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg.  Smoothgrad: removing noise by adding noise.  arXiv preprint arXiv:1706.03825, 2017.
  * [53] J Springenberg, Alexey Dosovitskiy, Thomas Brox, and M Riedmiller.  Striving for simplicity: The all convolutional net.  In International Conference on Learning Representations Workshops (ICLRW), 2015.
  * [54] Alexandros Stergiou.  The mind’s eye: Visualizing class-agnostic features of cnns.  In International Conference on Image Processing (ICIP), pages 2738–2742. IEEE, 2021.
  * [55] Alexandros Stergiou, Georgios Kapidis, Grigorios Kalliatakis, Christos Chrysoulas, Ronald Poppe, and Remco Veltkamp.  Class feature pyramids for video explanation.  In International Conference on Computer Vision Workshop (ICCVW), pages 4255–4264. IEEE, 2019.
  * [56] Alexandros Stergiou, Georgios Kapidis, Grigorios Kalliatakis, Christos Chrysoulas, Remco Veltkamp, and Ronald Poppe.  Saliency tubes: Visual explanations for spatio-temporal convolutions.  In International Conference on Image Processing (ICIP), pages 1830–1834. IEEE, 2019.
  * [57] Mukund Sundararajan, Ankur Taly, and Qiqi Yan.  Axiomatic attribution for deep networks.  In International Conference on Machine Learning (ICML), pages 3319–3328. PMLR, 2017.
  * [58] Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feiszli.  Video classification with channel-separated convolutional networks.  In International Conference on Computer Vision(ICCV), pages 5552–5561. IEEE, 2019.
  * [59] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri.  A closer look at spatiotemporal convolutions for action recognition.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 6450–6459. IEEE, 2018.
  * [60] Tomoki Uchiyama, Naoya Sogi, Koichiro Niinuma, and Kazuhiro Fukui.  Visually explaining 3d-cnn predictions for video classification with an adaptive occlusion sensitivity analysis.  In Winter Conference on Applications of Computer Vision (WACV), pages 1513–1522. IEEE, 2023.
  * [61] Feng Wang, Haijun Liu, and Jian Cheng.  Visualizing deep neural network by alternately image blurring and deblurring.  Neural Networks, 97:162–172, 2018.
  * [62] Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu.  Score-cam: Score-weighted visual explanations for convolutional neural networks.  In Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 24–25, 2020.
  * [63] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool.  Temporal segment networks: Towards good practices for deep action recognition.  In European Conference on Computer Vision (ECCV), pages 20–36. Springer, 2016.
  * [64] Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid.  Multiview transformers for video recognition.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 3333–3343. IEEE, 2022.
  * [65] Hongxu Yin, Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz.  Dreaming to distill: Data-free knowledge transfer via deepinversion.  In Conference on Computer Vision and Pattern Recognition (CVPR), pages 8715–8724. IEEE, 2020.
  * [66] Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson.  Understanding neural networks through deep visualization.  In International Conference of Machine Learning Workshops (ICML). PMLR, 2015.
  * [67] Matthew D Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In European conference on computer vision (ECCV), pages 818–833. Springer, 2014.

Leaping Into Memories: Space-Time Deep Feature Synthesis
Supplementary material

##  S1-0 Additional Qualitative results

We demonstrated and overviewed qualitative results of inverted spatiotemporal
models with LEAPS in Section 4.2. Supplementary to Figure 4, we provide
additional results in order to visualize the features synthesized from
different encoders when using the same action labels and stimuli. As shown in
Figures S1-0, S3-0, S5-0 and S7-0 LEAPS can synthesize coherent visual
features and effectively invert learned representations, independently of the
spatiotemporal architecture used. Similar to the synthesized videos in Figure
4, for actions that are best described by the objects used e.g. _juggling
balls_ , _dribbling basketball_ , and _playing trumpet_ , all models optimize
the input video to represent both class-relevant objects as well as actor-
object interaction. Importantly, the synthesized videos show that video models
learn motions with respect to both objects as well as actors. For the
synthesized videos of _juggling balls_ in Figure S1-0, the balls are primarily
shown to be thrown upwards. In contrast, for the _dribbling basketball_ videos
in Figure S5-0, basketballs are bouncing on the side of the actor. In
addition, evidence of LEAPS’s ability to synthesize class-relevant features
can be seen in Figure S7-0 where at the _playing trumpet_ stimulus used, the
better half of the trumpet is occluded. For actions that do not include
objects; e.g. _baby crawling_ in Figure S3-0, the synthesized videos primarily
focus on the actor.

Based on the videos from inverted models presented in Figures S1-0, S3-0, S5-0
and S7-0 there are no significant differences as to the objects and actors
that are synthesized. However, the level of detail in the synthesized videos
is shown to correlate with the model complexity. Specifically for _baby
crawling_ and _playing trumpet_ synthesized videos from inverted models of
increased capacities; e.g. X3D, Swin, and MViTv2 contain more visually
distinct concepts than those of smaller architectures; e.g. 3D/(2+1)D
Resnet-50. The effect is in line with the resulting synthesized videos from
inverted models in Figure 6. Overall, LEAPS can invert models of varying
complexities while also visualizing feature details based on the model’s
feature space capacity.

Model | $\lambda_{1}$ | $\lambda_{L}$ | $r$ |  $\mathcal{L}$
---|---|---|---|---
3D R50 | 1.0 | 0.3 | $7.5e^{-3}$ | 7.892
(2+1)D R50 | 0.75 | 0.1 | $5e^{-3}$ | 6.421
CSN R50 | 1.0 | 0.2 | $5e^{-3}$ | 6.603
X3D${}_{\text{XS}}$ | 1.0 | 0.2 | $1e^{-3}$ | 5.175
X3D${}_{\text{S}}$ | 1.0 | 0.1 | $1e^{-3}$ | 5.538
X3D${}_{\text{M}}$ | 0.75 | 0.1 | $1e^{-3}$ | 6.387
X3D${}_{\text{L}}$ | 0.75 | 0.1 | $1e^{-3}$ | 7.190
TimeSformer | 1.0 | 0.2 | $2.5e^{-3}$ | 5.629
Video Swin-T | 0.75 | 0.2 | $1e^{-3}$ | 6.527
Video Swin-S | 0.75 | 0.1 | $1e^{-3}$ | 7.508
Video Swin-B | 0.625 | 0.1 | $1e^{-3}$ | 8.841
MViTv2-S | 0.75 | 0.1 | $2.5e^{-3}$ | 7.356
MViTv2-B | 0.75 | 0.1 | $1e^{-3}$ | 8.195
rev-MViT | 0.625 | 0.1 | $5e^{-3}$ | 7.227
UniFormerv2-B | 1.0 | 0.2 | $2.5e^{-3}$ | 6.053
UniFormerv2-L | 1.0 | 0.1 | $1e^{-3}$ | 7.415
Table S1: LEAPS optimization hyperparameters based on grid search. We
additionally report the average loss on synthesized videos from the Kinetics
validation set. \begin{overpic}[trim=0.0pt 0.0pt 0.0pt
0.0pt,clip,width=496.85625pt]{supplementary/qualitative_sup_juggling_1.pdf}
\par\put(0.0,94.4){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;\\!$stimulus
video$\;\\!$}}}}}
\par\put(0.0,83.3){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;\;$3D
R50~{}\cite[cite]{[\@@bibref{}{hara2018can}{}{}]}$\;\;$}}}}}
\par\put(0.0,72.2){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\footnotesize{{\color[rgb]{1,1,1}$\;\\!$(2+1)D
R50~{}\cite[cite]{[\@@bibref{}{tran2018closer}{}{}]}$\;\\!$}}}}}
\par\put(0.0,60.9){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;$CSN
R50~{}\cite[cite]{[\@@bibref{}{tran2019video}{}{}]}$\;$}}}}}
\par\put(0.0,49.7){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;\;$X3D${}_{XS}$~{}\cite[cite]{[\@@bibref{}{feichtenhofer2020x3d}{}{}]}$\;\;$}}}}}
\par\put(0.0,38.5){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\quad$X3D${}_{S}$~{}\cite[cite]{[\@@bibref{}{feichtenhofer2020x3d}{}{}]}$\quad\\!$}}}}}
\par\put(0.0,27.3){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;\;$X3D${}_{M}$~{}\cite[cite]{[\@@bibref{}{feichtenhofer2020x3d}{}{}]}$\;\;\;$}}}}}
\par\put(0.0,16.1){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;\;\;$X3D${}_{L}$~{}\cite[cite]{[\@@bibref{}{feichtenhofer2020x3d}{}{}]}$\;\;\;$}}}}}
\par\put(0.0,4.9){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\footnotesize{{\color[rgb]{1,1,1}TimeSformer~{}\cite[cite]{[\@@bibref{}{bertasius2021space}{}{}]}$\,$}}}}}
\par\end{overpic} Figure S1-0: Qualitative examples of synthesized features
with LEAPS for action label _juggling_. \begin{overpic}[trim=0.0pt 0.0pt 0.0pt
0.0pt,clip,width=496.85625pt]{supplementary/qualitative_sup_juggling_2.pdf}
\par\put(0.0,94.4){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;\\!$stimulus
video$\;\\!$}}}}}
\par\put(0.0,83.3){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\scriptsize{{\color[rgb]{1,1,1}$\;$Video
Swin-T~{}\cite[cite]{[\@@bibref{}{liu2022video}{}{}]}$\;$}}}}}
\par\put(0.0,72.2){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\scriptsize{{\color[rgb]{1,1,1}$\;$Video
Swin-S~{}\cite[cite]{[\@@bibref{}{liu2022video}{}{}]}$\;$}}}}}
\par\put(0.0,61.0){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\scriptsize{{\color[rgb]{1,1,1}$\,$Video
Swin-B~{}\cite[cite]{[\@@bibref{}{liu2022video}{}{}]}$\;$}}}}}
\par\put(0.0,49.7){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}MViTv2-S~{}\cite[cite]{[\@@bibref{}{li2022mvitv2}{}{}]}}}}}}
\par\put(0.0,38.5){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}MViTv2-B~{}\cite[cite]{[\@@bibref{}{li2022mvitv2}{}{}]}}}}}}
\par\put(0.0,27.3){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}rev-
MViT~{}\cite[cite]{[\@@bibref{}{mangalam2022reversible}{}{}]}$\,$}}}}}
\par\put(0.0,16.2){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\scriptsize{$\\!${\color[rgb]{1,1,1}UniFormerv2-B~{}\cite[cite]{[\@@bibref{}{li2022uniformerv2}{}{}]}$\\!$}}}}}
\par\put(0.0,5.0){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\scriptsize{$\\!${\color[rgb]{1,1,1}UniFormerv2-L~{}\cite[cite]{[\@@bibref{}{li2022uniformerv2}{}{}]}$\\!$}}}}}
\par\end{overpic} Figure S2-0: Qualitative examples of synthesized features
with LEAPS for action label _juggling_. \begin{overpic}[trim=0.0pt 0.0pt 0.0pt
0.0pt,clip,width=496.85625pt]{supplementary/qualitative_sup_crawling_1.pdf}
\par\put(0.0,94.4){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;\\!$stimulus
video$\;\\!$}}}}}
\par\put(0.0,83.3){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;\;$3D
R50~{}\cite[cite]{[\@@bibref{}{hara2018can}{}{}]}$\;\;$}}}}}
\par\put(0.0,72.2){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\footnotesize{{\color[rgb]{1,1,1}$\;\\!$(2+1)D
R50~{}\cite[cite]{[\@@bibref{}{tran2018closer}{}{}]}$\;\\!$}}}}}
\par\put(0.0,60.9){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;$CSN
R50~{}\cite[cite]{[\@@bibref{}{tran2019video}{}{}]}$\;$}}}}}
\par\put(0.0,49.7){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;\;$X3D${}_{XS}$~{}\cite[cite]{[\@@bibref{}{feichtenhofer2020x3d}{}{}]}$\;\;$}}}}}
\par\put(0.0,38.5){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\quad$X3D${}_{S}$~{}\cite[cite]{[\@@bibref{}{feichtenhofer2020x3d}{}{}]}$\quad\\!$}}}}}
\par\put(0.0,27.3){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;\;$X3D${}_{M}$~{}\cite[cite]{[\@@bibref{}{feichtenhofer2020x3d}{}{}]}$\;\;\;$}}}}}
\par\put(0.0,16.1){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;\;\;$X3D${}_{L}$~{}\cite[cite]{[\@@bibref{}{feichtenhofer2020x3d}{}{}]}$\;\;\;$}}}}}
\par\put(0.0,4.9){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\footnotesize{{\color[rgb]{1,1,1}TimeSformer~{}\cite[cite]{[\@@bibref{}{bertasius2021space}{}{}]}$\,$}}}}}
\par\end{overpic} Figure S3-0: Qualitative examples of synthesized features
with LEAPS for action label _baby crawling_. \begin{overpic}[trim=0.0pt 0.0pt
0.0pt
0.0pt,clip,width=496.85625pt]{supplementary/qualitative_sup_crawling_2.pdf}
\par\put(0.0,94.4){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;\\!$stimulus
video$\;\\!$}}}}}
\par\put(0.0,83.3){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\scriptsize{{\color[rgb]{1,1,1}$\;$Video
Swin-T~{}\cite[cite]{[\@@bibref{}{liu2022video}{}{}]}$\;$}}}}}
\par\put(0.0,72.2){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\scriptsize{{\color[rgb]{1,1,1}$\;$Video
Swin-S~{}\cite[cite]{[\@@bibref{}{liu2022video}{}{}]}$\;$}}}}}
\par\put(0.0,61.0){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\scriptsize{{\color[rgb]{1,1,1}$\,$Video
Swin-B~{}\cite[cite]{[\@@bibref{}{liu2022video}{}{}]}$\;$}}}}}
\par\put(0.0,49.7){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}MViTv2-S~{}\cite[cite]{[\@@bibref{}{li2022mvitv2}{}{}]}}}}}}
\par\put(0.0,38.5){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}MViTv2-B~{}\cite[cite]{[\@@bibref{}{li2022mvitv2}{}{}]}}}}}}
\par\put(0.0,27.3){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}rev-
MViT~{}\cite[cite]{[\@@bibref{}{mangalam2022reversible}{}{}]}$\,$}}}}}
\par\put(0.0,16.2){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\scriptsize{$\\!${\color[rgb]{1,1,1}UniFormerv2-B~{}\cite[cite]{[\@@bibref{}{li2022uniformerv2}{}{}]}$\\!$}}}}}
\par\put(0.0,5.0){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\scriptsize{$\\!${\color[rgb]{1,1,1}UniFormerv2-L~{}\cite[cite]{[\@@bibref{}{li2022uniformerv2}{}{}]}$\\!$}}}}}
\par\end{overpic} Figure S4-0: Qualitative examples of synthesized features
with LEAPS for action label _baby crawling_. \begin{overpic}[trim=0.0pt 0.0pt
0.0pt
0.0pt,clip,width=496.85625pt]{supplementary/qualitative_sup_dribbling_1.pdf}
\par\put(0.0,94.4){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;\\!$stimulus
video$\;\\!$}}}}}
\par\put(0.0,83.3){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;\;$3D
R50~{}\cite[cite]{[\@@bibref{}{hara2018can}{}{}]}$\;\;$}}}}}
\par\put(0.0,72.2){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\footnotesize{{\color[rgb]{1,1,1}$\;\\!$(2+1)D
R50~{}\cite[cite]{[\@@bibref{}{tran2018closer}{}{}]}$\;\\!$}}}}}
\par\put(0.0,60.9){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;$CSN
R50~{}\cite[cite]{[\@@bibref{}{tran2019video}{}{}]}$\;$}}}}}
\par\put(0.0,49.7){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;\;$X3D${}_{XS}$~{}\cite[cite]{[\@@bibref{}{feichtenhofer2020x3d}{}{}]}$\;\;$}}}}}
\par\put(0.0,38.5){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\quad$X3D${}_{S}$~{}\cite[cite]{[\@@bibref{}{feichtenhofer2020x3d}{}{}]}$\quad\\!$}}}}}
\par\put(0.0,27.3){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;\;$X3D${}_{M}$~{}\cite[cite]{[\@@bibref{}{feichtenhofer2020x3d}{}{}]}$\;\;\;$}}}}}
\par\put(0.0,16.1){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;\;\;$X3D${}_{L}$~{}\cite[cite]{[\@@bibref{}{feichtenhofer2020x3d}{}{}]}$\;\;\;$}}}}}
\par\put(0.0,4.9){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\footnotesize{{\color[rgb]{1,1,1}TimeSformer~{}\cite[cite]{[\@@bibref{}{bertasius2021space}{}{}]}$\,$}}}}}
\par\end{overpic} Figure S5-0: Qualitative examples of synthesized features
with LEAPS for action label _dribbling basketball_. \begin{overpic}[trim=0.0pt
0.0pt 0.0pt
0.0pt,clip,width=496.85625pt]{supplementary/qualitative_sup_dribbling_2.pdf}
\par\put(0.0,94.4){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;\\!$stimulus
video$\;\\!$}}}}}
\par\put(0.0,83.3){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\scriptsize{{\color[rgb]{1,1,1}$\;$Video
Swin-T~{}\cite[cite]{[\@@bibref{}{liu2022video}{}{}]}$\;$}}}}}
\par\put(0.0,72.2){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\scriptsize{{\color[rgb]{1,1,1}$\;$Video
Swin-S~{}\cite[cite]{[\@@bibref{}{liu2022video}{}{}]}$\;$}}}}}
\par\put(0.0,61.0){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\scriptsize{{\color[rgb]{1,1,1}$\,$Video
Swin-B~{}\cite[cite]{[\@@bibref{}{liu2022video}{}{}]}$\;$}}}}}
\par\put(0.0,49.7){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}MViTv2-S~{}\cite[cite]{[\@@bibref{}{li2022mvitv2}{}{}]}}}}}}
\par\put(0.0,38.5){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}MViTv2-B~{}\cite[cite]{[\@@bibref{}{li2022mvitv2}{}{}]}}}}}}
\par\put(0.0,27.3){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}rev-
MViT~{}\cite[cite]{[\@@bibref{}{mangalam2022reversible}{}{}]}$\,$}}}}}
\par\put(0.0,16.2){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\scriptsize{$\\!${\color[rgb]{1,1,1}UniFormerv2-B~{}\cite[cite]{[\@@bibref{}{li2022uniformerv2}{}{}]}$\\!$}}}}}
\par\put(0.0,5.0){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\scriptsize{$\\!${\color[rgb]{1,1,1}UniFormerv2-L~{}\cite[cite]{[\@@bibref{}{li2022uniformerv2}{}{}]}$\\!$}}}}}
\par\end{overpic} Figure S6-0: Qualitative examples of synthesized features
with LEAPS for action label _dribbling basketball_. \begin{overpic}[trim=0.0pt
0.0pt 0.0pt
0.0pt,clip,width=496.85625pt]{supplementary/qualitative_sup_trumpet_1.pdf}
\par\put(0.0,94.4){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;\\!$stimulus
video$\;\\!$}}}}}
\par\put(0.0,83.3){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;\;$3D
R50~{}\cite[cite]{[\@@bibref{}{hara2018can}{}{}]}$\;\;$}}}}}
\par\put(0.0,72.2){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\footnotesize{{\color[rgb]{1,1,1}$\;\\!$(2+1)D
R50~{}\cite[cite]{[\@@bibref{}{tran2018closer}{}{}]}$\;\\!$}}}}}
\par\put(0.0,60.9){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;$CSN
R50~{}\cite[cite]{[\@@bibref{}{tran2019video}{}{}]}$\;$}}}}}
\par\put(0.0,49.7){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;\;$X3D${}_{XS}$~{}\cite[cite]{[\@@bibref{}{feichtenhofer2020x3d}{}{}]}$\;\;$}}}}}
\par\put(0.0,38.5){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\quad$X3D${}_{S}$~{}\cite[cite]{[\@@bibref{}{feichtenhofer2020x3d}{}{}]}$\quad\\!$}}}}}
\par\put(0.0,27.3){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;\;$X3D${}_{M}$~{}\cite[cite]{[\@@bibref{}{feichtenhofer2020x3d}{}{}]}$\;\;\;$}}}}}
\par\put(0.0,16.1){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;\;\;$X3D${}_{L}$~{}\cite[cite]{[\@@bibref{}{feichtenhofer2020x3d}{}{}]}$\;\;\;$}}}}}
\par\put(0.0,4.9){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\footnotesize{{\color[rgb]{1,1,1}TimeSformer~{}\cite[cite]{[\@@bibref{}{bertasius2021space}{}{}]}$\,$}}}}}
\par\end{overpic} Figure S7-0: Qualitative examples of synthesized features
with LEAPS for action label _playing trumpet_. \begin{overpic}[trim=0.0pt
0.0pt 0.0pt
0.0pt,clip,width=496.85625pt]{supplementary/qualitative_sup_trumpet_2.pdf}
\par\put(0.0,94.4){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}$\;\\!$stimulus
video$\;\\!$}}}}}
\par\put(0.0,83.3){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\scriptsize{{\color[rgb]{1,1,1}$\;$Video
Swin-T~{}\cite[cite]{[\@@bibref{}{liu2022video}{}{}]}$\;$}}}}}
\par\put(0.0,72.2){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\scriptsize{{\color[rgb]{1,1,1}$\;$Video
Swin-S~{}\cite[cite]{[\@@bibref{}{liu2022video}{}{}]}$\;$}}}}}
\par\put(0.0,61.0){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\scriptsize{{\color[rgb]{1,1,1}$\,$Video
Swin-B~{}\cite[cite]{[\@@bibref{}{liu2022video}{}{}]}$\;$}}}}}
\par\put(0.0,49.7){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}MViTv2-S~{}\cite[cite]{[\@@bibref{}{li2022mvitv2}{}{}]}}}}}}
\par\put(0.0,38.5){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}MViTv2-B~{}\cite[cite]{[\@@bibref{}{li2022mvitv2}{}{}]}}}}}}
\par\put(0.0,27.3){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\small{{\color[rgb]{1,1,1}rev-
MViT~{}\cite[cite]{[\@@bibref{}{mangalam2022reversible}{}{}]}$\,$}}}}}
\par\put(0.0,16.2){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\scriptsize{$\\!${\color[rgb]{1,1,1}UniFormerv2-B~{}\cite[cite]{[\@@bibref{}{li2022uniformerv2}{}{}]}$\\!$}}}}}
\par\put(0.0,5.0){\rotatebox[origin={c}]{90.0}{\hbox{\pagecolor{black}\scriptsize{$\\!${\color[rgb]{1,1,1}UniFormerv2-L~{}\cite[cite]{[\@@bibref{}{li2022uniformerv2}{}{}]}$\\!$}}}}}
\par\end{overpic} Figure S8-0: Qualitative examples of synthesized features
with LEAPS for action label _playing trumpet_.

##  S2-0 Hyperparameter settings

As described in Section 4.1, we discover the optimal $\lambda$ and $r$
hyperparameters for each model through grid search. To limit the search space
and computational overhead of hyperparameter tuning, we define
$\lambda_{1}\\!\in\\!\\{0.5,0.625,0.75,0.875,1.0\\}$,
$\lambda_{L}\\!\in\\!\\{0.1,0.2,0.3,0.4,0.5\\}$,
$r\\!\in\\!\\{1e^{-3},2.5e^{-3},5e^{-3},7.5^{e-3},1e^{-2}\\}$, where
$\lambda_{1}$ is the priming weight for the first layer of the network,
$\lambda_{L}$ is the priming weight for the final layer of the network. Based
on $\lambda_{1}$ and $\lambda_{L}$, we use a linear (decreasing) function for
the remaining $\lambda\in\\{2,...,L-1\\}$ layer priming weights. Table S1
provides a full list of the hyperparameters discovered and used for inverting
each model. We note that the loss shows to increase in larger models due to
the number of layers used for priming.

\begin{overpic}[width=433.62pt]{supplementary/UMAP.pdf}
\par\put(30.0,0.0){\small{First principal component}}
\put(3.5,35.0){\rotatebox[origin={c}]{90.0}{\small{Second principal
component}}} \put(82.5,16.0){\scriptsize{$\underset{feat}{\mathcal{R}}$}}
\par\end{overpic} Figure S9-0: Projection of X3D${}_{\textbf{M}}$’s final
encoder layer embeddings onto two principal components for Kinetic’s class
_tai chi_. Embeddings of videos from Kinetics are in blue, from LEAPS w/o
$\underset{feat}{\mathcal{R}}$ in orange, and from LEAPS in red.

##  S3-0 Embedding space visualizations

LEAPS aims to synthesize visually coherent representations of inverted models.
To better understand the relationship between the inverted model’s features
and real videos from the Kinetics-400 train set, we provide UMAP [35]
visualizations of their feature embeddings for action _tai chi_. We use the
spatiotemporally averaged feature vectors from the final convolution block in
X3D${}_{\text{M}}$ (s5.pathway0_res6.branch2.c).

As illustrated from the results in Figure S9-0, inverted model embeddings are
within the distribution of embeddings from Kinetics videos. While this is true
for both embeddings from LEAPS synthesized videos as well as LEAPS synthesized
videos without feature diversity regularization, LEAPS videos show a greater
level of variation without being as closely concentrated as the embeddings of
LEAPS w/o $\underset{feat}{\mathcal{R}}$.

Priming | top-1 (%) | Inception Score (IS)
---|---|---
layers (%) | model | ver. | model | verifier
3D R50
20 | 19.0 | 4.1 | 1.3 $\pm$ 0.2 | 1.1 $\pm$ 0.1
40 | 23.4 | 9.5 | 1.8 $\pm$ 0.4 | 1.4 $\pm$ 0.4
60 | 41.8 | 23.4 | 2.5 $\pm$ 0.6 | 1.6 $\pm$ 0.5
80 | 69.3 | 54.6 | 4.2 $\pm$ 1.3 | 2.0 $\pm$ 0.4
100 (LEAPS) | 86.7 | 68.5 |  9.0 $\pm$ 1.0 |  5.7 $\pm$ 0.7
X3D${}_{\text{M}}$
20 | 15.8 | 3.9 | 1.1 $\pm$ 0.1 | 1.0
40 | 18.3 | 5.4 | 1.4 $\pm$ 0.4 | 1.0
60 | 32.6 | 18.7 | 2.1 $\pm$ 0.8 | 1.2 $\pm$ 0.2
80 | 55.0 | 37.2 | 3.8 $\pm$ 0.7 | 2.1 $\pm$ 0.6
100 (LEAPS) | 90.3 | 82.5 |  11.4 $\pm$ 0.9 |  8.0 $\pm$ 1.4

Table S2: Ablation on the percentage of model’s layers used for priming. The
best results per metric are in bold.

##  S4-0 Priming layers

We further ablate over the number of layers used by the priming loss
$\underset{prim}{\mathcal{L}}$. We select embeddings from the first 20%, 40%,
60%, and 80% of the total network layers for our priming loss. Given our
proposed LEAPS uses embeddings from all network layers; i.e.
$\mathbf{\Lambda}=\\{1,...,L\\}$, each setting in turn uses
$\mathbf{\Lambda}_{20}=\\{1,...,\lfloor\frac{L}{5}\rfloor\\}$,
$\mathbf{\Lambda}_{40}=\\{1,...,\frac{2L}{5}\rfloor\\}$,
$\mathbf{\Lambda}_{60}=\\{1,...,\lfloor\frac{3L}{5}\rfloor\\}$, and
$\mathbf{\Lambda}_{20}=\\{1,...,\lfloor\frac{4L}{5}\rfloor\\}$, where
$\lfloor\cdot\rfloor$ denotes the floor function. As shown in Table S2, for
both 3D R50 and X3D${}_{\text{M}}$, priming layer reductions also correspond
to large decreases in top-1 accuracies and inception scores. The degradation
in accuracy and IS is observed for both the inverted models as well as the
verifier.

(a) 3D R50

(b) X3D${}_{\text{M}}$

Figure S10-0: Top-1 inverted model accuracy (%) with priming over stimuli
videos. The area between the lower and upper class-accuracy bounds achieved by
videos from LEAPS is shown in gray.

##  S5-0 Multi-stimuli priming

Our proposed video model inversion method is based on the approximation of
embeddings that are relevant to specific actions. LEAPS uses the embeddings
from a single priming example as stimulus. As an alternative, one may use
additional stimuli videos to recall the learned preconscious of models
associated with a class. We show in Figure S10-0 the top-1 accuracies achieved
by 3D R50 and X3D${}_{\text{M}}$ when priming is performed with multiple
stimuli instead of using LEAPS regularizers. As observed, the use of temporal
coherence and feature diversity regularizers terms can perform favorably over
internal representations from a small number of multiple stimuli. However,
increasing the number of stimuli used show comparable performance to that
achieved by LEAPS, thus advocating for an alternative to regularizers when
access to more data is available.