# Cross-Domain Video Anomaly Detection without Target Domain Adaptation

Abhishek Aich⋆, Kuan-Chuan Peng†, Amit K. Roy-Chowdhury⋆
⋆University of California, Riverside, USA, †Mitsubishi Electric Research
Laboratories, USA
{aaich001<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Most cross-domain unsupervised Video Anomaly Detection (VAD) works assume that
at least few task-relevant target domain training data are available for
adaptation from the source to the target domain. However, this requires
laborious model-tuning by the end-user who may prefer to have a system that
works “out-of-the-box.” To address such practical scenarios, we identify a
novel target domain (inference-time) VAD task where no target domain training
data are available. To this end, we propose a new ‘Zero-shot Cross-domain
Video Anomaly Detection (zxVAD)’ framework that includes a future-frame
prediction generative model setup. Different from prior future-frame
prediction models, our model uses a novel Normalcy Classifier module to learn
the features of normal event videos by learning how such features are
different “relatively” to features in pseudo-abnormal examples. A novel
Untrained Convolutional Neural Network based Anomaly Synthesis module crafts
these pseudo-abnormal examples by adding foreign objects in normal video
frames with no extra training cost. With our novel relative normalcy feature
learning strategy, zxVAD generalizes and learns to distinguish between normal
and abnormal frames in a new target domain without adaptation during
inference. Through evaluations on common datasets, we show that zxVAD
outperforms the state-of-the-art (SOTA), regardless of whether task-relevant
(i.e., VAD) source training data are available or not. Lastly, zxVAD also
beats the SOTA methods in inference-time efficiency metrics including the
model size, total parameters, GPU energy consumption, and GMACs.

##  1 Introduction

Figure 1: Problem overview. Current unsupervised cross-domain VAD works (xVAD)
entail adapting to the target domain, assuming access to at least a few
training examples [1, 2]. We relax this assumption of having such access to
training data from the target domain and tackle a more stringent, yet
practical, case using our proposed zero-shot xVAD or zxVAD framework.

Unsupervised Video Anomaly Detection (VAD) methods [3, 4, 5, 6, 7, 8, 9, 10,
11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29,
30, 31, 32, 33] have been widely used in security and surveillance
applications [34, 35, 36, 37] over the supervised or weakly-supervised VAD
methods [38, 39, 40, 41, 42, 43, 44, 45, 46]. This is mainly because
unsupervised VAD methods do not need training videos containing abnormal
events which are rare and laborious to annotate [35, 36]. Hence, with only
normal events in training videos, the unsupervised VAD methods mark the
activities unexplained by the trained model as anomalies during testing.
Recently, unsupervised VAD works under cross-domain settings have been
introduced [47, 48, 1, 2]. Given the video data containing only normal events
from a source domain, the goal is to perform VAD in a different target domain.
However, these cross-domain VAD (xVAD) works [47, 48, 1, 2] are methods which
need access to either the source and target domain VAD training data [1, 2] or
strong supervision from pre-trained object detectors (_e.g_., YOLOv3 [49] in
[48]). Collecting such data in the target domain and adapting or tuning the
model may not be feasible by the end-user who may want a system that works
“out-of-the-box” [50, 51]. Moreover, granting access to such video data may be
time-consuming to third-party corporations due to intellectual property and
security concerns [52, 53]. This renders the current xVAD works ineffective as
they assume access to at least some target domain training data.

#### Problem Statement.

Based on the aforesaid issues, we formally identify the following new
unsupervised xVAD problem of detecting anomalies in the target domain with
strictly _no access to target domain training data and no prior knowledge of
its anomaly types_. More specifically, our goal is to detect anomalies in the
target domain’s testing set, without having any training data on the target
side. Fig. 1 contrasts this problem setup with prior xVAD problem definitions.

#### Proposed framework.

We tackle this new problem using a novel xVAD framework, namely ‘Zero-shot
Cross-domain Video Anomaly Detection’ (zxVAD). The term zero-shot implies no
training videos available from the target domain for adaptation to perform
anomaly detection. zxVAD has a generator [54, 55, 56] in a future-frame
prediction setup [3] similar to xVAD approaches [2, 1]. However different from
these methods, zxVAD’s generator training is assisted by a novel Normalcy
Classifier (NC) module and an Untrained Convolutional Neural Network (CNN)
Anomaly Synthesis ($\bm{\mathcal{O}}$) module. Prior unsupervised xVAD works
learn features from only videos with normal events. This leads to overfitting
to the source domain distribution and the poor generalizing ability for target
domain VAD [2]. In contrast, zxVAD’s generator uses NC and $\bm{\mathcal{O}}$
modules to learn features of normal activities in input videos, by focusing on
how such features are relatively different from features of abnormal frames.
This “relative” learning strategy enhances the generator’s ability in
identifying anomalies in target domain without any adaptation at test time.

Normalcy, by definition, is always contextually dependent [43, 57] (_e.g_.,
running in playgrounds vs highways). Hence, to generalize across new target
domains (without its training data) where we have no prior knowledge about
anomaly types, we propose to learn normal event features that consider the
contextual or relative difference between “normal” and “abnormal” patterns.
More concretely, rather than learning only the normalcy features (_i.e_.
features of normal video frames), our model learns the relative normalcy
features (_i.e_. difference between features of normal and abnormal video
frames) using our proposed NC module. These pseudo-abnormal frames are created
through our proposed $\bm{\mathcal{O}}$ module which is capable of localizing
objects from both Task-Relevant or VAD data and Task-Irrelevant (TI) or non-
VAD data (_i.e_., data irrelevant to the VAD task). The $\bm{\mathcal{O}}$
module crafts pseudo-abnormal frames by localizing objects in input TI or VAD
video frames and pasting them (with random location and size) on normal VAD
video frames. Furthermore, a major advantage of introducing TI data to our
problem setup is that they can be treated as video distributions for learning
patterns of normal activities and also assist in creating diverse anomalies.
Hence, along with the strategy of learning the relative normalcy difference,
zxVAD aims to mitigate the generalizing issue via learning this relative
normalcy with respect to abnormal frames having different kinds of foreign
objects (either from VAD or TI frames). This allows zxVAD to avoid being
limited to specific anomaly types in the source domain, making it
fundamentally different from supervised specific anomaly learning.

Our NC module is designed to distinguish between a pseudo-abnormal and the
predicted normal future-frame through novel loss functions. The highlighting
attribute of these functions is to consider different properties of normal and
abnormal frames through our NC’s logit predictions and derived attention maps.
Our $\bm{\mathcal{O}}$ module is uniquely capable of using VAD or TI data with
an untrained randomly initialized CNN to create anomalies at no extra training
cost. To sum up, we make the following key contributions:

  1. 1.

We formally introduce a new problem setup in xVAD where the model is trained
on the source domain to detect anomalies (at test time) in a different target
domain without any adaptation via target-domain training data or using any
supervision from pre-trained models (_e.g_. YOLOv3).

  2. 2.

A novel xVAD method namely zxVAD is proposed, where the model learns the
relative difference between normal and abnormal frames in source domain and
generalize VAD to target without needing target domain training data or any
external support from pre-trained models.

  3. 3.

This “relative” difference learning is achieved via a novel Normalcy
Classifier that uses a new pseudo-anomaly synthesis module based on an
untrained CNN where anomalies are created with no extra training cost.

  4. 4.

Notably, for the first time in VAD literature (to the best of our knowledge),
we also show that zxVAD outperforms the SOTA xVAD works in the proposed
problem setup when trained only with TI data, in four common benchmarks.

  5. 5.

zxVAD beats the SOTA xVAD works in the proposed problem setup both in AUC on
most benchmarks, and in inference-time efficiency metrics (_e.g_., model size,
model parameters, GPU energy consumption, and GMACs).

Table 1: Characteristic comparison. Better than prior unsupervised VAD works
(_e.g_., $\mathcal{C}_{0}$: [58, 59, 60, 61, 62, 63, 57], $\mathcal{C}_{1}$:
[3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23,
24, 25], $\mathcal{C}_{2}$: [47, 48, 64], $\mathcal{C}_{3}$ (our baselines):
[1, 2]), zxVAD needs no prior knowledge (_e.g_. object extraction from VAD
videos), can perform cross-domain VAD with no VAD training data, and uses an
untrained CNN to create anomalies.

Unsupervised VAD Method Conditions | Unsupervised VAD Categories
---|---
$\mathcal{C}_{0}$ | $\mathcal{C}_{1}$ | $\mathcal{C}_{2}$ | $\mathcal{C}_{3}$ | zxVAD (ours)
no prior knowledge required? | ✗ | ${{\color[rgb]{0.25,0.51171875,0.42578125}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.51171875,0.42578125}\large\checkmark}}$ | ✗ | ${{\color[rgb]{0.25,0.51171875,0.42578125}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.51171875,0.42578125}\large\checkmark}}$ | ${{\color[rgb]{0.25,0.51171875,0.42578125}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.51171875,0.42578125}\large\checkmark}}$
show efficacy in cross-domain VAD? | ✗ | ✗ | ${{\color[rgb]{0.25,0.51171875,0.42578125}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.51171875,0.42578125}\large\checkmark}}$ | ${{\color[rgb]{0.25,0.51171875,0.42578125}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.51171875,0.42578125}\large\checkmark}}$ | ${{\color[rgb]{0.25,0.51171875,0.42578125}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.51171875,0.42578125}\large\checkmark}}$
works with no source domain VAD training data? | ✗ | ✗ | ✗ | ✗ | ${{\color[rgb]{0.25,0.51171875,0.42578125}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.51171875,0.42578125}\large\checkmark}}$
create pseudo-anomalies with an untrained network? | ✗ | ✗ | ✗ | ✗ | ${{\color[rgb]{0.25,0.51171875,0.42578125}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.51171875,0.42578125}\large\checkmark}}$

##  2 Related Works

#### Unsupervised VAD works.

Early unsupervised VAD works formulated the anomaly detection using
handcrafted features to characterize the normal event or regular pattern
distribution [4, 5, 6, 7, 8, 9, 10, 11, 12]. However, these methods were
outperformed by the CNN approaches [21, 13, 20, 14, 15, 22, 16, 17, 18, 3, 19,
23, 24, 25] (both categorized as $\mathcal{C}_{1}$ in Tab. 1). Some of these
CNN based unsupervised VAD works use generators [54] to model the normal frame
distributions [3, 20, 21, 22, 18, 61], and further introduce memory modeling
networks to record various normal event patterns in videos [1, 61, 2, 21, 18].
Another category of works ($\mathcal{C}_{0}$ in Tab. 1) [58, 57, 59, 60, 61,
62, 63] proposed computationally heavy approaches that used strong priors like
object extraction (using pre-trained object-detectors [57, 60]) for VAD, in
order to focus only on specific objects to detect anomalies. Compared to
aforesaid VAD works in $\mathcal{C}_{0}$ and $\mathcal{C}_{1}$, zxVAD (a) is
designed to tackle unsupervised cross-domain VAD problem, (b) is a future-
frame prediction method with a memory module, and (c) needs no strong prior
knowledge from object extraction. Finally, few works [65, 66, 57, 48, 63, 67]
have shown different VAD strategies where pseudo-anomalies are used. For
example, [57, 67] uses a generator to create fake anomaly data. [65, 66]
propose two different temporal pseudo anomaly synthesizers to craft anomalies
from normal videos. Needing no aforesaid extra training efforts, zxVAD uses a
novel strategy to create anomalies using an untrained randomly initialized CNN
(details in Sec. 3) at no extra training cost.

#### Cross-domain setting.

The cross-domain scenario in unsupervised VAD has been introduced in [47, 64,
2, 1, 68]. These works operate under the regime of few-shot target domain
scene adaptation. For example, [1, 2] ($\mathcal{C}_{3}$ in Tab. 1) use meta-
learning approaches [69] and adapt to the target domain with few scenes for
anomaly detection. In contrast, zxVAD is specifically designed for cross-
domain VAD without any target domain adaptation. [47, 48, 64]
($\mathcal{C}_{2}$ in Tab. 1) provide prior knowledge based methods where
videos are subject to object-extraction using pre-trained object detectors
[49, 70]. However, zxVAD needs no strong priors like object extraction using
pre-trained detectors. zxVAD is also capable of solely using TI data and
outperforming the SOTA in proposed cross-domain VAD setup. Finally, zxVAD uses
a simple training strategy (details in Sec. 3) rather than using a meta-
learning approach to avoid non-trivial computational and memory burdens, as
well as vanishing gradients issues [71, 47].

##  3 Proposed zxVAD Framework

#### Method Overview.

We are provided with source domain VAD normal videos to learn features that
should ideally transfer across different target domains without needing target
domain adaptation. To achieve this no adaptation-based cross-domain VAD
property, we introduce a novel zxVAD framework (illustrated in Fig. 2) based
on a future-frame prediction setup that can be trained end-to-end. It consists
of an untrained CNN based pseudo-anomaly Synthesis module (Sec. 3.1) where an
untrained randomly initialized CNN helps in creating pseudo-anomalies without
any extra training burden. These pseudo-abnormal frames along with predicted
future-frame are utilized in our novel Normalcy Classifier module (Sec. 3.2)
to regularize the backbone generator to learn relative normalcy features. This
learning strategy makes zxVAD capable of more generalizable VAD performance
across different target domains than existing xVAD methods.

#### Notations.

We denote a sample video from VAD datasets as
$[\bm{v}_{1},\bm{v}_{2},\cdots,\bm{v}_{L_{v}}]\in\mathbb{R}^{L_{v}\times
C\times H\times W}$, and TI datasets as
$[\bm{u}_{1},\bm{u}_{2},\cdots,\bm{u}_{L_{u}}]\in\mathbb{R}^{L_{u}\times
C\times H\times W}$, where each video contains $L_{v}$ and $L_{u}$ number of
frames, and each frame is of height $H$, width $W$, and $C$ channels. Our
future-frame prediction framework zxVAD contains a memory-augmented generator
[18] $\mathcal{G}(\cdot)$ with weights $\bm{\theta}_{\mathcal{G}}$ and memory
module $\mathcal{M}$, and a discriminator $\mathcal{D}(\cdot)$ with weights
$\bm{\theta}_{\mathcal{D}}$. As shown in [18, 72, 73], the memory module
$\mathcal{M}\in\mathbb{R}^{K\times Q}$ is a matrix with
$\bm{m}_{i}\in\mathbb{R}^{Q},\forall i\in[K]$ vectors (or memory items) that
learns to register the prototypical normal features during training.
$\mathcal{M}$ takes the output vector $\bm{z}\in\mathbb{R}^{Q}$ from
$\mathcal{G}(\cdot)$’s encoder and outputs
$\widehat{\bm{z}}=\bm{w}\mathcal{M}\in\mathbb{R}^{Q}$ that is forwarded to
$\mathcal{G}(\cdot)$’s decoder. Here, $\bm{w}\in\mathbb{R}^{1\times K}$ is
termed as a soft addressing vector [18]. Each element $w_{i}$ of $\bm{w}$ is
computed using softmax operation on the cosine similarity between $\bm{z}$ and
$\bm{m}_{i}$ [18, 74]. Our proposed anomaly synthesis module is denoted as
$\bm{\mathcal{O}}$ and contains a CNN denoted as $\mathcal{R}(\cdot)$ with
weights $\bm{\theta}_{\mathcal{R}}$. Further, our proposed normalcy classifier
module contains a CNN classifier denoted as $\mathcal{N}(\cdot)$ with weights
$\bm{\theta}_{\mathcal{N}}$. We denote the expectation operator, $l_{p}$-norm
operator, and element-wise multiplication by $\mathbb{E}[\cdot]$,
$\|\cdot\|_{p}$, and $\odot$, respectively.

Figure 2: Framework overview. Our zxVAD framework contains a Future-Frame
Prediction backbone (top-left) guided by our Normalcy Classifier module
(right). To enforce the prediction backbone to learn generalizable features
from source domain normal videos and avoid overfitting, we encourage the
generative model to learn normalcy features relative to pseudo-abnormal frames
using four novel loss functions. These abnormal frames are created using an
untrained randomly-initialized CNN through our novel anomaly synthesis module
$\bm{\mathcal{O}}$ (bottom-left).

#### Backbone description.

Given $N$ source domain training videos (with only normal events), we aim to
learn a future-frame prediction generator that takes in $T$ input frames and
predicts a future frame $\widehat{\bm{v}}_{T+1}$, _i.e_.,
$\mathcal{G}\big{(}[\bm{v}_{1},\bm{v}_{2},\cdots,\bm{v}_{T}]\big{)}$ =
$\widehat{\bm{v}}_{T+1}$. $\mathcal{G}(\cdot)$ is adversarially trained
against $\mathcal{D}(\cdot)$ in the Least-Square GAN [75, 54] setup where
$\mathcal{D}(\cdot)$ aims to distinguish between $\widehat{\bm{v}}_{T+1}$ and
the ground truth frame $\bm{v}_{T+1}$. Similar to [18, 2], we introduce a
memory module $\mathcal{M}$. In zxVAD, $\mathcal{G}(\cdot)$ is further
regularized using $\mathcal{N}(\cdot)$ with our proposed four novel objectives
(explained in Sec. 3.2) which uses pseudo-anomaly examples generated using an
untrained CNN based strategy. Following prior works [76, 1], we optimize
$\mathcal{G}(\cdot)$ with the mean square error loss
$\mathcal{L}_{\text{MSE}}=\|\widehat{\bm{v}}_{T+1}-\bm{v}_{T+1}\|^{2}_{2}$,
structure similarity loss
$\mathcal{L}_{\text{SSM}}=1-\text{SSIM}\big{(}\widehat{\bm{v}}_{T+1},\bm{v}_{T+1}\big{)}$,
where SSIM represents the structural similarity index measure [77] between
$\widehat{\bm{v}}_{T+1}$ and $\bm{v}_{T+1}$, and Gradient loss
$\mathcal{L}_{\text{GD}}$ [76, 3]. To optimize $\mathcal{M}$ and encourage
modeling normal videos using sparse but most relevant memory slots, we follow
[18] and apply a hard-shrinkage on $\mathcal{M}$’s memory addressing vectors
$w_{i}$ using continuous ReLU activation function with a shrinkage factor
$\lambda$ set as 0.0005. Next, we normalize each element
$\widehat{w}_{i}\leftarrow\nicefrac{{\widehat{w}_{i}}}{{\|\bm{\widehat{w}}\|_{1}}}\forall
i$ and get $\widehat{\bm{z}}=\widehat{\bm{w}}\mathcal{M}$. We also apply a
sparsity regularizer on $\widehat{\bm{w}}$ by minimizing its entropy as
$\mathcal{L}_{\text{MEM}}=\sum_{i=1}^{N}-\widehat{w}_{i}\log\big{(}\widehat{w}_{i}\big{)}$
[18]. We combine these losses as

$\displaystyle\mathcal{L}_{\text{BB}}=\mathcal{L}_{\text{REC}}+\alpha_{\text{MEM}}\mathcal{L}_{\text{MEM}},$
(1)

where the reconstruction loss is
$\mathcal{L}_{\text{REC}}=\mathcal{L}_{\text{MSE}}+\mathcal{L}_{\text{SSM}}+\mathcal{L}_{\text{GD}}$.
We set the loss weight $\alpha_{\text{MEM}}=0.0025$ following [18]. Totally,
the weights $\bm{\theta}_{\mathcal{G}}$, $\bm{\theta}_{\mathcal{D}}$ and
$\bm{\theta}_{\mathcal{N}}$ are updated during training, while
$\bm{\theta}_{\mathcal{R}}$ is randomly initialized before training and
remains fixed. Better than prior works which do not consider the relative
difference between normal and abnormal events, zxVAD introduces a novel
strategy to regularize this backbone generator by learning normal features
with respect to pseudo-abnormal features. As our normalcy classifier module
utilizes pseudo-anomalies to learn the relative normalcy features, we first
present our pseudo-anomaly creation strategy.

###  3.1 Pseudo-Anomaly Synthesis via Untrained CNN

Prior works [57] have focused on creating anomalies using pre-trained object
detectors (_i.e_., YOLOv3 [49] in [57]) that result in issues like additional
training overheads. Different from such methods, we present a training-free
strategy to extract objects from video frames. These objects can be obtained
on both VAD and TI video frames (_i.e_., $\bm{v}_{t}$ and $\bm{u}_{t}$). For
brevity, we refer to the input frame as $\bm{x}$. Given an input frame
$\bm{x}\in\mathbb{R}^{C\times H\times W}$, we denote the output of a CNN
$\mathcal{R}(\cdot)$ (before the classification layer) as tensor
$\bm{G}\in\mathbb{R}^{d\times h\times w}$. For example, if
$\mathcal{R}(\cdot)$ is ResNet152 [78], $\bm{G}$ is the output of
‘$conv5\\_x$’ with size ${2048\times 8\times 8}$ if input size is ${3\times
256\times 256}$. We employ SCDA [79] to perform channel-wise summation on
$\bm{G}$ to obtain an attention map $\bm{A}\in\mathbb{R}^{h\times w}$. We then
obtain a binary mask $\bm{M}$ from $\bm{A}$ as follows. We set
$\bm{M}_{(i,j)}=1\text{ if }\bm{A}_{(i,j)}>\varsigma$, or 0 otherwise. Here,
$(i,j)$ represents position in $h\times w$ locations. We empirically set
$\varsigma=0.1$. $\bm{M}_{(i,j)}=1$ indicates the foreground objects. Finally,
$\bm{M}$ is resized from $h\times w$ to $H\times W$. As noted in [80], the
idea behind this surprising property that randomly initialized CNN can
localize objects is: because the background in the input frame $\bm{x}$ is
relatively texture-less in comparison to the foreground objects in the scene,
these background regions have higher chances to be deactivated by nonlinear
activation functions like ReLU [81]. The object is finally localized as
$\bm{M}_{\bm{x}}=\bm{M}\odot\bm{x}$. To create pseudo-abnormal frame
$\tilde{\bm{v}}$, we combine $\bm{M}_{\bm{x}}$ and one of the input frames to
$\mathcal{G}(\cdot)$, _i.e_.,
$\bm{v}_{t}\in\left\\{\bm{v}_{1},\bm{v}_{2},\cdots,\bm{v}_{T}\right\\}$ by
pasting $\bm{M}_{\bm{x}}$ on $\bm{v}_{t}$ at random location $r_{z}$ with
random size $r_{x}\times r_{y}$. We discuss the method to choose the location
$r_{z}$ and size $r_{x}\times r_{y}$ in Supplementary Material. Note that most
of the video frames used for creating pseudo-anomalies happen to contain at
least one foreground object for the untrained CNN to extract. Even if there
are no such objects, our untrained CNN will still focus on some patches (on
the input frame) and treat them as anomaly on normal event VAD frame.

Figure 3: Pseudo-abnormal examples. We show pseudo-abnormal frames (marked as
(b)) created using our pseudo-anomaly synthesis strategy. The untrained
randomly initialized CNN is ResNet50 [78] which localizes objects in the TI
frames (marked as (a)). We also show examples of ground-truth masks
$\tilde{\bm{M}}$ used in loss $\mathcal{L}_{\text{RAA}}$ (marked as (c)). See
more examples in Supplementary Material.

###  3.2 Learning Normality w.r.t. Abnormality

Our Normalcy Classifier Module is a classifier $\mathcal{N}(\cdot)$ that is
optimized by the following four loss functions. These loss functions are
complementary to each other as follows: normalcy loss and attention
affirmation loss focus on the difference between normal and abnormal frames,
whereas relative normalcy loss and relative attention affirmation loss focus
on how relatively different are normal frames from abnormal frames (and vice-
versa). For clarity, we drop the subscript of the predicted frame
$\widehat{\bm{v}}_{T+1}$ and mark it as $\widehat{\bm{v}}$. The data
distribution of normal and pseudo-abnormal frames are denoted as $\rho$ and
$\kappa$, respectively.
Normalcy loss $\mathcal{L}_{\text{N}}$. Given the predicted future-frame
$\widehat{\bm{v}}$ and pseudo-abnormal frame $\tilde{\bm{v}}$,
$\mathcal{L}_{\text{N}}$ optimizes $\mathcal{N}(\cdot)$ to increase the
probability that $\widehat{\bm{v}}$ is ‘normal’ (label set as 1) and
$\tilde{\bm{v}}$ is ‘abnormal’ (label set as 0), using following loss
function.

$\displaystyle\small{\textstyle{\mathcal{L}_{\text{N}}=\nicefrac{{1}}{{2}}\mathbb{E}_{\widehat{\bm{v}}\sim\rho}\Big{[}\big{(}\mathcal{N}(\widehat{\bm{v}})-1\big{)}^{2}\Big{]}+\nicefrac{{1}}{{2}}\mathbb{E}_{\tilde{\bm{v}}\sim\kappa}\Big{[}\big{(}\mathcal{N}(\tilde{\bm{v}})\big{)}^{2}\Big{]}}}$
(2)

Relative normalcy loss $\mathcal{L}_{\text{RN}}$. Abnormal events can be
viewed as deviation with respect to normal events. We argue that the key
missing attribute of (2) is that the probability of normal data being normal
($\mathcal{N}(\widehat{\bm{v}})$) should increase as the probability of
abnormal data being normal ($\mathcal{N}(\tilde{\bm{v}})$) decreases and vice-
versa. Rather than just maximizing $\mathbb{P}[\widehat{\bm{v}}\text{ is
normal}]$, we also ask $\mathcal{N}(\cdot)$ to maximize
$\mathbb{P}[\widehat{\bm{v}}\text{ is more normal than }\tilde{\bm{v}}]$
($\mathbb{P}[\cdot]$ denotes probability operator). We define this novel
relative normalcy loss below:

$\displaystyle{\begin{gathered}\mathcal{L}_{\text{RN}}=\nicefrac{{1}}{{2}}\mathbb{E}_{\widehat{\bm{v}}\sim\rho}\Big{[}\big{(}\mathcal{N}(\widehat{\bm{v}})-\mathbb{E}_{\tilde{\bm{v}}\sim\kappa}[\mathcal{N}(\tilde{\bm{v}})]-1\big{)}^{2}\Big{]}+\\\
\nicefrac{{1}}{{2}}\mathbb{E}_{\tilde{\bm{v}}\sim\kappa}\Big{[}\big{(}\mathcal{N}(\tilde{\bm{v}})-\mathbb{E}_{\widehat{\bm{v}}\sim\rho}[\mathcal{N}(\widehat{\bm{v}})]+1\big{)}^{2}\Big{]}\end{gathered}}$
(5)

Attention affirmation loss $\mathcal{L}_{\text{AA}}$. The decision of
$\mathcal{N}(\cdot)$ on the normal frame $\widehat{\bm{v}}$ and abnormal frame
$\tilde{\bm{v}}$ should be based on the following information: (1)
$\mathcal{N}(\cdot)$ should consider the whole scene in $\widehat{\bm{v}}$ to
classify it as ‘normal,’ and (2) $\mathcal{N}(\cdot)$ should consider the
foreign object (introduced by our module $\bm{\mathcal{O}}$ in
$\tilde{\bm{v}}$) to classify it as ‘abnormal.’ Our strategy in Sec. 3.1
allows us to obtain the exact location of foreign objects in $\tilde{\bm{v}}$.
Hence, we leverage this knowledge and create ground-truth masks of
$\tilde{\bm{v}}$. We first initialize a tensor $\tilde{\bm{M}}$ with zeroes.
Next, we update this tensor by pasting $\bm{M}$ after resizing to $r_{x}\times
r_{y}$ at location $r_{z}$ (obtained from $\bm{\mathcal{O}}$ in Sec. 3.1). We
show examples of $\tilde{\bm{M}}$ in Fig. 3. We extract feature maps from the
last convolutional layer of $\mathcal{N}(\cdot)$ and apply SCDA [79] to obtain
attention maps $\mathcal{A}(\widehat{\bm{v}})$ and
$\mathcal{A}(\tilde{\bm{v}})$ for normal and abnormal frames, respectively.
$\mathcal{A}(\cdot)$ denotes the operation to extract attention maps from
$\mathcal{N}(\cdot)$. We enforce this constraint via the attention affirmation
loss $\mathcal{L}_{\text{AA}}$ as ($\mathbb{1}$ is a tensor of the same size
as $\mathcal{A}(\widehat{\bm{v}})$ filled with ones):

$\displaystyle\mathcal{L}_{\text{AA}}=\nicefrac{{1}}{{2}}\big{(}\mathbb{1}-\mathcal{A}(\widehat{\bm{v}})\big{)}^{2}+\nicefrac{{1}}{{2}}\big{(}\tilde{\bm{M}}-\mathcal{A}(\tilde{\bm{v}})\big{)}^{2},$
(6)

#### Relative attention affirmation loss $\mathcal{L}_{\text{RAA}}$.

Similar to the concept of $\mathcal{L}_{\text{RN}}$, we argue that
$\mathcal{L}_{\text{AA}}$ does not consider the relative difference of
attention maps from normal frames with respect to attention maps from abnormal
frames. Hence, we propose a relative attention affirmation loss
$\mathcal{L}_{\text{RAA}}$ that aims to learn this difference. We create two
attention map pairs: (Pair-1) $\mathcal{A}(\widehat{\bm{v}})$ and
$\mathcal{A}(g(\widehat{\bm{v}}))$, and (Pair-2)
$\mathcal{A}(\widehat{\bm{v}})$ and $\mathcal{A}(\tilde{\bm{v}})$. The
function $g(\cdot)$ denotes a series of transformations (Color Jitter, Random
Affine, and Random Perspective) applied to $\widehat{\bm{v}}$ using the
package Kornia [82] (related parameters are provided in Supplementary
Material). The relative difference between the attention on ‘augmented normal’
frame should be smaller than that of the ‘pseudo-abnormal’ frame with respect
to the ‘normal’ frame. We enforce this difference with a margin $m$ that
simultaneously enhances the intra-class compactness between normal and
augmented-normal frames and inter-class discrepancy between normal and pseudo-
abnormal frames. We design $\mathcal{L}_{\text{RAA}}$ using the ArcFace loss
[83] enforcing this margin as follows.

$\small{\mathcal{L}_{\text{RAA}}=\dfrac{-1}{N}\sum_{i=0}^{N-1}\log\Bigg{(}\dfrac{e^{s(\cos(\omega_{y_{i}}+m))}}{e^{s(\cos(\omega_{y_{i}}+m))}+\sum_{j=0,j\neq
y_{i}}^{1}e^{s\cos(\omega_{j})}}\Bigg{)},}$ (7)

where label $y_{i}$ is set as 1 for normal frame $\widehat{\bm{v}}$ and
augmented frame $g(\widehat{\bm{v}})$, and 0 for pseudo-abnormal frame
$\tilde{\bm{v}}$. We transform $\mathcal{A}(\bm{x})$ with
$\psi_{y_{i}}=\|\bm{W}_{y_{i}}\|\|\texttt{vec}(\mathcal{A}(\bm{x}))\|\cos(\omega_{y_{i}})$
(with $\omega_{y_{i}}\in[0,\pi]$ as the angle between $\bm{W}_{y_{i}}$ and
$\texttt{vec}(\mathcal{A}(\bm{x}))$). Here, $\texttt{vec}(\cdot)$ is a
vectorizing operation. $\|\bm{W}_{y_{i}}\|$ and
$\|\texttt{vec}(\mathcal{A}(\bm{x}))\|$ are normalized to 1 which leads to
$\psi_{y_{i}}=\cos(\omega_{y_{i}})$. With ArcFace loss, $\bm{W}_{y_{i}}$
behaves as a centre for each class (_i.e_. normal and abnormal) [83] which
creates a distance margin penalty of $m$. We set scaling factor $s$ = 64 and
margin $m$ = 28.6 degrees following [84]. $\mathcal{L}_{\text{RAA}}$ can be
implemented as any triplet metric learning loss [84]. However, we choose the
ArcFace loss as it has been shown to perform well in recent non-VAD works [85,
86, 87].

#### Final learning objectives.

To summarize, zxVAD is trained end-to-end with $\mathcal{G}(\cdot)$ learning
loss $\mathcal{L}_{\mathcal{G}}$, $\mathcal{D}(\cdot)$ learning loss
$\mathcal{L}_{\mathcal{D}}$, and $\mathcal{N}(\cdot)$ learning loss
$\mathcal{L}_{\mathcal{N}}$ as follows:

$\displaystyle\begin{gathered}\mathcal{L}_{\mathcal{G}}=\mathcal{L}_{\text{BB}}+\alpha_{\mathcal{D}}\mathbb{E}_{\widehat{\bm{v}}\sim\rho}\big{[}\nicefrac{{1}}{{2}}\big{(}\mathcal{D}(\widehat{\bm{v}})-1\big{)}^{2}\big{]}+\\\
\alpha_{\mathcal{N}}\mathbb{E}_{\widehat{\bm{v}}\sim\rho}\big{[}\nicefrac{{1}}{{2}}\big{(}\mathcal{N}(\widehat{\bm{v}})-1\big{)}^{2}\big{]},\\\
\mathcal{L}_{\mathcal{D}}=\mathbb{E}_{\widehat{\bm{v}}\sim\rho}\big{[}\nicefrac{{1}}{{2}}\big{(}\mathcal{D}(\widehat{\bm{v}})\big{)}^{2}\big{]}+\mathbb{E}_{\widehat{\bm{v}}\sim\rho}\big{[}\nicefrac{{1}}{{2}}\big{(}\mathcal{D}(\bm{v})-1\big{)}^{2}\big{]},\\\
\mathcal{L}_{\mathcal{N}}=\alpha_{\text{n}}\mathcal{L}_{\text{N}}+\alpha_{\text{rn}}\mathcal{L}_{\text{RN}}+\alpha_{\text{aa}}\mathcal{L}_{\text{AA}}+\alpha_{\text{raa}}\mathcal{L}_{\text{RAA}}~{}~{}~{}~{}\end{gathered}$
(12)

We set $\alpha_{\mathcal{D}}=0.05$ following [3]. The rest of loss weights
$\alpha_{\mathcal{N}}=0.5,\alpha_{\text{n}}=1,\alpha_{\text{rn}}=0.01,\alpha_{\text{aa}}=1$,
and $\alpha_{\text{raa}}=1$ are set empirically.

#### Discussion on why

zxVAD works. zxVAD trains $\mathcal{G}(\cdot)$ in predicting the future-frame
$\widehat{\bm{v}}$ of input normal video by considering the difference with
respect to pseudo-abnormal frames (via our normalcy classifier module). Not
considered in prior works, this strategy specifically helps
$\mathcal{G}(\cdot)$ to learn contextual relative difference between normal
and abnormal frames to alleviate overfitting to source domain normal video
features. The overfitting issue is further mitigated when the abnormal
examples created from our pseudo-anomaly module contains various kinds of
objects as “foreign entities” in VAD normal frames. This allows
$\mathcal{G}(\cdot)$ to learn the relative normalcy difference from extremely
diverse kinds of pseudo-anomalies, making it capable of detecting different
anomaly types (in inference-time) in multiple target domains without any prior
knowledge.

To make the further discussion concise, we show the statistics and acronyms of
the VAD and TI datasets in Tab. 2.

###  3.3 Introduction to Task-Irrelevant (TI) Datasets

Table 2: Dataset statistics. We highlight the difference in amount of training
data between VAD and TI datasets. ⋆: the train/test disjoint camera (dc) split
is provided by [1]. As stated in [1], UCFC dataset does not contain ground
truth frame-level labels and hence is not considered for evaluation.

| Task-Relevant/VAD Datasets |  | Task-Irrelevant/non-VAD Datasets
---|---|---|---
Property $\backslash$ Dataset | Shanghai-Tech [16] | Shanghai-Tech⋆ [1] | UCF-Crime [38] | Ped1 [10] | Ped2 [10] | CUHK-Avenue [9] |  | HMDB51 [88] | UCF101 [89] | 20BN-JESTER [90]
Acronym | SHT | SHT${}_{\text{dc}}$ | UCFC | Ped1 | Ped2 | Ave |  | HMDB | UCF101 | Jester
# of training / testing videos | 330 / 107 | 147 / 33 | 950 / - | 34 / 36 | 16 / 12 | 16 / 21 |  | 6,766 / - | 13,320 / - | 50,420 / -
# of abnormal instances | 47 | 33 | – | 40 | 12 | 21 |  | N/A | N/A | N/A

In this section, we discuss the utilities of task-irrelevant or non-VAD videos
for unsupervised VAD. Task-relevant or VAD datasets provided by VAD research
community are known to be limited in scale as shown in Tab. 2 and [43, 91, 1].
(_e.g_. Ave [9], Ped1, Ped2 [10] datasets have $<100$ training videos).
Further, it is difficult to collect different kinds of scenarios of normal
activities with such limited scale. Hence, we propose to introduce the utility
of Task-Irrelevant (TI) datasets to the task of VAD.

We define a dataset as ‘Task-Irrelevant’ which is freely available from
different other video downstream or non-VAD tasks (_e.g_., video
classification, action recognition, _etc_.). Examples of such datasets are
UCF101 [89] and HMDB [88] (see Tab. 2). Such datasets were originally
introduced for non-VAD works, specifically curated for large-scale deep
learning-based tasks. For example, Jester was originally introduced for video
classification of 25 hand gesture classes [90]. To show the performance using
diverse types of datasets in our zxVAD task, we choose Jester, UCF101, and
HMDB to be our TI datasets. Please see Supplementary Material for dataset
examples. Next, we discuss how the task-relevancy of these datasets is
measured with respect to the VAD task, followed by two simple strategies to
use these datasets in the proposed problem scenario. Note that zxVAD needs
nothing from the TI-VAD relevancy measure to operate. The purpose is to only
validate TI data’s irrelevancy to the VAD task.

UCF101JesterUCFCPed1/Ped2SHT$\longleftarrow$ TI Datasets
$\longrightarrow$$\longleftarrow$ VAD Datasets $\longrightarrow$Relevancy
Matrix$0$$0.2$$0.4$$0.6$$0.8$$1$Reference
Values$\begin{gathered}\underbrace{\overline{\mathcal{S}}(``husband",``wife")=0.829}_{\textrm{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}high
relevancy}}}\\\
\underbrace{\overline{\mathcal{S}}(``fox",``paper")=0.102}_{\textrm{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}low
relevancy}}}\\\
\underbrace{\overline{\mathcal{S}}(``bag",``dime")=0.209}_{\textrm{{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}example
w.r.t. max score}}}\end{gathered}$

($0:$ ‘lower’, $1:$ ‘higher’ semantic similarity)

Figure 4: Relevancy measure between VAD and TI labels. Using the relevancy
score matrix between the TI and VAD labels, we find that TI datasets have low
semantic similarity with the VAD datasets. The maximum score occurs between
HMDB and Ave.

#### Measuring relevancy of non-VAD datasets.

Following [92, 93], we use word2vec [94] (pre-trained on Google News dataset
[94]) to measure the task-relevancy of our introduced TI datasets: Jester,
UCF101, and HMDB. We first compute an embedding vector of the input labels (in
case the label contains more than one word, we average the embedding). Next,
we compute the mean absolute cosine similarity
$\overline{\mathcal{S}}\in[0,1]$ of the embedding for all possible pairs of
labels between the TI datasets and abnormal classes of VAD datasets. This is
denoted as
$\overline{\mathcal{S}}=\nicefrac{{1}}{{\ell_{P}\ell_{Q}}}\sum_{p=1}^{\ell_{P}}\sum_{q=1}^{\ell_{Q}}\big{|}\text{cos-
sim}\big{(}\bm{\pi}_{p},\bm{\pi}_{q}\big{)}\big{|},$ where $\ell_{P}$ and
$\ell_{Q}$ are the total number of labels in the TI and VAD dataset,
$\bm{\pi}_{p}$ and $\bm{\pi}_{q}$ are the word2vec representation of the
$p$-th and $q$-th label of the TI and VAD dataset, respectively. $\text{cos-
sim}(\cdot)$ denotes the cosine similarity operation on input vectors. A value
of $\overline{\mathcal{S}}$ closer to 0 indicates a higher degree of
irrelevancy (or lower degree of relevance). In Fig. 4, we show the mean cosine
similarity $\overline{\mathcal{S}}$ for all the TI (_i.e_., HMDB, UCF101,
Jester) and VAD (_i.e_., SHT, Ped1/Ped2, UCFC, Ave) datasets used in this
paper. Following reference values:
$\overline{\mathcal{S}}(``object",``scene")=0.829$,
$\overline{\mathcal{S}}(``bag",``dime")=0.209$, and
$\overline{\mathcal{S}}(``fox",``paper")=0.109$, we find that the maximum
semantic similarity $\overline{\mathcal{S}}=0.207$ occurs between Ave and HMDB
indicating that all the TI datasets are quite irrelevant to the task of VAD
problem.

#### Methods to use TI datasets.

We provide two methods to use TI datasets. Firstly, unsupervised VAD methods
learn features from normal events during training. These events are
particularly marked by continuous activities without any sudden disruption
from alien objects. Such kinds of videos are readily available in other video
downstream tasks like action recognition, where a sample video only contains
frames from a continuous activity. In cases where there is no VAD training
data available in the source domain (worst-case scenario), we show later in
Sec. 4 that training zxVAD solely with TI datasets reaches the SOTA results
across 3 different target domain datasets. We hypothesize that TI datasets
represent the recording of normal activities as in VAD training data with
normal videos. Hence, learning from such TI data helps in modeling features
similar to normal videos. Secondly, we recommend using TI frames to create
anomalies containing diverse types of objects (see Fig. 3). Using our proposed
method to create pseudo-anomaly frames using TI data (details in Sec. 3.1),
our generator learns features from normal frames relative to abnormal frames.
Such pseudo-anomalies contain diverse foreign entities extracted from TI video
frames allowing our generator to learn relative normalcy difference in a broad
manner.

##  4 Experiments and Results

#### Implementation details.

We implement our framework in PyTorch [95]. The generator $\mathcal{G}(\cdot)$
is an U-Net [96] adapted from [3] with a memory module at its bottleneck
similar to [20]. The discriminator $\mathcal{D}(\cdot)$ and normalcy
classifier $\mathcal{N}(\cdot)$ are Patch-GAN discriminators [97]. We provide
more details of our implementation in Supplementary Material.

Evaluation details. We evaluate zxVAD under three training scenarios with
respect to types of available source data: (1) Both VAD and TI data are
available: $\mathcal{G}(\cdot)$ takes VAD videos and $\bm{\mathcal{O}}$ takes
TI frames as input, (2) Only one of the VAD or TI data are available: Both
$\mathcal{G}(\cdot)$ and $\bm{\mathcal{O}}$ take VAD or TI videos as input. We
did not observe any performance gain empirically when $\mathcal{G}(\cdot)$
takes both VAD and TI videos as input, so we drop this case as it adds a
computational burden. We compare zxVAD with [1, 2] using the area under ROC
curve (AUC), model storage, total parameters, GPU energy consumption,
inference time FPS, and GMACs.

Baselines. Since the problem of ‘cross-domain VAD without target domain
training data adaptation’ is identified by us, we cannot find other methods
which are designed for such a setup. The latest and closest baselines we found
are rGAN [1] and MPN [2], which are designed for the xVAD task without needing
strong priors from VAD frame object extraction. Since both methods report
their performance under the proposed problem setup, we use them as our
baselines. Even though we outperform strong prior based xVAD methods [47, 48,
57] without any such computationally expensive operation under our problem
setup, we do not consider them as part of our baselines for fair comparisons
with respect to [1, 2]. In Tab. 3, and 5, paper denotes results as reported
and code denotes results computed using official code, if available.

Loss Functions | AUC (%)
---|---
$\mathcal{L}_{\text{BB}}$ | $\mathcal{L}_{\text{N}}$ | $\mathcal{L}_{\text{RN}}$ | $\mathcal{L}_{\text{AA}}$ | $\mathcal{L}_{\text{RAA}}$ | on SHT${}_{\text{dc}}$
${{\color[rgb]{0.25,0.51171875,0.42578125}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.51171875,0.42578125}\large\checkmark}}$ |  |  |  |  | 68.32
${{\color[rgb]{0.25,0.51171875,0.42578125}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.51171875,0.42578125}\large\checkmark}}$ | ${{\color[rgb]{0.25,0.51171875,0.42578125}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.51171875,0.42578125}\large\checkmark}}$ |  |  |  | 68.99
${{\color[rgb]{0.25,0.51171875,0.42578125}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.51171875,0.42578125}\large\checkmark}}$ | ${{\color[rgb]{0.25,0.51171875,0.42578125}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.51171875,0.42578125}\large\checkmark}}$ | ${{\color[rgb]{0.25,0.51171875,0.42578125}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.51171875,0.42578125}\large\checkmark}}$ |  |  | 69.33
${{\color[rgb]{0.25,0.51171875,0.42578125}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.51171875,0.42578125}\large\checkmark}}$ | ${{\color[rgb]{0.25,0.51171875,0.42578125}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.51171875,0.42578125}\large\checkmark}}$ | ${{\color[rgb]{0.25,0.51171875,0.42578125}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.51171875,0.42578125}\large\checkmark}}$ | ${{\color[rgb]{0.25,0.51171875,0.42578125}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.51171875,0.42578125}\large\checkmark}}$ |  | 69.61
${{\color[rgb]{0.25,0.51171875,0.42578125}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.51171875,0.42578125}\large\checkmark}}$ | ${{\color[rgb]{0.25,0.51171875,0.42578125}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.51171875,0.42578125}\large\checkmark}}$ | ${{\color[rgb]{0.25,0.51171875,0.42578125}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.51171875,0.42578125}\large\checkmark}}$ | ${{\color[rgb]{0.25,0.51171875,0.42578125}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.51171875,0.42578125}\large\checkmark}}$ | ${{\color[rgb]{0.25,0.51171875,0.42578125}\definecolor[named]{pgfstrokecolor}{rgb}{0.25,0.51171875,0.42578125}\large\checkmark}}$ | 70.85

(a) Loss ablation

AE-MAEGANGAN-M$75.00$$80.00$$85.00$$90.00$$95.00$AUC (%)Ped2AvePed1 (b) Impact
of Backbones

CutMixMixUp[66]$40.00$$50.00$$60.00$$70.00$$80.00$$90.00$$100.00$AUC
(%)Ped2AvePed1 (c) Mixing methods

Res50Res152Dense161AlexNetMnasNet$74.00$$76.00$$78.00$$80.00$$82.00$$84.00$AUC
(%)HMDBUCF101Jester (d) Impact of $\mathcal{R}(\cdot)$

Figure 5: Component Analysis of zxVAD. Fig. 5(a) shows the loss ablation using
SHT${}_{\text{dc}}$; Fig. 5(b) compares the cross-domain performance of zxVAD
with different future-frame prediction backbones on three datasets (source:
SHT); Fig. 5(c) compares the impact of different mixing strategies in module
$\bm{\mathcal{O}}$ with SOTA method [66] that also presents a pseudo-anomaly
method on three datasets (source: SHT); Fig. 5(d) compares the impact of
network $\mathcal{R}(\cdot)$ in module $\bm{\mathcal{O}}$ on three TI datasets
(source: SHT, target: Ave). Dotted lines in Fig. 5(b), 5(c) (three datasets),
and 5(d) (one dataset) show SOTA (MPN [2] with Ped1: 74.45%, Ped2: 90.17%,
Ave: 74.06%) in respective cross-domain VAD when source is SHT. Table 3:
Comparison in Efficiency and Same-dataset testing. We beat our baselines in
most of the same-dataset testing, and outperform them in the listed efficiency
metrics. ⋆: GPU energy consumption is measured by testing on Ped2. †: rGAN [1]
does not provide its official testing code for inference-time metric
evaluation.

| Efficiency Metrics |  | Same Dataset Testing
---|---|---|---
Method |  |  Parameters ($\downarrow$)
---
(millions)
GMACs ($\downarrow$) |  |  Energy ($\downarrow$)
---
(Joules)⋆
|  Storage ($\downarrow$)
---
(MegaByte)
FPS ($\uparrow$) |  | SHT${}_{\text{dc}}$ | Ped2 | SHT
rGAN [1] | 19.0 | 1384.52 | –† | 79.85 | 2.1 |  | 70.11 | 96.90 | 77.90
MPN [2] | 12.7 | 55.09 | 10.65 | 53.14 | 166.8 |  | 67.47 | 96.20 | 73.80
zxVAD | 8.73 | 43.10 | 6.81 | 34.92 | 208.5 |  | 70.85 | 96.95 | 71.60

Table 4: Cross-dataset testing. Comparison with xVAD works that need no
background-subtraction. The best and second best AUC are marked in bold and
underline, respectively. ‡: For MPN [2], the publicized code [98] gives lower
AUC than what was reported in their paper.

|  |  | VAD Testing Data
---|---|---|---
VAD Training Data (Input to $\mathcal{G}(\cdot)$) |  Auxiliary Data (Input to $\bm{\mathcal{O}}$) | Method | Ped1 | Ped2 | Ave
SHT | N/A | rGAN [1] (paper) | 73.10 | 81.95 | 71.43
SHT | N/A | MPN [2] (paper) | 74.45 | 90.17 | 74.06
SHT | N/A | MPN [2] (code)‡ | 66.05 | 84.73 | 74.06
SHT | SHT | zxVAD (ours) | 76.14 | 95.78 | 82.28
SHT | HMDB | zxVAD (ours) | 75.62 | 95.74 | 83.19
SHT | UCF101 | zxVAD (ours) | 75.41 | 95.80 | 82.25
SHT | Jester | zxVAD (ours) | 75.93 | 95.62 | 82.49
UCFC | N/A | rGAN [1] (paper) | 66.87 | 62.53 | 64.32
UCFC | N/A | MPN [2] (paper) | 75.52 | 86.04 | 82.26
UCFC | UCFC | zxVAD (ours) | 78.61 | 91.65 | 81.11
UCFC | HMDB | zxVAD (ours) | 78.02 | 87.66 | 81.50
UCFC | UCF101 | zxVAD (ours) | 76.27 | 86.80 | 81.45
UCFC | Jester | zxVAD (ours) | 78.39 | 88.71 | 81.55

Table 5: Cross-dataset testing. Cross-domain performance when zxVAD model only
trained with TI datasets.

|  | VAD Testing Data
---|---|---
TI Training Data (Input to $\mathcal{G}(\cdot)$) |  Auxiliary Data (Input to $\bm{\mathcal{O}}$) | Ped1 | Ped2 | Ave
HMDB | HMDB | 76.66 | 91.53 | 81.92
UCF101 | UCF101 | 75.67 | 85.84 | 81.78
Jester | Jester | 78.12 | 91.23 | 78.06

[47][57][48]zxVAD$70.00$$72.00$$74.00$$76.00$$78.00$N/P$71.6$$74.4$$76.14$AUC
(%)SHT $\rightarrow$ Ped1

[47][57][48]zxVAD$74.00$$76.00$$78.00$$80.00$$82.00$$84.00$$74.7$$78.8$$81$$82.28$AUC
(%)SHT $\rightarrow$ Ave

[47][57][48]zxVAD$90.00$$92.00$$94.00$$96.00$$98.00$$100.00$$90.3$$95.7$$95.78$$95.9$AUC
(%)SHT $\rightarrow$ Ped2

Figure 6: Cross-dataset testing. Comparison with VAD works that need
background-subtraction. The best AUC is marked in red. ‘N/P’ in leftmost plot
means ‘Not Provided.’

$0$$25$$50$$75$$100$$125$$150$$175$$200$$0.0$$0.2$$0.4$$0.6$$0.8$$1.0$Frame
IndexAnomaly ScorePed1 (video #03)Ground-TruthMPN [2]zxVAD (ours)

$0$$25$$50$$75$$100$$125$$150$$175$$0.0$$0.2$$0.4$$0.6$$0.8$$1.0$Frame
IndexAnomaly ScorePed2 (video #04)Ground-TruthMPN [2]zxVAD (ours)

Figure 7: Anomaly detection curve. We compare our cross-domain (source: SHT)
anomaly detection on Ped1/Ped2 against MPN [2]. Larger value in curves
indicates possible anomalies.

#### Ablation study.

We show the ablation study of our proposed loss functions in zxVAD in Fig.
5(a) on the SHT${}_{\text{dc}}$ dataset. Fig. 5(a) shows that each of our
proposed loss functions contributes to the AUC, and jointly training with them
all achieves the best AUC. In Fig. 5(b), we analyze different combinations of
an autoencoder and generative adversarial network with (AE-M,GAN-M) and
without our memory module (AE, GAN) as our zxVAD backbone. In Fig. 5(c), we
analyze the impact of different mixing strategies (MixUp [99], CutMix [100]
within our module $\bm{\mathcal{O}}$ and compare with recent SOTA strategy
called Patch [66] that proposes a pseudo-anomaly method. In Fig. 5(d), we
analyze impact of changing $\mathcal{R}(\cdot)$ with (ResNet50, ResNet152
[78], DenseNet161 [101], AlexNet [102], MnasNet [103]) in zxVAD. Fig. 5 shows
that regardless of the backbone choice, the pseudo-anomaly strategy, and the
architecture of $\mathcal{R}(\cdot)$, zxVAD still outperforms the SOTA
baselines in most settings, which supports that zxVAD is flexible w.r.t. these
factors.

Same-dataset experiments. We compare zxVAD with [1, 2] on the
SHT${}_{\text{dc}}$, SHT and Ped2 datasets. Tab. 3 shows that zxVAD
outperforms both baselines in AUC in such experiment. For example, zxVAD shows
better generalization ability across different camera angles than the
baselines in the SHT${}_{\text{dc}}$ dataset with the least efficiency metrics
like model parameters and GMACs. We also find that using extra TI data (HMDB
and UCF101) can improve the AUC further compared to baselines (results in
Supplementary Material).

Cross-dataset experiments. We compare zxVAD with [1, 2] under the cross-
dataset setting. In the top two sections of Tab. 5, we train zxVAD with either
the SHT or UCFC dataset with optional TI data and test it on the Ped1, Ped2,
and Ave datasets. Tab. 5 shows that zxVAD outperforms both baselines in AUC
under most settings, regardless of whether the extra TI data are used, which
supports that zxVAD has better generalization ability across different
datasets (with different types of anomalies under different scenes) than the
baselines. For example, when our model is trained on the SHT dataset [16], it
outperforms existing xVAD methods in the proposed problem setup in detecting
anomalies like “chasing” and “brawling” in SHT’s test set as well as anomalies
like “bicycles” and “ cars” in Ped1/Ped2’s test set without performing any
kind of adaptation on Ped1/Ped2’s training set. This shows that our method is
not specific to anomalies in the source domain, but generalizes well to target
domain scenes during inference without adaptation. Tab. 5 shows that even
without using any source domain VAD training data at all, zxVAD still
outperforms [1, 2] in most settings by training with only TI data, which
supports our proposed mechanism of using the TI data under the proposed
problem setup. These encouraging results suggest that making use of TI data is
a promising research direction for the zxVAD problem. Interestingly, when
either $\mathcal{G}$ or $\bm{\mathcal{O}}$ or both use TI data, it’s not
surprising to see slightly lower AUC than if both $\mathcal{G}$ and
$\bm{\mathcal{O}}$ use VAD relevant data, _i.e_. more relevant source data
lead to less source-target domain gap, resulting in better AUC. This is
confirmed by average AUC (Tab. 5 and 5) when source is only VAD: 84.26%, VAD
w/ TI: 83.46%, and only TI: 82.30%. We also analyzed the impact of the amount
videos needed when solely training with TI data with HMDB and UCF101 in zxVAD
setup and found that even as little as $\sim$1.25% of UCF101 or $\sim$8% of
HMDB is enough to outperform the SOTA (details in Supplementary Material).
Following [1], we do not perform a cross-domain evaluation with Ped1/Ped2 as a
source as the training dataset is too small to make reasonable conclusions. In
Fig. 6, we show that zxVAD outperforms existing strong prior based
unsupervised xVAD methods [47, 48, 57] that report cross-domain VAD testing
performance when source domain data is SHT. This implies that zxVAD provides a
computationally efficient and reduced supervision approach with no need for
object extraction from videos (using YOLOv3 [49] in [48, 57] and CenterNet
[104] in [47]) both in source and target domain, under the proposed problem
setup. Compared to [48] (in Fig. 6) and [66] (in Fig. 5(c)), our untrained CNN
based abnormal example generation strategy results in superior VAD for the
proposed problem setup. Our “relative normalcy” learning approach optimizes
the VAD model to learn features that differentiate normal events from
(pseudo)-abnormal events, rather than focusing on learning only patterns of
normal events as in prior xVAD works. Results in Tab. 5 (when zxVAD uses only
TI data) validate this claim as zxVAD still outperforms SOTA on target VAD by
learning such differentiating features from TI videos. [105] is a few-shot VAD
method that puts together three off-the-shelf pre-trained models (YOLOv4
[106], AlphaPose [107], Flownet2 [108]) to perform xVAD. Even with such
costlier storage, high training overhead, and strong priors from different
distributions, zxVAD easily beats [105] by 11.76% (Ave), 13.85% (Ped2) with
source as SHT, and 10.12% (Ave), 29.12% (Ped2) with source as UCFC with
extremely less parameters and no initial priors. Finally, we provide
qualitative evaluation under the cross-domain setting with anomaly curves of
two testing videos of Ped1 and Ped2 when trained with SHT in Fig. 7, where
zxVAD provides better cross-domain detection ability than MPN [2]. We also
visualize difference maps in Fig. 8 (absolute error between ground truth and
the predicted frame) that indicate the presence of anomalies by zxVAD in three
datasets under cross-domain setting after training with SHT. We show more such
qualitative results of zxVAD in Supplementary Material. In addition to the
above, zxVAD achieves such results with much better inference-time efficiency
than the baselines. Tab. 3 shows that zxVAD outperforms [1, 2] in model size,
total parameters, GPU energy consumption (computed by pyJoules [109] following
[110, 111]), and GMACs by 34.3%, 31.3%, 36.1%, and 21.76%, respectively.

##  5 Conclusion

We identify a new unsupervised xVAD problem of detecting anomalies in the
target domain where no target domain training data are available. To tackle
this problem, we propose a novel framework named ‘Zero-shot Cross-domain Video
Anomaly Detection’ (zxVAD). zxVAD aims to learn features of normal activities
in input videos by learning how such features are relatively different from
features of pseudo-abnormal frames.

Figure 8: Difference maps. We show examples of cross-domain frame prediction
comparison on three datasets (source: SHT). The lighter colors in difference
map mean larger prediction error indicating anomalies. Red boxes indicate
ground truth anomalies.

Finally, zxVAD outperforms the SOTA baselines in most settings under common
benchmarks not only in AUC but also in four efficiency metrics, regardless of
whether the source domain VAD training data are available or not. Our results
demonstrates the potential of task-irrelevant data as a promising direction
for addressing the xVAD problem. As part of our future work, we will extend
our method to enhance it’s ability in directly localizing the anomaly in the
videos. Another important aspect of VAD models to explore is sensitivity to
input manipulations [112, 113, 114, 115] in order to test their robustness.
Acknowledgement. AA and KP were supported by Mitsubishi Electric Research
Laboratories. AA and ARC were partially supported by ONR grants
N00014-19-1-2264 and N00014-18-1-2252.

Supplementary material for “Cross-Domain Video Anomaly Detection
without Target Domain Adaptation”

CONTENTS

toc

List of Tables

lot

List of Figures

lof

##  A Additional Details for zxVAD

#### Additional Implementation Details.

Table 1: Augmentation parameters. K denotes kornia.augmentation.

Operation |  | Kornia Parameters
---|---|---
K.ColorJitter |  | $0.1,0.1,0.1,0.1$
K.RandomAffine |  | $degrees=360$
K.RandomPerspective |  | $distortion\\_scale=0.2$

We implement zxVAD in PyTorch [95]. We resize the input frames to $256\times
256$ and normalize them to the range of $[-1,1]$. The generator, the
discriminator, and the normalcy classifier are trained with the learning rates
of $0.0002$, $0.00002$, and $0.00002$, respectively with the Adam [116]
optimizer ($\beta_{1}=0.5,\beta_{2}=0.999$), following [3]. The generator
takes 4 frames as input and outputs one frame. We drop the last sigmoid layer
of $\mathcal{N}(\cdot)$ as suggested in [117]. We extracted the frames of all
TI datasets at 30 frames/sec. The batch size is set as 8. The training
iterations for both SHT and UCFC are set as 5000 in all settings of
combinations with TI datasets. Unless otherwise specified, we use the default
PyTorch parameters. The average training time is $\sim$2 hours for VAD
datasets and $\sim$24 hours for the experiments involving TI datasets on the
Nvidia Titan Xp GPUs.

#### Augmentation Parameters.

Our relative attention affirmation loss $\mathcal{L}_{\text{RAA}}$ requires
augmentation of normal frames $\bm{v}$ using Kornia [82] to create augmented
normal frames $g(\bm{v})$. We use kornia.augmentation.AugmentationSequential
to apply these augmentation operations sequentially whose parameters are
listed in Tab. 1. kornia.augmentation.ColorJitter has four parameter values
that represent factors of “brightness,” “contrast,” “saturation,” and “hue.”
All the operations have probability parameter $p=1.0$.

#### Location and Size Parameters in Pseudo-Anomaly Synthesis Module.

Our untrained CNN based Pseudo-anomaly synthesis module $\bm{\mathcal{O}}$
creates pseudo-anomalies $\tilde{\bm{v}}$ by pasting cropped object
$\bm{M}_{\bm{x}}$ at random location $r_{z}$ with random size $r_{x}\times
r_{y}$. We start by initializing a temporary tensor $\overline{\bm{v}}$ with
$\bm{v}$. The random location $r_{z}$ is a rectangular box with coordinates
$(b_{1},b_{2},b_{3},b_{4})$ [100]. These are computed as
$b_{1}=b_{x}-\nicefrac{{b_{w}}}{{2}},b_{2}=b_{x}+\nicefrac{{b_{w}}}{{2}},b_{3}=b_{y}-\nicefrac{{b_{h}}}{{2}}$,
and $b_{4}=b_{y}+\nicefrac{{b_{h}}}{{2}}$, where $(b_{x},b_{y},b_{w},b_{h})$
are uniformly sampled as follows. If $H$ and $W$ are height and width of
$\bm{v}$ respectively, then
$b_{x}\sim\text{Unif}\big{(}0,W\big{)},b_{y}\sim\text{Unif}\big{(}0,H\big{)},b_{w}=W\sqrt{1-\beta},b_{h}=H\sqrt{1-\beta}$.
Here, $b_{2}>b_{1}$ and $b_{4}>b_{3}$. We then resize $\bm{M}_{\bm{x}}$ and
$\bm{M}$ to size $(b_{2}-b_{1})\times(b_{4}-b_{3})$. Finally, only the pixels
corresponding to regions where $\bm{M}_{(i,j)}=1$ are replaced in
$\overline{\bm{v}}$ to create anomaly frame $\tilde{\bm{v}}$. To handle
boundary conditions where $0\leq b_{x},b_{w}\leq W$ and $0\leq b_{x},b_{w}\leq
H$, we clip the values to be in the range of $[0,W]$ and $[0,H]$,
respectively. Here, $\beta\sim\text{Unif}\big{(}0,1\big{)}$.

#### Evaluation criteria.

For anomaly scores, we follow [3, 2] and compute Peak Signal to Noise Ratio
(PSNR) [76] scores per frame and normalize PSNR of all frames in each testing
video to the range [0, 1] in order to compare with ground-truth binary labels.
Note that we observed such normalization practice (adopted from [3]) impacts
anomaly scores.

##  B Additional Results on zxVAD

#### Impact of the amount of TI Data.

We analyzed the impact of the amount of TI data on our zxVAD framework in
extreme settings. Particularly, we evaluated zxVAD when the amount of videos
of TI datasets (HMDB and UCF101) is close to the number of training videos
available in the VAD datasets. With 0.5%, 1%, 2%, 4%, and 8% of HMDB data, we
observed an average cross-domain AUC performance of 74.99% on Ped1, 93.82% on
Ped2, and 79.49% on Ave. A similar observation was made on UCF101 (0.0625%,
0.125%, 0.315%, 0.63%, and 1.25% of data resulted in average cross-domain AUC
performance of 74.61% on Ped1, 94.17% on Ped2, and 79.46% on Ave). This
demonstrates that almost SOTA cross-domain performance on the current VAD
datasets is achievable even with an extremely low amount of TI data.

#### Relevancy among VAD data.

We followed [92, 93] for the relevancy analysis between the TI to target
domain (Ave, Ped1/2) VAD data. We observed higher relevancy scores among SHT
(to Ave: 0.241, to Ped1/2: 0.250) and UCFC (to Ave: 0.201, to Ped1/2: 0.167)
compared to average TI (to Ave: 0.186, to Ped1/2: 0.138). This confirms: TI
data is indeed less relevant to VAD data.

#### More results on the impact of randomly initialized networks for Pseudo-
Anomaly Synthesis.

We analyzed the impact of the randomly initialized network
$\mathcal{R}(\cdot)$ on our untrained CNN based pseudo-anomaly synthesis
module. In Fig. 1, it can be observed that our zxVAD method outperforms the
state-of-the-art (SOTA) xVAD works on the Ped1 and Ped2 datasets in the zero-
shot settings when the source is SHT irrespective of kind of randomly
initialized network $\mathcal{R}(\cdot)$ employed to extract objects from all
our TI datasets.

Res50Res152Dense161AlexNetMnasNet$74.50$$75.00$$75.50$$76.00$$76.50$$77.00$SOTAAUC
(%)Ped1HMDB51UCF101Jester

Res50Res152Dense161AlexNetMnasNet$90.00$$91.50$$93.00$$94.50$$96.00$SOTAAUC
(%)Ped2

Figure 1: Impact of $\mathcal{R}(\cdot)$ in zxVAD. The source is SHT.

#### More results on same-dataset testing.

We beat our baselines in the same-dataset testing in all VAD and TI
combination scenarios as shown in Tab. 3. We also compare with more state-of-
the-art unsupervised VAD methods under the same-dataset setting in Tab. 3.

Table 2: Same-dataset testing on the SHT${}_{\text{dc}}$ dataset. We beat our
baselines in all the source domain data settings.

VAD Training data | Input to $\bm{\mathcal{O}}$ | Method |  AUC (%) on SHT${}_{\text{dc}}$
---|---|---|---
SHT${}_{\text{dc}}$ | N/A |  rGAN [1] (paper) | 70.11
SHT${}_{\text{dc}}$ | N/A |  MPN [2] (code) | 67.47
SHT${}_{\text{dc}}$ | SHT${}_{\text{dc}}$ | zxVAD (ours) | 70.73
SHT${}_{\text{dc}}$ | HMDB | zxVAD (ours) | 70.85
SHT${}_{\text{dc}}$ | UCF101 | zxVAD (ours) | 70.80
SHT${}_{\text{dc}}$ | Jester | zxVAD (ours) | 70.50

Table 3: Additional same dataset testing comparison. The best and second best
AUC are marked in bold and underline, respectively.

Methods | Ped2 | Ave | SHT
---|---|---|---
MPPCA [8] | 69.3 | - | -
MPPC+SFA [8] | 61.3 | - | -
MDT [10] | 82.9 | - | -
ConvAE [22] | 85.0 | 80.0 | 60.9
TSC [16] | 91.0 | 80.6 | 67.9
StackRNN [16] | 92.2 | 81.7 | 68.0
MT-FRCN [58] | 92.2 | - | -
Unmasking [14] | 82.2 | 80.6 | -
Frame-Pred [3] | 95.4 | 85.1 | 72.8
AMC [15] | 96.2 | 86.9 | -
MemAE [18] | 94.1 | 83.3 | 71.2
SDOR [118] | 83.2 | - | -
rGAN [1] | 96.2 | 85.8 | 77.9
LMN [21] | 97.0 | 88.5 | 70.5
MPN [2] | 96.9 | 89.5 | 73.8
zxVAD | 96.95 | 83.8 | 71.6

Ratios | SHTdc
---|---
MPN [2] | 67.47
$(\alpha_{\text{n}},\alpha_{\text{rn}})=(1,0.01)$ | 70.85
$(\alpha_{\text{n}},\alpha_{\text{rn}})=(1,0.1)$ | 69.49
$(\alpha_{\text{n}},\alpha_{\text{rn}})=(0.1,0.1)$ | 69.95
$(\alpha_{\text{n}},\alpha_{\text{rn}})=(0.01,0.01)$ | 70.37

#### Ablation analysis.

zxVAD is not too sensitive to the loss ratios and Table (on right) validates
this point. For our backbone GAN, we use exact same ratios as suggested in
[3]. For the proposed normalcy classifier, we do not use ratios for our losses
$\mathcal{L}_{\text{AA}}$ and $\mathcal{L}_{\text{RAA}}$ (_i.e_. set as 1).
Finally, the effect of ratios $\alpha_{\text{n}}$ on $\mathcal{L}_{\text{N}}$
and $\alpha_{\text{rn}}$ on $\mathcal{L}_{\text{RN}}$ is shown. All cases show
better AUC than SOTA MPN [2].

##  C Examples from Datasets

We provide some video examples of the VAD datasets (SHT, UCFC, Ped1, Ped2, and
Ave in Fig.2(a)) and TI datasets (HMDB, UCF101, and Jester in Fig.2(b)) listed
in Tab.2 of the main manuscript.

(a) VAD datasets

(b) TI datasets

Figure 2: Examples from VAD and TI datasets. We visualize some examples of
videos used for experiments in our paper.

##  D More Qualitative Results

We show additional examples of pseudo-abnormal frames created using our
pseudo-anomaly module in Fig. 3 and difference maps from three different
datasets indicating anomalies in Fig. 4.

Figure 3: Pseudo-abnormal frames. We present examples of pseudo-abnormal
frames generated using our proposed untrained CNN based pseudo-anomaly
synthesis module. Figure 4: Difference maps. We show more examples of
difference maps obtained from zxVAD (source: SHT). The lighter colors in
difference map mean larger prediction error indicating anomalies. Red boxes
indicate ground truth anomalies. Best viewed in color.

## References

  * [1] Yiwei Lu, Frank Yu, Mahesh Kumar Krishna Reddy, and Yang Wang.  Few-shot scene-adaptive anomaly detection.  In European Conference on Computer Vision, pages 125–141. Springer, 2020.
  * [2] Hui Lv, Chen Chen, Zhen Cui, Chunyan Xu, Yong Li, and Jian Yang.  Learning normal dynamics in videos with meta prototype network.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15425–15434, 2021.
  * [3] Wen Liu, Weixin Luo, Dongze Lian, and Shenghua Gao.  Future frame prediction for anomaly detection–a new baseline.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6536–6545, 2018.
  * [4] Frederick Tung, John S Zelek, and David A Clausi.  Goal-based trajectory analysis for unusual behaviour detection in intelligent surveillance.  Image and Vision Computing, 29(4):230–240, 2011.
  * [5] Shandong Wu, Brian E Moore, and Mubarak Shah.  Chaotic invariants of lagrangian particle trajectories for anomaly detection in crowded scenes.  In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 2054–2060. IEEE, 2010.
  * [6] Dong Zhang, Daniel Gatica-Perez, Samy Bengio, and Iain McCowan.  Semi-supervised adapted hmms for unusual event detection.  In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), volume 1, pages 611–618. IEEE, 2005.
  * [7] Amit Adam, Ehud Rivlin, Ilan Shimshoni, and Daviv Reinitz.  Robust real-time unusual event detection using multiple fixed-location monitors.  IEEE transactions on pattern analysis and machine intelligence, 30(3):555–560, 2008.
  * [8] Jaechul Kim and Kristen Grauman.  Observe locally, infer globally: a space-time mrf for detecting abnormal activities with incremental updates.  In 2009 IEEE conference on computer vision and pattern recognition, pages 2921–2928. IEEE, 2009.
  * [9] Cewu Lu, Jianping Shi, and Jiaya Jia.  Abnormal event detection at 150 fps in matlab.  In Proceedings of the IEEE international conference on computer vision, pages 2720–2727, 2013.
  * [10] Vijay Mahadevan, Weixin Li, Viral Bhalodia, and Nuno Vasconcelos.  Anomaly detection in crowded scenes.  In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 1975–1981. IEEE, 2010.
  * [11] Yang Cong, Junsong Yuan, and Ji Liu.  Sparse reconstruction cost for abnormal event detection.  In CVPR 2011, pages 3449–3456. IEEE, 2011.
  * [12] Bin Zhao, Li Fei-Fei, and Eric P Xing.  Online detection of unusual events in videos via dynamic sparse coding.  In CVPR 2011, pages 3313–3320. IEEE, 2011.
  * [13] Dan Xu, Yan Yan, Elisa Ricci, and Nicu Sebe.  Detecting anomalous events in videos by learning deep representations of appearance and motion.  Computer Vision and Image Understanding, 156:117–127, 2017.
  * [14] Radu Tudor Ionescu, Sorina Smeureanu, Bogdan Alexe, and Marius Popescu.  Unmasking the abnormal events in video.  In Proceedings of the IEEE international conference on computer vision, pages 2895–2903, 2017.
  * [15] Trong-Nguyen Nguyen and Jean Meunier.  Anomaly detection in video sequence with appearance-motion correspondence.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1273–1283, 2019.
  * [16] Weixin Luo, Wen Liu, and Shenghua Gao.  A revisit of sparse coding based anomaly detection in stacked rnn framework.  In Proceedings of the IEEE International Conference on Computer Vision, pages 341–349, 2017.
  * [17] Mahdyar Ravanbakhsh, Moin Nabi, Enver Sangineto, Lucio Marcenaro, Carlo Regazzoni, and Nicu Sebe.  Abnormal event detection in videos using generative adversarial nets.  In 2017 IEEE International Conference on Image Processing (ICIP), pages 1577–1581. IEEE, 2017.
  * [18] Dong Gong, Lingqiao Liu, Vuong Le, Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton van den Hengel.  Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1705–1714, 2019.
  * [19] Bharathkumar Ramachandra and Michael Jones.  Street scene: A new dataset and evaluation protocol for video anomaly detection.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2569–2578, 2020.
  * [20] Fei Dong, Yu Zhang, and Xiushan Nie.  Dual discriminator generative adversarial network for video anomaly detection.  IEEE Access, 8:88170–88176, 2020.
  * [21] Hyunjong Park, Jongyoun Noh, and Bumsub Ham.  Learning memory-guided normality for anomaly detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14372–14381, 2020.
  * [22] Mahmudul Hasan, Jonghyun Choi, Jan Neumann, Amit K Roy-Chowdhury, and Larry S Davis.  Learning temporal regularity in video sequences.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 733–742, 2016.
  * [23] Guang Yu, Siqi Wang, Zhiping Cai, Xinwang Liu, Chuanfu Xu, and Chengkun Wu.  Deep anomaly discovery from unlabeled videos via normality advantage and self-paced refinement.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13987–13998, 2022.
  * [24] Keval Doshi and Yasin Yilmaz.  Multi-task learning for video surveillance with limited data.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3889–3899, 2022.
  * [25] Chaewon Park, MyeongAh Cho, Minhyeok Lee, and Sangyoun Lee.  Fastano: Fast anomaly detection via spatio-temporal patch transformation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2249–2259, 2022.
  * [26] M Zaigham Zaheer, Arif Mahmood, M Haris Khan, Mattia Segu, Fisher Yu, and Seung-Ik Lee.  Generative cooperative learning for unsupervised video anomaly detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14744–14754, 2022.
  * [27] MyeongAh Cho, Taeoh Kim, Woo Jin Kim, Suhwan Cho, and Sangyoun Lee.  Unsupervised video anomaly detection via normalizing flows with implicit latent features.  Pattern Recognition, 129:108703, 2022.
  * [28] Xiangru Lin, Yuyang Chen, Guanbin Li, and Yizhou Yu.  A causal inference look at unsupervised video anomaly detection.  Proceedings of the 6th AAAI Conference on Artificial Intelligence, 2022.
  * [29] Yuxin Zhang, Jindong Wang, Yiqiang Chen, Han Yu, and Tao Qin.  Adaptive memory networks with self-supervised learning for unsupervised anomaly detection.  IEEE Transactions on Knowledge and Data Engineering, 2022.
  * [30] Lin Wang, Haishu Tan, Fuqiang Zhou, Wangxia Zuo, and Pengfei Sun.  Unsupervised anomaly video detection via a double-flow convlstm variational autoencoder.  IEEE Access, 10:44278–44289, 2022.
  * [31] Guodong Shen, Yuqi Ouyang, and Victor Sanchez.  Video anomaly detection via prediction network with enhanced spatio-temporal memory exchange.  In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3728–3732. IEEE, 2022.
  * [32] Yuqi Ouyang, Guodong Shen, and Victor Sanchez.  Look at adjacent frames: Video anomaly detection without offline training.  arXiv preprint arXiv:2207.13798, 2022.
  * [33] Joo-Yeon Lee, Woo-Jeoung Nam, and Seong-Whan Lee.  Multi-contextual predictions with vision transformer for video anomaly detection.  arXiv preprint arXiv:2206.08568, 2022.
  * [34] Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel.  Deep learning for anomaly detection: A review.  ACM Computing Surveys (CSUR), 54(2):1–38, 2021.
  * [35] Saikiran Bulusu, Bhavya Kailkhura, Bo Li, Pramod K Varshney, and Dawn Song.  Anomalous example detection in deep learning: A survey.  IEEE Access, 8:132330–132347, 2020.
  * [36] Raghavendra Chalapathy and Sanjay Chawla.  Deep learning for anomaly detection: A survey.  arXiv preprint arXiv:1901.03407, 2019.
  * [37] Abhishek Aich, Meng Zheng, Srikrishna Karanam, Terrence Chen, Amit K. Roy-Chowdhury, and Ziyan Wu.  Spatio-temporal representation factorization for video-based person re-identification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 152–162, October 2021.
  * [38] Waqas Sultani, Chen Chen, and Mubarak Shah.  Real-world anomaly detection in surveillance videos.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6479–6488, 2018.
  * [39] Jia-Chang Feng, Fa-Ting Hong, and Wei-Shi Zheng.  Mist: Multiple instance self-training framework for video anomaly detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14009–14018, 2021.
  * [40] Wen Liu, Weixin Luo, Zhengxin Li, Peilin Zhao, Shenghua Gao, et al.  Margin learning embedded prediction for video anomaly detection with a few anomalies.  In IJCAI, pages 3023–3030, 2019.
  * [41] Yu Tian, Guansong Pang, Yuanhong Chen, Rajvinder Singh, Johan W Verjans, and Gustavo Carneiro.  Weakly-supervised video anomaly detection with robust temporal feature magnitude learning.  arXiv preprint arXiv:2101.10030, 2021.
  * [42] Didik Purwanto, Yie-Tarng Chen, and Wen-Hsien Fang.  Dance with self-attention: A new look of conditional random fields on anomaly detection in videos.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 173–183, 2021.
  * [43] Boyang Wan, Wenhui Jiang, Yuming Fang, Zhiyuan Luo, and Guanqun Ding.  Anomaly detection in video sequences: A benchmark and computational model.  IET Image Processing, 2021.
  * [44] Mariana-Iuliana Georgescu, Antonio Barbalau, Radu Tudor Ionescu, Fahad Shahbaz Khan, Marius Popescu, and Mubarak Shah.  Anomaly detection in video via self-supervised and multi-task learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12742–12752, 2021.
  * [45] Jaeyoo Park, Junha Kim, and Bohyung Han.  Learning to adapt to unseen abnormal activities under weak supervision.  In Proceedings of the Asian Conference on Computer Vision, 2020\.
  * [46] Antonio Barbalau, Radu Tudor Ionescu, Mariana-Iuliana Georgescu, Jacob Dueholm, Bharathkumar Ramachandra, Kamal Nasrollahi, Fahad Shahbaz Khan, Thomas B Moeslund, and Mubarak Shah.  Ssmtl++: Revisiting self-supervised multi-task learning for video anomaly detection.  arXiv preprint arXiv:2207.08003, 2022.
  * [47] Pankaj Raj Roy, Guillaume-Alexandre Bilodeau, and Lama Seoud.  Predicting next local appearance for video anomaly detection.  arXiv preprint arXiv:2106.06059, 2021.
  * [48] Mariana-Iuliana Georgescu, Radu Tudor Ionescu, Fahad Shahbaz Khan, Marius Popescu, and Mubarak Shah.  A background-agnostic framework with adversarial training for abnormal event detection in video.  arXiv preprint arXiv:2008.12328, 2020.
  * [49] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  arXiv preprint arXiv:1804.02767, 2018.
  * [50] Takafumi Moriya, Tomohiro Tanaka, Takahiro Shinozaki, Shinji Watanabe, and Kevin Duh.  Evolution-strategy-based automation of system development for high-performance speech recognition.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(1):77–88, 2018.
  * [51] Jiayi Liu, Samarth Tripathi, Unmesh Kurup, and Mohak Shah.  Auptimizer-an extensible, open-source framework for hyperparameter tuning.  In 2019 IEEE International Conference on Big Data (Big Data), pages 339–348. IEEE, 2019.
  * [52] Alevtina Krotova, Armin Mertens, and Marc Scheufen.  Open data and data sharing: An economic analysis.  Technical report, IW-Policy Paper, 2020.
  * [53] Michael Mattioli.  Disclosing big data.  Minn. L. Rev., 99:535, 2014.
  * [54] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  Communications of the ACM, 63(11):139–144, 2020.
  * [55] Abhishek Aich, Akash Gupta, Rameswar Panda, Rakib Hyder, M. Salman Asif, and Amit K. Roy-Chowdhury.  Non-adversarial video synthesis with learned priors.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [56] Akash Gupta, Abhishek Aich, and Amit K Roy-Chowdhury.  Alanet: Adaptive latent attention network for joint video deblurring and interpolation.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 256–264, 2020.
  * [57] Radu Tudor Ionescu, Fahad Shahbaz Khan, Mariana-Iuliana Georgescu, and Ling Shao.  Object-centric auto-encoders and dummy anomalies for abnormal event detection in video.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7842–7851, 2019.
  * [58] Ryota Hinami, Tao Mei, and Shin’ichi Satoh.  Joint detection and recounting of abnormal events by learning deep generic knowledge.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3619–3627, 2017.
  * [59] Che Sun, Yunde Jia, Yao Hu, and Yuwei Wu.  Scene-aware context reasoning for unsupervised abnormal event detection in videos.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 184–192, 2020.
  * [60] Guang Yu, Siqi Wang, Zhiping Cai, En Zhu, Chuanfu Xu, Jianping Yin, and Marius Kloft.  Cloze test helps: Effective video anomaly detection via learning to complete video events.  In Proceedings of the 28th ACM International Conference on Multimedia, pages 583–591, 2020.
  * [61] Zhian Liu, Yongwei Nie, Chengjiang Long, Qing Zhang, and Guiqing Li.  A hybrid video anomaly detection framework via memory-augmented flow reconstruction and flow-guided frame prediction.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13588–13597, 2021.
  * [62] Chun-Yu Chen and Yu Shao.  Crowd escape behavior detection and localization based on divergent centers.  IEEE Sensors Journal, 15(4):2431–2439, 2014.
  * [63] Muhammad Zaigham Zaheer, Jin-ha Lee, Marcella Astrid, and Seung-Ik Lee.  Old is gold: Redefining the adversarially learned one-class classifier training paradigm.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14183–14193, 2020.
  * [64] Keval Doshi and Yasin Yilmaz.  Any-shot sequential anomaly detection in surveillance videos.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 934–935, 2020.
  * [65] Marcella Astrid, Muhammad Zaigham Zaheer, and Seung-Ik Lee.  Synthetic temporal anomaly guided end-to-end video anomaly detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshop, pages 207–214, 2021.
  * [66] Marcella Astrid, Muhammad Zaigham Zaheer, Jae-Yeong Lee, and Seung-Ik Lee.  Learning not to reconstruct anomalies.  arXiv preprint arXiv:2110.09742, 2021.
  * [67] Masoud Pourreza, Bahram Mohammadi, Mostafa Khaki, Samir Bouindour, Hichem Snoussi, and Mohammad Sabokrou.  G2d: Generate to detect anomaly.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2003–2012, 2021.
  * [68] Yutao Hu, Xin Huang, and Xiaoyan Luo.  Adaptive anomaly detection network for unseen scene without fine-tuning.  In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pages 311–323. Springer, 2021.
  * [69] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey.  Meta-learning in neural networks: A survey.  arXiv preprint arXiv:2004.05439, 2020.
  * [70] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.
  * [71] Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine.  Meta-learning with implicit gradients.  In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 113–124, 2019.
  * [72] Jason Weston, Sumit Chopra, and Antoine Bordes.  Memory networks.  arXiv preprint arXiv:1410.3916, 2014.
  * [73] Jack W Rae, Jonathan J Hunt, Tim Harley, Ivo Danihelka, Andrew Senior, Greg Wayne, Alex Graves, and Timothy P Lillicrap.  Scaling memory-augmented neural networks with sparse reads and writes.  arXiv preprint arXiv:1610.09027, 2016.
  * [74] Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap.  One-shot learning with memory-augmented neural networks.  arXiv preprint arXiv:1605.06065, 2016.
  * [75] Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley.  Least squares generative adversarial networks.  In Proceedings of the IEEE international conference on computer vision, pages 2794–2802, 2017.
  * [76] Michael Mathieu, Camille Couprie, and Yann LeCun.  Deep multi-scale video prediction beyond mean square error.  arXiv preprint arXiv:1511.05440, 2015.
  * [77] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612, 2004.
  * [78] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [79] Xiu-Shen Wei, Jian-Hao Luo, Jianxin Wu, and Zhi-Hua Zhou.  Selective convolutional descriptor aggregation for fine-grained image retrieval.  IEEE Transactions on Image Processing, 26(6):2868–2881, 2017.
  * [80] Yun-Hao Cao and Jianxin Wu.  A random cnn sees objects: One inductive bias of cnn and its applications.  Proceedings of the 6th AAAI Conference on Artificial Intelligence, 2022.
  * [81] Richard HR Hahnloser, Rahul Sarpeshkar, Misha A Mahowald, Rodney J Douglas, and H Sebastian Seung.  Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit.  Nature, 405(6789):947–951, 2000.
  * [82] Edgar Riba, Dmytro Mishkin, Daniel Ponsa, Ethan Rublee, and Gary Bradski.  Kornia: an open source differentiable computer vision library for pytorch.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3674–3683, 2020.
  * [83] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690–4699, 2019.
  * [84] Kevin Musgrave, Serge Belongie, and Ser-Nam Lim.  Pytorch metric learning, 2020.
  * [85] Hui Wu, Min Wang, Wengang Zhou, and Houqiang Li.  Learning deep local features with multiple dynamic attentions for large-scale image retrieval.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11416–11425, 2021.
  * [86] Min Yang, Dongliang He, Miao Fan, Baorong Shi, Xuetong Xue, Fu Li, Errui Ding, and Jizhou Huang.  Dolg: Single-stage image retrieval with deep orthogonal fusion of local and global features.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11772–11781, 2021.
  * [87] Arman Afrasiyabi, Jean-Francois Lalonde, and Christian Gagne.  Mixture-based feature space learning for few-shot image classification.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9041–9051, 2021.
  * [88] Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre.  Hmdb: a large video database for human motion recognition.  In 2011 International conference on computer vision, pages 2556–2563. IEEE, 2011.
  * [89] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  arXiv preprint arXiv:1212.0402, 2012.
  * [90] Joanna Materzynska, Guillaume Berger, Ingo Bax, and Roland Memisevic.  The jester dataset: A large-scale video dataset of human gestures.  In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [91] Andra Acsintoae, Andrei Florescu, Mariana-Iuliana Georgescu, Tudor Mare, Paul Sumedrea, Radu Tudor Ionescu, Fahad Shahbaz Khan, and Mubarak Shah.  Ubnormal: New benchmark for supervised open-set video anomaly detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20143–20153, 2022.
  * [92] Kuan-Chuan Peng, Ziyan Wu, and Jan Ernst.  Zero-shot deep domain adaptation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 764–781, 2018.
  * [93] Zhenyong Fu, Tao Xiang, Elyor Kodirov, and Shaogang Gong.  Zero-shot object recognition by semantic manifold distance.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2635–2644, 2015.
  * [94] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  In Advances in neural information processing systems, pages 3111–3119, 2013.
  * [95] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems, 32:8026–8037, 2019.
  * [96] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [97] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
  * [98] The official codes of the CVPR21 paper: Learning Normal Dynamics in Videos with Meta Prototype Network.  https://github.com/ktr-hubrt/MPN.
  * [99] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  arXiv preprint arXiv:1710.09412, 2017.
  * [100] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6023–6032, 2019.
  * [101] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017.
  * [102] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems, 25:1097–1105, 2012.
  * [103] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le.  Mnasnet: Platform-aware neural architecture search for mobile.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2820–2828, 2019.
  * [104] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl.  Objects as points.  arXiv preprint arXiv:1904.07850, 2019.
  * [105] Keval Doshi and Yasin Yilmaz.  A modular and unified framework for detecting and localizing video anomalies.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3982–3991, 2022.
  * [106] Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao.  Yolov4: Optimal speed and accuracy of object detection.  arXiv preprint arXiv:2004.10934, 2020.
  * [107] Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu.  Rmpe: Regional multi-person pose estimation.  In Proceedings of the IEEE international conference on computer vision, pages 2334–2343, 2017.
  * [108] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox.  Flownet 2.0: Evolution of optical flow estimation with deep networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2462–2470, 2017.
  * [109] pyJoules.  https://pypi.org/project/pyJoules/.
  * [110] Krishnateja Killamsetty, Xujiang Zhao, Feng Chen, and Rishabh Iyer.  Retrieve: Coreset selection for efficient and robust semi-supervised learning.  Proceedings of the 33rd International Conference on Neural Information Processing Systems, 2021.
  * [111] Krishnateja Killamsetty, Durga Sivasubramanian, Baharan Mirzasoleiman, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer.  Grad-match: A gradient matching based data subset selection for efficient learning.  Proceedings of the 38th International Conference on Machine Learning, 2021.
  * [112] Furkan Mumcu, Keval Doshi, and Yasin Yilmaz.  Adversarial machine learning attacks against video anomaly detection systems.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 206–213, 2022.
  * [113] Abhishek Aich, Calvin-Khang Ta, Akash A Gupta, Chengyu Song, Srikanth Krishnamurthy, M. Salman Asif, and Amit Roy-Chowdhury.  Gama: Generative adversarial multi-object scene attacks.  In Thirty-Sixth Conference on Neural Information Processing Systems, 2022.
  * [114] Abhishek Aich, Shasha Li, Chengyu Song, M Salman Asif, Srikanth V Krishnamurthy, and Amit K Roy-Chowdhury.  Leveraging local patch differences in multi-object scenes for generative adversarial attacks.  arXiv preprint arXiv:2209.09883, 2022.
  * [115] Shasha Li, Abhishek Aich, Shitong Zhu, Salman Asif, Chengyu Song, Amit Roy-Chowdhury, and Srikanth Krishnamurthy.  Adversarial attacks on black box video classifiers: Leveraging the power of geometric transformations.  Advances in Neural Information Processing Systems, 34, 2021.
  * [116] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [117] Alexia Jolicoeur-Martineau.  The relativistic discriminator: a key element missing from standard gan.  arXiv preprint arXiv:1807.00734, 2018.
  * [118] Guansong Pang, Cheng Yan, Chunhua Shen, Anton van den Hengel, and Xiao Bai.  Self-trained deep ordinal regression for end-to-end video anomaly detection.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12173–12182, 2020.


$p_{0}<j\leq p$ since $\lambda_{j}<a_{n}$. Then by the Central Limit Theorem,
Condition 3 and using Lemma 1 and the linearity of the expectation operator we
know $E\left(\frac{\partial R(\beta_{0}|x^{n})}{\partial\beta_{j}}\right)=0$,
so we have

$\sqrt{n}\left(\frac{1}{n}\frac{\partial
R(\beta_{0}|x^{n})}{\partial\beta_{j}}-E\left(\frac{\partial
R(\beta_{0}|x^{n})}{\partial\beta_{j}}\right)\right)\overset{D}{\longrightarrow}\mathcal{N}(0,I^{*}_{1}(\beta_{10})).$

Thus,

$\frac{1}{\sqrt{n}}\frac{\partial
R(\beta_{0}|x^{n})}{\partial\beta_{j}}\overset{D}{\longrightarrow}\mathcal{N}(0,I^{*}_{1}(\beta_{10})).$

By equality in (7.1), it follows that

$\sqrt{n}I^{*}_{1}(\beta_{10}|x^{n})(\hat{\beta_{1}}-\beta_{10})\overset{D}{\longrightarrow}\mathcal{N}(0,I^{*}_{1}(\beta_{10})).$

∎

## References

## References

  * [1] A. Hoerl, Application of ridge analysis to regression problems, Chemical Engineering Progress 58 (1962) 54–59.
  * [2] J. Sjöburg, L. Ljung, Overtraining, regularization, and searching for minimum in neural networks, in: Proceedings of the 4th IFAC Symposium on Adaptive Systems in Control and Signal Processing, Vol. 25, 1992, pp. 73–78.
  * [3] R. Tibshirani, Regression shrinkage and selection via the lasso, J. Roy. Statist. Soc., Ser. B 58 (1996) 267–288.
  * [4] J. Fan, R. Li, Variable selection via concave penalized likelihood and its oracle properties, J. Amer. Statist. Assoc. 96 (2001) 1348–1360.
  * [5] C.-H. Zhang, Nearly unbiased variable selection under minimax concave penalty, Ann. Statist. 38 (2010) 894–942.
  * [6] H. Zou, T. Hastie, Regularization and variable selection via the elastic net, J. Roy. Stat. Soc., Ser. B. 67 (2005) 301–320.
  * [7] H. Zou, H. H. Zhang, On the adaptive elastic-net with a diverging number of parameters, Ann. Statist. 37 (2009) 1733 – 1751.
  * [8] H. Zou, The adaptive lasso and its oracle properties, J. Amer. Statist. Assoc. 101 (2006) 1418–1429.
  * [9] H. Wang, G. Li, G. Giang, Robust regression shrinkage and consistent variable selection through the lad-lasso., J. business and Economic Stat. 25 (2007) 347–355.
  * [10] W. Gian, Y. Yang, Model selection via standard error adjusted adaptive lasso, Ann. Inst. Stat. Math 65 (2013) 295–318.
  * [11] J. Fan, J. Lv, Asymptotic equivalence of regularization methods in thresholded parameter spaces, J. Amer. Statist. Assoc. 108 (2013) 1044–1061.
  * [12] X. Luo, L. Stefanski, D. Boos, Variable selection via concave penalized likelihood and its oracle properties, Technometrics 48 (2006) 165–175.
  * [13] J. Friedman, T. Hastie, R. Tibshirani, Regularization paths for generalized linear models via coordinate descent, Journal of Statistical Software 33 (1) (2010) 1–22.
URL https://www.jstatsoft.org/v33/i01/

  * [14] B. Sherwood, A. Maidman, rqPen: Penalized Quantile Regression, r package version 2.2.2 (2020).
URL https://CRAN.R-project.org/package=rqPen

  * [15] J. Fan, L. Xue, H. Zou, Strong oracle optimality of folded concave penalized estimation, Ann. Stat. 42 (2014) 819–849.
  * [16] P. Zhao, B. Yu, On model selection consistency selection of lasso, J. M. L. R. 7 (2006) 2541–2563.
  * [17] K. Hamidieh, A data-driven statistical model for predicting the critical temperature of a superconductor, J. Amer. Statist. Assoc. 96 (2018) 1348–1360.
  * [18] G. Givens, J. Hoeting, Computational Statistics, 2nd Edition, Wiley, Hoboken, NJ, 2013.
  * [19] G. Rudolph, Convergence of evolutionary algorithms in general search spaces, in: Proc. of IEEE International Conference on Evolutionary Computation, 1996, pp. 50–54.  doi:10.1109/ICEC.1996.542332.
  * [20] E. Willighagen, M. Ballings, genalg: R Based Genetic Algorithm, r package version 0.2.0 (2015).
URL https://CRAN.R-project.org/package=genalg

  * [21] H. Royden, P. FitzPatrick, Real Analysis, 4th Edition, Prentice-Jall, Hoboken, NJ, 2010.
  * [22] B. Hoadley, Asymptotic properties of maximum likelihood estimators for the independent not identically distributed case., Ann. math. Statsit. 40 (1971) 1977–1991.