# LLMem: Estimating GPU Memory Usage for Fine-Tuning Pre-Trained LLMs

Taeho Kim1111This work is based on Taeho Kim’s internship at AWS. Yanming
Wang2 Vatshank Chaturvedi2 Lokesh Gupta2 Seyeon Kim1 Yongin Kwon3 Sangtae Ha1
1University of Colorado Boulder
2Amazon Web Services222This work is unconnected to current role at AWS.
3Electronics and Telecommunications Research Institute
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Fine-tuning pre-trained large language models (LLMs) with limited hardware
presents challenges due to GPU memory constraints. Various distributed fine-
tuning methods have been proposed to alleviate memory constraints on GPU.
However, determining the most effective method for achieving rapid fine-tuning
while preventing GPU out-of-memory issues in a given environment remains
unclear. To address this challenge, we introduce LLMem, a solution that
estimates the GPU memory consumption when applying distributed fine-tuning
methods across multiple GPUs and identifies the optimal method. We conduct GPU
memory usage estimation prior to fine-tuning, leveraging the fundamental
structure of transformer-based decoder models and the memory usage
distribution of each method. Experimental results show that LLMem accurately
estimates peak GPU memory usage on a single GPU, with error rates of up to
1.6%. Additionally, it shows an average error rate of 3.0% when applying
distributed fine-tuning methods to LLMs with more than a billion parameters on
multi-GPU setups.

##  1 Introduction

Since the introduction of the Transformer model Vaswani et al. (2017),
researchers have proposed numerous language models based on it. As the model’s
performance has improved, its size has grown exponentially, necessitating a
substantial dataset for training. However, training emerging large language
models (LLMs) is infeasible without a dedicated infrastructure with high-
performance hardware due to memory constraints. Instead, it is preferred to
utilize a small dataset to fine-tune a pre-trained model for a specific
application.

To efficiently handle small datasets and reduce training time, the
conventional method of data parallelism places the entire model on each GPU,
splits the dataset, and trains simultaneously. Nevertheless, the model size
remains huge, potentially causing GPU out-of-memory (OOM) issues. Therefore,
it is necessary to reduce the amount of memory a GPU uses by splitting the
model and distributing it to each GPU.

ZeRO Rajbhandari et al. (2020) Stage 3 is an advanced data parallelism method
that partitions the model parameters, gradients, and optimizer states to each
GPU for memory advantage while maintaining the distribution of the dataset
across GPUs. Although ZeRO Stage 3 saves memory by using only partitioned
model data on each GPU during non-computation phases, there are limitations in
preventing GPU OOM issues because partitioned parameters/gradients must be
all-gathered during computation.

Tensor parallelism divides each parameter tensor in the model into rows or
columns and distributes them to each GPU, using only partitioned parameters on
each GPU during computation. For example, Megatron-LM Shoeybi et al. (2019), a
representative tensor parallelism method, splits a tensor along its rows or
columns considering the position and connection of operators. By doing so, it
can reduce GPU memory usage more than data parallelism when the model size is
large.

As we described above, various distributed fine-tuning methods have been
proposed, but the GPU memory usage and fine-tuning time required for each are
different. For instance, conventional data parallelism provides the shortest
fine-tuning time but requires the highest GPU memory usage. On the other hand,
tensor parallelism has no benefit in saving fine-tuning time but can
significantly reduce GPU memory usage. Users may want to select an appropriate
method that avoids GPU OOM and has a short fine-tuning time. However, it is
difficult to determine in advance whether there is enough GPU memory to fine-
tune a given pre-trained LLM.

DNNMem Gao et al. (2020) is the most recent work detailing procedures to
estimate GPU memory usage on a single GPU. DNNMem provides key equations for
GPU memory estimation when training various DNN models by analyzing the
connections between operators and live tensors in the forward and backward
passes. However, it has limitations for fine-tuning LLMs. GPU memory
estimation for fine-tuning transformer-based LLM is challenging for two
reasons.

First, when fine-tuning an LLM in multi-GPU, distributed fine-tuning methods
should be used to overcome GPU memory constraints due to large model sizes.
Depending on the method used, the distribution of parameters, gradients, and
optimizer states to each GPU is different, as is the amount of GPU memory used
during the calculation process. Therefore, GPU memory usage estimates from a
single GPU cannot be used in a multi-GPU environment.

Second, GPU memory consumption must be predicted by distinguishing between
transformer and language modeling head (lm_head) parts. The transformer part
is the central part of fine-tuning, where chunk memory management for memory
sharing of model parameters and gradients is applied, and parameters are
updated. On the other hand, the lm_head part requires separate analysis
because it does not apply distributed methods directly and consumes a lot of
memory due to its large dictionary size.

To address these challenges, we propose LLMem that estimates the GPU memory
consumption when applying distributed fine-tuning methods to multiple GPUs.
LLMem considers several factors to estimate GPU memory usage for each method,
including recombining parameters prior to computation when applying advanced
data parallelism and the output driven by all-gather in the backward pass when
using tensor parallelism. Additionally, LLMem analyzes the difference in
memory allocation method between the transformer and the lm_head part and
reflects it in GPU memory estimation. To the best of our knowledge, this is
the first work to estimate the peak GPU memory consumption for LLM fine-
tuning.

In summary, our contributions are:

  * •

We propose a GPU memory usage estimation method for LLM fine-tuning on single
and multiple GPUs.

  * •

We provide an algorithm to determine the most efficient distributed fine-
tuning method based on GPU memory usage estimation.

  * •

Experimental results show that LLMem estimates peak GPU memory usage to fine-
tune LLM on a single GPU with error rates of up to 1.6%, which is
significantly smaller than the state-of-the-art DNNMem’s average error rate of
42.6%. When applying distributed fine-tuning methods to LLMs with over a
billion parameters on multiple GPUs, LLMem successfully estimates GPU memory
usage with an average error rate of 3.0%.

Our source code repository can be found at
https://github.com/taehokim20/LLMem.

##  2 Related Works

###  2.1 GPU Memory Estimation

There have been several attempts to avoid GPU OOM issues by predicting the GPU
memory usage that will be used to train a given model in advance. DNNMem Gao
et al. (2020) sequentially traverses the computation graph of a DL model and
computes the GPU memory consumption by taking into account previously
allocated but still in-use tensors, newly allocated tensors for the currently
visited operator, and resident buffers of the CUDA context and allocator
reservation. Our LLMem is inspired by DNNMem, whose mechanism is described in
more detail in Section 3. TSplit Nie et al. (2022) also calculates the total
size of live tensors for the visiting operator. However, TSplit lacks an
explanation of the detailed memory estimation process and its accuracy.
SchedTune Albahar et al. (2022) predicts GPU memory usage not only based on DL
model characteristics but also on different GPU types running the job.
However, using measured GPU memory as data for prediction does not align with
the purpose of estimating memory usage before fine-tuning a model.

###  2.2 Distributed Fine-Tuning with GPUs

Data parallelism can enhance fine-tuning speed in proportion to the number of
GPUs. However, LLM often runs into memory constraints, so the ZeRO optimizer
Rajbhandari et al. (2020), described in Section 1, is widely used as an
alternative. The ZeRO optimizer selectively gathers only the model parameters
or gradients required during the computation process and utilizes reduce-
scatter after the computation to maintain their partitioning on each GPU.

Tensor parallelism can further reduce peak GPU memory usage by sharding
tensors under certain conditions, eliminating the need to gather all model
parameters and gradients even during computation. Tensor parallelism results
in each GPU producing only partial results, necessitating that all GPUs
receive the same input data. The widely adopted tensor parallelism method,
Megatron-LM Shoeybi et al. (2019), splits each model parameter tensor by row
or column. Other proposed methods Xu et al. (2021) Wang et al. (2021) Bian et
al. (2021) achieve additional memory savings by sharding both input and model
parameters.

If GPU memory constraints cannot be met with any distributed fine-tuning
method on GPUs alone, we can use heterogeneous fine-tuning utilizing CPU
memory. ZeRO-offload Ren et al. (2021) manages gradients, optimizer states,
and optimizer computation on the CPU while retaining parameters and forward
and backward computation on the GPU.

##  3 Motivation

To select distributed fine-tuning methods, it is crucial to estimate GPU
memory usage accurately. Existing approaches for estimating GPU memory usage
do not consider scenarios where advanced data parallelism, such as the ZeRO
Stage 3 optimizer Rajbhandari et al. (2020), or tensor parallelism is applied
across multiple GPUs. Relying solely on estimated GPU memory usage on a single
GPU when estimating on multiple GPUs can lead to significant errors. In this
section, we implement the existing DNNMem Gao et al. (2020), validate the
implementation results, and discuss factors causing substantial GPU memory
estimation errors during the fine-tuning of pre-trained transformer-based
language models.

###  3.1 DNNMem Implementation

DNNMem source codes are not publicly available and are mainly based on
TensorFlow, so we implement DNNMem based on the description in the paper Gao
et al. (2020). First, we extract the corresponding computation graph from a
given pre-trained DL model to identify the output size in each operator based
on parameters, batch size ($bs$), and sequence length ($sl$). We also compute
pre-allocated GPU memory, including CUDA context and weight tensors of the
model, before operator execution. In particular, since PyTorch does not
release the loaded model parameters until the end of the fine-tuning, the
initial GPU memory is retained throughout the fine-tuning process. The next
step is to compute peak GPU memory usage at each operator while traversing the
graph. We compute additional GPU memory with the input/output tensors and
previously unreleased tensors in each operator during the forward propagation.
Additionally, we reflect that PyTorch aligns with multiples of 512 bytes for
internal tensor fragmentation, and DNNMem treats the buffer size as a constant
(64 MB by default) as memory block management.

To validate our DNNMem implementation, we compare GPU memory estimation
results for the BERT Devlin et al. (2018) model on the GLUE benchmark Wang et
al. (2018) with the experimental results from the paper. The environment we
used in the experiment was PyTorch 2.0.1 with CUDA 11.7 on NVIDIA RTX2060,
which differs from PyTorch 1.2.0 with CUDA 9.0 on NVIDIA Tesla P40 used in the
DNNMem paper. In $bs=32,sl=32$, our DNNMem shows 34.38%, and the DNNMem shows
31.42% error rates. In $bs=64,sl=32$, our DNNMem shows 20.48%, and the DNNMem
shows 19.12% error rates. Considering similar error rates, we use our DNNMem
implementation for single-GPU comparisons.

Figure 1: Peak GPU memory estimates per total parameter size on a single GPU

###  3.2 Limitations of DNNMem for LLM Fine-Tuning Memory Estimation

DNNMem Gao et al. (2020) does not handle mixed precision, which is commonly
used in fine-tuning pre-trained language models. In addition, it does not
consider how memory chunks are managed to ensure that forward pass parameters
and backward pass gradients share the same GPU memory space Fang et al.
(2022). Furthermore, DNNMem overlooks extra GPU memory usage during the
initial fine-tuning iteration due to optimizer states.

Comparison results for estimating peak GPU memory usage of our proposed LLMem
and DNNMem on a single GPU are shown in Figure 1. The experimental environment
is summarized in Section 7.1. LLMem predicts peak GPU memory usage with
minimal error rates compared to ground truth, outperforming DNNMem. DNNMem
exhibits larger errors as the total parameter size increases. Furthermore,
DNNMem fails to predict GPU memory consumption in the context of distributed
fine-tuning methods across multiple GPUs. As a result, existing approaches for
estimating GPU memory usage face challenges when using the current
transformer-based LLM for distributed fine-tuning.

Table 1: Notation Symbol |  Description
---|---
$m_{base}$ |  the initially used GPU memory
$embed_{p}$ |  the input embedding param size
$lm_{p}$ |  the language modeling head param size
$cs,bs,sl$ |  chunk size, batch size, sequence length
$other_{p}$ |  the remaining param size w/o $embed_{p}$ and $lm_{p}$
$B_{16}$, $B_{32}$ |  2 bytes for fp16, 4 bytes for fp32
$m_{p}$ |  the GPU memory used by param fp16 and fp32
$m_{p,16}$ |  the GPU memory used by param fp16
$m_{p,32}$ |  the GPU memory used by param fp32
$cu_{p}$ |  the CUDA memory page size
$m_{os}$ |  the GPU memory used by momentum fp32 and variance fp32
$e_{n},l_{n}$ |  the number of Embedding or layers
$o_{n}$ |  the number of model’s output features
$m_{out}$ |  the peak GPU memory usage due to output tensors
$dict_{n}$ |  the size of the embedding dictionary
$m_{lm}$ |  the GPU memory used in the lm_head with the loss calculation
$m_{peak}^{s}$ |  the peak GPU memory usage on a single GPU
$gpu_{n}$ |  the number of GPUs in use
$m_{peak}^{dp}$ |  the peak GPU memory usage with the advanced DP on multiple GPUs
$m_{back}^{tp}$ |  the additional GPU memory usage due to the temporary buffer through the backward all-gather
$dp_{n},tp_{n}$ |  the number of GPUs used for DP or TP
$m_{peak}^{tp}$ |  the peak GPU memory usage with 1D TP on multiple GPUs
$m_{peak}^{dp+tp}$ |  the peak GPU memory usage with the combination of DP+TP on multiple GPUs
$m_{total}$ |  the total GPU memory capacity

##  4 Single-GPU Memory Usage Estimation

This section outlines considerations for estimating GPU memory usage of
transformer-based language models on a single GPU. The symbols used in the
explanation are organized in Table 1.

Figure 2: Illustration of tensors using GPU memory while fine-tuning the pre-
trained model Ren et al. (2021)

###  4.1 Workflow for Fine-Tuning Pre-Trained Models

Initialization phase. The initialization phase preceding fine-tuning involves
allocating memory for the CUDA context, responsible for managing information
to control GPU devices, and memory for applying chunk-based memory management
Fang et al. (2022). The initially used GPU memory is denoted as $m_{base}$.
The chunk manager determines the optimal chunk size to minimize GPU memory
waste based on the parameters of the provided pre-trained language model. GPU
memory spaces for param fp16 (float-16) and param fp32 (float-32) are
allocated in units of the chunk size ( 1 in Figure 4).

Fine-tuning phase. During the fine-tuning phase, param fp16 goes through
forward and backward passes, and param fp16 is converted to gradient fp16, as
illustrated in Figure 2. Consequently, param fp16 and gradient fp16 share the
same GPU memory space. After the backward pass, the ADAM optimizer updates
parameters using optimizer states, including param fp32, momentum fp32, and
variance fp32 tensors. Momentum fp32 and variance fp32 tensors, which are not
allocated memory during the initialization process before fine-tuning, consume
GPU memory based on the actual tensor size, not the chunk size. GPU memory
occupied by these tensor types is allocated in the first iteration for fine-
tuning ( 5 in Figure 4). Subsequently, similar to chunk-based parameters, the
GPU memory is retained until the fine-tuning process is complete.

Figure 3: Basic structure of transformer-based decoder model Vaswani et al.
(2017). As shown in Figure 2, the parameters in the transformer part are
managed using chunk-based memory, while the lm_head part, responsible for
deriving the output, consumes GPU memory based on its actual size.

###  4.2 Memory Consumption with Structure of Transformer-based Decoder Model

The peak GPU memory usage on a single GPU (${m}_{peak}^{s}$) is

$\displaystyle m_{peak}^{s}=m_{base}+m_{p}+m_{os}+m_{out}+m_{lm}$

Each variable in this formula, except $m_{base}$ described in Section 4.1, is
calculated as follows.

First, $m_{p}$ is the GPU memory used by param/gradient fp16 and param fp32.
Considering that param fp16 and fp32 use 2 bytes ($B_{16}$) and 4 bytes
($B_{32}$) per value, respectively, $m_{p}$ is

$\displaystyle
m_{p}=\left\lceil(embed_{p}+\left\lceil\frac{other_{p}}{cs}\right\rceil\times
cs)\times\frac{B_{16}+B_{32}}{cu_{p}}\right\rceil\times cu_{p}$

, where $embed_{p}$ is the input embedding param size, $cs$ is the chunk size,
$other_{p}$ is the remaining param size, and $cu_{p}$ is the CUDA memory page
size, typically $2\times 1024^{2}$ bytes. Transformer-based decoder models
Vaswani et al. (2017) are largely divided into a transformer model for fine-
tuning and lm_head for output, as shown in Figure 3. The part that uses the
chunk memory is the transformer model in which the parameters are updated. The
$embed_{p}$ is huge because the input embedding has a large dictionary.
Therefore, $embed_{p}$ is managed separately.

Second, $m_{os}$ is the GPU memory used by momentum fp32 and variance fp32 of
optimizer states. $m_{os}$ is

$\displaystyle m_{os}=\sum_{t\in\\{E,L\\}}\left\lceil
t_{p}\times\frac{B_{32}+B_{32}}{cu_{p}}\right\rceil\times cu_{p}$

, where $t$ is the operator of the given transformer model, $E$ is Embedding,
$L$ is Linear, and $t_{p}$ is the parameter size of $t$. The system allocates
GPU memory based on the actual size of each momentum fp32 and variance fp32,
so GPU memory must be calculated for each tensor of each operator. Since the
amount of GPU memory consumed by Bias or LayerNorm is very small, they can use
space with other memory fragmentation. Therefore, we only calculate the GPU
memory usage due to Embedding or Linear operator parameters.

Third, $m_{out}$ is the peak GPU memory usage due to output tensors. If the
number of Embedding, layers, and model’s output features are $e_{n}$, $l_{n}$,
and $o_{n}$, respectively, then $m_{out}$ is

$\displaystyle m_{out}=\left\lceil(e_{n}+l_{n})\times(bs\times sl\times
o_{n})\times\frac{B_{16}}{cu_{p}}\right\rceil\times cu_{p}$

PyTorch provides gradient checkpointing333Gradient checkpointing reduces GPU
memory usage by clearing specific outputs and recomputing them during a
backward pass. as an option to save memory during fine-tuning. Therefore, we
support estimating GPU memory usage due to each operator’s input/output
tensors considering gradient checkpointing. Since the output tensors of the
current operator are the input tensors of the next operator, we focus on the
output. It is challenging to accurately predict GPU memory consumption due to
the outputs of operators within a model. We observed that the layer and
embedding outputs of the transformer model are kept in GPU memory for
efficient gradient checkpointing, which minimizes the increase in fine-tuning
time. The estimation error rate is reduced using the $m_{out}$ equation, which
accounts for our observation.

Lastly, $m_{lm}$ is the GPU memory used in the lm_head part including the loss
calculation part. If the size of the embedding dictionary is $dict_{n}$,
$m_{lm}$ is

$\displaystyle m_{lm}=\left\lceil bs\times sl\times
dict_{n}\times\frac{B}{cu_{p}}\right\rceil\times cu_{p}+$ $\displaystyle
2\times\left\lceil bs\times(sl-1)\times
dict_{n}\times\frac{B}{cu_{p}}\right\rceil\times cu_{p}+lm_{p}$

, where $lm_{p}$ is the lm_head param size, and $B$ is either $B_{16}$ or
$B_{32}$ depending on the model type. The lm_head converts the transformer
outputs into logits. Then, the value obtained by shifting the sequence length
of the logits by one space is stored in a separate temporary variable and used
for the loss calculation. $m_{out}+m_{lm}$ is the output phase ( 3 in Figure
4).

Figure 4: Peak GPU memory computation for different distributed fine-tuning
methods.

(a) Advanced DP

(b) TP

Figure 5: Advanced DP gathers the entire param fp16, while TP maintains the
sharded param fp16 intact before entering the computation process. Figure 6:
Performing the linear operation in the forward and backward passes when
employing TP. During the collection of partial outputs from each GPU after the
backward pass, an additional GPU memory is consumed by a temporary buffer.

##  5 Multi-GPU Memory Usage Estimation

This section outlines the factors for estimating peak GPU memory usage during
distributed fine-tuning on multiple GPUs and summarizes the estimation
process.

Conventional data parallelism (CDP). Since CDP places the entire model on each
GPU, its peak GPU memory usage estimation equals the peak single-GPU memory
usage $m_{peak}^{s}$, as shown in Figure 4.

Advanced data parallelism (ADP). The peak GPU memory usage with ADP on
multiple GPUs ($m_{peak}^{dp}$) is

$\displaystyle
m_{peak}^{dp}=m_{base}+m_{p,16}+\frac{m_{p,32}+m_{os}}{gpu_{n}}+m_{out}+m_{lm}$

, where $m_{p,16}$ and $m_{p,32}$ are the GPU memory consumed by the entire
param fp16/fp32, and $gpu_{n}$ is the number of GPUs in use. ZeRO-3 optimizer,
a method of advanced data parallelism, evenly distributes parameters,
gradients, and optimizer states by $gpu_{n}$, reducing GPU memory usage. Among
these, gradient fp16 shares GPU memory with param fp16 as explained in Section
4, so we only need to divide the GPU memory usage of parameters and optimizer
states by $gpu_{n}$. However, during the calculation process, each GPU must
have all the values of param fp16 (Figure 5(a) and 2 in Figure 4), so the
whole $m_{p,16}$ is allocated to the GPU memory.

Tensor parallelism (TP). The peak GPU memory usage with 1D TP on multiple GPUs
($m_{peak}^{tp}$) is

$\displaystyle
m_{peak}^{tp}=m_{base}+\frac{m_{p}+m_{os}}{gpu_{n}}+m_{out}+m_{lm}+m_{back}^{tp}$

, where $m_{back}^{tp}$ ( 4 in Figure 4 and Figure 6) is the additional GPU
memory usage due to the temporary buffer through the backward all-gather. If
the number of GPUs used for tensor parallelism is $tp_{n}$, $m_{back}^{tp}$ is

$\displaystyle m_{back}^{tp}=\left\lceil l_{n}\times(bs\times sl\times
o_{n})\times\frac{tp_{n}-1}{tp_{n}}\times\frac{B_{16}}{cu_{p}}\right\rceil\times
cu_{p}$

Tensor parallelism divides the parameter values of each operator by $gpu_{n}$
and does not combine them again, as shown in Figure 5(b). It splits each model
parameter tensor by row or column to apply tensor parallelism to multiple pre-
trained language models. We call this one-dimension tensor parallelism (1D
TP). Let us assume that we apply 1D TP to a linear operation on four GPUs. The
linear operator’s equation is $y=xA^{T}+b$, where $y$ is output, $x$ is input,
$A$ is params/gradients, and $b$ is bias. The linear matrix multiplication
process when each parameter tensor is split into columns is shown in Figure 6.
We shard parameters by column because the output size after multiplication is
the same as the size of the bias without sharding, so it is not affected by
the use of bias. In the backward pass, the fine-tuning goes through an all-
gather process. $m_{back}^{tp}$ is the total temporary buffer size for tensors
imported from the other GPUs, calculated by multiplying the output size of
each layer by the number of layers.

Combination of DP+TP. The peak GPU memory usage with the combination of DP+TP
on multiple GPUs ($m_{peak}^{dp+tp}$) is

$\displaystyle m_{peak}^{dp+tp}=m_{peak}^{dp}-\frac{m_{p,16}\times
tp_{n}}{gpu_{n}}+m_{back}^{tp}$

, as shown in Figure 4. It is possible to achieve hybrid parallelism by fine-
tuning through a combination of data and tensor parallelism.

##  6 Distributed Fine-Tuning Method Decision

Algorithm 1 describes the process for selecting the optimal method to fine-
tune a pre-trained model based on the results of estimating the peak GPU
memory usage. In Sections 4 and 5, We estimated $m_{peak}^{s}$,
$m_{peak}^{dp}$, $m_{peak}^{tp}$, and $m_{peak}^{dp+tp}$. Here, $m_{peak}^{s}$
represents CDP, and the remaining estimations are connected to ADP, TP, and
DP+TP, respectively. Of these methods, the optimal one is the method that
requires the shortest time for fine-tuning while avoiding GPU OOM.

LLMem takes a pre-trained model $M$, the total number of GPUs to fine-tune
$gpu_{n}$, and the maximum sequence length $sl$. $eval$ is a list that stores
the performance evaluation score of each method. $eval[0]$, $eval[1]$,
$eval[2]$, and $eval[3]$ correspond to CDP, ADP, TP, and DP+TP, respectively.
LLMem increments the batch size $bs$ for each method and gets the value of
$bs$ when it reaches the total GPU memory capacity. Then, $bs-1$ is the
largest batch size to avoid GPU OOM. CDP uses $(bs-1)\times gpu_{n}$ amount of
data for fine-tuning in one iteration. In addition, since the ZeRO-3 optimizer
increases the total communication volume of a baseline DP to $1.5\times$
Rajbhandari et al. (2020), the performance score of CDP is $(bs-1)\times
gpu_{n}\times 1.5$. In one iteration, ADP uses $(bs-1)\times gpu_{n}$, TP uses
$bs-1$, and DP+TP uses $(bs-1)\times dp_{n}$ of data for fine-tuning. $dp_{n}$
is the number of GPUs used for DP. These values become the performance scores
of each method. Finally, LLMem selects the method with the highest performance
score (If the scores are tied, select CDP, ADP, TP, and DP+TP in that order).
If the performance scores of all methods are 0, heterogeneous training using
CPU memory is selected as an alternative to avoid GPU OOM.

Algorithm 1 Distributed Fine-Tuning Method Decision

Input: Pre-trained model $M$, $gpu_{n}$, and $sl$
Output: Selected fine-tuning method and the optimal $bs$

1:$eval=[0,0,0,0]$ and $bs_{list}=[0,0,0,0]$

2:Measure total GPU memory capacity ($m_{total}$)

3:for $i$ in range(4) do

4: Set up the configure of the $i^{th}$ method

5: $bs=1$ and compute $m_{base}$, $m_{p}$, and $m_{os}$

6: [1]: Compute $m_{out}$ and $m_{lm}$

7: if $i=0$ then

8: Repeat $bs=bs+1$ until $m_{peak}^{s}>m_{total}$ after [1]

9: $eval[i]=(bs-1)\times gpu_{n}\times 1.5$

10: else if $i=1$ then

11: Repeat $bs=bs+1$ until $m_{peak}^{dp}>m_{total}$ after [1]

12: $eval[i]=(bs-1)\times gpu_{n}$

13: else if $i=2$ then

14: Repeat $bs=bs+1$ until $m_{peak}^{tp}>m_{total}$ after [1]

15: $eval[i]=bs-1$

16: else if $i=3$ then

17: Repeat $bs=bs+1$ until $m_{peak}^{dp+tp}>m_{total}$ after [1]

18: $eval[i]=(bs-1)\times dp_{n}$

19: end if

20: $bs_{list}[i]=bs-1$

21:end for

22:Save the index of the maximum score to $idx$

23:return $idx$, $bs_{list}[idx]$ if $eval[2]\neq 0$ else 4, 0

##  7 Experiments

In this section, we compare the peak GPU memory usage estimate of LLMem with
the ground truth data when applying various distributed fine-tuning methods.
In addition, our DNNMem implementation is included in comparing GPU memory
usage estimation on a single GPU.

###  7.1 Experimental Setup

For a multi-GPU environment, we use a Tesla V100 (total GPU memory capacity:
16384 MB) with 4 GPUs in CloudLab CloudLab (2023). We also use the Colossal-AI
Li et al. (2023), a widely used framework for applying distributed fine-tuning
methods, and PyTorch 2.0.1 with CUDA 11.7. The models we used in the
experiment are OPT Zhang et al. (2022), BLOOM Workshop et al. (2022), CodeGen
Nijkamp et al. (2022), BioGPT Luo et al. (2022), GPTBigCode Allal et al.
(2023), GPT Neo Black et al. (2021), and LLaMA Touvron et al. (2023). The
dataset used is alpaca data Taori et al. (2023), which is 52K instruction-
following data. For the ground truth data, we measure peak GPU memory usage
using only the maximum sequence length of 512.

Figure 7: Comparison of peak GPU memory usage estimates between LLMem and DNNMem for models experiencing GPU OOM during fine-tuning. Table 2: Estimating GPU memory usage on a single GPU. The values in parentheses represent the comparisons between the LLMem estimate and the ground truth, or the DNNMem estimate and the ground truth. Model (MB) | LLMem | DNNMem | Ground truth
---|---|---|---
OPT-125m | 16314 (0.4) | 10402 (36.5) | 16378
OPT-350m | 16004 (1.6) | 9354 (42.5) | 16264
bloom-560m | 16578 (1.6) | 10726 (34.3) | 16324
codegen-350M | 16236 (0.8) | 6910 (57.1) | 16100

###  7.2 Estimation of Single-GPU Memory Usage

First, we compare the peak GPU memory usage estimate from LLMem for a single
GPU with the DNNMem estimate and the actual peak GPU memory usage. Since we
used gradient checkpointing for LLM fine-tuning, the same approach was applied
to DNNMem. Figure 7 compares the peak GPU memory usage estimation results of
LLMem and DNNMem for various pre-trained LLMs that cause GPU OOM during fine-
tuning on a single GPU. LLMem predicts GPU OOM for all models, while DNNMem
predicts peak GPU memory usage that falls short of $m_{total}$. Table 2 shows
the predicted and actual GPU memory usage peaks when applying the maximum
batch size to obtain the ground truth data for each model during fine-tuning
on a single GPU. DNNMem underestimates the peak GPU memory usage for all
models because it does not account for factors considered when fine-tuning
Transformer-based LLM, as explained in Section 3.2. LLMem’s GPU memory
estimation helps approximate the peak GPU memory usage close to the ground
truth.

Figure 8: Estimating GPU memory usage for ADP on four GPUs at each model’s
maximum batch size to prevent GPU OOM. OOM in codegen-2b indicates running out
of memory even at batch size=1.

###  7.3 Estimation of Multi-GPU Memory Usage

CDP. The experimental results are the same as the memory usage estimation
results on a single GPU in Section 7.2.

ADP. Figure 8 shows the predicted and actual GPU memory usage peaks when
applying the maximum batch size to obtain ground truth data for each model
during fine-tuning with ADP on four GPUs. The error rate between the predicted
value of LLMem and the actual GPU memory usage tends to increase on multi-GPU
setups. One reason is the gap in memory usage between the GPUs. ADP places
tensors separately on each GPU instead of being sharded, so not all GPUs can
use precisely the same amount of memory. Second, the error tends to be
slightly larger when the model size is large. A larger number of layers and
outputs in large models can lead to larger error rates due to memory allocator
characteristics.

Figure 9: Estimating GPU memory usage for 4TP and 2DP+2TP at each model’s
maximum batch size to avoid OOM.

TP and DP+TP. Figure 9 shows the predicted and actual GPU memory usage peaks
when applying the maximum batch size to obtain ground truth data for each
model during fine-tuning with 4TP or 2DP+2TP on four GPUs. 4TP uses 4 GPUs in
TP, and 2DP+2TP uses 2 GPUs in DP and 2 GPUs in TP for hybrid parallelism. We
focus on estimating the peak GPU memory usage of the large-size model for TP
because LLMem can select DP for quick fine-tuning of models that are small and
do not have OOM problems. TP applies the all-gather operation in the backward
pass, as shown in Figure 6. The all-gather operation allocates temporary
buffers in GPU memory and collects values in those buffers, consuming
additional GPU memory. If the model size is large and the possible batch size
is small, the system can use the allocated but currently empty memory space
for a temporary buffer. Therefore, the GPU memory consumed due to the
temporary buffer does not increase excessively, leading to smaller errors as
shown in Figure 9. 2DP+2TP shows slightly larger errors than 4TP in most
cases. This is because GPU memory usage due to the temporary buffer may be
additionally affected in 2 and 4 of Figure 4 while applying both DP and TP.

Table 3: Distributed fine-tuning method selection by LLMem on four GPUs and the actual amount of time it takes to fine-tune each method to the largest possible batch size (s) Model (s) | Selection | 4DP | 2DP+2TP | 4TP
---|---|---|---|---
OPT-1.3b | 4DP | 688 | 1616 | 2186
OPT-2.7b | 4TP | OOM | 8174 | 6038
bloom-1b1 | 4DP | 680 | 1724 | 2631
bloom-3b | 4TP | OOM | OOM | 14495
BioGPT-Large | 4DP | 1022 | 3315 | 4773
codegen-2B-nl | 4TP | OOM | 6314 | 6244
gpt_bigcode | 4DP | 651 | 1292 | 1652
gpt-neo-1.3B | 4DP | 768 | 1686 | 2372
llama-7b | CPU offloading | OOM | OOM | OOM

###  7.4 Fine-Tuning Method Selection with LLMem

Table 3 assesses whether LLMem finds the optimal fine-tuning method to achieve
the fastest fine-tuning while avoiding GPU OOM for various models. When
measuring the time taken for each method, we applied the maximum batch size
that can prevent GPU OOM. LLMem typically selects TP when DP causes GPU OOM.
It is challenging for LLMem to choose DP+TP because only 4 GPUs were used in
the experiment. DP+TP allows for more diverse combinations depending on the
number of GPUs used and is more likely to be selected. LLMem also suggests CPU
offloading when GPU memory is insufficient.

##  8 Conclusion

This paper introduces LLMem, a method for estimating GPU memory consumption
during fine-tuning of large language models (LLMs) on multi-GPU setups. We
analyze factors affecting GPU memory usage, considering different memory
allocation methods for the transformer and output sections. Experimental
results demonstrate that LLMem achieves accurate peak GPU memory usage
estimation on both single and multiple GPUs with minimal error rates.

## References

  * Albahar et al. [2022] Hadeel Albahar, Shruti Dongare, Yanlin Du, Nannan Zhao, Arnab K Paul, and Ali R Butt.  Schedtune: A heterogeneity-aware gpu scheduler for deep learning.  In 2022 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid), pages 695–705. IEEE, 2022.
  * Allal et al. [2023] Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al.  Santacoder: don’t reach for the stars!  arXiv preprint arXiv:2301.03988, 2023.
  * Bian et al. [2021] Zhengda Bian, Qifan Xu, Boxiang Wang, and Yang You.  Maximizing parallelism in distributed training for huge neural networks.  arXiv preprint arXiv:2105.14450, 2021.
  * Black et al. [2021] Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman.  Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow.  https://doi.org/10.5281/zenodo.5297715, 2021.
  * CloudLab [2023] CloudLab.  https://www.cloudlab.us/, 2023.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * Fang et al. [2022] Jiarui Fang, Zilin Zhu, Shenggui Li, Hui Su, Yang Yu, Jie Zhou, and Yang You.  Parallel training of pre-trained models via chunk-based dynamic memory management.  IEEE Transactions on Parallel and Distributed Systems, 34(1):304–315, 2022.
  * Gao et al. [2020] Yanjie Gao, Yu Liu, Hongyu Zhang, Zhengxian Li, Yonghao Zhu, Haoxiang Lin, and Mao Yang.  Estimating gpu memory consumption of deep learning models.  In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 1342–1352, 2020.
  * Li et al. [2023] Shenggui Li, Hongxin Liu, Zhengda Bian, Jiarui Fang, Haichen Huang, Yuliang Liu, Boxiang Wang, and Yang You.  Colossal-ai: A unified deep learning system for large-scale parallel training.  In Proceedings of the 52nd International Conference on Parallel Processing, pages 766–775, 2023.
  * Luo et al. [2022] Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu.  Biogpt: generative pre-trained transformer for biomedical text generation and mining.  Briefings in Bioinformatics, 23(6):bbac409, 2022.
  * Nie et al. [2022] Xiaonan Nie, Xupeng Miao, Zhi Yang, and Bin Cui.  Tsplit: Fine-grained gpu memory management for efficient dnn training via tensor splitting.  In 2022 IEEE 38th International Conference on Data Engineering (ICDE), pages 2615–2628. IEEE, 2022.
  * Nijkamp et al. [2022] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong.  Codegen: An open large language model for code with multi-turn program synthesis.  arXiv preprint arXiv:2203.13474, 2022.
  * Rajbhandari et al. [2020] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He.  Zero: Memory optimizations toward training trillion parameter models.  In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020.
  * Ren et al. [2021] Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He.  $\\{$ZeRO-Offload$\\}$: Democratizing $\\{$Billion-Scale$\\}$ model training.  In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pages 551–564, 2021.
  * Shoeybi et al. [2019] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro.  Megatron-lm: Training multi-billion parameter language models using model parallelism.  arXiv preprint arXiv:1909.08053, 2019.
  * Taori et al. [2023] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * Wang et al. [2018] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  arXiv preprint arXiv:1804.07461, 2018.
  * Wang et al. [2021] Boxiang Wang, Qifan Xu, Zhengda Bian, and Yang You.  2.5-dimensional distributed model training.  arXiv e-prints, pages arXiv–2105, 2021.
  * Workshop et al. [2022] BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al.  Bloom: A 176b-parameter open-access multilingual language model.  arXiv preprint arXiv:2211.05100, 2022.
  * Xu et al. [2021] Qifan Xu, Shenggui Li, Chaoyu Gong, and Yang You.  An efficient 2d method for training super-large deep learning models.  arXiv e-prints, pages arXiv–2104, 2021.
  * Zhang et al. [2022] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.  Opt: Open pre-trained transformer language models.  arXiv preprint arXiv:2205.01068, 2022.


$\displaystyle\Pr[v\in V_{j}^{\perp}\,{\rm and}\,v\notin V_{j}|v\in
V_{j-1}^{\perp}\setminus
V_{j-1}]\leq\frac{\zeta(n,m,k_{j-1}+1)}{\zeta(n,m,k_{j-1})}\leq\frac{p}{2p-1}\leq\frac{3}{5}$
(E.13)

because $w_{j}$ has to be orthogonal to $\text{span}(v)+V_{j-1}$, which is
null and is a proper superset of $V_{j-1}$. Thus, for any $t$,

$\displaystyle\Pr[v\in V_{t}^{\perp}\,{\rm and}\,v\not\in V_{t}]$
$\displaystyle=\Pr[v\in V_{1}^{\perp}\setminus V_{1}]\prod_{j=2}^{t}\Pr[v\in
V_{j}^{\perp}\setminus V_{j}\leavevmode\nobreak\ |\leavevmode\nobreak\ v\in
V_{j-1}^{\perp}\setminus V_{j-1}]$
$\displaystyle\leq\prod_{j=2}^{t}\frac{\zeta(n,m,k_{j-1}+1)}{\zeta(n,m,k_{j-1})}$
$\displaystyle\leq\left(\frac{3}{5}\right)^{t-1}.$ (E.14)

For $t=c$, we find in particular that

$\displaystyle\Pr[v\in V_{c}^{\perp}\,{\rm and}\,v\not\in
V_{c}]\leq\left(\frac{3}{5}\right)^{c-1}.$ (E.15)

Now assume $v\cdot v\neq 0$. The event that $v\in V_{j}^{\perp}$ happens only
if $w_{j}$ is chosen from $v^{\perp}$. We bound the decomposition $\Pr[v\in
V_{c}^{\perp}]=\Pr[v\in V_{1}^{\perp}]\prod_{j=2}^{c}\Pr[v\in
V_{j}^{\perp}|v\in V_{j-1}^{\perp}]$. The first term is bounded by 1
trivially. For other factors, we observe that the dimension of $v^{\perp}$ is
$n-1$, and a maximal null subspace in $v^{\perp}$ has dimension
$m^{\prime}\leq m$. Under the conditioning $v\in V_{j-1}^{\perp}$, the null
space $V_{j-1}$ is a subspace of $v^{\perp}$, and
$\\#\mathcal{Z}(v^{\perp},V_{j-1})=\zeta(n-1,m^{\prime},k_{j-1})\leq\zeta(n-1,m,k_{j-1})$.
Hence,

$\displaystyle\Pr[v\in V_{c}^{\perp}]$ $\displaystyle=\Pr[v\in
V_{1}^{\perp}]\prod_{j=2}^{c}\Pr[v\in V_{j}^{\perp}|v\in V_{j-1}^{\perp}]$
$\displaystyle\leq\prod_{j=2}^{c}\frac{\zeta(n-1,m,k_{j-1})}{\zeta(n,m,k_{j-1})}$
$\displaystyle\leq\left(\frac{p^{2}+p-1}{2p^{2}-p}\right)^{c-1}\leq\left(\frac{11}{15}\right)^{c-1}$
(E.16)

where in the second inequality, we used the assumption that $k_{j-1}\leq
c<(n-2)/2\leq m$.

Let us turn to the second event, assuming $v\cdot v=0$. Note that if $v\in
V_{c}$, there is a least $j$ such that $v\in V_{j}$. So,

$\displaystyle\Pr[v\in V_{c}^{\perp}\,{\rm and}\,v\in
V_{c}]\leq\sum_{j=2}^{c}\Pr[v\in V_{j}\,{\rm and}\,v\in V_{j-1}^{\perp}\,{\rm
and}\,v\not\in V_{j-1}].$ (E.17)

We have

$\displaystyle\Pr[v\in V_{j}\,{\rm and}\,v\in V_{j-1}^{\perp}\,{\rm
and}\,v\not\in V_{j-1}]$ $\displaystyle=\Pr[v\in V_{j-1}^{\perp}\,{\rm
and}\,v\not\in V_{j-1}]\cdot\Pr[v\in V_{j}|v\in V_{j-1}^{\perp}\,{\rm
and}\,v\not\in V_{j-1}]$
$\displaystyle\leq\left(\frac{3}{5}\right)^{j-2}\Pr[v\in V_{j}|v\in
V_{j-1}^{\perp}\,{\rm and}\,v\not\in V_{j-1}],$ (E.18)

where we used Eq. (E). The second factor is bounded as

$\displaystyle\Pr[v\in V_{j}|v\in V_{j-1}^{\perp}\,{\rm and}\,v\not\in
V_{j-1}]\leq\frac{p^{1+k_{j-1}}}{\zeta(n,m,k_{j-1})}<\frac{3}{2p^{n-2j}}$
(E.19)

because $w_{j}$ belongs to $\text{span}(v)+V_{j-1}$. Hence, $\Pr[v\in
V_{j}\,{\rm and}\,v\in V_{j-1}^{\perp}\,{\rm and}\,v\notin V_{j-1}]\leq
5(\frac{3}{5})^{n-j}$. So by Eq. (E.17),

$\displaystyle\Pr[v\in V_{c}^{\perp}\,{\rm and}\,v\in
V_{c}]\leq\sum_{j=2}^{c}5\left(\frac{3}{5}\right)^{n-j}<20\left(\frac{3}{5}\right)^{n-c}.$
(E.20)

Summing the probabilities of (E.16) and (E.20), we conclude the proof. ∎


# Audio-visual Representation Learning for
Anomaly Events Detection in Crowds

Junyu Gao, Maoguo Gong, and Xuelong Li  J. Gao is with the Academy of Advanced
Interdisciplinary Research, Xidian University, Xi’an 710071, Shaanxi, China,
and the School of Artificial Intelligence, Optics and Electronics (iOPEN),
Northwestern Polytechnical University, Xi’an 710072, P. R. China. E-mail:
<EMAIL_ADDRESS>M. Gong is with the Key Laboratory of Intelligent
Perception and Image Understanding of Ministry of Education, International
Research Center for Intelligent Perception and Computation, Xidian University,
Xi’an 710071, Shaanxi, China. E-mail<EMAIL_ADDRESS>X. Li is with the School
of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern
Polytechnical University, Xi’an 710072, P. R. China. E-mail<EMAIL_ADDRESS>Corresponding author: X. Li.

###### Abstract

In recent years, anomaly events detection in crowd scenes attracts many
researchers’ attentions, because of its importance to public safety. Existing
methods usually exploit visual information to analyze whether any abnormal
events have occurred due to only visual sensors are generally equipped in
public places. However, when an abnormal event in crowds occurs, sound
information may be discriminative to assist the crowd analysis system to
determine whether there is an abnormality. Compare with vision information
that is easily occluded, audio signals has a certain degree of penetration.
Thus, this paper attempt to exploit multi-modal learning for modeling the
audio and visual signals simultaneously. To be specific, we design a two-
branch network to model different types of information. The first is a typical
3D CNN model to extract temporal appearance feature from video clips. The
second is an audio CNN for encoding Log Mel-Spectrogram of audio signals.
Finally, by fusing the above features, the more accurate prediction will be
produced. We conduct the experiments on SHADE dataset, a synthetic audio-
visual dataset in surveillance scenes, and find introducing audio signals
effectively improves the performance of anomaly events detection and
outperforms other state-of-the-art methods. Furthermore, we will release the
code and the pre-trained models as soon as possible.

###### Index Terms:

Crowd analysis, Anomaly events detection, Audio-visual representation
learning, Multi-modal learning

##  I Introduction

Crowd analysis is an essential task in the field of public safety including
crowd counting [1, 2], localization [3, 4, 5], anomaly events detection [6,
7], flow/motion analysis [8, 9], segmentation [10, 11], group detection [12,
13], _etc._ Especially, anomaly events detection is a fundamental task for
safety warning in crowd scenes. Timely alarming of anomaly events that are
occurring is essential to ensure public safety. By our observation, we find
that when an abnormal event occurs, it is often accompanied by some special
sounds. Unfortunately, since the common surveillance cameras are not equipped
with audio collection recorders, many researchers only pay attention to the
visual information and ignore the audio signal. However, there are many cases
that occur in the blind spot of surveillance cameras, resulting in the
security system not being able to recognize and warn for police or others in
advance. Thus, in this paper, we attempt to explore the effect of ambient
sound on the task of anomaly event detection.

Figure 1: Some typical samples in SHADE dataset. The green box contains three
groups of image sequences of Shoot, Knockdown, and Arrest, of which red boxes
in frames highlight the incident location. The magenta box visualizes the
audio signals and Mel-Spectrograms for the above crowd scenes.

With the development of deep learning on computer vision, there are many
phenomenal works in the fields of image recognition [14, 15, 16], temporal
modeling [17, 18, 19], and audio signals processing [20, 21, 22]. In the field
of crowd analysis, most of the existing work focuses on the research of still
images [23, 24] and image sequences [25, 26], namely visual crowd analysis.
However, currently there is only one method that combines visual and auditory
information [27], which proposes a multi-modal learning to encode still images
and ambient sounds simultaneously. The method significantly reduces the
estimation errors for crowd counting in extreme conditions. Inspired this, we
investigate common abnormal events under video surveillance in crowd scenes,
such as fighting, fleeing, chasing, chaos, _etc._ and find that the crowd can
make some special sounds, which may aid model extract more discriminative
features to recognize the event. Fig. 1 illustrates the videos and the
corresponding audio data of some typical samples in SHADE dataset [7]. From
it, there are many differences among different events’ ambient sound. Besides,
the surveillance camera may be blocked by obstacles and cannot capture
effective content. In this case, the sound signal shows its unique advantage:
it can bypass obstacles and be collected by recording equipment.

Based on the aforementioned observation, this paper propose a multi-modal
learning to extract features from image sequences and audio data for anomaly
event detection in crowd scenes. It named as “Audio-visual Representation
Learning” (AVRL for short), which consists of two streams: a Residual 3D CNN
[28] is developed on image clips and a audio CNN on processed audio
information represented by Log Mel-Spectrogram algorithm [29]. After obtaining
two types of features, a feature-level fusion mechanism is presented to
integrate them and the event class is directly predicted.

In summary, this paper has three-fold contributions:

  1. 1)

Propose a new framework, Audio-visual Representation Learning (AVRL), to
extract appearance and sound features from image sequences and the
corresponding audio signals.

  2. 2)

Present an effective scheme to fuse the spatial-temporal 3D-CNN’s features and
temporal audio features, which can achieve the balanced trade-off between
visual and audio representations and reaches better anomaly detection
accuracy.

  3. 3)

Outperform the state-of-the-art method on the anomaly events detection task in
crowd scenes.

The rest of this paper is organized as follows: Section II briefly lists and
reviews the related works about crowd analysis and multi-modal learning;
Section III presents the proposed AVRL method for anomaly event detection;
Section IV and V conduct the extensive experiments and further analyze the the
key settings of the method. Finally, this work is summarized in Section VI.

Figure 2: AVRL for abnormal events detection in crowd scenes. The details of
3D ResNet and VGGish is the same as [28] and [29] respectively.

##  II Related Work

This section briefly reviews the related works about research fields in this
paper: abnormal events detection in crowd scenes and audio-visual
representation learning.

###  II-A Abnormal Events Detection in Crowds

The recent anomaly detection methods can be divided into two main categories:
local analysis and global analysis. As for local analysis methods, the
locations where abnormal events occur are detected. The typical local analysis
methods include [30, 31, 32, 33], _etc._ These methods detect the abnormal
objects, such as trucks or running people in crowds and localize the position
where anomalies occur. Liao _et al._ [34] use video descriptors to detect the
fighting event in video frames. The other category, global analysis, usually
analyze the whole video clips and predict whether there are abnormal events in
the video clips. The global analysis methods have three typical schemes:
trajectory points based, optical flow based, and classification based. [35,
36, 37] utilize the object trajectories extracted from crowd video clips for
anomaly detection. Helbing _et al._ [38] proposed is a typical trajectory
points based method named social force model for crowd interaction
description. Inspired by [38], [35] and [37] introduce the social force model
into the social events analysis in videos. Cui _et al._ [36] attempt to
cluster trajectories by Fuzzy C-Means Clustering [39] and predict the category
of input trajectory through clustering. Du _et al._ [33] propose a change
detector named DSFA, and it utilizes two symmetric streams and slow feature
analysis module to acquire better performance on changes in remote sensing
imagery.

Because the definition of abnormal events in statistic based methods is
ambiguous and inchoate, the VSD [40] and UCF-Crime [41] are proposed for
abnormal events detection in crowd scenes, containing 7 and 13 different
categories about anomalies respectively. The video clips in these two datasets
are gained from movies and videos on internet. [42] is a typical method for
video classification with VSD benchmark. In the meantime, [43] estimates the
crowd emotion for abnormal behavior analysis and understanding on dataset in
[44]. The datasets mentioned above define the abnormal events in crowd
detailedly, leading the abnormal detection task to the video classification
task. Therefore, we deploy the recent video classification methods into the
anomaly detection and receive excellent results.

###  II-B Audio-Visual Multi-Modal Learning

The joint audio-visual multi-modal learning attempts to the learn the
representation from visual and auditory modalities for special tasks. The
early researches on this field concentrate on the speech recognition task
[45]. The motivation of these works is that the visual modality provides
auxiliary information for audio analysis with noisy. A typical research on
audio-visual multi-modal learning is to model the movement of face and mouth
through the videos and corresponding audio signals [46, 47]. Similarly, the
multi-modal learning is introduced into other tasks [48, 49].

In recent years, the audio-visual multi-modal learning is utilized in more
general scenes. Owens _et al._ [50, 51] try to transfer the knowledge from
audio learning to visual learning. Arandjelovic _et al._ [52] analyze the
video through audio-visual relations. Meanwhile, this work is employed to
sound localization [53, 54] and audio-visual separation [55].

This paper is inspired by these audio-visual multi-modal learning methods and
we attempt to introduce the audio modality into the traditional visual-based
abnormal detection methods. The anomaly detection in crowd scenes benefits
from special ambient sounds in some events.

##  III Our Approach

The proposed framework AVRL contains three main modules: visual representation
learning, audio representation learning, and audio-visual representation
fusion, showing in Fig. 2. Compared with the conventional video anomaly
detection method, our framework has the unique audio representation learning,
and audio-visual representation fusion modules for abnormal events detection
in crowd analysis. As mentioned above, the motivation of our framework is that
some abnormal events have the extremely special ambient sounds, such as
shooting, scattering, _etc._ and these discriminating ambient sounds is
beneficial for abnormal events recognition. For example, when the shooting
rampage occurred, some people near the crime scene may not see the shooter at
all, but the loud gunfire helps the people in judging the abnormal event. To
introduce this judgment capacity, we design a audio representation learning
module in the traditional video abnormal events detection framework, gaining
the _Audio-visual Representation Learning (AVRL)_ framework.

###  III-A Visual Representation Learning

Compared with the images, videos have inherent temporal correlation, and video
data have extra time dimension. In order to extract the temporal information,
3D convolutional filter, a 2D convolutional filter with time dimension, is
proposed and introduced into the vision task on videos. Thus, the video
representation learning module in AVRL framework uses 3D-ResNet [28] to
extract the visual representation with temporal correlation among video
frames.

Here, the input videos with $l$ frames are sampled to a fixed length $T$
($T=50$ in this paper) with a fixed sample interval $\lfloor l/T\rfloor$. This
fixed length frame sequence $\mathcal{V}\in\mathbb{R}^{C\times T\times H\times
W}$ is passed to the visual representation learning module
$\mathcal{F}_{vid}$. The visual representation $f_{v}$ is extracted with the
equation:

$f_{v}=\mathcal{F}_{vid}(\mathcal{V}),$ (1)

where $v_{feat}\in\mathbb{R}^{C\times 1\times 1}$ is the finally output of
3D-ResNet-34, and $C$ is the number of channels ($C_{v}=512$ in 3D-ResNet).

In vanilla 3D-ResNet, the representation $v_{feat}$ is mapped to logits for
events classification, achieving the abnormal events detection. In our
proposed AVRL framework, this video representation do not use for
classification directly, it will be merged with audio representation to
realize the multi-modal learning system.

###  III-B Audio Representation Learning

With the development of deep learning, there are some CNN-based methods [29]
have applied on audio analysis successful. The CNN-based method are usually
utilized to the image data due to the characteristic of convolutional layer.
However, the audio signals from videos only has one-dimension, and a audio
signal preprocess is necessary for the CNN-based audio representation
learning.

In this paper, Log Mel Spectrogram (LMS) for audio preprocess is used in audio
representation learning module, because the LMS features from audio is used in
CNN-based audio task generally. According to the VGGish [29], the input audio
signal from video is first resampled with 16kHz, and then the time-frequency
map of resampled audio signal is gain through a Short Time Fourier Transform
(STFT) with Hann window, and the details of preprocess are same as VGGish. The
LMS feature $\mathcal{A}\in\mathbb{R}^{H\times W}$ has two-dimensiom, and the
CNN-based methods can easily extract the hidden feature from audio signal. In
AVRL, we use VGGish as audio representation learning module
$\mathcal{F}_{aud}$. The audio representation extraction is defined as:

$f_{a}=\mathcal{F}_{aud}(\mathcal{A}),$ (2)

where $f_{a}\in\mathbb{R}^{C_{a}\times 1}$ is the audio representation, and
$C_{a}=512$ in VGGish.

###  III-C Audio-visual Representation Fusion

The traditional abnormal events detection methods only analyze the feature
from videos, but some anomaly is not noticeable in crowd scene. Supposing that
two people are fighting in a congested crowd, this fight event is not obvious
in vision due to the dense crowd, but if the people in fight are roaring or
crying, the fight event will be judged easily. Under this circumstance, the
fusion of audio and visual representation has the advantages for events
detection in crowd analysis. To detect abnormal events effectively, we fuse
the audio and visual representation by audio-visual representation fusion
module. In order to select the better fusion methods, we design several fusion
strategy and compared their performances in Section IV-E. Finally, we select a
simple and effective fusion method for AVRL framework.

Based on the visual/audio representation learning above, we fuse the
representations through feature concatenation, and then pass this multi-modal
feature into classification network $\mathcal{F}_{cls}$ to detect whether
abnormal events occur in the video. The formula is shown here:

$p=\mathcal{F}_{cls}(Cat(f_{v},f_{a})),$ (3)

where $p\in\mathbb{R}^{N\times 1\times 1}$ is the event category prediction,
$N$ is the number of categories in SHADE, $Cat$ denotes the feature
concatenate manipulation. This simple fusion module achieve the better
performance in comparison with other fusion strategy we designed, and the
details of different fusion strategy is shown in Section IV-E. The
classification network $\mathcal{F}_{cls}$ in audio-visual representation
fusion module is a simple fully connected network.

Intuitively, the audio-visual representation fusion module combines the visual
representation and audio representation, and extracts the comprehensive
feature with fully connected network for abnormal events classification. With
the auxiliary information from audio, the audio-visual combined representation
is more discriminating compared with traditional visual methods. Although the
fusion module of AVRL framework is quite simple, the experiment results
illustrate that this simple strategy is effective in abnormal events
detection, and even surpasses more complex fusion methods.

###  III-D Loss Function

Anomaly events detection in crowd scenes is a video classification task. In
order to find the anomaly events, the model should judge whether the action in
video is belong to abnormal events, such as shoot, scatter, fight, _etc._
Given a video with category $y$ and a predict category $p$, we selected Cross
Entropy Loss, a typical loss function for classification task. The loss
function $\mathcal{L}$ is defined as:

$\mathcal{L}=-\frac{1}{M}\sum_{i=1}^{M}\sum_{j=1}^{N}y_{ij}log(p_{ij}),$ (4)

where $M$ is the mini-batch size, $N$ denotes the number of categories.

###  III-E Other Details

We select the 3D-ResNet-34 pre-trained on UCF-101 [56] as the backbone for
visual representation learning on video data. 3D-ResNet is a widely used deep
learning model for video vision task. 3D-ResNet replace the traditional 2D
convolutional layer in ResNet with 3D convolutional layer, introducing the
temporal correlation in CNN. 3D convolutional filter adds an extra length
dimension in 2D convolutional filter.

To be specific, suppose $F^{i,j}_{x,y,z}$ is the value at position $(x,y,z)$
on $j^{th}$ feature map from the output of $i^{th}$ 3D convolutional layer,
and the input feature of $i^{th}$ layer has $C$ channels. The 3D convolutional
filter has $H$ height, $W$ width, and $L$ length. The 3D convolution is shown
as follow:

$F^{ij}_{x,y,z}=b^{ij}+\sum^{C}_{c=1}\sum^{H}_{h=1}\sum^{W}_{w=1}\sum^{L}_{l=1}w^{ij}_{h,w,l}\cdot
F^{(i-1)m}_{x+w,y+h,z+l}\ ,$ (5)

where $w^{ij}$, and $b^{ij}$ are the weight and bias of convolutional filter
respectively.

In addition, we select a common audio analysis CNN, VGGish pre-trained on
AudioSet [57], to learn the audio representation. The PCA postprocess of
VGGish is removed from our audio representation learning module. Due to the
discrepancy of audio length in SHADE dataset, the LMS features have different
shapes. We impose Global Average Pooling (GAP) along channel on the output of
audio representation, so that the audio representation $f_{a}$ has a unified
size $C_{a}\times 1$.

The fused audio-visual representation is passed to a fully connected network
for event categories classification. We design a simple fully connected
network $\mathcal{F}_{cls}$, and the architecture of $\mathcal{F}_{cls}$ is
illustrated in Table I:

TABLE I: Classification network in Audio-visual representation fusion module. | Layer | Input Size | Output Size
---|---|---|---
0 | Linear | 1024 | 512
1 | ReLU | 512 | 512
2 | Linear | 512 | 256
3 | ReLU | 256 | 256
4 | Linear | 256 | # of categories
5 | Classification |  |

##  IV Experiments

The experiments are conducted on SHADE dataset. The experimental details and
discussions are shown in this section.

###  IV-A Evaluation Metrics

Abnormal events detection can be considered as a video classification task, so
that the classification accuracy is the core metric for this task. The Top-1
Accuracy is gauged to compare the performances among different methods:

$Acc_{i}=\frac{1}{N}\sum_{j=1}^{M}\mathbbm{1}\\{y_{i},p_{i}\\},$ (6)
${}\mathbbm{1}\\{y_{i},p_{i}\\}=\left\\{\begin{array}[]{rcl}1&\mbox{if}&p_{i}=y_{i}\\\
0&\mbox{if}&p_{i}\neq y_{i}\end{array}\right.,$ (7)

where i indicates the $i^{th}$ category, and $N$ is the total number of
categories in SHADE dataset. $M_{i}$ is the total number of samples belong to
the $i^{th}$ category. $p_{i}$ and $y_{i}$ is the prediction and ground-truth
of category label respectively. And the indicator function
$\mathbbm{1}\\{\cdot,\cdot\\}$ is defined as Eq. 6. In the experiments, we
gauge the Top-1 Accuracy on every category separately.

###  IV-B Dataset

Due to the lack of audio collection recorders on current surveillance cameras,
the recent abnormal events detection/classification datasets under real world
only contain visual information without ambient sounds. Under this condition,
the traditional action classification dataset or abnormal events detection
datasets are not suitable for our AVRL framework, so that we choose a
synthetic dataset: SHADE [7] which is generated in the video game named Grand
Theft Auto V (GTA5), including 2149 videos (879,932 frames with the size of
$1920\times 1080$). The videos in SHADE are from nine categories, _arrest_ ,
_chase_ , _fight_ , _knowkdown_ , _run_ , _shoot_ , _scatter_ , _normal type
1_ , and _normal type 2_. Every category contains about 200 videos with
different weather conditions (rain, foggy, sunny, _etc._) and different
happening time. There are 1701 videos (about 80%) for training, and the
remaining 448 videos (about 20%) for validation or testing. Meanwhile, we
extract the audio sounds from videos for the training of audio representation
learning module.

###  IV-C Experimental Settings

In this part, some experimental settings and details are explained. Both the
spatial transformations and temporal transformations are imposed on the input
video. The input frames are first resized to the size of $240\times 240$, and
a random horizontal flip with a probability of 0.5. The temporal
transformations sample $50$ frames from the whole frame sequence with a fixed
step. The audio signal only preprocess to gain the LMS feature without any
other transformations.

In the training phase, we select Stochastic Gradient Descent (SGD) method as
the optimizer. The momentum of SGD optimizer is 0.9, and the weight decay is
0.001. The learning rate is set as $1e-2$ at the beginning, and the learning
rate is updated automatically through minimize the validation loss. From the
experiment results, the final learning rate is $1e-7$. The batch size in
training is set as 12, and we train our model 80 epochs. The experiments run
on two NVIDIA GTX 1080TI GPUs.

TABLE II: Top-1 accuracy comparison of different approaches on SHADE dataset. Category |  MLP |  LSTM |  LRCN |  3D ResNet |  N3D ResNet |  AVRL (ours) | $\Delta$
---|---|---|---|---|---|---|---
shoot |  0.866 |  0.916 |  0.905 |  0.898 |  0.920 |  0.959 | $\uparrow$
scatter |  0.822 |  0.896 |  0.896 |  0.905 |  0.923 |  0.954 | $\uparrow$
fight |  0.572 |  0.612 |  0.630 |  0.640 |  0.577 |  0.644 | $\uparrow$
arrest |  0.643 |  0.703 |  0.714 |  0.823 |  0.837 |  0.900 | $\uparrow$
knokdown |  0.663 |  0.649 |  0.621 |  0.771 |  0.742 |  0.808 | $\uparrow$
run |  0.627 |  0.650 |  0.648 |  0.606 |  0.646 |  0.595 | $\downarrow$
chase |  0.582 |  0.619 |  0.641 |  0.578 |  0.660 |  0.551 | $\downarrow$
normal1 |  0.611 |  0.635 |  0.627 |  0.656 |  0.671 |  0.700 | $\uparrow$
normal2 |  0.712 |  0.718 |  0.750 |  0.787 |  0.804 |  0.804 | -

###  IV-D Performance on SHADE Dataset

Table II illustrates the comparisons among listed models. Specially, we
compare the performance with several visual-based video classification models,
such as MLP, LSTM [58], LRCN [59], 3D ResNet [28], and N3D ResNet [7]. The
comparison demonstrates that the audio information improves the classification
accuracy significantly. The model introducing 3D convolutional layer, _e.g._
3D ResNet and N3D ResNet, achieves the better performance among visual-based
methods. Compared to the state-of-the-art method, N3D ResNet, on SHADE
dataset, the proposed AVRL framework achieve higher Top-1 accuracy on majority
of the categories, and only two categories, run and chase, are lower than N3D
ResNet. Our AVRL embeds the audio representation learning module into the
visual-based method 3D ResNet with an extremely simple fusion module, reaching
the notable advance on Top-1 accuracy for events classification. The events
with special ambient sounds, such as _shoot_ , _scatter_ , _arrest_ , and
_knokdown_ , have the remarkable increase, which means the AVRL framework
learns the discriminating feature in audio signals and aids the classification
effectively. Meanwhile, only a simple concatenation method to fuse the audio
and visual representations can gain a great improvement, showing the
superiority of the multi-modal learning in abnormal events detection.

Figure 3: The compared Audio-Visual Representation Fusion modules. Through the
experimental comparison, we select the Concat strategy eventually in proposed
AVRL.

###  IV-E Comparison of Fusion Strategy

In Section III, we only introduce the concatenation fusion strategy. Actually,
we design several different fusion strategy and compare the performance, and
we choose the concatenation fusion strategy (Concat) finally. Besides the
Concat strategy, the other fusion strategies are concatenation with layer
normalization (Concat w/ LN), concatenation with unified magnitude (Concat w/
UM), addition fusion strategy (Add), and low-level feature fusion strategy
(LF).

From the experiments, we find that the audio representation has a lower
magnitude than video representation. The average visual representation is
tenfold to that of audio representation. We attempt to introduce the layer
normalization and a scale factor $S$ respectively to alleviate this
divergence, acquiring the Concat w/ LN and Concat w/ UM strategies. Meanwhile,
to find out the suitable fusion strategy, we design different prevalent
feature fusion strategies, Add and LF, to select a better fusion strategy.

The details of several fusion strategy mentioned above are shown in Fig. 3.
First, the Concat ((a) in Fig. 3) directly concatenates the visual and audio
representation along channel dimension. The Concat w/ LN introduces the Layer
Normalization after feature concatenation to balance the visual representation
and audio representation. In addition, we adjust the magnitude by multiplying
an hand-craft scale factor $S=10$ to the audio representation in Concat w/ UM.
Due to the same shape of visual and audio representations, we naturally add
the audio feature to visual feature, fusing the multi-modal features. The
former fusion strategies all fuse the representations in the high-level
feature. In contrast, we embed the audio representation in the low-level
feature from visual representation. We first calculate the inherent
correlation in audio representation by outer product, gaining the audio
correlation map $Corr_{a}\in\mathbb{R}^{1\times 512\times 512}$. And the
$Corr_{a}$ is resized to the same spatial shape with the input frames through
bilinear interpolation. Then, the resized $Corr_{a}$ is stacked with all of
input video frames along channel dimension.

Concat

Concat w/ LN

Concat w/ UM

Add

LF

Figure 4: The visualization of classification results from the compared fusion strategies. Rectangles consisting of 9$\times$9 grids are classification results in 9 categories. TABLE III: Top-1 accuracy comparison of different approaches on SHADE dataset. Category |  Concat |  Concat w/ LN |  Concat w/ UM |  Add |  LF
---|---|---|---|---|---
shoot |  0.959 |  0.898 |  0.918 |  0.918 |  0.918
scatter |  0.954 |  0.954 |  0.977 |  0.931 |  0.954
fight |  0.644 |  0.644 |  0.689 |  0.689 |  0.644
arrest |  0.900 |  0.783 |  0.833 |  0.800 |  0.783
knokdown |  0.808 |  0.723 |  0.766 |  0.851 |  0.745
run |  0.595 |  0.714 |  0.619 |  0.762 |  0.666
chase |  0.551 |  0.531 |  0.510 |  0.469 |  0.531
normal1 |  0.700 |  0.816 |  0.800 |  0.733 |  0.633
normal2 |  0.804 |  0.848 |  0.869 |  0.848 |  0.913

The comparison of fusion strategy introduced above is illustrated in Table
III, and the classification results are demonstrated in Fig. 4. From the
comparison, the following conclusions are obtained: (1) The Concat strategy
achieves relatively high performance among several fusion strategies, reaching
the top place on _shoot_ , _arrest_ , and _chase_. (2) Compared to Concat, the
Concat w/ LN reaches higher accuracy on _run_ , _normal1_ , and _normal2_ ,
but the performances on events with discriminating ambient sounds (_shoot_ and
_arrest_) have significant decrease, which means the Concat w/ LN can not
exploit the ambient sounds sufficiently. (3) The Concat w/ UM achieves better
performance than Concat w/ LN, and the accuracy on _run_ has increase, but the
accuracy on _chase_ decrease notably. We find that the Concat w/ UM strategy
tends to consider the events with running people as _run_ events, causing the
increase of accuracy on _run_. However, the _chase_ events contain two running
people in scenes, and the Concat w/ UM confuses _chase_ with _run_ , leading
to the decrease of accuracy on _chase_. (4) The Add also achieve high
performances on several events, but the improvements is not higher than
Concat, and there is a significant decrease of performance on _chase_ due to
the same reason as (3). Finally, the improvement of LF strategy is not obvious
than other fusion strategies. In summary, with the comprehensive
consideration, we select the Concat strategy in audio-video representation
fusion module.

##  V Discussion

To further verify the advantage of AVRL and elaborate the principle about
multi-modal learning in AVRL, some discussions are developed in this section.

###  V-A The performance on Dark Scenes

In the real world, the abnormal events usually happen in the dark environment.
One of the motivations of AVRL is to aid the abnormal events detection task,
especially in some degeneration scenes, such as low illumination, noise,
_etc._ In order to stand out the robustness of AVRL, we select a subset in
SHADE that only includes the dark scenes. The selected dark subset contains
147 events from 18:00 pm to 23:59 pm, 85 events from 0:00 am to 5:59 am, and
145 events occur in daytime with low illumination environments. Furthermore,
the comparison between the proposed AVRL and the state-of-the-art N3D ResNet
on SHADE dataset. Table IV illustrates the comparison on dark subset. From the
table, we find that the proposed AVRL framework surpasses the state-of-the-art
method N3D ResNet absolutely.

TABLE IV: Top-1 accuracy comparison on dark subset. Category | N3D ResNet |  AVRL (ours)
---|---|---
shoot | 0.832 | 0.977
scatter | 0.805 | 1.0
fight | 0.551 | 0.961
arrest | 0.635 | 1.0
knokdown | 0.495 | 1.0
run | 0.632 | 0.925
chase | 0.591 | 0.914
normal1 | 0.628 | 0.925
normal2 | 0.754 | 0.921

The nighttime events in the dark subset usually occur in quite environments,
so that the event sounds are discriminative, leading to a better distinguish
of anomalies. Due to this point, the classification on _run_ and _chase_
achieves better accuracy. And some events with notable ambient sounds, such as
_shoot_ , _scatter_ , _arrest_ , and _knokdown_ , have the accuracy extremely
close to 1.0. Interestingly, the AVRL does not reaches high performance on
_fight_ , _run_ , and _chase_ in SHADE dataset, but the performance on these
categories in dark subset increases significantly. The daytime samples have
noise from crowds involving chatting, bustle, _etc._ which do harm to the
assistance from audio. Overall, the AVRL framework addresses the anomaly
detection with dark scenes effectively, proving the advantage of audio-visual
multi-modal learning.

###  V-B Ablation Study

The ablation study is conduct to further compare the influences from audio
representation and visual representation. We consider three schemes: only
visual scheme (3D ResNet), only audio (VGGish), and audio-visual (the proposed
AVRL). The results of ablation study are shown in Table V.

TABLE V: The comparison on top-1 accuracy in ablation study. Category | 3D ResNet | VGGish |  AVRL (ours)
---|---|---|---
shoot | 0.898 | 0.918 | 0.959
scatter | 0.905 | 0.591 | 0.954
fight | 0.640 | 0.133 | 0.644
arrest | 0.823 | 0.417 | 0.900
knokdown | 0.771 | 0.638 | 0.808
run | 0.606 | 0.476 | 0.595
chase | 0.578 | 0.326 | 0.551
normal1 | 0.656 | 0.433 | 0.700
normal2 | 0.787 | 0.326 | 0.804

From the Table V, the following conclusion can be obtained: (1) The 3D ResNet
(only visual) achieves better accuracy than VGGish (only audio), which means
the visual representation makes heavier contributions on abnormal events
detection than audio representation. (2) VGGish (only audio) reaches high
classification accuracy on the events with special ambient sounds, especially
the _shoot_. (3) Although the performance of VGGish (only audio) is weak, the
AVRL (audio-visual) achieves better classification accuracy by combining the
audio and video representations.

(a) Temporal Patch

(b) Spatial Patch

Figure 5: The temporal and spatial patch applied in this section.

(a) Temporal Patch

(b) Spatial Patch

Figure 6: The influence on classification accuracy of temporal and spatial
patch ratio.

###  V-C Temporal and Spatial Patches

In order to probe the relationship between visual representation and audio
representation, we add patches randomly in videos, as shown in Fig. 5. As for
temporal patch, the video frames are randomly selected under a proportion and
colored by black (Fig. 5(a)). Moreover, the black patches with a proportion of
the full spatial area are put to random positions on every frames (Fig. 5(b)).
The comparison results are demonstrated in Fig. 6(a) and Fig. 6(b).

With the increase of patch ratio, the classification accuracy of both temporal
patch and spatial patch decreases unsurprisingly. The effect of temporal patch
is less than that of spatial patch, so that the accuracy of spatial patch have
fallen more against temporal patch, which means that the spatial correlation
in single frame provide more contributions than temporal correlation among
frames for events classification. From Fig. 6, we find that the accuracy on
some categories with 80% spatial patch ratio is lower than that of the method
with only audio representation. This probably because the high spatial patch
ratio lead to the extremely noisy distortion on content and this distortion
mislead the visual representation learning, causing the impediment to events
detection.

##  VI Conclusion

This paper proposes a novel multi-modal learning framework AVRL for abnormal
events detection. The AVRL framework attempts to improve the abnormal events
detection accuracy in crowd scenes through combining the visual representation
and audio representation from the surveillance videos. Specifically, a 3D
ResNet is applied for visual representation learning and a audio analysis
model, VGGish, is used for audio representation learning. On obtaining the
audio and visual representations, a simple concatenation feature fusion module
is exploited to fuse the multi-modal representations. Experiments show that
the proposed AVRL has significant superiority compared with state-of-the-art
visual-based methods.

In the AVRL, we attempt several feature fusion strategies preliminarily and
the fusion module still needs optimize to achieve better anomaly detection
accuracy. Meanwhile, due to the limitation of datasets, the experiments are
only conducted on a synthetic dataset, SHADE dataset, rather than real-world
dataset. With the emergence of surveillance video datasets with audio signals,
we will attempt to pursue this research on more datasets.

## References

  * [1] Q. Wang, J. Gao, W. Lin, and Y. Yuan, “Learning from synthetic data for crowd counting in the wild,” in _Proceedings of IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 8198–8207.
  * [2] Y. Zhou, J. Yang, H. Li, T. Cao, and S.-Y. Kung, “Adversarial learning for multiscale crowd counting under complex scenes,” _IEEE transactions on cybernetics_ , 2020.
  * [3] S. Abousamra, M. Hoai, D. Samaras, and C. Chen, “Localization in the crowd with topological constraints,” _arXiv preprint arXiv:2012.12482_ , 2020.
  * [4] Y. Wang, J. Hou, X. Hou, and L.-P. Chau, “A self-training approach for point-supervised object detection and counting in crowds,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 2876–2887, 2021.
  * [5] J. Gao, T. Han, Y. Yuan, and Q. Wang, “Learning independent instance maps for crowd localization,” _arXiv preprint arXiv:2012.04164_ , 2020.
  * [6] M. Thida, H.-L. Eng, and P. Remagnino, “Laplacian eigenmap with temporal constraints for local abnormality detection in crowded scenes,” _IEEE Transactions on Cybernetics_ , vol. 43, no. 6, pp. 2147–2156, 2013.
  * [7] W. Lin, J. Gao, Q. Wang, and X. Li, “Learning to detect anomaly events in crowd scenes from synthetic data,” _Neurocomputing_ , vol. 436, pp. 248–259, 2021.
  * [8] S. Ali and M. Shah, “A lagrangian particle dynamics approach for crowd flow segmentation and stability analysis,” in _2007 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2007, pp. 1–6.
  * [9] A. S. Rao, J. Gubbi, S. Marusic, and M. Palaniswami, “Crowd event detection on optical flow manifolds,” _IEEE transactions on cybernetics_ , vol. 46, no. 7, pp. 1524–1537, 2015.
  * [10] V. J. Kok and C. S. Chan, “Grcs: Granular computing-based crowd segmentation,” _IEEE transactions on cybernetics_ , vol. 47, no. 5, pp. 1157–1168, 2016.
  * [11] Q. Wang, J. Gao, W. Lin, and Y. Yuan, “Pixel-wise crowd understanding via synthetic data,” _International Journal of Computer Vision_ , vol. 129, no. 1, pp. 225–245, 2021.
  * [12] R. Mazzon, F. Poiesi, and A. Cavallaro, “Detection and tracking of groups in crowd,” in _2013 10th IEEE International Conference on Advanced Video and Signal Based Surveillance_. IEEE, 2013, pp. 202–207.
  * [13] X. Li, M. Chen, F. Nie, and Q. Wang, “A multiview-based parameter free framework for group detection,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 31, no. 1, 2017.
  * [14] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Advances in neural information processing systems_ , vol. 25, pp. 1097–1105, 2012.
  * [15] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [16] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [17] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei-Fei, “Large-scale video classification with convolutional neural networks,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2014, pp. 1725–1732.
  * [18] B. Zhao, H. Li, X. Lu, and X. Li, “Reconstructive sequence-graph network for video summarization,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , DOI:10.1109/TPAMI.2021.3072117, 2015.
  * [19] X. LI and B. ZHAO, “Video distillation,” _SCIENCE CHINA Information Sciences_ , 2021.
  * [20] S. Hershey, S. Chaudhuri, D. P. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold _et al._ , “Cnn architectures for large-scale audio classification,” in _2017 ieee international conference on acoustics, speech and signal processing (icassp)_. IEEE, 2017, pp. 131–135.
  * [21] J. F. Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in _2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2017, pp. 776–780.
  * [22] D. Hu, Z. Wang, H. Xiong, D. Wang, F. Nie, and D. Dou, “Curriculum audiovisual learning,” _arXiv preprint arXiv:2001.09414_ , 2020.
  * [23] J. Wan, W. Luo, B. Wu, A. B. Chan, and W. Liu, “Residual regression with semantic prior for crowd counting,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4036–4045.
  * [24] J. Gao, Q. Wang, and X. Li, “Pcc net: Perspective crowd counting via spatial convolutional network,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 30, no. 10, pp. 3486–3498, 2020.
  * [25] W. Sultani, C. Chen, and M. Shah, “Real-world anomaly detection in surveillance videos,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 6479–6488.
  * [26] A. Singh, D. Patil, and S. Omkar, “Eye in the sky: Real-time drone surveillance system (dss) for violent individuals identification using scatternet hybrid deep learning network,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , June 2018\.
  * [27] D. Hu, L. Mou, Q. Wang, J. Gao, Y. Hua, D. Dou, and X. X. Zhu, “Ambient sound helps: Audiovisual crowd counting in extreme conditions,” _arXiv preprint arXiv:2005.07097_ , 2020.
  * [28] K. Hara, H. Kataoka, and Y. Satoh, “Learning spatio-temporal features with 3d residual networks for action recognition,” in _2017 IEEE International Conference on Computer Vision Workshops, ICCV Workshops 2017, Venice, Italy, October 22-29, 2017_. IEEE Computer Society, 2017, pp. 3154–3160.
  * [29] S. Hershey, S. Chaudhuri, D. P. W. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seybold, M. Slaney, R. J. Weiss, and K. W. Wilson, “CNN architectures for large-scale audio classification,” in _2017 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2017, New Orleans, LA, USA, March 5-9, 2017_. IEEE, 2017, pp. 131–135.
  * [30] K. Cheng, Y. Chen, and W. Fang, “Abnormal crowd behavior detection and localization using maximum sub-sequence search,” in _Proceedings of the 4th ACM/IEEE international workshop on Analysis and retrieval of tracked events and motion in imagery stream, ARTEMIS@ACM Multimedia 2013, Barcelona, Spain, October 21, 2013_. ACM, 2013, pp. 49–58.
  * [31] B. Du, Y. Zhang, L. Zhang, and D. Tao, “Beyond the sparsity-based target detector: A hybrid sparsity and statistics-based detector for hyperspectral images,” _IEEE Trans. Image Process._ , vol. 25, no. 11, pp. 5345–5357, 2016.
  * [32] H. Chen, O. Nedzvedz, S. Ye, and S. Ablameyko, “Crowd abnormal behaviour identification based on integral optical flow in video surveillance systems,” _Informatica_ , vol. 29, no. 2, pp. 211–232, 2018.
  * [33] B. Du, L. Ru, C. Wu, and L. Zhang, “Unsupervised deep slow feature analysis for change detection in multi-temporal remote sensing images,” _IEEE Trans. Geosci. Remote. Sens._ , vol. 57, no. 12, pp. 9976–9992, 2019.
  * [34] H. Liao, J. Xiang, W. Sun, Q. Feng, and J. Dai, “An abnormal event recognition in crowd scene,” in _Sixth International Conference on Image and Graphics, ICIG 2011, Hefei, Anhui, China, August 12-15, 2011_. IEEE Computer Society, 2011, pp. 731–736.
  * [35] R. Mehran, A. Oyama, and M. Shah, “Abnormal crowd behavior detection using social force model,” in _2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA_. IEEE Computer Society, 2009, pp. 935–942.
  * [36] J. Cui, W. Liu, and W. Xing, “Crowd behaviors analysis and abnormal detection based on surveillance data,” _J. Vis. Lang. Comput._ , vol. 25, no. 6, pp. 628–636, 2014.
  * [37] Y. Zhang, L. Qin, R. Ji, H. Yao, and Q. Huang, “Social attribute-aware force model: Exploiting richness of interaction for abnormal crowd detection,” _IEEE Trans. Circuits Syst. Video Technol._ , vol. 25, no. 7, pp. 1231–1245, 2015.
  * [38] D. Helbing and P. Molnár, “Social force model for pedestrian dynamics,” _Phys. Rev. E_ , vol. 51, pp. 4282–4286, 1995.
  * [39] R. L. Cannon, J. V. Dave, and J. C. Bezdek, “Efficient implementation of the fuzzy c-means clustering algorithms,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 8, no. 2, pp. 248–255, 1986. [Online]. Available: https://doi.org/10.1109/TPAMI.1986.4767778
  * [40] C. Demarty, C. Penet, M. Soleymani, and G. Gravier, “Vsd, a public dataset for the detection of violent scenes in movies: design, annotation, analysis and evaluation,” _Multim. Tools Appl._ , vol. 74, no. 17, pp. 7379–7404, 2015\.
  * [41] W. Sultani, C. Chen, and M. Shah, “Real-world anomaly detection in surveillance videos,” in _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_. IEEE Computer Society, 2018, pp. 6479–6488.
  * [42] I. Mironica, I. C. Duta, B. Ionescu, and N. Sebe, “A modified vector of locally aggregated descriptors approach for fast video classification,” _Multim. Tools Appl._ , vol. 75, no. 15, pp. 9045–9072, 2016.
  * [43] H. R. Rabiee, J. Haddadnia, H. Mousavi, M. Nabi, V. Murino, and N. Sebe, “Emotion-based crowd representation for abnormality detection,” _CoRR_ , vol. abs/1607.07646, 2016.
  * [44] H. R. Rabiee, J. Haddadnia, H. Mousavi, M. Kalantarzadeh, M. Nabi, and V. Murino, “Novel dataset for fine-grained abnormal behavior understanding in crowd,” in _13th IEEE International Conference on Advanced Video and Signal Based Surveillance, AVSS 2016, Colorado Springs, CO, USA, August 23-26, 2016_. IEEE Computer Society, 2016, pp. 95–101.
  * [45] S. Dupont and J. Luettin, “Audio-visual speech modeling for continuous speech recognition,” _IEEE Trans. Multim._ , vol. 2, no. 3, pp. 141–151, 2000\.
  * [46] J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng, “Multimodal deep learning,” in _Proceedings of the 28th International Conference on Machine Learning, ICML 2011, Bellevue, Washington, USA, June 28 - July 2, 2011_. Omnipress, 2011, pp. 689–696.
  * [47] D. Hu, X. Li, and X. Lu, “Temporal multimodal learning in audiovisual speech recognition,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016_. IEEE Computer Society, 2016, pp. 3574–3582.
  * [48] S. Kettebekov, M. Yeasin, and R. Sharma, “Prosody based audiovisual coanalysis for coverbal gesture recognition,” _IEEE Trans. Multim._ , vol. 7, no. 2, pp. 234–242, 2005.
  * [49] Z. Zeng, J. Tu, M. Liu, T. S. Huang, B. Pianfetti, D. Roth, and S. E. Levinson, “Audio-visual affect recognition,” _IEEE Trans. Multim._ , vol. 9, no. 2, pp. 424–428, 2007.
  * [50] A. Owens, J. Wu, J. H. McDermott, W. T. Freeman, and A. Torralba, “Ambient sound provides supervision for visual learning,” in _Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I_ , ser. Lecture Notes in Computer Science, vol. 9905. Springer, 2016, pp. 801–816.
  * [51] A. Owens, J. Wu, J. McDermott, W. Freeman, and A. Torralba, “Learning sight from sound: Ambient sound provides supervision for visual learning,” _Int. J. Comput. Vis._ , vol. 126, no. 10, pp. 1120–1137, 2018.
  * [52] R. Arandjelovic and A. Zisserman, “Look, listen and learn,” in _IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017_. IEEE Computer Society, 2017, pp. 609–617.
  * [53] D. Hu, F. Nie, and X. Li, “Deep multimodal clustering for unsupervised audiovisual learning,” in _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_. Computer Vision Foundation / IEEE, 2019, pp. 9248–9257.
  * [54] D. Hu, Z. Wang, H. Xiong, D. Wang, F. Nie, and D. Dou, “Curriculum audiovisual learning,” _CoRR_ , vol. abs/2001.09414, 2020.
  * [55] C. Gan, H. Zhao, P. Chen, D. D. Cox, and A. Torralba, “Self-supervised moving vehicle tracking with stereo sound,” in _2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 \- November 2, 2019_. IEEE, 2019, pp. 7052–7061.
  * [56] K. Soomro, A. R. Zamir, and M. Shah, “UCF101: A dataset of 101 human actions classes from videos in the wild,” _CoRR_ , vol. abs/1212.0402, 2012\.
  * [57] J. F. Gemmeke, D. P. W. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter, “Audio set: An ontology and human-labeled dataset for audio events,” in _2017 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2017, New Orleans, LA, USA, March 5-9, 2017_. IEEE, 2017, pp. 776–780.
  * [58] H. Sak, A. W. Senior, and F. Beaufays, “Long short-term memory recurrent neural network architectures for large scale acoustic modeling,” in _INTERSPEECH 2014, 15th Annual Conference of the International Speech Communication Association, Singapore, September 14-18, 2014_ , H. Li, H. M. Meng, B. Ma, E. Chng, and L. Xie, Eds. ISCA, 2014, pp. 338–342.
  * [59] J. Donahue, L. A. Hendricks, M. Rohrbach, S. Venugopalan, S. Guadarrama, K. Saenko, and T. Darrell, “Long-term recurrent convolutional networks for visual recognition and description,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 39, no. 4, pp. 677–691, 2017.