#  Micro Batch Streaming: Allowing the Training of DNN Models to Use a large
Batch Size in Memory Constrained Environments

XinYu Piao1 *, DoangJoo Synn1 *, JooYoung Park2, Jong-Kook Kim1, 2 †
1 Department of Electrical and Computer Engineering
2 School of Electrical Engineering
Korea University
{xypiao97, alansynn, nehalem<EMAIL_ADDRESS>

###### Abstract

Recent deep learning models are difficult to train using a large batch size,
because commodity machines may not have enough memory to accommodate both the
model and a large data batch size. The batch size is one of the hyper-
parameters used in the training model, and it is dependent on and is limited
by the target machine memory capacity because the batch size can only fit into
the remaining memory after the model is uploaded. Moreover, the data item size
is also an important factor because if each data item size is larger then the
batch size that can fit into the remaining memory becomes smaller. This paper
proposes a framework called Micro-Batch Streaming (MBS) to address this
problem. This method helps deep learning models to train by providing a batch
streaming method that splits a batch into a size that can fit in the remaining
memory and streams them sequentially. A loss normalization algorithm based on
the gradient accumulation is used to maintain the performance. The purpose of
our method is to allow deep learning models to train using larger batch sizes
that exceed the memory capacity of a system without increasing the memory size
or using multiple devices (GPUs).

00footnotetext: These authors contributed equally.00footnotetext:
Corresponding author.

##  1 Introduction

Recently, many research use heavy Deep Neural Network (DNN) models that need a
lot of memory to execute. Plus, data sizes are increasing such that it is
difficult to increase the mini-batch size which is used for the training of
the models. The mini-batch size is an important hyper-parameter that
determines the number of data set items that are used in one iteration of a
training process and the mini-batch size may affect the overall performance of
the DNN as shown in [1]. The results in table 1 show the comparison between a
large mini-batch and a small mini-batch used for the image classification and
semantic segmentation problems. Utilizing a large mini-batch shows 21.88% and
2.01% higher than a small mini-batch when using higher resolution images for
image classification and semantic segmentation problems, respectively.

The image size may also affect the model performance, because the image size
may limit the mini-batch size. Also, higher resolution images (i.e., larger
image size) contain more information about objects. Table 1 shows the
comparison of results between the higher resolution images and the lower
resolution images that are used for the image classification and semantic
segmentation problems. Using the higher resolution image data shows 21.64% and
2.01% higher than the lower resolution image data in a large mini-batch for
image classification and semantic segmentation problems, respectively. This
aspect is also similar when using a small mini-batch size.

Table 1:  The effect of batch size and image size for a image classification model (ResNet-50) using Flower-102 dataset and a semantic segmentation model (U-Net) using Carvana dataset.  Model | ResNet-50 | U-Net
---|---|---
Metric | Max. acc (%) | Max. IoU (%)
Image size | 32$\times$32 | 224$\times$224 | 96$\times$96 | 384$\times$384
Batch | 2 | 48.66 | 61.86 | 92.30 | 93.61
Size | 16 | 62.10 | 83.74 | 93.61 | 95.62

The total memory size of a mini-batch can only be increased to the remaining
memory after the model is loaded. Thus, there is a limit on the size of the
mini-batch and the number of data items included in the mini-batch. If the
memory requirement of a certain mini-batch size is larger than the free
remaining memory size, the mini-batch cannot be allocated to the GPU memory
and the model cannot be trained. If the optimal mini-batch size for a
particular resolution dataset is larger than the device memory, then the mini-
batch size must be reduced when increasing the image resolution or use low-
resolution image data to increase the mini-batch size. Therefore, the
significant increase in image size makes it more challenging to train DNN
models.

Many researchers tried various techniques such as data parallelism and/or
model parallelism to alleviate the problems that deep learning methods face.
Data parallelism (e.g., [2, 3, 4]) is usually used when the mini-batch size is
too large to be fit into a single device’s memory. A mini-batch is partitioned
for computation and scattered to multiple devices and each device has a full
copy of the learning model. When all data within a mini-batch is processed,
weights are updated across devices through communications. Model parallelism
partitions the learning model into cells and distributes the cells to multiple
devices. It is usually used when the model is too large to fit into a device’s
memory (e.g., [5, 6, 7]). Other methods that use pipeline parallelism are
proposed (e.g., [8, 9]). These methods employ Data-Parallel Synchronous
Stochastic Gradient Descent (SGD), which distributes mini-batches across many
machines in unit of micro-batches and executes them in a pipelined fashion.
Although, all of these research improve the learning and performance of the
models, they still have the problem of the mini-batch size being limited by
the device memory size.

This paper proposes the Micro-Batch Streaming (MBS), which is a method that
can fetch a large batch of data using the stream-based pipeline scheme to
train models even if the batch cannot fit into the memory without increasing
the device memory or using multiple devices (GPUs). Thus, allowing researchers
to experiment using large mini-batch sizes on a single device. The idea is to
split a mini-batch into $n$ micro-batches ([8, 9])and stream them sequentially
to a GPU. Results show that MBS can increase the training batch size to the
full size of the training set regardless of the model type, dataset, and data
size. To maintain the performance, MBS computes the gradient for a large batch
using loss normalization based on micro-batch gradient accumulation, a method
that accumulates the gradient of multiple micro-batches.

This paper is organized as follows. Related work is described in Section 2.
Section 3 introduces MBS and terminologies used throughout this paper is
described, and the loss normalization algorithm used to maintain the
performance is introduced. Section 4 depicts the results. Finally, Section 5
concludes the paper.

##  2 Related work

If model parallelism is done naively, GPUs suffer from idle time overhead
because a worker have to wait for previous or successor worker to finish its
job. To minimize this overhead, GPipe [8], a model-parallelism library for
training large neural networks, presented a novel pipeline parallelism with
batch splitting. GPipe splits mini-batch into smaller batches, called micro-
batch, for pipeline execution across multiple GPUs. This enables each GPU to
work on different micro-batch simultaneously, and gradient and model
parameters are updated synchronously after all micro-batches are processed.
However, synchronizing at the end of every mini-batch still causes GPU to
stall. Pipedream [9] used asynchronous communication method and corresponding
work scheduling and learning method to handle this problem. But neither of
these solutions are adequate for large mini-batch training because they focus
on maximizing the learning performance or parallelization of the learning
process. Plus, they have a limitation where the batch size cannot exceed the
free memory space as the physical memory does not have infinite capacity. The
proposed method in this paper solves this limitation without increasing the
memory of a single device and without using multiple devices (GPUs)

##  3 Micro-Batch Streaming

(a) Processing flow over time

(b) A system perspective of MBS

Figure 1: Overview of Micro-Batch Streaming (MBS)

###  3.1 Stream-based Pipeline

The stream-based pipeline is a pipeline method that splits the data and
streams them sequentially to target devices. Usually, the batch data is pre-
loaded into the main memory before being loaded and trained in the GPU.
Because of the limited size of the GPU’s memory, it may not be possible to
load a large batch of data onto the GPU memory for training. The stream-based
pipeline used in the Micro-Batch Streaming (MBS) splits a large batch of data
into smaller batches that can be allocated to the remaining GPU memory after
the learning model is loaded. Therefore, the MBS enables the training of a DNN
model to use a larger mini-batch that exceeds the remaining memory space after
the loading of the learning model by training micro batches that fit into the
remaining memory streamed from the CPU.

###  3.2 Micro-Batch

In MBS, the micro-batch ($\mu$-batch) in [8] is extended and is defined as; 1)
a unit of streaming to the target device, and 2) a unit of computing on the
target device. The first definition presents the splitting of the input mini-
batch into micro batches that can fit onto the remaining memory space and
streaming them sequentially to the target device. The second statement
describes the unit size that target device uses to train the neural network.
The formulation of the micro-batch can be expressed as follows.

$mini\,batch=\bigcup_{i\in S_{\mu}}\mu\,batch_{i}$ (1)

$\mu\,batch_{i}\subset mini\,batch\;(i\in S_{\mu})\\\ \vspace{0.06cm}$ (2)

$\mu\,batch_{size}\leq mini\,batch_{size}$ (3)

where $S_{\mu}$ in equation 1 and 2 is the set of micro-batches for that
certain mini-batch. The micro-batch effectively removes the dependency between
the mini-batch size and the target device’s memory capacity. While MBS is
using micro-batch for training, this information is accrued such that the
model can be trained as if using a large mini-batch size.

###  3.3 MBS Process

Figure 1 and 2 show the overview of the process of MBS between CPU and GPU.
The input mini-batch is split into micro-batches in the CPU, and these micro-
batches are sequentially streamed by the CPU to the GPU for computation. When
the learning model starts to train, the GPU memory is split into two domains
(shown in figure 2); one is the model parameter space where model parameters
and gradients reside, and the other is the data space where tensorized input
batch and intermediate outcomes are populated. The data in the model parameter
space is used to determine the updates of the deep learning model’s
parameters. The intermediate outcomes that are computed by the forward pass of
the model and in the data space are used to calculate the gradients in the
following processes.

The training process using the MBS is as follows (figure 2). First, MBS loads
the mini-batch dataset to the CPU memory space and then splits the mini-batch
into $n$ micro-batches in preparation to stream to the GPU (❶). Then, the
micro-batches are streamed sequentially to the GPU and the GPU starts to train
the model using each micro-batch (❷). The GPU executes the forward and
backward steps and stores the gradient in the model parameter space (❸). Only
when all the micro-batches in the mini-batch are used for training, the model
parameters are updated. The gradients computed by the forward and backward
propagation are accumulated until all micro-batch is processed (❹). When the
final micro-batch is completed, MBS updates the model parameters using the
accumulated gradient, similar to the gradient computed by the mini-batch (❺).
Therefore, it is as though the model’s parameters are updated using the mini-
batch, while the gradients are computed from micro-batches.

The accumulated gradient is computed using the gradient accumulation method
that adds the earlier gradients and current gradient until a update. This
results in the accumulated gradient being larger than the gradient computed by
using a mini-batch. This is because the training process of deep learning
model using the MBS and using the mini-batch is different. Thus, the resulting
performance can be different. This paper introduces a loss normalization
method that normalizes the loss calculated using MBS by a loss function at
every micro-batch iteration. The necessity and appropriateness of loss
normalization is described in detail in the next section. In MBS, the loss
normalization method is used for the automatic update of gradients and MBS
waits until all micro-batches, split from a mini-batch, are completed before
the update. The timing of the update will seem to be the same as an update
after a mini-batch is completed.

Figure 2: The training process using MBS.

###  3.4 Loss Normalization

Stochastic Gradient Descent (SGD) [10] is one of the widely used method for
deep learning. SGD computes the loss from the difference between outputs
computed from the forward pass in the model and targets. MBS adopted the
gradient accumulation to derive the gradient of the mini-batch using gradients
calculated and accumulated from using micro-batches. If loss accumulation is
performed without loss normalization in the micro-batch loss calculation, loss
calculated at every micro-batch unit will be accumulated, leading to a
different loss calculation compared to the loss calculated without MBS. This
section defines the problem of gradient accumulation in the case of training
using MBS and proves the necessity and appropriateness of the loss
normalization method introduced in this paper.

Gradient accumulation calculates the sum of the previous gradient and the
current gradient until updated
($\nabla\omega_{accum}=\sum_{i=0}^{n}\nabla\omega_{i}$). The formulation of
loss ($\mathcal{L}_{B}$) and gradient ($\nabla\omega_{B}$) for a mini-batch
can be expressed as follows:

$\mathcal{L}_{B}={\frac{1}{N_{B}}}\sum_{i=1}^{N_{B}}\mathcal{L}(output_{i},target_{i})={\frac{1}{N_{B}}}\sum_{i=1}^{N_{B}}\mathcal{L}_{i}$
(4)
$\nabla\omega_{B}=\partial(\mathcal{L}_{B})=\partial({\frac{1}{N_{B}}}\sum_{i=1}^{N_{B}}\mathcal{L}_{i})$
(5) $N_{B}=N_{S_{\mu}}\times N_{\mu}$ (6)

where $\mathcal{L}_{i}$ is loss that is calculated between output and target
for the $i$-th batch in one mini-batch, $N_{S_{\mu}}$ is the size of set of
micro-batches in a mini-batch, $N_{\mu}$ is the size of the micro-batch, and
$N_{B}$ is the size of the mini-batch. The formulation of loss between a mini-
batch and micro-batches can be defined as follows (equation 7 and 8):

$\displaystyle\sum_{i=1}^{N_{B}}\mathcal{L}_{i}$
$\displaystyle=\sum_{j=1}^{N_{S_{\mu}}}\sum_{k=1}^{N_{\mu}}\mathcal{L}_{j,k}$
(7) $\displaystyle\frac{1}{{N_{B}}}\sum_{i=1}^{N_{B}}\mathcal{L}_{i}$
$\displaystyle=\frac{1}{N_{S_{\mu}}\cdot
N_{\mu}}\sum_{j=1}^{N_{S_{\mu}}}\sum_{k=1}^{N_{\mu}}\mathcal{L}_{j,k}$ (8)

The relationship of the gradient between a mini-batch and micro-batches can be
expressed as follows:

$\displaystyle\nabla\omega_{B}$
$\displaystyle=\partial({\frac{1}{N_{B}}}\sum_{i=1}^{N_{B}}\mathcal{L}_{i})$
(9) $\displaystyle=\partial({\frac{1}{N_{S_{\mu}}\cdot
N_{\mu}}}\sum_{j=1}^{N_{S_{\mu}}}\sum_{k=1}^{N_{\mu}}\mathcal{L}_{j,k})$ (10)
$\displaystyle=\partial({\frac{1}{N_{S_{\mu}}}}\sum_{k=1}^{N_{S_{\mu}}}({\frac{1}{N_{\mu}}}\sum_{j=1}^{N_{\mu}}\mathcal{L}_{j})_{k})$
(11)
$\displaystyle=\partial({\frac{1}{N_{S_{\mu}}}}\sum_{k=1}^{N_{S_{\mu}}}(\mathcal{L}_{\mu})_{k})$
(12)

where $\nabla\omega_{B}$ is the gradient of one mini-batch, and
$\mathcal{L}_{\mu}$ is the loss of one micro-batch. The accumulated gradient
($\nabla\omega_{accum}$) can be expressed as follows:

$\nabla\omega_{accum}=\sum_{k=1}^{N_{S_{\mu}}}(\nabla\omega_{\mu})_{k}=\sum_{k=1}^{N_{S_{\mu}}}\partial(\mathcal{L}_{\mu})_{k}=\partial(\sum_{k=1}^{N_{S_{\mu}}}(\mathcal{L}_{\mu})_{k})$
(13)

The above equations (from equation 12 to equation 13) prove that the
accumulated gradient is not equal to the gradient of a mini-batch. This means
that there is a need for a method to normalize the gradients of a mini-batch
that uses the MBS.

To calculate the gradient of mini-batch, we considered two normalization
methods; one is normalizing the accumulated gradient for each layer, the other
is normalizing loss computed by the loss function. These two methods provide
the same effect of normalizing the gradient. However, the former method uses a
complex algorithm to normalize the accumulated gradients for each layer, and
it will cause heavy overhead at run-time. The latter method does not have a
heavy overhead at run-time because it is a simple method that normalizes the
accumulated loss (equation 14). Therefore, the latter method to normalize the
gradient is used for MBS because this method provides the similar effect of
normalizing and is very simple.

$\displaystyle\mathcal{L}_{norm_{j}}$
$\displaystyle=\frac{1}{N_{S_{\mu}}}\times\mathcal{L}_{\mu_{j}}$ (14)
$\displaystyle\sum_{j=1}^{N_{S_{\mu}}}\partial(\mathcal{L}_{norm_{j})}$
$\displaystyle=\sum_{j=1}^{N_{S_{\mu}}}\partial(\frac{1}{N_{S_{\mu}}}\times\mathcal{L}_{\mu_{j}})$
(15)
$\displaystyle=\sum_{j=1}^{N_{S_{\mu}}}\partial(\frac{1}{N_{S_{\mu}}}\times\frac{1}{N_{\mu}}\sum_{k=1}^{N_{\mu}}\mathcal{L}_{j,k})$
(16) $\displaystyle=\partial(\frac{1}{N_{S_{\mu}}\cdot
N_{\mu}}\sum_{j=1}^{N_{S_{\mu}}}\sum_{k=1}^{N_{\mu}}\mathcal{L}_{j,k})$ (17)

The above equations (from equation 15 to equation 17) prove that the loss
normalization provides the same calculation compared to equation 10. The loss
normalization algorithm dynamically determines the normalizing factor and
normalizes the loss considering the current input mini-batch size because the
mini-batch size is not guaranteed to be uniform for all iterations (shown in
Algorithm 1).

0: The size of micro-batch $N_{\mu}$, a mini-batch $B$

1: Count the size of mini-batch $N_{B}$ from $B$

2: if $N_{B}<N_{\mu}$ then

3: $N_{\mu}\leftarrow N_{B}$

4: end if

5: $N_{S_{\mu}}$ $\leftarrow$ Round-up($N_{B}$ / $N_{\mu}$)

6: List of micro-batches ($\mu$) $\leftarrow$ Split(B, $N_{\mu}$)

7: $i$ $\leftarrow$ 0

8: while $i<N_{S_{\mu}}$ do

9: Forward-propagation using micro-batch ($\mu$i)

10: $\mathcal{L}_{\mu_{i}}\leftarrow\mathcal{L}(output_{i},target_{i})$

11: $\mathcal{L}_{norm}\leftarrow\mathcal{L}_{\mu_{i}}/N_{S_{\mu}}$

12: Backward-propagation and accumulate gradients using $\mathcal{L}_{norm}$

13: $i\leftarrow i+1$

14: end while

15: Update the DNN model parameters using accumulated gradients by optimizer

Algorithm 1 Loss Normalization Algorithm

##  4 Evaluation

###  4.1 Overview

Experiments show that MBS is an efficient method to train deep learning models
with a large batch regardless of models, datasets, data size and capacity of
GPU memory in limited memory environments. The loss normalization method
presented in this paper is used and it seems to be a good method that
maintains performance when training using MBS.

###  4.2 Experimental Setup

####  4.2.1 Systems

The experiments are run on a system consisting of GeForce RTX 3090 GPU with
24GB GDDR6 memory, intel i7 3.7GHz 6-Core processor, and 64GB main memory. MBS
was implemented by using PyTorch version 1.10.2 and CUDA version 11.3.

####  4.2.2 Models

We evaluate MBS performance with three different types of model architectures.
For image classification tasks, two ResNet models [11] (ResNet-50 and
ResNet-101)111https://github.com/weiaicunzai/pytorch-cifar100 and the
AmoebaNet [12] model that uses the architecture search algorithm are used for
comparison. Specifically, the AmoebaNet-D model that is used consists of 6
layers and 190 filters and is implemented in
[8]222https://github.com/kakaobrain/torchgpipe. For semantic segmentation
tasks, the U-Net [13] model, commonly used in many research, is used for
testing MBS. The batch normalization layers, which compute the statistics
along the input mini-batch dimension, are included in the experiment models.

####  4.2.3 Datasets

To evaluate the performance of the proposed MBS, two datasets are used that
consist of large image size data. One is the Flower-102 dataset [14] which
includes 8,189 color images in 102 classes, where each class consists of
between 40 and 258 images, for image classification tasks. The image size of
Flower-102 dataset varies, ranging from 500 $\times$ 500 up to 1168 $\times$
500\. The other is the Carvana333https://www.kaggle.com/c/carvana-image-
masking-challenge dataset which consists of both color images and binary
images of 5,088 1918$\times$1280 image resolution for semantic segmentation
tasks.

The original image size is not appropriate for comparison between the learning
models that use or not use MBS. Because if the image size is too large, the
baseline models that do not use MBS will fail even after 2 mini-batch size
because of the memory limitations. Thus, the image size of the given datasets
are resized. The image size of Flower-102 dataset was fixed to 224 $\times$
224, which is widely used image size is other research such as [8, 15, 16] in
image classification tasks, for the ResNet models. For the AmoebaNet-D model,
the image size of Flower-102 dataset was fixed at 416 $\times$ 416 image size
which is used in [8]2. The image size of Carvana dataset was fixed at 384
$\times$ 384 as used in [17] with the method of random crop from a resized 959
$\times$ 640 image, which is a method to crop the given image at random
location, for U-Net models.

####  4.2.4 Methods

ResNet-50 and ResNet-101 models are both trained using the same cross-entropy
loss function and the SGD optimizer with parameters set to 0.01 learning rate,
0.9 momentum, and 0.0005 decay. AmoebaNet-D model is trained using the cross-
entropy loss function, SGD optimizer with parameters set to 0.1 learning rate,
0.9 momentum, and 0.0001 decay. AmoebaNet-D uses a learning rate scheduler
that reduces the learning rate linearly during training.

U-Net model is trained using binary cross-entropy (BCE) loss with dice
coefficient (DC), Adam optimizer with parameters set to 0.01 learning rate,
0.0005 decay, and without learning rate scheduler. The dice coefficient [18]
is one of widely used metrics to measure the similarity between images for
semantic segmentation, and represents the value between 0 and 1. The dice
coefficient can be expressed as follows:

$DC=\frac{2|A\cap B|}{|A|+|B|}$ (18)

where $A$ means the ground-truth image, and $B$ means the binary image
predicted/computed by the learning model. And then, we use the dice
coefficient as loss function and combine it with binary cross entropy loss.
The loss of dice coefficient($Loss_{dc}$) and the combined
loss($Loss_{total}$) can be expressed as follows:

$\mathcal{L}_{dc}=1-DC=1-\frac{2|A\cap B|}{|A|+|B|}$ (19)

$\mathcal{L}_{total}=\mathcal{L}_{bce}+\mathcal{L}_{dc}$ (20)

Note that MBS uses the default training algorithm in PyTorch [19] to evaluate
performance and pre-trained models are not used in the experiments.

###  4.3 Experimental Results

####  4.3.1 Training Performance

The training performance of MBS in all models is shown using the size of mini-
batch and micro-batch shown in table 2. The size of mini-batch is determined
to be the maximum size that can be computed on the GPU and the size of micro-
batch for MBS is determined to be half the size of mini-batch for comparison.

Table 2:  Initial size of mini-batch and micro-batch for different models.  Tasks | Model | mini-batch |  $\mu$-batch
---|---|---|---
Classification | ResNet-50 | 16 | 8
ResNet-101 | 8 | 4
AmoebaNet-D | 32 | 16
Segmentation | U-Net | 16 | 8

Figure 3 shows the overall performance of training between models with and
without MBS for image classsification tasks. The loss and accuracy at each
epoch are very similar for all methods with or without MBS. This shows that
the model using MBS is trained similarly to the model not using MBS and this
means that the loss normalization method used in this paper is appropriate.
This also shows that the MBS method does not affect the model performance
during training like other hyper-parameter such as (mini) batch size that is
used as the unit of computation/streaming on GPU. Maximum accuracy for each
model is shown in Table 4.

Figure 3: Performance for each model in image classification models.

(a) ResNet-50

(b) ResNet-101

(c) AmoebaNet-D

To evaluate the results of U-Net, we use a metric called intersection over
union (IoU) [20], which is one of metrics to evaluate performance in semantic
segmentation tasks and shows the degree of the area of overlap between the
predicted image and the ground-truth image. The result in table 3 shows the
performance of training between with and without MBS for U-Net. The initial
experiment shows that the performance is comparable.

Table 3: Performance for a semantic segmentation model. Metrics | w/o MBS | w/ MBS
---|---|---
IoU (%) | 95.48 $\pm$0.13 | 95.45 $\pm$0.26
Table 4: Accuracy and training time of larger batch sizes than the remaining GPU memory for image classification tasks Model | Batch size | $\mu$-batch size | Accuracy (%) | Training time (sec)
---|---|---|---|---
w/o MBS | w/ MBS | w/o MBS | w/ MBS
ResNet-50 | 16 | 8 | 87.16 $\pm$0.33 | 87.04 $\pm$1.04 | 226.8 $\pm$0.01 | 233.0 $\pm$0.01
32 | 16 | Failed | 86.51 $\pm$2.14 | Failed | 229.4 $\pm$0.02
64 | 16 | Failed | 88.96 $\pm$0.41 | Failed | 227.4 $\pm$0.02
128 | 16 | Failed | 89.61 $\pm$0.14 | Failed | 227.5 $\pm$0.02
256 | 16 | Failed | 88.18 $\pm$0.64 | Failed | 229.4 $\pm$0.02
512 | 16 | Failed | 85.95 $\pm$0.11 | Failed | 232.6 $\pm$0.03
1024 | 16 | Failed | 77.63 $\pm$0.86 | Failed | 238.2 $\pm$0.02
ResNet-101 | 8 | 4 | 85.82 $\pm$1.84 | 83.54 $\pm$2.51 | 385.5 $\pm$0.02 | 405.0 $\pm$0.01
16 | 8 | Failed | 86.35 $\pm$2.19 | Failed | 381.9 $\pm$0.01
32 | 8 | Failed | 86.59 $\pm$1.70 | Failed | 381.3 $\pm$0.01
64 | 8 | Failed | 85.94 $\pm$1.02 | Failed | 381.5 $\pm$0.01
128 | 8 | Failed | 88.14 $\pm$0.42 | Failed | 382.1 $\pm$0.01
256 | 8 | Failed | 87.90 $\pm$0.28 | Failed | 383.8 $\pm$0.02
512 | 8 | Failed | 82.72 $\pm$0.67 | Failed | 386.4 $\pm$0.03
1024 | 8 | Failed | 75.14 $\pm$0.67 | Failed | 392.8 $\pm$0.02
AmoebaNet-D | 32 | 16 | 74.51 $\pm$0.67 | 75.92 $\pm$1.59 | 95.2 $\pm$0.01 | 100.1 $\pm$0.22
64 | 32 | Failed | 73.78 $\pm$0.43 | Failed | 95.7 $\pm$0.04
128 | 32 | Failed | 73.23 $\pm$1.59 | Failed | 96.4 $\pm$0.02
256 | 32 | Failed | 75.80 $\pm$0.17 | Failed | 98.9 $\pm$0.17
512 | 32 | Failed | 74.69 $\pm$0.49 | Failed | 103.9 $\pm$0.19
1024 | 32 | Failed | 72.49 $\pm$1.10 | Failed | 111.1 $\pm$0.06
Table 5: IoU and training time of larger batch sizes than the remaining GPU memory for semantic segmentation task Model | Batch size | $\mu$-batch size | IoU (%) | Training time (sec)
---|---|---|---|---
w/o MBS | w/ MBS | w/o MBS | w/ MBS
U-Net | 16 | 8 | 95.48 $\pm$0.13 | 95.45 $\pm$0.26 | 180.7 $\pm$0.18 | 183.2 $\pm$0.20
32 | 16 | Failed | 96.13 $\pm$0.13 | Failed | 180.5 $\pm$0.24
64 | 16 | Failed | 96.83 $\pm$0.12 | Failed | 182.8 $\pm$0.28
128 | 16 | Failed | 97.51 $\pm$0.06 | Failed | 187.0 $\pm$0.24
256 | 16 | Failed | 96.96 $\pm$0.16 | Failed | 194.2 $\pm$0.21
512 | 16 | Failed | 94.18 $\pm$0.44 | Failed | 207.3 $\pm$0.21
1024 | 16 | Failed | 94.18 $\pm$0.44 | Failed | 228.0 $\pm$0.07

####  4.3.2 Maximum Batch Size

If MBS is used, one can compute the entire batch by streaming small batches
regardless of the type of models, datasets, entire batch size and the capacity
of a device memory, if necessary. Experimental results are shown in table 4
and 5. Three different random seeds are used for the experimental results, and
the mean and standard deviation of maximum accuracy/IoU for each random seed
are calculated. The size of mini-batch starts from the maximum size that can
be computed on the GPU (without using MBS). As the mini-batch sizes of each of
the learning models are increased, the model that uses MBS can continue to
train while previous methods that does not use MBS fail because of limited
memory size.

ResNet-50 model uses 7 different mini-batch sizes, ResNet-101 model uses 8
different mini-batch sizes, and AmoebaNet-D model uses 6 different mini-batch
sizes to evaluate MBS performance in this experiment. The U-Net model use 7
different mini-batch sizes to evaluate MBS performance, and the result shows
the maximum IoU value for each mini-batch sizes and micro-batch sizes in table
5. The micro-batch size is experimentally determined; 1) the half of mini-
batch size in the smallest mini-batch, 2) the maximum size that can compute on
GPU in other large mini-batches.

Table 4 and 5 shows the accuracy or IoU for each mini-batch and micro-batch.
Without MBS, all models fail to train models using a large mini-batch because
the capacity of GPU memory is not enough to allocate a large mini-batch. When
using MBS, all models are trained using a large mini-batch up to a given
maximum size on the same GPU. Compared to models without MBS, deep learning
models are trained using 64$\times$ larger mini-batch in ResNet-50 model,
128$\times$ larger mini-batch in ResNet-101 model, and 32$\times$ larger mini-
batch in AmoebaNet-D model. Moreover, the U-Net can train using 64$\times$
larger mini-batch sizes compared to the same model without MBS. Although the
input data size is 3.45$\times$ larger for AmoebaNet-D model and 2.94$\times$
larger for U-Net model than the input data size for ResNet models, AmoebaNet-D
and U-Net models can increase the size of mini-batch up to the given maximum
size when using MBS. Theoretically, MBS allows the increase of the mini-batch
size up to the total size of the dataset, if necessary.

It is observed that MBS can be used to determine or find the optimal (mini)
batch size for each model. Determining the optimal batch size for a deep
learning model will affect that model’s performance. When using MBS, ResNet-50
model achieved 2.44% higher accuracy in 128 mini-batch sizes, and ResNet-101
model achieved 2.32% higher accuracy in 128 mini-batche sizes and it is shown
in table 4. Although AmeobaNet-D model using MBS shows 1.41% higher accuracy
in 32 mini-batch size, comparable performance of 1.29% higher is shown in 256
mini-batch size. The U-Net model shows higher percentage of IoU score between
32 and 256 mini-batch sizes as shown in table 5. The U-Net model achieved
97.51% the highest percentage of IoU score in 128 mini-batch size, and it is
2.03% higher compared to the model without MBS. Previously, research on deep
learning models had difficulty searching for the optimal batch size and train
using a large batch, due to the limitation of hardware resource such as the
GPU memory capacity and had to consider multiple GPU(s) and/or virtual memory
for GPUs. MBS allows large mini-batch size even on a single memory limited
device.

####  4.3.3 Training time

Deep learning models using MBS process multiple back-propagations for multiple
micro-batches until updated for one mini-batch. This is because MBS splits one
mini-batch into several micro-batches and streams micro-batches to GPU
sequentially. To update the model parameters for one mini-batch, MBS process
multiple back-propagations for micro-batches. This process will cause the
overhead for training time. Table 4 and 5 show the results for average
training time for one epoch between with and without MBS. Deep learning models
without MBS show almost less training time compared to models with MBS except
for the ResNet-101 model. The results show that average training time will
increase as the mini-batch size is increased. This is because the overhead of
loading images from secondary memory to CPU memory affects training time more
than MBS process overhead. On the other hand, the overhead of training time is
relatively small when using the mini-batch size that shows the best
performance. The ResNet-50 model shows 1.1% and 0.3% training overhead in 32
and 128 mini-batch size, AmoebaNet-D model shows 5.1% and 4.0% training
overhead in 32 and 256 mini-batch size, and U-Net models shows 3.5% training
overhead in 128 mini-batch size.

##  5 Conclusion

This paper proposes and introduces Micro-Batch Streaming (MBS), an efficient
and novel method based on stream-based pipeline computation to allow deep
learning models to train with large batches on limited memory size devices.
The loss normalization scheme provides a method to maintain the performance
while using a micro batch size that is a subset of a larger mini-batch size
and that fits into the remaining memory space after the upload of the learning
model. All this without increasing the memory and without using multiple GPUs.
MBS allows the training of a learning model to use a large mini-batch size up
to the maximum number of training samples. MBS is applied to classification
models and a semantic segmentation model, showed overall comparable results,
some results show higher performance, and allowed larger mini-batch sizes than
previous methods.

## References

  * [1] Chao Peng, Tete Xiao, Zeming Li, Yuning Jiang, Xiangyu Zhang, Kai Jia, Gang Yu, and Jian Sun.  Megdet: A large mini-batch object detector.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6181–6189, 2018.
  * [2] Qirong Ho, James Cipar, Henggang Cui, Seunghak Lee, Jin Kyu Kim, Phillip B Gibbons, Garth A Gibson, Greg Ganger, and Eric P Xing.  More effective distributed ml via a stale synchronous parallel parameter server.  Advances in neural information processing systems, 26, 2013.
  * [3] Zhouyuan Huo, Bin Gu, Heng Huang, et al.  Decoupled parallel backpropagation with convergence guarantee.  In International Conference on Machine Learning, pages 2098–2106. PMLR, 2018.
  * [4] Zhihao Jia, Matei Zaharia, and Alex Aiken.  Beyond data and model parallelism for deep neural networks.  Proceedings of Machine Learning and Systems, 1:1–13, 2019.
  * [5] Tal Ben-Nun and Torsten Hoefler.  Demystifying parallel and distributed deep learning: An in-depth concurrency analysis.  ACM Computing Surveys (CSUR), 52(4):1–43, 2019.
  * [6] Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc’aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, et al.  Large scale distributed deep networks.  Advances in neural information processing systems, 25, 2012.
  * [7] Seunghak Lee, Jin Kyu Kim, Xun Zheng, Qirong Ho, Garth A Gibson, and Eric P Xing.  On model parallelization and scheduling strategies for distributed machine learning.  Advances in neural information processing systems, 27, 2014.
  * [8] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al.  Gpipe: Efficient training of giant neural networks using pipeline parallelism.  Advances in neural information processing systems, 32, 2019.
  * [9] Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R Devanur, Gregory R Ganger, Phillip B Gibbons, and Matei Zaharia.  Pipedream: generalized pipeline parallelism for dnn training.  In Proceedings of the 27th ACM Symposium on Operating Systems Principles, pages 1–15, 2019.
  * [10] Jack Kiefer and Jacob Wolfowitz.  Stochastic estimation of the maximum of a regression function.  The Annals of Mathematical Statistics, pages 462–466, 1952.
  * [11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [12] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le.  Regularized evolution for image classifier architecture search.  In Proceedings of the aaai conference on artificial intelligence, volume 33, pages 4780–4789, 2019.
  * [13] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [14] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722–729. IEEE, 2008.
  * [15] Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani.  Bottleneck transformers for visual recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16519–16529, 2021.
  * [16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [17] Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang.  Swin-unet: Unet-like pure transformer for medical image segmentation.  arXiv preprint arXiv:2105.05537, 2021.
  * [18] Lee R Dice.  Measures of the amount of ecologic association between species.  Ecology, 26(3):297–302, 1945.
  * [19] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  Advances in neural information processing systems, 32, 2019.
  * [20] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese.  Generalized intersection over union: A metric and a loss for bounding box regression.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 658–666, 2019.
  * [21] Yang You, Zhao Zhang, Cho-Jui Hsieh, James Demmel, and Kurt Keutzer.  Imagenet training in minutes.  In Proceedings of the 47th International Conference on Parallel Processing, pages 1–10, 2018.
  * [22] Yang You, Igor Gitman, and Boris Ginsburg.  Large batch training of convolutional networks.  arXiv preprint arXiv:1708.03888, 2017.
  * [23] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al.  $\\{$TensorFlow$\\}$: A system for $\\{$Large-Scale$\\}$ machine learning.  In 12th USENIX symposium on operating systems design and implementation (OSDI 16), pages 265–283, 2016.
  * [24] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [25] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [26] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [27] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International conference on machine learning, pages 448–456. PMLR, 2015.
  * [28] Youjie Li, Amar Phanishayee, Derek Murray, and Nam Sung Kim.  Doing more with less: Training large dnn models on commodity servers for the masses.  In Proceedings of the Workshop on Hot Topics in Operating Systems, pages 119–127, 2021.
  * [29] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [30] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems, 25, 2012.
  * [31] Alex Krizhevsky.  One weird trick for parallelizing convolutional neural networks.  arXiv preprint arXiv:1404.5997, 2014.
  * [32] Zhihao Jia, Sina Lin, Charles R Qi, and Alex Aiken.  Exploring hidden dimensions in parallelizing convolutional neural networks.  In ICML, pages 2279–2288, 2018.
  * [33] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He.  Accurate, large minibatch sgd: Training imagenet in 1 hour.  arXiv preprint arXiv:1706.02677, 2017.
  * [34] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang.  On large-batch training for deep learning: Generalization gap and sharp minima.  arXiv preprint arXiv:1609.04836, 2016.
  * [35] Mu Li, Tong Zhang, Yuqiang Chen, and Alexander J Smola.  Efficient mini-batch training for stochastic optimization.  In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 661–670, 2014.
  * [36] Jianmin Chen, Xinghao Pan, Rajat Monga, Samy Bengio, and Rafal Jozefowicz.  Revisiting distributed synchronous sgd.  arXiv preprint arXiv:1604.00981, 2016.
  * [37] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [38] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He.  Accurate, large minibatch sgd: Training imagenet in 1 hour.  arXiv preprint arXiv:1706.02677, 2017.
  * [39] Ibrahem Kandel and Mauro Castelli.  The effect of batch size on the generalizability of the convolutional neural networks on a histopathology dataset.  ICT express, 6(4):312–315, 2020.
  * [40] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [41] Ross Girshick.  Fast r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 1440–1448, 2015.
  * [42] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [43] M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman.  The pascal visual object classes challenge: A retrospective.  International Journal of Computer Vision, 111(1):98–136, January 2015.
  * [44] Torsten Rohlfing.  Image similarity and tissue overlaps as surrogates for image registration accuracy: widely used but unreliable.  IEEE transactions on medical imaging, 31(2):153–163, 2011.


11institutetext: Nagoya Institute of Technology, Nagoya, Japan
11email<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Temporal Cross-attention for Action Recognition

Ryota Hashiguchi  Toru Tamaki  0000-0001-9712-7777

###### Abstract

Feature shifts have been shown to be useful for action recognition with CNN-
based models since Temporal Shift Module (TSM) was proposed. It is based on
frame-wise feature extraction with late fusion, and layer features are shifted
along the time direction for the temporal interaction. TokenShift, a recent
model based on Vision Transformer (ViT), also uses the temporal feature shift
mechanism, which, however, does not fully exploit the structure of Multi-head
Self-Attention (MSA) in ViT. In this paper, we propose _Multi-head Self/Cross-
Attention_ (MSCA), which fully utilizes the attention structure. TokenShift is
based on a frame-wise ViT with features temporally shifted with successive
frames (at time $t+1$ and $t-1$). In contrast, the proposed MSCA replaces MSA
in the frame-wise ViT, and some MSA heads attend to successive frames instead
of the current frame. The computation cost is the same as the frame-wise ViT
and TokenShift as it simply changes the target to which the attention is
taken. There is a choice about which of key, query, and value are taken from
the successive frames, then we experimentally compared these variants with
Kinetics400. We also investigate other variants in which the proposed MSCA is
used along the patch dimension of ViT, instead of the head dimension.
Experimental results show that a variant, MSCA-KV, shows the best performance
and is better than TokenShift by 0.1% and then ViT by 1.2%.

##  1 Introduction

Recognizing the actions of people in videos is an important topic in computer
vision. After the emergence of Vision Transformer (ViT) [1] which has been
shown to be effective for various image recognition tasks [2, 3], research
extending ViT to video has become active [4, 5, 6, 7, 8].

In recognition of video, unlike images, it is necessary to model the temporal
information across frames of a given video clip. While most early works of
action recognition were frame-wise CNN models followed by temporal aggregation
[9, 10], 3D CNN models [11, 12, 13] were shown to be effective and well
generalized when they were trained on large-scale datasets [14] and
transferred to smaller datasets [15].

However, the computational cost of 3D convolution is usually high. To mitigate
the issue of the trade-off between the ability of temporal modeling and the
high computational cost, Temporal Shift Module (TSM) [16] was proposed by
extending spatial feature shifting [17, 18] to the temporal dimension. TSM
uses late fusion by applying 2D ResNet [19] to each frame, while it exploits
the temporal interaction between adjacent frames by shifting the features in
each layer of ResNet. The feature shift operation is computationally
inexpensive, nevertheless it has been shown to perform as well as 3D CNN.
Therefore, many related CNN models [20, 21, 22] have been proposed. TokenShift
[23] is a TSM-like 2D ViT-based model that shifts a fraction of ViT features,
showing that shifting only the class tokens is enough to achieve a good
performance.

However, TokenShift doesn’t fully exploit the advantage of the architecture of
the attention mechanism of ViT, instead it simply shifts features like as in
TSM. We propose a method that seamlessly utilizes the ViT structure to
interact with features across adjacent frames. The original TSM or TokenShift
uses a 2D model applied to each frame, while the output features of layers of
the 2D model are then shifted from the current time $t$ to one time step
forward $t+1$ and backward $t-1$. In the encoder architecture of ViT, Multi-
head Self-Attention (MSA) computes attentions between patches at a given frame
$t$. The proposed method utilizes and modifies it for the temporal
interaction; some patches at frame $t$ attends to not only patches at the
current frame $t$ but also patches at the neighbor frames $t+1$ and $t-1$. We
call this mechanisms _Multi-head Self/Cross-Attention_ (MSCA). This allows the
temporal interaction to be computed within the Transformer block without
additional shift modules. Furthermore, the computational cost remains the same
because some parts of self-attention computation is simply replaced with the
temporal cross-attention. The contributions of this paper are as follows:

  * •

We propose a new action recognition model with the proposed MSCA, which
seamlessly combines the concepts of ViT and TSM. This avoids computing spatio-
temporal attention directly in the 3D video volume.

  * •

MSCA uses cross-attention to perform temporal interactions inside the
Transformer block, unlike TokenShift that uses additional shifting modules.
This allows for temporal interaction without increasing computational
complexity nor changing model architecture.

  * •

Experiments on Kinetics400 show that the proposed method improves performance
compared to ViT and TokenShift.

##  2 Related Work

###  2.1 2D/3D CNN, and TSM

Early approaches of deep learning to action recognition are 2D-based methods
[9, 10] that apply 2D CNN to videos frame by frame, or Two-Stream methods [24]
that use RGB frames and optical flow stacks. However, it is difficult to model
long-term temporal information beyond several time steps, and this is where
3D-based methods [11, 12, 13] came in. These models simultaneously consider
spatial and temporal information, but their computational cost is relatively
high compared to 2D-based methods. Therefore, mixed models of 2D and 3D
convolution [25, 26, 27] have been proposed by separating convolution in the
spatial and temporal dimensions.

TSM [16] is an attempt to model temporal information for action recognition
without increasing parameters and computational complexity while maintaining
the architecture of 2D CNN-based methods. TSM shifts intermediate features
between neighbor frames, allowing 2D CNNs of each frame to interact with other
frames in the temporal direction. Many CNN-based variants have followed, such
as Gated-Shift Network [20] that learns temporal features using only 2D CNN
with the gate shift module, and RubiksNet [22] which uses three learnable
shifts (spatial, vertical, and temporal) for end-to-end learning.

###  2.2 ViT-based video models

ViT [1] performs very well in image recognition tasks, and its application to
action recognition tasks has been actively studied [4, 5, 6, 7, 8, 28, 29]. It
is known that the computational cost of self-attention in Transformer is
$O(N^{2})$ where $N$ is the number of tokens, or the number of patches $N$ for
ViT. For videos, the number of frames $T$ also contributes to the
computational cost of spatio-temporal self-attention. A simple extension of
ViT with full spatio-temporal attention gives a computational cost of
$O(N^{2}T^{2})$, since the number of tokens increases in proportion to the
temporal extent. To alleviate this computational issue, TimeSformer [7]
reduces the computational complexity to $O(T^{2}N+TN^{2})$ by applying
temporal and spatial attention separately, TokenLeaner [28] to $O(S^{2}T^{2})$
by selecting $S(<N)$ patches that are important for the representation, and
Space-time Mixing Attention [29] to $O(TN^{2})$ by attending tokens in
neighbor frames only.

TokenShift [23] is a TSM-like model that shifts a fraction of ViT features in
each Transformer block. It computes the attention only in the current frame,
hence the complexity is $O(TN^{2})$ with fewer FLOPs than [29]. TokenShift
shifts the class token only based on experimental results, however it doesn’t
fully exploit the attention mechanism. In contrast, the proposed method makes
use of the attention mechanism for feature shift; we replace some of the self-
attention modules with the proposed cross-attention modules that shift key,
query, and value to neighbor frames, instead of naively shifting features.

##  3 Method

(a)

(b)

(c)

Figure 1:  Encoder blocks of (a) ViT, (b) TokenShift with two shift modules
(in blue), and (c) the proposed method with MSCA (in green).

Figure 1 shows the original ViT encoder block that has the Multi-head Self-
Attention (MSA) module. Fig.1 shows the encoder block of TokenShift. Two
modules for shifting intermediate features are added to the original ViT
block. These modules shift a portion of the class tokens in the temporal
direction by one time step forward and backward, similar to TSM. Fig.1 shows
the encoder block of the proposed MSCA. The difference is that MSA is replaced
with the Multi-head Self/Cross-Attention (MSCA) module, and no additional
modules exist.

###  3.1 TokenShift and ViT

In this section, we briefly review TokenShift [23] which is based on ViT [1].

####  3.1.1 Input Patch Embedding

Let an input video be $x\in\mathbb{R}^{T\times 3\times H\times W}$, where $T$
is the number of frames in the video clip, and $H,W$ are the height and width
of the frame. Each frame is divided into patches of size $P\times P$ pixels
and transformed into a tensor
$\hat{x}=[x_{0}^{1},\ldots,x_{0}^{N}]\in\mathbb{R}^{T\times N\times d}$, where
$x_{0}^{i}\in\mathbb{R}^{T\times d}$ denotes the $i$-th patch,
$N=\frac{HW}{P^{2}}$ is the number of patches of dimension $d=3P^{2}$.

The input patch $x_{0}^{i}$ is then transformed by the embedding matrix
$E\in\mathbb{R}^{d\times D}$ and the positional encoding $E_{\mathrm{pos}}$ as
follows:

$\displaystyle z_{0}$
$\displaystyle=[c_{0},x_{0}^{1}E,x_{0}^{2}E,\ldots,x_{0}^{N}E]+E_{\mathrm{pos}},$
(1)

where $c_{0}\in\mathbb{R}^{T\times D}$ is the class token. This patch
embedding $z_{0}\in\mathbb{R}^{T\times(N+1)\times D}$ is the input to the
first encoder block.

####  3.1.2 Encoder Block

Let $z_{\ell}$ be the input to the $\ell$-th encoder block. The output
$z_{\ell}$ of the block can be expressed as follows:

$\displaystyle z^{\prime}_{\ell}$
$\displaystyle=\mathrm{MSA}(\mathrm{LN}(z_{\ell-1}))+z_{\ell-1}$ (2)
$\displaystyle z_{\ell}$
$\displaystyle=\mathrm{MLP}(\mathrm{LN}(z^{\prime}_{\ell}))+z^{\prime}_{\ell},$
(3)

where LN is the layer normalization, MSA is the multi-head self-attention, and
MLP is the multi-layer perceptron. In the following, $z_{\ell,t,n,d}$ denotes
the element at $(t,n,d)$ in $z_{\ell}$.

####  3.1.3 Shift Modules

TokenShift inserts two shift modules in the block as follows;

$\displaystyle z^{\prime}_{\ell-1}$ $\displaystyle=\mathrm{Shift}(z_{\ell-1})$
(4) $\displaystyle z^{\prime\prime}_{\ell}$
$\displaystyle=\mathrm{MSA}(\mathrm{LN}(z^{\prime}_{\ell-1}))+z^{\prime}_{\ell-1}$
(5) $\displaystyle z^{\prime\prime\prime}_{\ell}$
$\displaystyle=\mathrm{Shift}(z^{\prime\prime}_{\ell})$ (6) $\displaystyle
z_{\ell}$
$\displaystyle=\mathrm{MLP}(\mathrm{LN}(z^{\prime\prime\prime}_{\ell}))+z^{\prime\prime\prime}_{\ell}.$
(7)

The shift modules take the input $z_{\mathrm{in}}\in R^{T\times(N+1)\times D}$
and compute the output $z_{\mathrm{out}}$ of the same size by shifting the
part of $z_{\mathrm{in}}$ corresponding to the class tokens
($z_{\mathrm{in},t,0,d}$, the first elements of the second dimension of
$z_{\mathrm{in}}$) while leaving the other parts untouched. This is
implemented by the following assignments;

$\displaystyle z_{\mathrm{out},t,0,d}$
$\displaystyle=\begin{cases}z_{\mathrm{in},t-1,0,d},&\text{$1<t\leq T,1\leq
d<D_{b}$}\\\ z_{\mathrm{in},t+1,0,d},&\text{$1\leq t<T,D_{b}\leq
d<D_{b}+D_{f}$}\\\ z_{\mathrm{in},t,0,d},&\text{$\forall t,D_{b}+D_{f}\leq
d\leq D$}\\\ \end{cases}$ (8) $\displaystyle z_{\mathrm{out},t,n,d}$
$\displaystyle=z_{\mathrm{in},t,n,d},\hskip 27.0pt\text{$\forall t,1\leq
n<N,\forall d$}$ (9)

The first equation shifts the class tokens; along the channel dimension $D$,
the backward shift to $t-1$ is done for the first $D_{\mathrm{b}}$ channels,
the forward shift to $t+1$ is done for the next $D_{\mathrm{f}}$, and no shift
for the rest channels. The second equation passes through features other than
the class tokens.

###  3.2 MSCA

The proposed method replaces MSA with MSCA in the original block;

$\displaystyle z^{\prime}_{\ell}$
$\displaystyle=\mathrm{MSCA}(\mathrm{LN}(z_{\ell-1}))+z_{\ell-1}$ (10)
$\displaystyle z_{\ell}$
$\displaystyle=\mathrm{MLP}(\mathrm{LN}(z^{\prime}_{\ell}))+z^{\prime}_{\ell}.$
(11)

In the following, we first describe MSA, and then define MSCA.

####  3.2.1 MSA

The original MSA computes the key $K^{(t)}$, query $Q^{(t)}$, and value
$V^{(t)}$ for the portion $z^{(t)}\in\mathbb{R}^{(N+1)\times D}$ of the input
feature $z\in\mathbb{R}^{T\times(N+1)\times D}$ at time $t$, as follows;

$\displaystyle K^{(t)},Q^{(t)},V^{(t)}$
$\displaystyle=z^{(t)}[W_{k},W_{q},W_{v}],$ (12)

where $W_{k},W_{q},W_{v}\in\mathbb{R}^{D\times D}$ are embedding matrices, and
$z=[z^{(1)},\ldots,z^{(T)}]$. These are used to compute the $i$-th attention
head;

$\displaystyle\operatorname{head}_{i}^{(t)}$
$\displaystyle=a(Q^{(t)}_{i},K^{(t)}_{i})V^{(t)}_{i}\in\mathbb{R}^{(N+1)\times
D/h},$ (13)

at time $t$ for $i=1,\ldots,h$, where the attention $a$ is

$\displaystyle a(Q,K)$
$\displaystyle=\operatorname{softmax}(QK^{T}/\sqrt{D}),$ (14)

and $Q_{i}^{(t)}\in\mathbb{R}^{(N+1)\times D/h}$ is the part of $Q^{(t)}$
corresponding to $i$-th head

$\displaystyle Q^{(t)}=[Q_{1}^{(t)},\ldots,Q_{i}^{(t)},\ldots,Q_{h}^{(t)}],$
(15)

and $K_{i}^{(t)},V_{i}^{(t)}$ are the same. These heads are finally stacked to
form

$\displaystyle\mathrm{MSA}(z^{(t)})$
$\displaystyle=[\operatorname{head}_{1}^{(t)},\ldots,\operatorname{head}_{h}^{(t)}].$
(16)

In MSA, patches in $t$-th frame are attended from other patches of the same
frame at time $t$, which means that there are no temporal interactions between
frames.

Figure 2:  Shit operation at $T=t$ in the MSCA-KV model

(a)

(b)

(c)

Figure 3: Shit operations of (a) MSCA-V, (b) MSCA-pKV, and (c) MSCA-pV.

####  3.2.2 MSCA-KV

The proposed MSCA computes the attention across frames, that is, patches in
$t$-th frame are also attended from patches in frames at time $t+1$ and $t-1$.
This can be done with shift operations; after generating $Q,K,V$ at each
frame, these are exchanged with the neighbor frames.

There are choices of which of $Q,K$, or $V$ to be shifted. A possible choice
is to shift $K$ and $V$, and the query in the current frame is attended by
key-value pairs in other frames. This is expressed as follows;

$\displaystyle\operatorname{head}_{i}^{(t)}$
$\displaystyle=\begin{cases}a(Q^{(t)}_{i},K^{(t-1)}_{i})V^{(t-1)}_{i}&1\leq
i<h_{b}\\\ a(Q^{(t)}_{i},K^{(t+1)}_{i})V^{(t+1)}_{i}&h_{b}\leq
i<h_{b}+h_{f}\\\ a(Q^{(t)}_{i},K^{(t)}_{i})V^{(t)}_{i}&h_{b}+h_{f}\leq i\leq
h.\end{cases}$ (17)

Here, we have the first $h_{b}$ heads with the backward shift, the next
$h_{f}$ heads with the forward shift, and the rest heads with no shift. We
call this MSCA-KV, and Figure 2 shows the diagram of the shift operation.
First, queries, keys and values are computed at each frame, and then some of
them are shifted before computing the attention. The solid arrows indicate the
key-value shift from time $t-1$ and $t+1$. As in the same way, the key-value
shift from the current frame $t$ to $t-1$ and $t+1$ are shown in dotted
arrows. There are shifts in all other frames at the same time in the same
manner.

####  3.2.3 MSCA-V

Another choice of shift is shown in Fig. 3. Here, $Q$ and $K$ are not shifted,
but only $V$ is shifted as follows;

$\displaystyle\operatorname{head}_{i}^{(t)}$
$\displaystyle=\begin{cases}a(Q^{(t)}_{i},K^{(t)}_{i})V^{(t-1)}_{i}&1\leq
i<h_{b}\\\ a(Q^{(t)}_{i},K^{(t)}_{i})V^{(t+1)}_{i}&h_{b}\leq i<h_{b}+h_{f}\\\
a(Q^{(t)}_{i},K^{(t)}_{i})V^{(t)}_{i}&h_{b}+h_{f}\leq i\leq h.\end{cases}$
(18)

This might not be common because the key and value are now separated and taken
from different frames. However, this makes sense for modeling temporal
interactions because the values (which are mixed by the attention weights)
come from different frames while the attention is computed in the current
frame. We call this version MSCA-V. Therefore, in addition to shifting $V$ in
this way, there are seven possible combinations; Q, K, V, QK, KV, QV, and QKV.
We compared these variants in the experiments. Note that MSCA-QKV is
equivalent to a simple feature shifting because attended features are computed
in each frame and then shifted.

####  3.2.4 MSCA-pKV

All of the above seven variants perform shift operations along the head (or
channel) dimension $D$, but similar variants of shift are also possible for
the patch dimension $N+1$.

The shapes of $K^{(t)},Q^{(t)},V^{(t)}$ are $\mathbb{R}^{(N+1)\times D}$, and
the first dimension is for patches while the second is for heads. As in the
same way the the shift along the head dimension, we have a variation of shift
operations along the patch dimension as shown in Fig. 3.

First, $K$ and $V$ are expressed as stacks of keys and values of patches at
different frames as follows;

$\displaystyle K^{(t)}$
$\displaystyle=[K^{(t)}_{0},K^{(t)}_{1},\ldots,K^{(t)}_{N}]$ (19)
$\displaystyle V^{(t)}$
$\displaystyle=[V^{(t)}_{0},V^{(t)}_{1},\ldots,V^{(t)}_{N}],$ (20)

where $K^{(t)}_{n},V^{(t)}_{n}\in\mathbb{R}^{D}$ are the key and value of
patch $n$ at time $t$.

Then, keys of some patches in the current frame are shifted to form
$K^{\prime}$;

$\displaystyle K^{\prime(t)}_{n}$
$\displaystyle=\begin{cases}K^{(t-1)}_{n}&0\leq n<N_{b}\\\
K^{(t+1)}_{n}&N_{b}\leq n<N_{b}+N_{f}\\\ K^{(t)}_{n}&N_{b}+N_{f}\leq n\leq
N,\end{cases}$ (21)

and also $V^{\prime}$ in the same way. Finally, the $i$-th head is computed as
follows

$\displaystyle\operatorname{head}_{i}^{(t)}$
$\displaystyle=a(Q^{(t)}_{i},K^{\prime(t)}_{i})V^{\prime(t)}_{i}.$ (22)

We refer to this version as MSCA-pKV.

####  3.2.5 MSCA-pV

Like as MSCA-V, a variant of the shift in the patch direction with $V$ only
can be also considered as shown in Fig. 3, by the following shift;

$\displaystyle\operatorname{head}_{i}^{(t)}$
$\displaystyle=a(Q^{(t)}_{i},K^{(t)}_{i})V^{\prime(t)}_{i}.$ (23)

As before, there are seven variants, and we call these MSCA-pV, and so on.

##  4 Experimental results

###  4.1 Setup

Kinetics400 [14] was used to train and evaluate the proposed method. This
dataset consists of a training set of 22k videos, a validation set of 18k
videos, with 400 categories of human actions. Each video was collected from
Youtube, and the portion corresponding to each category was cropped to a
length of 10 seconds.

We used a 2D ViT pre-trained on ImageNet21k [30] with $h=12$ heads, 12 encoder
blocks, and patches of size $P=16$ and $D=768=3\times 16\times 16$ (these
parameters are the same for TokenShift and MSCA models). For action
recognition, ViT was applied to each frame and resulting frame-wise features
were aggregated by temporal averaging (this is referred to as ViT in the
experiment and in [23]). We compared this ViT, TokenShift and the proposed
method. Note that we report the performance of TokenShift based on our
reproduction using the author’s
code.111https://github.com/VideoNetworks/TokShift-Transformer

For training, we used the same settings as in [23]. Input clips were of 8
frames with stride of 32 frames (starting frames were randomly chosen). Frames
were flipped horizontally at a probability of 50%, and the short side was
randomly resized in the range of [244, 330] pixels while maintaining the
aspect ratio, then a random $224\times 224$ pixel rectangle was cropped
(therefore the number of patches is $N=196=14\times 14$). In addition,
brightness change, saturation change, gamma correction, and hue correction
were applied to frames, each at the probability of 10%. The number of epochs
was set to 12, the optimizer to SDG with momentum of 0.9 and no weight decay.
The initial learning rate was set to 0.1, and decayed by a factor of 10 at
10th epoch. The batch size was set to 42, and 21 batches were trained on each
of two GPUs. Gradient updates were performed once every 10 iterations, so the
effective batch size was 420.

We used the multi-view test [31]. From a validation video, one clip was
sampled as in training, and this was repeated 10 times to sample 10 clips.
Each clip was resized to 224 pixels on its short side while maintaining the
aspect ratio, and cropped to $224\times 224$ at the right, center, and left.
The results of these 30 clips (views) were averaged to compute a single
prediction score.

###  4.2 The amount of shift of MSCA-KV

Table 1:  The performance of MSCA-KV for the validation sets of Kinetics400. Note that the zero shift means a naive frame-wise ViT. The column “shift” means the percentage of the shifted dimensions to the total dimensions $D$.  model | heads | $h_{b},h_{f}$ | shift % | top-1 | top-5
---|---|---|---|---|---
ViT | 0 | 0 | 0 (ViT) | 75.65 | 92.19
MSCA-KV | 2 | 1 | 16.7 | 76.47 | 92.88
| 4 | 2 | 33.3 | 76.07 | 92.61
| 6 | 3 | 50.0 | 75.66 | 92.30
| 8 | 4 | 66.7 | 74.72 | 91.91

We first investigate the effect of the number of heads to be shifted. Table 1
shows the performance of MSCA-KV. The best performance was obtained when only
two heads (each for forward and backward) which corresponds to shifting
$2/12=16.7\%$ of the channels. As the number of shifted heads increased, the
performance decreased, suggesting that shifting a few heads is sufficient
while most heads need not to be shifted. This observation coincides with the
conclusions of TokenShift [23], which shows that the shift of the class token
only is enough, and also TSM [16], which used the shift of $1/4=25\%$ of the
channels (each 1/8 for forward and backward).

In the following experiments, we used shifting two heads for all MSCA
variations.

###  4.3 Comparison of MSCA variations shifting in the head direction

Table 2:  The performance of MSCA variants shifting in the head direction.  model | top-1 | top-5
---|---|---
TokenShift | 76.37 | 92.82
ViT | 75.65 | 92.19
MSCA-Q | 75.84 | 92.57
MSCA-K | 75.76 | 92.13
MSCA-V | 75.58 | 92.39
MSCA-QK | 75.47 | 92.32
MSCA-KV | 76.47 | 92.88
MSCA-QV | 75.57 | 92.43
MSCA-QKV | 75.78 | 92.37

Table 3:  The performance with different numbers of blocks with MSCA modules of KV shift.  model | # MSCA | # MSA | top-1 | top-5
---|---|---|---|---
ViT | 0 | 12 | 75.65 | 92.19
| 4 | 8 | 75.67 | 92.19
| 8 | 4 | 76.40 | 92.77
MSCA-KV | 12 | 0 | 76.47 | 92.88

Table 3 shows the performance of the MSCA model variants with shift operations
in the head direction. Among them, MSCA-KV performed the best, outperforming
others by at least 0.5%. Other variants performed as same as the baseline ViT,
indicating that the shift operation is not working effectively in such
variations.

###  4.4 The number of encoder blocks with MSCA

The proposed method replaces MSA modules in 12 encoder blocks in ViT with MSCA
modules. However, it is not obvious that modules of all blocks need to be
replaced. Table 3 shows the results when we replaced MSA modules with MSCA in
some blocks near the end (or top) of the network. All 12 replacements
correspond to MSCA-KV, and 0 (no MCSA) is ViT. Using four MSCA modules showed
no improvement, while using 8 MSCA modules performed almost the same as MSCA-
KV with all 12 MSCA modules. This is because the input video clip consists of
8 frames, and shifting more than 8 times with MSCA modules ensures that the
temporal information from all frames is available for the entire network.
Therefore, it would be necessary to use at least as many MSCA modules as the
number of frames in the input clip.

###  4.5 Comparison of MSCA variations shifting in the patch direction

Table 4:  The performance of MSCA-pKV. The column “shift” means the percentage of the shifted patches to the total patches $N+1$.  patches | $N_{b},N_{b}$ | shift % | top-1 | top-5
---|---|---|---|---
0 | 0 | 0 (ViT) | 75.65 | 92.19
8 | 4 | 4.1 | 76.28 | 92.49
16 | 8 | 8.1 | 76.35 | 92.91
32 | 16 | 16.2 | 76.07 | 92.77
48 | 24 | 24.4 | 75.84 | 92.49
64 | 32 | 32.5 | 75.29 | 92.04

Table 5:  The performance of MSCA models shifting in the patch direction.  model | top-1 | top-5
---|---|---
TokenShift | 76.37 | 92.82
ViT | 75.65 | 92.19
MSCA-pQ | 75.75 | 92.10
MSCA-pK | 75.69 | 92.24
MSCA-pV | 75.84 | 92.43
MSCA-pQK | 75.50 | 91.99
MSCA-pKV | 76.35 | 92.91
MSCA-pQV | 75.83 | 92.35
MSCA-pQKV | 75.73 | 92.21

Table 5 shows the performance of MSCA-pKV with different amount of shift. The
results indicate that, again, a small amount of shift is enough for a better
performance and the best performance was obtained when 16 patches were
shifted. The performance decreases for a smaller amount of shift, approaching
the performance of ViT with no shift. In the experiments below, we used the
shift of 16 patches.

Table 5 shows the performance of MSCA variants with shifting in the patch
direction. Just like as MSCA-KV was the best for shifting in the head
direction, MSCA-pKV has the best performance here, while it is just comparable
to TokenShift. An obvious drawback of this approach is the first layer; the
patches to be shifted were fixed to the first $N_{b}$ and $N_{f}$ patches in
the order of the patch index, which is irrelevant to the content of the frame.
This might be mitigated by not using MSCA modules in the first several layers.

##  5 Conclusions

In this paper, we proposed MSCA, a ViT-based action recognition model that
replaces MSA modules in the encoder blocks. The MSCA modules compute the
attention by shifting of the key, query, and value for temporal interaction
between frames. Experimental results using Kinetics400 showed that the
proposed method is effective for modeling spatio-temporal features and
performs better than a naive ViT and TokenShift. Future work includes
evaluations on other datasets and comparisons with similar methods such as
Space-time Mixing Attention [29].

#### Acknowledgements.

This work was supported in part by JSPS KAKENHI Grant Number JP22K12090.

## References

  * [1] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.:  An image is worth 16x16 words: Transformers for image recognition at scale.  In: International Conference on Learning Representations. (2021)
  * [2] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.:  Learning transferable visual models from natural language supervision.  CoRR abs/2103.00020 (2021)
  * [3] Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, I.:  Zero-shot text-to-image generation.  In Meila, M., Zhang, T., eds.: Proceedings of the 38th International Conference on Machine Learning. Volume 139 of Proceedings of Machine Learning Research., PMLR (2021) 8821–8831
  * [4] Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., Schmid, C.:  Vivit: A video vision transformer.  In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). (2021) 6836–6846
  * [5] Li, X., Zhang, Y., Liu, C., Shuai, B., Zhu, Y., Brattoli, B., Chen, H., Marsic, I., Tighe, J.:  Vidtr: Video transformer without convolutions.  CoRR abs/2104.11746 (2021)
  * [6] Girdhar, R., Carreira, J., Doersch, C., Zisserman, A.:  Video action transformer network.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). (2019)
  * [7] Bertasius, G., Wang, H., Torresani, L.:  Is space-time attention all you need for video understanding?  In: Proceedings of the International Conference on Machine Learning (ICML). (2021)
  * [8] Sharir, G., Noy, A., Zelnik-Manor, L.:  An image is worth 16x16 words, what is a video worth?  CoRR abs/2103.13915 (2021)
  * [9] Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., Fei-Fei, L.:  Large-scale video classification with convolutional neural networks.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2014)
  * [10] Donahue, J., Anne Hendricks, L., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., Darrell, T.:  Long-term recurrent convolutional networks for visual recognition and description.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2015)
  * [11] Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.:  Learning spatiotemporal features with 3d convolutional networks.  In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). (2015)
  * [12] Carreira, J., Zisserman, A.:  Quo vadis, action recognition? a new model and the kinetics dataset.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2017)
  * [13] Hara, K., Kataoka, H., Satoh, Y.:  Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2018) 6546–6555
  * [14] Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., Suleyman, M., Zisserman, A.:  The kinetics human action video dataset.  CoRR abs/1705.06950 (2017)
  * [15] Soomro, K., Zamir, A.R., Shah, M.:  Ucf101: A dataset of 101 human actions classes from videos in the wild.  CoRR abs/1212.0402 (2012)
  * [16] Lin, J., Gan, C., Han, S.:  Tsm: Temporal shift module for efficient video understanding.  In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). (2019)
  * [17] Chen, W., Xie, D., Zhang, Y., Pu, S.:  All you need is a few shifts: Designing efficient convolutional neural networks for image classification.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). (2019)
  * [18] Wu, B., Wan, A., Yue, X., Jin, P., Zhao, S., Golmant, N., Gholaminejad, A., Gonzalez, J., Keutzer, K.:  Shift: A zero flop, zero parameter alternative to spatial convolutions.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2018)
  * [19] He, K., Zhang, X., Ren, S., Sun, J.:  Deep residual learning for image recognition.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2016)
  * [20] Sudhakaran, S., Escalera, S., Lanz, O.:  Gate-Shift Networks for Video Action Recognition.  In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). (2020)
  * [21] Chang, Y.L., Liu, Z.Y., Lee, K.Y., Hsu, W.:  Learnable gated temporal shift module for deep video inpainting”.  BMVC (2019)
  * [22] Fan, L., Buch, S., Wang, G., Cao, R., Zhu, Y., Niebles, J.C., Fei-Fei, L.:  Rubiksnet: Learnable 3d-shift for efficient video action recognition.  In Vedaldi, A., Bischof, H., Brox, T., Frahm, J.M., eds.: Computer Vision – ECCV 2020, Cham, Springer International Publishing (2020)
  * [23] Zhang, H., Hao, Y., Ngo, C.W.  In: Token Shift Transformer for Video Classification. Association for Computing Machinery, New York, NY, USA (2021) 917–925
  * [24] Simonyan, K., Zisserman, A.:  Two-stream convolutional networks for action recognition in videos.  In Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., Weinberger, K.Q., eds.: Advances in Neural Information Processing Systems. Volume 27., Curran Associates, Inc. (2014)
  * [25] Qiu, Z., Yao, T., Mei, T.:  Learning spatio-temporal representation with pseudo-3d residual networks.  In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). (2017)
  * [26] Zhang, D., Dai, X., Wang, X., Wang, Y.F.:  S3d: Single shot multi-span detector via fully 3d convolutional network.  In: Proceedings of the British Machine Vision Conference (BMVC). (2018)
  * [27] Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., Paluri, M.:  A closer look at spatiotemporal convolutions for action recognition.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2018)
  * [28] Ryoo, M.S., Piergiovanni, A., Arnab, A., Dehghani, M., Angelova, A.:  Tokenlearner: Adaptive space-time tokenization for videos.  In: Advances in Neural Information Processing Systems (NeurIPS). (2021)
  * [29] Bulat, A., Perez-Rua, J.M., Sudhakaran, S., Martinez, B., Tzimiropoulos, G.:  Space-time mixing attention for video transformer.  In Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W., eds.: Advances in Neural Information Processing Systems. (2021)
  * [30] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.:  Imagenet: A large-scale hierarchical image database.  In: 2009 IEEE Conference on Computer Vision and Pattern Recognition. (2009) 248–255
  * [31] Wang, X., Girshick, R., Gupta, A., He, K.:  Non-local neural networks.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). (2018)