∎

11institutetext:  Huabin Liu 22institutetext: Weiyao Lin 33institutetext:
Tieyuan Chen 44institutetext: Yuxi Li 55institutetext: Shuyuan Li
66institutetext: Department of Electronic Engineering, Shanghai Jiao Tong
University, Shanghai, China
66email<EMAIL_ADDRESS>77institutetext:
Tieyuan Chen 88institutetext: Shanghai Jiao Tong University, Shanghai, China
88email<EMAIL_ADDRESS>99institutetext: John See 1010institutetext:
Heriot-Watt University, Malaysia
1010email<EMAIL_ADDRESS>
# Few-shot Action Recognition via Intra- and Inter-Video Information
Maximization

Huabin Liu  Weiyao Lin  Tieyuan Chen  Yuxi Li
Shuyuan Li  John See

(Received: date / Accepted: date)

###### Abstract

Current few-shot action recognition involves two primary sources of
information for classification:(1) _intra_ -video information, determined by
frame content within a single video clip, and (2) _inter_ -video information,
measured by relationships (e.g., feature similarity) among videos. However,
existing methods inadequately exploit these two information sources. In terms
of intra-video information, current sampling operations for input videos may
omit critical action information, reducing the utilization efficiency of video
data. For the inter-video information, the action misalignment among videos
makes it challenging to calculate precise relationships. Moreover, how to
jointly consider both inter- and intra-video information remains under-
explored for few-shot action recognition. To this end, we propose a novel
framework, Video Information Maximization (VIM), for few-shot video action
recognition. VIM is equipped with an adaptive spatial-temporal video sampler
and a spatial-temporal action alignment model to maximize intra- and inter-
video information, respectively. The video sampler adaptively selects
important frames and amplifies critical spatial regions for each input video
based on the task at hand. This preserves and emphasizes informative parts of
video clips while eliminating interference at the data level. The alignment
model performs temporal and spatial action alignment sequentially at the
feature level, leading to more precise measurements of inter-video similarity.
Finally, These goals are facilitated by incorporating additional loss terms
based on mutual information measurement. Consequently, VIM acts to maximize
the distinctiveness of video information from limited video data. Extensive
experimental results on public datasets for few-shot action recognition
demonstrate the effectiveness and benefits of our framework. 111This
manuscript is an extended version of two conference papers (_TA 2N_ ta2n2022
and _Sampler_ sampler2022 ). We have cited them in the manuscript and extended
them substantially but not limited to the following aspects: (1) We provide a
novel insight from the perspective of video information maximization to
address the few-shot action recognition problem. (2) Based on mutual
information measurement, we newly introduce auxiliary loss terms to facilitate
the maximization of intra- and inter-video information, respectively. (3) More
ablation studies on both the video sampler and action alignment model. (4)
More analysis about the interaction between video sampler and action alignment
(5) More experiments are appended to demonstrate the generality and
effectiveness of VIM, including upper bound analysis, limit test, many-way
few-shot experiment, results of domain generalization, and more
visualizations.

††journal: International Journal of Computer Vision

Figure 1: Overall pipeline of VIM. The few-shot action recognition involves
two primary sources of information: (1) Intra-video information and (2) Inter-
video information. Our proposed VIM newly introduces an adaptive sampling and
action alignment, which seeks to maximize the inter- and intra-video
information, respectively.

Figure 2: (a) Uniform sampler may overlook frames containing key actions.
Critical regions involving the actors and objects may be too small to be
properly recognized. (b) Our sampler is able to (I) select frames from an
entire video that contribute most to recognition, (II) amplify discriminative
regions in each frame. This sampling strategy is also dynamically adjusted
according to the episode task at hand.

Figure 3: Illustration of action misalignment. (a)&(b): action duration
misalignment. The action duration is highlighted with blue rectangles.
(b)&(c): action evolution misalignment in temporal (left) and spatial (right)
aspects. The red dashed connected lines indicate pairs of temporal or spatial
positions are consistent in action content. The action category of these
presented videos is ‘Hammer throw’.

##  1 Introduction

Recent years have witnessed spectacular developments in video action
recognition hieve2020 ; keypoint_comp2020 ; yufeng . Most current approaches
for action recognition employ deep learning models c3d2015 ; I3d2017 ; TSN2016
; survery_action , which are expected to achieve higher performance but
require large-scale labeled video data for training. Due to the expensive cost
of collecting and annotating video data, sometimes very few video data samples
are available in real-world applications. Consequently, the _few-shot_ video
recognition task, which aims to learn a robust action classifier with very few
training samples, has attracted much attention.

The majority of existing approaches cmn-j2020 ; trx2021 ; tarn2019  for few-
shot action recognition are built upon the metric learning paradigm.
Specifically, it first performs uniform frame sampling over each video to
generate input frames. Then, a feature extractor (e.g., ResNet) is leveraged
to obtain the feature representation of each video individually. Subsequently,
relationships among videos are measured using specific metrics (e.g.,
Euclidean or cosine distance) at the feature level. The categories of query
videos can then be inferred based on their relationships with support videos.
Under this pipeline, we establish that the information contributing to the
final action recognition can be summarized into two distinct types (as shown
in Fig. 3): (1) Intra-video information, which is determined by input frame
contents within a single video clip and then encoded into the feature
representation, and (2) Inter-video information, which is measured by the
relationships among videos and directly impacts prediction.

This naturally raises the question: How can we obtain more valuable intra- and
inter-video information for recognition from a limited amount of data? Driven
by this inquiry, we further observe that the existing approaches have not
fully exploited these two types of video information in the following aspects:

  * •

For intra-video information, most approaches primarily focus on feature
representation learning cmn-j2020 ; ARN2020 ; HCL , while little attention is
paid to the processing of input videos. As illustrated in Fig. 3(a), they
typically employ uniform frame sampling along both spatial and temporal
dimensions over each video to produce its corresponding frame input. This may
omit critical action content and is vulnerable to some redundancy and
interference from irrelevant parts in the video clip, thereby reducing the
utilization efficiency of intra-video information.

  * •

For inter-video information, since different action instances show distinct
spatial and temporal distributions, severe action misalignment issues exist
between videos (illustrated in Fig. 3). This makes it difficult to measure the
relationship among videos accurately. Although some approaches otam2020 ;
tarn2019 ; trx2021  have tried to solve this issue, they lack specific
analysis or guidance on how to cope with different types of misalignment.
Additionally, there has been little exploration into spatial action alignment.

  * •

Additionally, to our knowledge, no research in this field has jointly
considered intra- and inter-video information simultaneously, despite the fact
that they are complementary and may benefit from each other.

To overcome these limitations, we devise a novel framework called Video
Information Maximization (VIM) for few-shot action recognition. As illustrated
in Fig. 3, VIM introduces two innovative steps to the conventional recognition
pipeline: video adaptive sampling and action alignment. The former targets
intra-video information maximization at the data level, while the latter
focuses on inter-video information maximization at the feature level. In this
framework, we implement adaptive sampling by a task-specific spatial-temporal
video sampler, which improves the utilization efficiency of videos via
temporal selection and spatial amplification over frames. Meanwhile, action
alignment is achieved through a spatial-temporal action alignment model. It
learns to address misalignment issues in both spatial and temporal dimensions,
resulting in more accurate measurements of the relationships among videos. In
this way, the video sampler and action alignment steps are complementary to
each other in VIM to maximize the overall video information.

Specifically, in the VIM, input videos are first processed by the video
sampler, which consists of a temporal selector (TS) and a spatial amplifier
(SA). The TS selects $T$ frames from the entire video that contribute most to
the recognition, while SA learns to amplify critical action regions for each
selected frame. Moreover, the above sampling strategy can be dynamically
adjusted conditioned on the task at hand. Then, the action alignment is
applied between paired video features, sequentially performing a temporal and
spatial alignment. The temporal coordination (TC) module temporally aligns the
paired videos into a consistent action duration and evolution. The spatial
coordination (SC) module further ensures the action-specific regions (e.g.,
the actor or interacting objects) of paired frames are spatially consistent in
their positions. Finally, based on these well-aligned features, we can
calculate accurate relationships between videos. Furthermore, inspired by the
application mi_representation ; vsd  of mutual information (MI) in feature
representation learning, we further introduce new auxiliary loss terms based
on MI measurement to facilitate the optimization of VIM. They play their roles
by (1) encouraging the video sampler to preserve more class-related
information from the entire video and (2) enforcing the alignment process to
provide additional class-specific information than those without performing an
alignment. Under such guidance, our VIM can be optimized toward intra- and
inter-video information maximization.

In summary, our main contributions are as follows:

  * •

We analyze the few-shot action recognition problem from the perspective of
intra- and inter-video information. Based on this, we propose a novel
framework VIM that jointly maximizes these two types of video information.

  * •

Within VIM, we devise a task-specific spatial-temporal video sampler for few-
shot action recognition, which significantly improves utilization efficiency
in video data by adaptive spatial-temporal sampling over frames.

  * •

We resolve the misalignment issues from both temporal and spatial aspects for
few-shot action recognition by introducing a spatial-temporal action alignment
model that addresses distinctive action misalignment sequentially.

  * •

Under the criterion of video information maximization, we design two auxiliary
loss terms based on the Mutual Information (MI) measurement to facilitate the
overall optimization of VIM.

##  2 Related works

###  2.1 Few-shot image classification

A primary challenge faced in few-shot learning (FSL) is the insufficiency of
data in novel classes. The direct approach to address this is to enlarge the
sample size by data augmentation. Some approaches fsl-augment1 ; fsl-augment2
were proposed to generate unseen data with labels to enrich the feature spaces
of novel classes. Autoaugment autoaug2018  further learns the augmentation
policy automatically to improve the generalization on various few-shot
datasets. Besides, learning metrics to compare the seen and novel classes is
another popular way of handling FSL. Matching network matching2016  is an end-
to-end trainable kNN model using cosine as the metric, with an attention
mechanism over a learned embedding of the labeled samples to predict the
categories of the unlabeled data. The Prototypical Network prototypical2017
uses a feed-forward neural network to embed the task examples and perform
nearest neighbor classification with the class centroids. Relation Net
relationNet2018  proposed a novel network that concatenates the feature maps
of two images and proceeds to employ the concatenated features to a relation
net to learn the similarity between features. While these methods perform well
on image recognition tasks, it is less optimal to transfer them directly to
action recognition.

###  2.2 Few-shot action recognition

From our perspective of video information, the majority of current studies on
few-shot action recognition can be interpreted as solutions to improve either
inter-video or intra-video information.

Alignment-based methods are commonly used to address temporal action
misalignment, enabling more accurate measurements of inter-video information.
TARN tarn2019  devises an attentive relation network to perform implicit
temporal alignment at the video segment level, while OTAM otam2020  explicitly
aligns video sequences with a variant of the Dynamic Time Warping (DTW)
algorithm. TRX trx2021  represents video by exhaustive pairs and triplets of
frames for further alignment and matching, which is computationally expensive
but leads to significant improvement. Recently, some methods further perform
alignment in the spatial dimension for videos. ATA ATA2022  conducts
appearance alignment using frame-level feature matching to calculate
appearance similarity scores. MetaUVFS metaUVFS2021  obtains the action-
specific spatial representation by aligning the frame-wise appearance features
produced by 2D-CNNs to the video-wise motion features produced by 3D-CNNs.
Some methods focus on metric design to facilitate the measurement of inter-
video information. HyRSM HyRSM2022  reformulates the distance calculation
between videos as a set matching problem and adopts the Mean Hausdorff Metric
for measurement. However, the issue of action misalignment among videos has
not been thoroughly analyzed from both temporal and spatial perspectives.
Furthermore, most of these alignment-based methods lack specific guidance on
how to align actions accurately. Additionally, there has been little
exploration into spatial action alignment, despite its potential benefits.

Another line of methods can be regarded as approaches that aim to maximize
intra-video information. The early work CMN compound2018  achieves this
objective using a memory network to store matrix representations of videos.
ARN ARN2020  utilizes a self-supervised training strategy to improve the
robustness of video representation. All the above methods focus on feature
representation enhancement, but little attention is paid to the input video
data. AmeFu AmeFu2020  alleviates the extreme data-scarcity problem by
introducing an additional depth modality for videos. However, to our
knowledge, no attempt has been made to improve the utilization of intra-video
information from the root – the input videos.

More importantly, none of the existing methods can jointly maximize both
inter- and intra-video information, both of which are deemed critical for
improving few-shot action recognition.

###  2.3 Frame sampling in action recognition

Recently, some methods have attempted frame selection on input videos to
improve inference efficiency. Since the selection operation is discrete, many
methods utilize reinforcement learning adaframe2018 ; adafocus2021 ; marl2019
or the Gumbel trick arnet2020  to address this issue. Adaframe adaframe2018
adaptively selects a small number of frames for each video for recognition,
which is trained with the policy gradient method. MARL marl2019  adopted
multi-agent reinforcement learning to select multiple frames in parallel
fashion. FrameExit frameexit2021  introduced the early stop strategy for frame
sampling whereby frame selection stops when a pre-defined criterion is
satisfied. Most of these works only focus on temporal selection. A recent work
AdaFocus adafocus2021  leverages reinforcement learning to locate a specific
sub-region for each frame and passes it to classification.

While most of these methods aim to reduce the number of frames to improve
inference speed, we focus on selecting a fixed number but more informative
frames to improve the utilization efficiency of video (which can be regarded
as a fixed-size subset selection problem). It is also worth noting that the
success of these methods depends on having sufficient training data, whereas
only a limited number of training videos are available in few-shot
recognition. Additionally, the sampling operation on the spatial dimension,
which plays a vital role in understanding video content, has not been
thoroughly explored in conventional and few-shot action recognition.

Figure 4: The overall framework of video sampler in VIM. Specifically, videos
are first sent to the Scan Network with dense frame sampling to get their
global perception. Based on it, TS (temporal selector) evaluates frames and
selects $T$ frames that contribute most to recognition. Then selected frames
are processed by SA (spatial amplifier), which amplifies discriminative sub-
regions under the guidance of saliency maps. An encoder aggregates the task
information and generates task-specific parameters for TS and SA to conduct a
dynamic sampling according to the episode task at hand.

##  3 Methods

###  3.1 Problem formulation

Following standard episode training in few-shot action recognition, the
dataset is divided into three distinct parts: training set
$\mathcal{C}_{train}$, validation set $\mathcal{C}_{val}$, and test set
$\mathcal{C}_{test}$. The training set contains sufficient labeled data for
each class while there exist only a few labeled samples in the test set. The
validation set is only used for model evaluation during training. Moreover,
there are no overlapping categories between these three sets. Generally, _few-
shot_ action recognition aims to train a classification network that can
generalize well to novel classes in the test set. Consider an $N$-way $K$-shot
episode training, each episode contains a support set $\mathcal{S}$ sampled
from the training set $\mathcal{C}_{train}$. It contains $N\times K$ samples
from $N$ different classes where each class contains $K$ support samples.
Likewise, $Q$ samples from each class are then selected to form the query set
$\mathcal{Q}$ which contains $N\times Q$ samples. The goal is to classify
$N\times Q$ query samples using only $N\times K$ support samples.

###  3.2 Overall pipeline of VIM

Fig. 3 illustrates the overall pipeline of our VIM. Specifically, support and
query videos are first fed into the video sampler. It conducts adaptive
temporal-spatial frame sampling over video clips and outputs $T$ frames for
each video. Then, we produce the feature representation for each video using
the feature extractor (i.e., backbone). Based on these features, the action
alignment performs alignment between query and support videos, which generates
well-aligned features for video pairs. Finally, the relationship (e.g.,
distance, similarity) among videos are calculated using a specific metric.

###  3.3 Task-adaptive spatial-temporal video sampler

Fig. 4 presents the architecture of our proposed video sampler in VIM. We will
elaborate on each module in the following.

####  3.3.1 Video scanning

To get a global perception of video, we first densely sample $M$ frames from
each video. Thus, each video is represented by a dense frame sequence
$\mathbf{X_{D}}=\\{X_{1},X_{2},...,X_{M}\\}$, where $|\mathbf{X_{D}}|=M$. Our
sampler performs both temporal and spatial sampling among frames in
$\mathbf{X}_{\text{D}}$ to obtain a subset
$\mathbf{X},\text{where}~{}|\mathbf{X}|=T<M$. Given the dense sequence, a
lightweight Scan Network $f(\cdot)$ takes the frames as input and obtains
corresponding frame-level features $\displaystyle
f_{\mathbf{X_{\text{D}}}}=\\{f(X_{1}),\dots,f(X_{M})\\}\in\mathbb{R}^{c\times
M\times h\times w}$. To reduce computation, each frame is down-scaled to a
smaller size ($64\times$64 by default) before feeding into $f(\cdot)$. The
video-level representation is determined as the average of all the frames
$g_{\mathbf{X_{D}}}=\frac{1}{M}\sum_{i}^{M}f(X_{i})\in\mathbb{R}^{c\times
h\times w}$, for the subsequent sampling procedure.

####  3.3.2 Temporal selector

A simple way to improve the data coverage and utilization efficiency is to
feed all $M$ frames into few-shot learners without any filtering. However,
this is intractable since it requires enormous computations and memory,
especially under episode training (each episode contains $N(Q+K)$ videos).
Therefore, we aim to select $T$ informative frames that contribute the most to
few-shot recognition from the set of dense frames $\mathbf{X}$, which will
improve the data utilization without introducing further overhead. To this
end, a temporal selector (TS) is devised to conduct this selection.

Evaluation Based on frame-level features, an evaluator $\Phi_{\text{eval}}$
predicts the importance score for each frame. Specifically, it receives frame
features as input and outputs their corresponding importance scores
$\mathbf{S}\in\mathbb{R}^{M\times 1}$. Meanwhile, the global information
$g_{\mathbf{X}}$ is concatenated with each frame-level feature to provide a
global perception:

$\displaystyle
s_{i}=\Phi_{\text{eval}}(\texttt{Avg}(\texttt{Cat}(f(X_{i}),g_{\mathbf{X}}))+\texttt{PE}(i))$
(1) $\displaystyle\Phi_{\text{eval}}=w_{2}(\texttt{ReLU}(w_{1}(\cdot))$ (2)

where $w_{1}\in\mathbb{R}^{c\times d}$, $w_{2}\in\mathbb{R}^{d\times 1}$
denotes the weights of linear layers, and Avg denotes spatial global average
pooling, PE indicates the Position Embedding transfomer2017  of position $i$.
Specifically, the value of $w_{2}$ is dynamically adjusted among different
episode tasks, which will be further elaborated in Sec. 3.3.4. Then, scores
are normalized to $[0,1]$ to stabilize the training process. Based on the
importance scores, we can pick the $T$ highest scores by a Top-k operation
(set $k=T$), which returns the indices $\\{i_{1},i_{2},\dots,i_{T}\\}\in[0,M)$
of these $T$ frames. To keep the temporal order of selected frames, the
indices are sorted s.t., $i_{1}<i_{2}<\dots<i_{T}$. Further, the indices are
converted to one-hot vectors
$\mathbf{I}=\\{I_{i_{1}},I_{i_{2}},\dots,I_{i_{T}}\\}\in\\{0,1\\}^{M\times
T}$. This way, the selected frame subset $X^{\prime}$ could be simply
extracted by matrix tensor multiplication:

$\displaystyle\mathbf{X}=\mathbf{I}^{\top}\mathbf{X_{D}}$ (3)

where $\mathbf{I}^{\top}$ is the transpose of $\mathbf{I}$. Nevertheless, the
above operations pose a great challenge in that the Top-k and one-hot
operations are non-differentiable for end-to-end training.

Differentiable top-$k$ selection We adopt the _perturbed maximum method_
introduced in perturb2020 ; differential2021  to differentiate the sampling
process during training. Specifically, the above temporal selection process is
equivalent to solving a linear program of:

$\mathop{\arg\max}_{\mathbf{I}\in\mathcal{C}}\langle\mathbf{I},\mathbf{S}\mathbf{1}^{T}\rangle$
(4)

Here, input $\mathbf{S}\mathbf{1}^{T}\in\mathbb{R}^{M\times T}$ denotes
repeating score $\mathbf{S}$ by $T$ times, while $\mathcal{C}$ indicates a
convex constraint set containing all possible $\mathbf{I}$.

Under this equivalence, the linear program of Eq. 4 can be solved by the
perturbed maximum method, which performs forward and backward operations for
differentiation as described below:

Forward This step forwards a smoothed version of Eq. 4 by calculating
expectations with random perturbations on input:

$\mathbf{I}_{\sigma}=\mathbb{E}_{Z}\left[\underset{\mathbf{I}\in\mathcal{C}}{\arg\max}\left\langle\mathbf{I},\mathbf{S1}^{T}+\sigma\mathbf{Z}\right\rangle\right]$
(5)

where $\sigma$ is a temperature parameter and $\mathbf{Z}$ is a random noise
sampled from the uniform Gaussian distribution. In practice, we conduct the
Top-k ($k=T$ in our case) algorithm with perturbed importance scores for $n$
(i.e., $n=500$ in our implementation) times and compute their expectations.

Backward Following perturb2020 , the Jacobian of the above forward pass can be
calculated as:

$J_{\mathbf{s}}\mathbf{I}=\mathbb{E}_{Z}\left[\underset{\mathbf{I}\in\mathcal{C}}{\arg\max}\left\langle\mathbf{I},\mathbf{S}\mathbf{1}^{\mathbf{T}}+\sigma\mathbf{Z}\right\rangle\mathbf{Z}^{T}/\sigma\right]$
(6)

Based on this, we can back-propagate the gradients through the Top-k ($k=T$)
operation.

During inference, we leverage hard Top-k to boost computational efficiency.
However, applying hard Top-k during evaluation also creates inconsistencies
between training and testing. To this end, we linearly decay $\sigma$ to zero
during training. When $\sigma=0$, the differentiable Top-k is identical with
the hard Top-k.

####  3.3.3 Spatial amplifier

Critical actions tend to occur in the selected regions of the frame, such as
around the actors or objects. However, current approaches in few-shot action
recognition treat these regions as equal to other areas (with non-critical
actions) during data processing, thus reducing the efficiency of data
utilization. Some works address this similar issue in image recognition by
locating an important sub-region and then cropping it out from the whole image
crop1 ; racnn2017 ; scapnet2021 . However, directly replacing the original
image with its partial region may further impact the data utilization
efficiency under the few-shot setting. Inspired by the application of
attention-based non-uniform sampling learningzoom2018 ; STN2015 ;
trilinear2019 , we adopt a spatial amplifier (SA) for few-shot action
recognition, which emphasizes the discriminative spatial regions while
maintaining relatively complete frame information.

Saliency map We then introduce how the informative spatial regions of each
frame are estimated. Since each feature map channel in CNN models may
characterize a specific recognition pattern, we could estimate areas that
contribute significantly to few-shot recognition by feature map aggregation.
From the Scan Network $f(\cdot)$, we resort to using its output feature maps
$f(X)\in\mathbb{R}^{c\times h\times w}$ to generate spatial saliency maps.
However, there may exist activations in some action-irrelevant regions (e.g.,
background). Hence, directly aggregating all the channels will introduce some
noises. Therefore, to enhance the discriminative patterns while dismissing
these noises, we incorporate the self-attention mechanism transfomer2017  on
the channels of $f(X)$, such that

$\alpha=\frac{f(X)f(X)^{T}}{\sqrt{h\times w}}\in\mathbb{R}^{c\times
c},~{}~{}f(X)^{\prime}=\alpha f(X)\in\mathbb{R}^{c\times h\times w}$ (7)

Then, we can calculate a saliency map $\mathbf{M_{s}}$ for frame $i$ by
aggregating the activations of all channels of the feature map:

$\mathbf{M_{s}}\in\mathbb{R}^{h\times w}=\frac{1}{c}\sum_{i}^{c}w_{s_{i}}\cdot
f(X)^{\prime}$ (8)

where $w_{s}\in\mathbb{R}^{c\times 1}$ are learnable parameters that aggregate
all the channels. Note that $w_{s}$ is also dynamically adjusted according to
the episode task at hand. We elaborate more details in Sec. 3.3.4. Finally,
saliency map $\mathbf{M}_{s}$ will be up-sampled to the same size of frame
$X$.

Amplification Based on the saliency map, our rule for spatial sampling is that
an area with a large saliency value should be given a larger probability to be
sampled (i.e., in this context, we say that this area will be ‘ _amplified_ ’
compared to other regions). We implement the above amplification process by
the _inverse-transform sampling_ used in inverse_sample1986 ; trilinear2019 ;
saliencysampling2022 .

Figure 5: Illustration of the 2D inverse-transform sampling.

As illustrated in Fig. 5, we first decompose the saliency map $\mathbf{M}_{s}$
into $x$ and $y$ dimensions by calculating the maximum values over axes
following trilinear2019  for stabilization:

$\displaystyle M_{x}=\max_{1\leq i\leq
W}(\mathbf{M_{s}})_{i,j},~{}~{}~{}~{}M_{y}=\max_{1\leq j\leq
H}(\mathbf{M_{s}})_{i,j}$ (9)

Then, we consider the Cumulative Distribution Function (cdf), which is non-
uniform and monotonically-increasing, to obtain their respective distributions
(Fig. 5 b1&b2):

$\displaystyle
D_{x}=\texttt{cdf}(M_{x}),~{}~{}~{}~{}D_{y}=\texttt{cdf}(M_{y})$ (10)

Therefore, the sampling function for frame $X$ under saliency map
$\mathbf{M}_{s}$ can be calculated by the inverse function of Eq. 10:

$X^{{}^{\prime}}_{i,j}=\texttt{Func}(F,\mathbf{M_{s}},i,j)=X_{{D}_{x}^{-1}(i),{D}_{y}^{-1}(j)}$
(11)

In practice, we implement Eq. 11 by uniformly sampling points over the $y$
axis, and projecting the values to the $x$ axis to obtain sampling points
(depicted by the green arrows shown in Fig. 5 b1&b2). The sampling points
obtained from Fig. 5 b1&b2 form the 2D sampling grid. Finally, we can conduct
an affine transformation above the original image based on the grid to obtain
the final amplified image.

The SA performs on all frames selected by TS. In this way, the amplified frame
can emphasize the discriminative regions while maintaining complete
information of the original one. Finally, the video sampler outputs $T$ frames
$\mathbf{X}=\\{X^{{}^{\prime}}_{1},\dots,X^{{}^{\prime}}_{T}\\}$ for each
input video.

Figure 6: Illustration of the task-adaptive sampling learner.

####  3.3.4 Task-adaptive learner

In general action recognition, once a sampler is well-trained, it samples each
testing video with a fixed strategy and criterion adaframe2018 ; arnet2020 .
Nevertheless, in the few-shot episode paradigm, the query video relies on
videos in the support set to perform classification. Thus, our testing videos
are not independent compared to general action recognition. Therefore, fixing
the sampling strategy for each video among episodes is not ideal in few-shot
recognition. To this end, we adopt a task-adaptive learner for our sampler,
which generates task-specific parameters for layers in TS and SA to
dynamically adjust the sampling strategy according to the episode task at
hand.

Given support set $\mathcal{S}=\\{(\mathbf{X}^{i})_{i}^{N\times K}\\}$ and its
corresponding video-level features $\\{(g_{\mathbf{X}^{i}})_{i}^{N\times
K}\\}$ (extracted by Scan Net) at hand, we can estimate the summary statistics
of this task by parameterizing it as a conditional multivariate Gaussian
distribution with a diagonal covariance static2016 ; LGMNet2019 . Therefore, a
task encoder $E$ is employed to estimate its summary statistics:

$\mu,\sigma=\frac{1}{N\times K}\sum_{i=1}^{N\times
K}E(\texttt{Avg}(g_{\mathbf{X}^{i}}))$ (12)

where $E$ consists of two linear layers, Avg is global average pooling, $\mu$
and $\sigma$ are 128-$dim$ mean and variance estimated by $E$. Then, we denote
the probability distribution of task summary feature as:

$\small
p(f_{t}\mid\mathcal{S})=\mathcal{N}\left(\mu,\operatorname{diag}\left(\sigma^{2}\right)\right)$
(13)

We could sample a task summary feature $f_{t}$ that satisfies the above
distribution with the re-parameterization trick vae2013 :
$f_{t}\in\mathbb{R}^{128}=\mu+\sigma\varepsilon$, where $\varepsilon$ is a
random variable that $\varepsilon\sim\mathcal{N}(0,1)$. In this way, each task
is encoded into a fixed-length representation, which reflects more
consistently across the data distribution of task.

Based on the summary of current task, a task-specific sampling strategy can be
implemented by adjusting the parameters that influence the criterion of
sampling. Specifically, we generate task-specific parameters for
$w_{2}\in\mathbb{R}^{d\times 1}$ in TS (Eq. 2) and
$w_{s}\in\mathbb{R}^{c\times 1}$ in SA (Eq. 8) where the former decides how to
evaluate the importance score of frames while the latter determines the
generation of saliency maps. The above process is illustrated in Fig. 6.

$\theta_{ts}=G_{t}(f_{t}),~{}~{}\theta_{sa}=G_{s}(f_{t})$ (14)

where $G_{t}\in\mathbb{R}^{128\times d}$ and $G_{s}\in\mathbb{R}^{128\times
c}$ denote the weights of the parameter generator. Finally, the generated
parameters are normalized and filled into corresponding layers
($w_{2}\leftarrow\frac{\theta_{ts}}{\|\theta_{ts}\|}$,
$w_{s}\leftarrow\frac{\theta_{sa}}{\|\theta_{sa}\|}$).

Figure 7: Overall framework of the action alignment model in VIM.
Specifically, the embedded video features are first fed into the Temporal
Coordination (TC) module, which addresses action duration misalignment by
performing temporal transformations on each feature. Next, TC performs
temporal rearrangement to align the action evolution between the support and
query features. Finally, the Spatial Coordination (SC) module ensures that the
support and query features are consistent in spatial position before
conducting metric learning.

###  3.4 Spatial-temporal action alignment

####  3.4.1 Delve into the misalignment

We observe that the action misalignment among videos can be summarized into
two distinctive types:

  * (1)

The relative temporal location of action is usually inconsistent between
videos due to different start time and duration (as shown in Fig. 3(a)&(b)),
we call the issue of location inconsistency as action duration misalignment.

  * (2)

Since action often evolves in a non-linear manner, the discriminative spatial-
temporal part within the action process varies from action instances (as shown
in Fig. 3(b)&(c)), even though they share the same semantic category and
duration. We define this internal spatial-temporal malposition among action
instances as action evolution misalignment.

To cope with these two types of misalignment, we devise an action alignment
model that performs temporal and spatial alignment sequentially. The overall
framework of our model is presented in Fig. 4.

####  3.4.2 Feature embedding

After our sampler pre-processes the raw video into $T$ frames
$\mathbf{X}=\\{X_{1}^{{}^{\prime}},...,X_{T}^{{}^{\prime}}\\}$, a strong
feature embedding network $F(\cdot)$ (e.g., ResNet-50) receives these frames
to generate sequential features:

$\displaystyle
F_{\mathbf{X}}=\\{F(X_{1}^{{}^{\prime}}),F(X_{2}^{{}^{\prime}}),\dots,F(X_{T}^{{}^{\prime}})\\}$
(15)

where $F_{\mathbf{X}}\in\mathbb{R}^{\rm C\times T\times H\times W}$ denotes
the video representation. From this point, we will use $F_{s}$, $F_{q}$ to
represent the video-level feature of the support sample and query sample,
respectively.

####  3.4.3 Temporal Coordination

We perform the temporal alignment with a temporal coordination (TC) module. It
consists of a temporal transformation and rearrangement operation, aiming to
address these two types of misalignment in the temporal dimension.

Figure 8: Illustration of temporal transformation.

Temporal transformation To address the action duration misalignment, we aim to
locate the action temporally, then the duration feature could be located and
emphasized while dismissing the action-irrelevant feature. In this way, the
duration misalignment could be eliminated. We achieve this objective through a
temporal transformation module. It consists of two parts: a localization
network $\mathbf{L}$ and a temporal affine transformation $\mathbf{T}$.

Specifically, given an input frame-level feature sequence $F_{\mathbf{X}}$,
the localization network generates warping parameters
$\phi=(a,b)=\mathbf{L}(F_{\mathbf{X}})$ firstly. Then the input feature
sequence is warped by the affine transformation $\mathbf{T}_{\phi}$.
Succinctly, the temporal transform process is defined as:

$\hat{F_{\mathbf{X}}}=\mathbf{T}_{\phi}(F_{\mathbf{X}}),~{}\phi=\mathbf{L}(F_{\mathbf{X}})$
(16)

where $\hat{f_{X}}$ indicates the feature sequence aligned to the action
duration period, $\mathbf{L}$ consists of several light trainable layers in
our implementation. Since the action duration misalignment is
characteristically linear among frame sequences, the warping is represented
using linear temporal interpolation.

The structure of temporal transformation is illustrated in Fig. 8. During the
episode training and testing, all the support and query features are first fed
into this transformation to perform a preliminary temporal alignment, where
their video features could be roughly aligned to their action periods. In this
way, the temporal transformation operation caters towards relieving the action
duration misalignment problem.

Temporal rearrangement To temporally align the action evolution among videos,
similar motion patterns between videos should be aggregated to the same
temporal location. We treat this as a global rearrangement task where the
motion evolution of query video can be temporally rearranged to match the
support ones. Accordingly, we model the evolution correlation
$\mathbf{M}_{e}\in\mathbb{R}^{T\times T}$ between support and query as:

$\mathbf{M}_{e}=\texttt{softmax}(\frac{(W_{k}\cdot G(\hat{F_{s}}))(W_{q}\cdot
G(\hat{F_{q}}))^{T}}{\sqrt{dim}})$ (17)

where $W_{k},W_{q}$ indicate linear projection layer, $dim$ is the dimension
of feature $G(\hat{F})$, $G$ is the global average pooling in spatial
dimensions whose output tensor shape is $C\times T\times 1\times 1$, i.e., the
correlation is only calculated in the temporal dimension,
$\operatorname{Softmax}$ limits the values in $\mathbf{M}_{e}$ to $[0,1]$.

Then, we could temporally rearrange the query feature by calculating the
matrix multiplication between the normalized motion correlation matrix
$\mathbf{M}_{e}$ and the query features:

$\displaystyle\tilde{F_{q}}=\mathbf{M}_{e}\cdot(W_{v}\cdot G(\hat{F_{q}}))$
(18)

Similar to $W_{k},W_{q}$, $W_{v}$ denotes a linear projection layer. In order
to keep feature space consistent, this projection is also applied to support
feature $\hat{F_{s}}$: $\tilde{F_{s}}=W_{v}\cdot G(\hat{F_{s}})$. This way,
the same temporal location is in the consistent evolution, and thus evolution
misalignment in the temporal aspect can be relieved. The illustration of TC is
presented in Fig. 9.

Figure 9: Illustration of temporal rearrangement.

####  3.4.4 Spatial Coordination

The TC ensures the action evolves with the same process over the duration
period. However, the spatial variation of actor evolution, such as the
positions of actors, also being critical for action recognition, which cannot
be modeled by TC. Thus, we further devise the spatial coordination (SC) to
reduce the spatial variations in action evolution. On the basis of temporally
well-aligned features, we aim to predict a spatial offset for each paired
frame and then measure their similarity in the intersection area only, as
shown in the top of Fig. 10.

Specifically, SC consists of two steps: lightweight offset prediction and
offset mask generation. First, given the temporally well-aligned feature
$\tilde{F_{s}}$ and $\tilde{F_{q}}$, they are feed into the offset predictor
$S$ to predict spatial offset $O\in\mathbb{R}^{T\times 2}$ in $x$ and $y$
coordinates for all timestamps:

$\displaystyle O=S(\texttt{Cat}(\tilde{F_{s}},\tilde{F_{q}}))$ (19)

where $\texttt{Cat}(\cdot)$ denotes concatenation along the channel. As shown
in Fig. 10, the predicted offset indicates the relative position vector of the
action-specific region between query and support frames. Then, the
intersection area can be located by its corresponding spatial offset.

To calculate the similarity in the intersection area in a differentiable way,
for each frame, we generate an offset mask $I_{o}$ to calculate the average
feature in the intersection on each feature. Moreover, $I_{o}$ is set to be
smooth for differentiability, where the value of the mask is $1$ inside the
intersection area and it gradually decreases to $0$ on the edge. Then the
masks are performed over the query and support features simultaneously as
weights of the weighted average:

$\displaystyle\overline{F}_{s,i}=\sum_{HW}{(I_{o_{i}}*\tilde{F_{s}})}/\sum{I_{o_{i}}},~{}i=0,\dots,T$
(20)
$\displaystyle\overline{F}_{q,i}=\sum_{HW}{(I_{-o_{i}}*\tilde{F_{q}})}/\sum{I_{-o_{i}}},~{}i=0,\dots,T$
(21)

In order to expand the exploration space of the offset predictor, we further
add some perturbations on the predicted offset and use the average of
corresponding features of different perturbations.

Undergoing TC and SC, the action evolution misalignment among videos is
dismissed in spatial-temporal aspect. The well-aligned paired features
$\overline{F_{s}}$ and $\overline{F_{q}}$ are used in final distance
measurement and classification.

Figure 10: Spatial coordination (SC). top: illustration of how to measure the
distance under the offset. bottom: details of offset prediction and mask
generation.

###  3.5 Optimization of VIM

####  3.5.1 Classification

We can easily train our VIM in an end-to-end manner with standard softmax
cross-entropy. Given the aligned feature of query sample $\overline{F}_{q}$,
and the support prototype $\overline{P}_{s}^{c}$ for class $c$ (the prototype
is calculated as the same as ProtoNet prototypical2017 ), we can obtain the
classification probability as:

${\rm P}(x_{q}\in
c_{i})=\frac{\exp(-d(\overline{F_{q}},\overline{P}_{s}^{c_{i}}))}{\sum_{c_{j}\in
C}{\exp(-d(\overline{F_{q}},\overline{P}_{s}^{c_{j}}))}}$ (22)
$d(F,P)=\sum_{t=1}^{T}{1-\frac{<F_{[t]},P_{[t]}>}{\|F_{[t]}\|_{2}\|P_{[t]}\|_{2}}},$
(23)

where $d(F,P)$ is the frame-wise cosine distance metric. Then the
classification loss is calculated as:

$\mathcal{L}_{\text{cls}}=-\sum_{q\in Q}{\mathbf{Y}_{q}\cdot\log{{\rm
P}(X_{q}\in c_{i})}},$ (24)

where $\mathbf{Y}_{q}\in\\{0,1\\}$ indicates whether $X_{q}\in c_{i}$, while
$Q$ and $C$ represent the query set and its corresponding class label set,
respectively.

####  3.5.2 Video information maximization

Furthermore, to ensure that our VIM could be well optimized towards the goal
of maximizing intra-and inter-video information, we design two auxiliary loss
terms based on the mutual information measurement as additional supervision.
In information theory, mutual information (MI) measures the amount of
information shared between two random variables. Formally, MI quantifies the
statistical dependency of two discrete random variables:

$\displaystyle\mathcal{I}(V_{1};V_{2})=D_{\text{KL}}(p(V_{1},V_{2})||p(V_{1})p(V_{2}))$
(25)

where $D_{\text{KL}}$ is the Kullback-Leibler (KL) divergency between joint
distribution $p(V_{1},V_{2})$ and the product of marginals $p(V_{1})p(V_{2})$.
Eq. 25 measures how much knowing variable $V_{2}$ reduces uncertainty about
variable $V_{1}$. Particularly, $\mathcal{I}(V_{1};V_{2})=0$ indicates that
$V_{1}$ and $V_{2}$ are independent (i.e., observing $V_{2}$ observing tells
nothing about the prediction $V_{1}$).

By virtue of the above characteristic of mutual information, some approaches
deep_infomax ; tschannen2019mutual ; mi_representation  learn to enhance the
feature representation by optimizing the MI between input and output in neural
networks. Moreover, some methods mutual_subset1 ; multual_subset2  also try to
solve the feature subset selection from the perspective of MI. Taking
inspiration from them, we introduce additional auxiliary loss terms based on
the mutual information for our VIM to facilitate its optimization.

Intra-video information loss. The video sampler aims to maximize the intra-
video information at the data level by adaptive sampling over frames.
Specifically, it is expected to preserve and emphasize more action-specific
information while eliminating interference from the entire video. From the
perspective of MI, this objective can be achieved by minimizing the margin
between $\mathcal{I}(\mathbf{Y};\mathbf{X})$ and
$\mathcal{I}(\mathbf{Y};\mathbf{X_{D}})$:

$\displaystyle\text{min}~{}||\mathcal{I}(\mathbf{Y};\mathbf{X})-\mathcal{I}(\mathbf{Y};\mathbf{X_{D}})||$
(26) $\displaystyle\text{where}~{}\mathbf{X}=\texttt{Sampler}(\mathbf{X_{D}})$
(27)

where $\mathbf{X}$ denotes the sampled frames, $\mathbf{X_{D}}$ denotes the
dense frame sequence for scanning, $\mathbf{Y}$ denotes the label. Here
$\mathcal{I}(\mathbf{Y};\mathbf{X})$ measures the class-specific information
of input videos. Eq. 26 encourages the sampled frames to keep as much class-
specific information as possible of the entire video clip. Therefore, we could
design an auxiliary loss term for sampler to facilitate the intra-video
information maximization as:

$\displaystyle\mathcal{L}_{\text{intra}}=||\mathcal{I}(\mathbf{Y};f_{\mathbf{X}})-\mathcal{I}(\mathbf{Y};f_{\mathbf{X_{D}}})||$
(28)

where $f_{\mathbf{X_{D}}}$ and $f_{\mathbf{X}}$ denotes the features of
$\mathbf{X_{D}}$ and $\mathbf{X}$, respectively. Both of them are produced by
the Scan Network $f(\cdot)$.

Inter-video information loss. The action alignment aims to provide more
accurate measurements of inter-video information, which is achieved by
aligning the query features to the support features. Therefore, the above
objective is equivalent to maximizing the following conditional mutual
information:

$\displaystyle\text{max}~{}\mathcal{I}(\mathbf{Y};\overline{F}_{q}|F_{q})$
(29)
$\displaystyle\text{where}~{}\overline{F}_{q}=\texttt{Alignment}(F_{q}\rightarrow
F_{s})$ (30)

where $F_{q}$ and $\overline{F}_{q}$ indicate the query features before/after
aligning it to support feature $F_{s}$. This way, Eq. 29 indicates the amount
of class-specific information in the aligned feature $\overline{F_{q}}$,
excluding the information from the raw feature $F_{q}$. Intuitively, it
measures the incremental amount of information that results from aligning
$F_{q}$ to support video $F_{s}$. Optimizing this objective will maximize the
additional information we can extract from the relationships between support
and query videos to improve the few-shot recognition. However, conditional MI
is hard to be estimated in neural networks. According to the formula of
conditional MI, we have:

$\displaystyle\mathcal{I}(\mathbf{Y};\overline{F}_{q}|F_{q})=\mathcal{I}(\mathbf{Y};\overline{F}_{q})-\mathcal{I}(\overline{F}_{q};F_{q})+\mathcal{I}(\overline{F}_{q};F_{q}|\mathbf{Y})$
(31)

The last term measures the class-irrelevant information shared by
$\overline{F}_{q}$ and $F_{q}$, which can be omitted under the supervision of
classification loss. In this way, the objective Eq. 29 can be simplified to:

$\displaystyle\text{max}~{}\mathcal{I}(\mathbf{Y};\overline{F}_{q})-\mathcal{I}(\overline{F}_{q};F_{q})$
(32)

Therefore, we could introduce an auxiliary loss for the alignment model to
facilitate the inter-video information maximization as:

$\displaystyle\mathcal{L}_{\text{inter}}=-\big{(}\mathcal{I}(\mathbf{Y};\overline{F}_{q})-\mathcal{I}(\overline{F}_{q};F_{q})\big{)}$
(33)

These MI terms can be estimated by existing MI estimators mine ; vsd . In this
paper, we employ the VSD vsd  to estimate the MI for each term.

Overall, the loss terms for training the VIM can be summarized as:

$\displaystyle\mathcal{L}=\mathcal{L}_{\text{cls}}+\mu_{1}\mathcal{L}_{\text{intra}}+\mu_{2}\mathcal{L}_{\text{inter}}$
(34)

where the hyper-parameters $\mu_{1}$ and $\mu_{2}$ balance different loss
terms.

##  4 Experiments

###  4.1 Datasets and baselines

Datasets Four popular datasets and their few-shot data splits are selected for
experiments:

  * •

UCF101 ucf101 : We follow the same protocol introduced in ARN ARN2020 , where
70/10/21 classes and 9154/1421/2745 videos are included for train/val/test
respectively.

  * •

HMDB51 HMDB : We follow the protocol of ARN ARN2020 , which takes 31/10/10
action classes with 4280/1194/1292 videos for train/val/test.

  * •

SSv2 ssv2 : For SSv2 benchmark, there are two splits proposed in CMN
compound2018  and OTAM otam2020  denoted as $\text{SSv2}^{{\dagger}}$ and
$\text{SSv2}^{*}$ respectively. We use the split of $\text{SSv2}^{{\dagger}}$
compound2018  for its amount of data is closer to the standard few-shot
setting, where 64/12/24 classes and 6400/1200/2400 videos are included for
train/val/test.

  * •

Kinetics compound2018  contains 100 classes selected from Kinetics-400, where
64/12/24 classes are split into train/val/test set with 100 videos for each
class.

Competitors The widely-used few-shot learning baseline ProtoNet
prototypical2017  is selected as the baseline of VIM. Besides, we choose
recent state-of-the-art few-shot action recognition approaches, including
CMN-J cmn-j2020 , ARN ARN2020 , TARN tarn2019 , OTAM otam2020 , ITANet itanet
, TRX trx2021 , HCL HCL , and HyRSM HyRSM2022 .

###  4.2 Implementation details

We select the ProtoNet prototypical2017  as our baseline to conduct few-shot
classification. For the video sampler, we use ShuffleNet-v2 shufflenet2018
pre-trained in ImageNet as the Scan Network $f(\cdot)$ for efficiency. In
addition, we remove its first max-pooling layer to improve the quality of
saliency maps. By default, we scan each video by densely sampling 20 frames at
$224\times 224$ pixels (i.e., $M=20$). Moreover, each frame is downscaled to
$64\times 64$ pixels before feeding into the Scan Network to reduce the
computational cost. Following previous works, our sampler outputs 8 frames
(i.e., $T=8$) at $224\times 224$ pixels for the subsequent few-shot learners
to ensure fair comparisons. The ImageNet pre-trained ResNet-50 he2016deep  is
selected as the feature extractor $F(\cdot)$ so that we could have a fair
comparison with most current approaches cmn-j2020 ; otam2020 ; trx2021 ;
HyRSM2022 . Specifically, the feature before the last average pooling layer
(stage-4) in ResNet-50 forms the frame-level input to our action alignment
model. For episode training, 5-way 1-shot and 5-way 5-shot classification
tasks are conducted. The whole meta-training contains 15,000 epochs, each with
200 episodes. In the testing phase, we sample 10,000 episodes in the meta-test
split and report the average result. We train our model using SGD optimizer,
with an initial learning rate of $1\times 10^{-2}$. The learning rate is
decayed (multiplied) by 0.9 after every 5000 epochs. The temperature $\sigma$
in video sampler is set to 0.1, decayed by 0.8 after 2000 epochs. The
coefficients of loss terms are $\mu_{1}=0.5,\mu_{2}=1$ in default.

Method | Publisher | Backbone | HMDB51 | UCF101 | Kinetics | SSv2
---|---|---|---|---|---|---
1-shot | 5-shot | 1-shot | 5-shot | 1-shot | 5-shot | 1-shot | 5-shot
MAML finn2017model  | ICML’17 | ResNet-50 | - | - | - | - | 54.2 | 75.3 | 31.3 | 45.5
TRN++ otam2020  | ECCV’18 | ResNet-50 | - | - | - | - | 68.4 | 82.0 | - | -
Plain CMN compound2018  | ECCV’18 | ResNet-50 | - | - | - | - | 57.3 | 76.0 | 33.4 | 46.5
ProtoGAN protogan2019  | CVPR’19 | C3D | 34.7 | 54.0 | 57.8 | 80.2 | - | - | - | -
TARN tarn2019  | BMVC’19 | C3D | - | - | - | - | 64.8 | 78.5 | - | -
ARN ARN2020  | ECCV’20 | C3D | 45.5 | 60.6 | 66.3 | 83.1 | 63.7 | 82.4 | - | -
CMN-J cmn-j2020  | TPAMI’20 | ResNet-50 | - | - | - | - | 60.5 | 78.9 | 36.2 | 48.8
ProtoNet* prototypical2017  | - | ResNet-50 | 54.2 | 68.4 | 78.7 | 89.6 | 64.5 | 77.9 | 34.2 | 46.5
OTAM otam2020  | CVPR’20 | ResNet-50 | 54.5 | 66.1 | 79.9 | 88.9 | 73.0 | 85.8 | 36.4 | 48.0
AMeFu AmeFu2020  | MM’20 | ResNet-50 | 54.5 | 66.1 | 85.1 | 95.5 | - | - | - | -
ITANet itanet  | IJCAI’21 | C3D | - | - | - | - | 73.6 | 84.3 | 39.8 | 53.7
TRX trx2021  | CVPR’21 | ResNet-50 | 53.1 | 75.6 | 78.6 | 96.1 | 63.6 | 85.9 | 36.0 | 59.1
HyRSM HyRSM2022  | CVPR’22 | ResNet-50 | 60.3 | 76.0 | 83.9 | 94.7 | 73.7 | 86.1 | 40.6 | 56.1
HCL HCL  | CVPR’22 | ResNet-50 | 59.1 | 76.3 | 82.5 | 93.9 | 73.7 | 85.8 | 38.7 | 55.4
VIM (Ours) | - | ResNet-50 | 61.1 | 76.5 | 86.2 | 96.1 | 73.8 | 87.1 | 40.8 | 54.9
Table 1: Few-shot action recognition results under standard 5-way k-shot setting. * refers to our re-implementation. Method | ActivityNet
---|---
1-shot | 3-shot | 5-shot
ProtoNet* | 66.5 | 73.4 | 80.3
OTAM* | 69.9 | 75.9 | 85.2
TRX* | 69.9 | 77.6 | 88.7
HyRSM* | 72.1 | 78.0 | 87.5
VIM (Ours) | 74.7 | 79.3 | 89.6
Table 2: Results on ActivityNet. * refers to results from our re-
implementation.

###  4.3 Main results

####  4.3.1 Quantitative results

Table 1 summarizes the results of state-of-the-art methods and our VIM on few-
shot action recognition. VIM outperforms various existing approaches and
achieves new state-of-the-art performance in most cases. It consistently
presents significant improvements compared to the ProtoNet baseline on which
it is built, across a range of benchmarks, from appearance-dominated video
datasets such as HMDB, UCF, and Kinetics, to temporal-sensitive datasets like
SSv2. In fact, VIM achieves the best 1-shot and 5-shot performance on UCF,
HMDB, and Kinetics datasets when compared to state-of-the-art methods.
Notably, the improvements in 1-shot performance are the most significant. To
our knowledge, VIM is the first method to improve the performance of UCF under
the 1-shot setting to 86+ and HMDB under the 1-shot setting to 61+. This
observation highlights VIM’s ability to learn more information for recognition
when fewer samples are available. It’s also worth noting that under the 5-shot
setting, VIM yields a performance of 54.9 on SSv2, which is behind TRX trx2021
and HCL HCL . Their superiority in many-shot can be attributed to the use of
attention mechanisms for ensemble and feature enhancement over samples within
the same category. However, this characteristic also imposes limitations on
their performance in 1-shot settings.

####  4.3.2 Effectiveness on untrimmed & long-term video

Current few-shot action recognition methods only conduct evaluations on short-
term videos (e.g., UCF, SSv2). In this paper, we aim to benchmark our VIM
against current methods on a long-term video dataset ActivityNet activity2015
, which has not been done before. Following the standard few-shot data split,
we randomly selected 100 classes from the whole dataset. The 100 classes are
then split into 64, 12, and 24 classes as the meta-training, meta-validation,
and meta-testing sets, respectively. 70 samples are randomly selected for each
class. Table 2 summarizes the experimental results on ActivityNet. We observe
that the existing state-of-the-art methods cannot achieve satisfactory
improvements on long-term videos compared to the simple baseline (ProtoNet),
especially on the 1-shot and 3-shot. This may be due to the attenuation of
critical actions caused by uniform sampling, which becomes more severe in
long-term videos. Moreover, each video in ActivityNet may contain multiple
action segments, further increasing the difficulty of metric learning. In
contrast, our VIM model outperforms other methods in all cases for
ActivityNet, highlighting the importance of considering both intra-video and
inter-video information for achieving optimal performance in long-term action
videos, particularly the intra-video information.

####  4.3.3 Extension to other methods

As illustrated in Fig. 3, our proposed VIM does not modify the backbone
architecture or metrics. As a result, it can be easily incorporated into
existing few-shot action recognition methods. To further evaluate the
effectiveness of VIM, we extend it into three existing approaches: OTAM
otam2020 , TRX trx2021 , and HyRSM HyRSM2022 . We implement these extensions
based on their official code 222https://github.com/tobyperrett/trx
333https://github.com/alibaba-mmai-research/HyRSM. The detailed modifications
of these extensions compared to vanilla VIM are as follows:

  * •

OTAM + VIM: It replaces the cosine distance metric with the Dynamic Time
Warping (DTW) metric.

  * •

TRX + VIM: It conducts an exhaustive comparison among videos based on frame
tuples and triplets.

  * •

HyRSM + VIM: Following the paradigm of HyRSM, it performs an attention
mechanism over all videos within the support set and query set. It also
replaces the cosine distance metric in VIM with the bidirectional Mean
Hausdorff Metric (Bi-MHM).

The results are summarized in Table 3. It can be seen that our VIM can still
greatly promote these powerful methods. Although TRX trx2021  and HyRSM
HyRSM2022  are currently considered the best methods in the 5-shot setting,
incorporating VIM with them still yields stable improvements in most 5-shot
cases. Furthermore, our VIM significantly improved their less-than-ideal
performances in the 1-shot setting across all datasets by a significant
margin.

Extension |  | OTAM otam2020
---
+VIM
| TRX trx2021
---
+VIM
| HyRSM HyRSM2022
---
+VIM
HMDB | 1-shot | 59.3 (4.8$\uparrow$) | 55.5 (2.4$\uparrow$) | 60.9 (0.6$\uparrow$)
5-shot | 74.2 (8.1$\uparrow$) | 76.3 (0.7$\uparrow$) | 75.6 (0.4$\downarrow$)
UCF101 | 1-shot | 83.6 (3.7$\uparrow$) | 83.7 (5.3$\uparrow$) | 85.3 (1.4$\uparrow$)
5-shot | 94.5 (5.6$\uparrow$) | 95.6 (0.5$\downarrow$) | 95.2 (0.5$\uparrow$)
Kinetics | 1-shot | 74.3 (0.7$\uparrow$) | 68.8 (5.2$\uparrow$) | 73.9 (0.2$\uparrow$)
5-shot | 86.1 (0.5$\uparrow$) | 86.0 (0.1$\uparrow$) | 86.6 (0.5$\uparrow$)
SSv2 | 1-shot | 38.9 (2.5$\uparrow$) | 39.6 (3.6$\uparrow$) | 41.0 (0.4$\uparrow$)
5-shot | 51.4 (3.4$\uparrow$) | 58.0 (1.1$\downarrow$) | 56.8 (0.7$\uparrow$)
ActivityNet | 1-shot | 75.4 (5.5$\uparrow$) | 72.2 (2.3$\uparrow$) | 73.8 (1.7$\uparrow$)
5-shot | 87.7 (2.5$\uparrow$) | 89.9 (1.2$\uparrow$) | 88.1 (0.6$\uparrow$)

Table 3: Extension of VIM to other methods. ${\color[rgb]{0,0,1}\uparrow}$
indicates the improvement compared to the original OTAM/TRX/HyRSM, while
${\color[rgb]{1,0,0}\downarrow}$ means decline.

###  4.4 Ablation study

####  4.4.1 Effectiveness of main components of VIM

We first conduct an ablation study where we isolate the core components (video
sampler, action alignment, $\mathcal{L}_{intra}$, and $\mathcal{L}_{inter}$ )
of VIM and quantify their effectiveness in few-shot action recognition.
Results are summarized in Table 4. As we observe, each component yields a
stable improvement over the baseline. Specifically, our experiments began by
only adopting the video sampler to process all input video data. This approach
demonstrated substantial improvements over the baseline on all datasets,
particularly on the Kinetics and HMDB datasets. Importantly, since this
setting does not apply any additional operations or modules on the feature
level compared to the baseline model, it showcases that our spatial-temporal
video sampler can improve the intra-video utilization efficiency at the data
level, thereby boosting few-shot recognition performance.

Video Sampler |  Action Alignment | $\mathcal{L}_{intra}$ | $\mathcal{L}_{inter}$ | HMDB51 | UCF101 | SSv2 | Kinetics
---|---|---|---|---|---|---|---
1-shot | 5-shot | 1-shot | 5-shot | 1-shot | 5-shot | 1-shot | 5-shot
|  |  |  | 54.2 | 68.4 | 78.7 | 89.6 | 34.2 | 46.5 | 64.5 | 77.9
✓ |  |  |  | 59.0 | 72.8 | 82.5 | 93.6 | 36.8 | 51.2 | 71.0 | 84.8
✓ |  | ✓ |  | 59.6 | 73.2 | 83.4 | 94.3 | 37.8 | 52.8 | 72.1 | 84.9
| ✓ |  |  | 59.7 | 73.9 | 82.8 | 95.1 | 38.6 | 53.1 | 72.8 | 85.8
| ✓ |  | ✓ | 60.4 | 75.6 | 85.4 | 95.7 | 39.6 | 53.7 | 73.0 | 86.2
✓ | ✓ |  |  | 59.9 | 73.5 | 83.5 | 96.0 | 38.9 | 53.8 | 73.6 | 86.2
✓ | ✓ | ✓ |  | 60.6 | 74.2 | 84.6 | 95.8 | 39.0 | 54.1 | 73.6 | 86.7
✓ | ✓ |  | ✓ | 60.8 | 76.0 | 83.5 | 96.0 | 40.3 | 54.3 | 73.4 | 86.7
✓ | ✓ | ✓ | ✓ | 61.1 | 76.5 | 86.2 | 96.1 | 40.8 | 54.9 | 73.8 | 87.1
Table 4: Ablation study on the effectiveness of main components of VIM.
✓indicates the component is adopted.

We then individually evaluate the action alignment model by replacing the
video sampler with a normal uniform sampling strategy. The results demonstrate
that the action alignment model contributes the most to performance
improvement, particularly on the SSv2 dataset (surpassing the baseline by 4.4%
accuracy). It demonstrates that the alignment among actions is critical for
few-shot action recognition. The reason for this is that classification in the
metric-learning paradigm is highly reliant on the accurate measurement of
inter-video information, such as feature distance or similarity. Additionally,
we observe that combining the video sampler and action alignment model
produces the best performance compared to a single application of them,
highlighting the importance of jointly considering intra- and inter-video
information maximization in few-shot action recognition.

Finally, we examined the effectiveness of our designed loss terms. Apart from
the traditional classification loss, we add the $\mathcal{L}_{intra}$ and
$\mathcal{L}_{inter}$ to optimize the video sampler and action alignment
during training, respectively. We observe a stable improvement in performance
using any of them. Meanwhile, when training our full VIM model with these two
losses, it yields the best performance over all settings and baselines. These
results prove that explicit guidance can facilitate the learning of input
sampling and feature alignment, especially when only limited training data is
available. Besides, it also proves that our devised losses do encourage our
VIM framework to optimize towards intra- and inter-video information
maximization.

Video Sampler | Action Alignment | HMDB | SSv2
---|---|---|---
TS | SA | Ada | TC | SC
✓ |  |  |  |  | 58.1 | 36.9
| ✓ |  |  |  | 57.3 | 35.7
✓ | ✓ |  |  |  | 59.1 | 37.5
✓ |  | ✓ |  |  | 58.7 | 37.6
| ✓ | ✓ |  |  | 57.6 | 36.1
✓ | ✓ | ✓ |  |  | 59.6 | 37.8
|  |  | ✓ |  | 58.8 | 37.9
|  |  |  | ✓ | 58.3 | 36.0
|  |  | ✓ | ✓ | 60.4 | 39.6
✓ |  | ✓ | ✓ | ✓ | 60.0 | 40.5
| ✓ | ✓ | ✓ | ✓ | 60.9 | 40.2
✓ | ✓ | ✓ | ✓ |  | 60.7 | 39.1
✓ | ✓ | ✓ |  | ✓ | 60.4 | 37.6
✓ | ✓ | ✓ | ✓ | ✓ | 61.1 | 40.8
Table 5: Breakdown study of each module. Results are reported under 1-shot
setting on HMDB and SSv2 datasets. TS: temporal selector, SA: spatial
amplifier, Ada: task-adaptive learner, TC: temporal coordination, SC: spatial
coordination. ✓indicates the module is adopted.

####  4.4.2 Breakdown analysis on each module

In this section, we further break down all the modules of each component in
VIM to analyze their impact on few-shot action recognition performance. Table
5 summarizes the results. In terms of the video sampler, we conduct
experiments with different combinations of its three modules: temporal
selector (TS), spatial amplifier (SA), and task-adaptive learner (Ada). As we
can see, the TS brings significant gains on HMDB and SSv2, indicating that
conducting temporal selection is crucial for few-shot action recognition,
which is also consistent with the conclusions derived from general action
recognition adaframe2018 ; adafocus2021 . Besides, we also observe that the SA
works better on HMDB than SSv2 dataset. Since the videos of SSv2 are first-
person scenes and the main object in frames are human hands, amplifying these
hands leads to limited improvement in recognition. Further, by introducing
task adaptation for TS and SA, they were able to reach their peak
performances. This demonstrates that dynamic sampling strategies among
episodic tasks are indeed beneficial and essential for few-shot action
recognition.

As for the action alignment, we conduct experiments by first applying temporal
coordination (TC) and spatial coordination (SC) separately. We observe that
both of them can boost the performance of few-shot action recognition since
each dimension plays an equally important role in action alignment. When TC
and SC are applied in a sequential manner, the performance can be further
improved, thereby supporting our notion of a spatiotemporal alignment design.

Finally, we further test different combinations of modules from both the video
sampler and action alignment. To our expectation, equipping all the modules
yields the best performance for VIM, fully justifying the relevance of each
module.

Figure 11: t-SNE feature embedding of all videos w/o and with our proposed
sampler. Reported using all videos in a 5-way 30-shot episode on ActivityNet.

Figure 12: t-SNE feature embedding of query videos (circles) and support
prototypes (triangles) w/o and with applying our alignment. Reported in a
4-way 20-shot episode on HMDB.

####  4.4.3 t-SNE visualization

To illustrate the effect of different components in our VIM more intuitively,
we visualize the feature embedding of videos under the following
circumstances: (1) with and w/o adaptive sampling by our sampler, (2) with and
w/o applying our action alignment. The results are presented in Fig. 12 and
Fig. 12, respectively. We can see that each cluster appears more concentrated
after both adaptive sampling and action alignment. It demonstrates that the
sampler and alignment can facilitate better feature metric learning. Moreover,
for action alignment, each cluster is placed closer to its prototype after
performing the alignment. It further proves that our action alignment model
can well-align the query videos to support videos, obtaining a more consistent
feature representation.

####  4.4.4 Analysis of video sampler

Frequency of scanning. To ensure that the Scan Network in the sampler can
cover more frames in the videos, we set $M=20$ by default to provide a trade-
off between computation and performance. To investigate the effect of video
scanning frequency in our sampler, we adjust different scanning frequencies
(i.e., value of M) for experiments. Results are shown in Table 6. Directly
increasing the frequency cannot guarantee a consistent improvement to the
baseline due to the existence of redundancy. When enlarging $M$ from 14 to 20,
the performance on UCF further improves and peaks at M=20, while it brings no
additional gains on the SSv2 dataset. The difference between UCF and SSv2 can
be attributed to the fact that videos in SSv2 are much shorter than in UCF.
When $M=32$, there exists a large decrease in the performance of UCF and SSv2.
The reason may be that learning an ideal subset selection from too many
candidates is difficult using only a few training samples.

M = | 14 | 16 | 18 | 20 | 24 | 28 | 32
---|---|---|---|---|---|---|---
UCF101 | 84.8 | 85.9 | 85.9 | 86.2 | 85.5 | 84.6 | 83.9
SSv2 | 40.8 | 40.4 | 39.8 | 39.7 | 39.4 | 38.6 | 38.3

Table 6: Ablation of frequency of dense sampling, results are reported under 5-way 1-shot setting and $T=8$. T = | 4 | 6 | 8 | 10 | 12 | 14
---|---|---|---|---|---|---
UCF101 | 84.9 | 85.3 | 86.2 | 86.4 | 86.2 | 86.7
SSv2 | 37.8 | 39.3 | 40.8 | 40.9 | 41.2 | 41.0
Table 7: Ablation of the number of sampled frames, reported under 5-way 1-shot
setting and $M=20$. Note that we only report the results under $T=8$ in Table
1 for fair comparisons, although it’s not necessarily the optimal setting.

Scanning input frame size | UCF101 | SSv2
---|---|---
1-shot | 5-shot | 1-shot | 5-shot
32$\times$32 | 85.1 | 94.8 | 39.6 | 54.1
64$\times$64 (default) | 86.2 | 96.1 | 40.8 | 54.9
128$\times$128 | 86.5 | 96.3 | 40.8 | 54.7
224$\times$224 | 86.5 | 96.4 | 41.1 | 54.9

Table 8: Results of different frame size for scanning

How many frames do we need? For fair comparisons, we follow existing methods
to sample $T=8$ frames as input for few-shot learners to report our results in
Table 1. By virtue of our sampler, increasing the number of sampled frames is
likely to introduce more informative frames for recognition instead of
interference. We adjust the value of $T$ from $4$ to $14$ to explore this
issue in Table 7. For SSv2, the performance grows with $T$ and peaks at
$T=12$. However, its performance slightly drops when $T$ increases to 14. For
UCF101, it is clear that increasing the number of sampled frames can
consistently boost performance. On the contrary, when reducing the frame
number to $4$, the SSv2 suffers a significant drop in performance while UCF101
only drops slightly. This is consistent with the observation that SSv2 is more
sensitive to temporal information.

Scan Network | Params | UCF101
---|---|---
1-shot | 5-shot
ShuffleNetV2 0.5$\times$ (default) | 1.4M | 86.2 | 96.1
ShuffleNetV2 1.5$\times$ | 3.5M | 86.3 | 95.9
ShuffleNetV1 2.0$\times$ | 5.4M | 85.9 | 95.5
MobileNetV2 | 3.4M | 86.1 | 96.1
0.75 $\times$ MobileNetV1 | 2.6M | 85.5 | 95.4
Table 9: Different model selection for scanning net. Style | UCF101 | SSv2
---|---|---
1-shot | 5-shot | 1-shot | 5-shot
None | 84.8 | 94.2 | 40.5 | 54.2
Amplifying (Ours) | 86.2 | 96.1 | 40.8 | 54.9
Cropping | 84.7 | 93.3 | 40.6 | 53.7
Table 10: Results of different solutions for spatial sampling in VIM. ‘None’
means no spatial operation is performed over video sampling.

Scanning input frame size. In our implementation, we first downsample each
frame to a relatively smaller size before feeding it into the scan network to
save computational costs. In this part, we study how the input size of the
scan network affects the performance of our sampler. Experimental results are
shown in Table 8. Increasing the input size to $224\times 224$ only improves
the overall performance by a slight margin, but it incurs significantly higher
computational costs. Meanwhile, the performance drops rapidly using our
smallest choice of $32\times 32$ for inputs. Therefore, we adopt $64\times 64$
as the default input size for the scan network to provide an optimum trade-off
between efficiency and performance in practice.

Model selection of sampler. To save computational costs and speed up the
scanning, we select ShuffleNet-V2 as our scan network. In this part, we
further study the impact of employing different scanning models444We adopt
public models from https://github.com/megvii-model/ShuffleNet-Series. Results
are summarized in Table 9. It can be seen that all of them share almost
similar performances. Specifically, the ShuffleNet surpasses the MobileNet by
a slight margin. From this, we select the ShuffleNet-V2 0.5×, which has
minimal FLOPs and parameters, to build our scan network.

Spatial amplifying or cropping? In image classification, there exists another
popular solution for spatial sampling, which is to locate a sub-region and
then crop it from the original image crop1 ; racnn2017 ; adafocusv2 . However,
this operation may not be ideal for few-shot learning since it reduces the
data utilization efficiency in the spatial dimension. Moreover, it requires
setting a fixed size for the sub-region to achieve differentiable
implementation and often involves multi-stage training. In contrast, our
spatial amplifier can emphasize the discriminative region while maintaining
most information in the whole image. Besides, our implementation is more
flexible to different degrees of amplification. To gain further insight, we
substitute our SA with a crop-based solution in adafocusv2 . In the summary of
results shown in Table 10, we observe that the crop-based solution only
improves the 1-shot performance on the SSv2 dataset by a slight margin, while
it leads to declines in other cases. This may be due to the removal of
valuable recognition information from the original image during cropping. In
contrast, our amplifier brings promising improvements in all settings. These
findings indicate that our spatial amplification is more suitable for the few-
shot circumstance, which preserves and emphasizes more intra-video information
in the spatial dimension.

Models | UCF101 | SSv2
---|---|---
ProtoNet | 78.7 | 34.2
ProtoNet + MGSampler mgsampler  | 80.5 | 36.8
ProtoNet + Our TS | 82.6 | 37.6
ProtoNet + Our Sampler (with $\mathcal{L}_{\text{intra}}$) | 83.4 | 37.8
Table 11: Comparison with existing frame sampler in general action
recognition. Reported under 1-shot. Figure 13: t-SNE visualization of task-
specific weights. Different tasks are indicated by different colors.

(a) Cliffdiving.

(b) BenchPress.

Figure 14: Visualization results of the temporal selector on two selected
action classes. The green bar indicates that our sampler selects the frame.
Figure 15: Histogram of sampled frame locations on the videos of action
‘CliffDiving’ and ‘BenchPress’. Reported with $T=8$ and $M=20$, using 50
videos per class.

Comparison to other sampling approaches. Most existing frame selection methods
cannot be directly applied to address our issues (fixed-size subset
selection). To this end, we apply the recently proposed MGSampler mgsampler
to few-shot action recognition, which can be regarded as an independent pre-
processing operation for video frame sampling. Specifically, it first
calculates the feature difference between adjacent frames to estimate the
motion process for the whole video. Then, frame selection is performed based
on this estimated motion information. In this way, we feed the $T=8$ frames
selected by MGSampler based on the calculated motion difference among $M=20$
frames into ProtoNet to report its performance. As shown in Table 11, the
MGSampler can also improve the baselines, especially for SSv2 dataset.
However, our single temporal selector (TS) still outperforms it on both
datasets. Further, when our complete sampler (i.e., TS + SA) is adopted, we
surpass its performance by a significant margin. The above results prove that
(1) spatial sampling is necessary and beneficial, especially for appearance-
dominated datasets like UCF and HMDB, (2) Directly applying current methods in
general action recognition is not ideal, learning specific sampling strategies
for few-shot action recognition is obviously a better solution.

Study on the task-adaptive learner. To gain further insight into our task-
adaptive learner, we visualize the generated dynamic parameters for different
tasks with t-SNE. Specifically, we randomly select three different tasks and
proceed to sample 7 task summary features that had undergone re-
parameterization (in Eq. 14). We then obtain 7 task-specific parameters for
$w_{2}\in\mathbb{R}^{d\times 1}$ by our generator. Finally, we visualize the
generated weights of these three tasks with t-SNE. In results presented in
Fig. 13, it can be observed that task-specific weights of the same task are
situated closely in feature space, while weights of different tasks appear to
have their very own distributions. It demonstrates that our task-adaptive
learner is capable of generating task-specific weights of layers in our
sampler network for effective sampling strategies.

Distributions of sampling. To gain more intuitive insights into temporal frame
selection, we focus on specific action classes that tend to follow a
predictable distribution over time. Specifically, we select the ‘CliffDiving’
and ‘BenchPress’ classes from UCF dataset. For videos belonging to these
categories, we keep count of the temporal locations of the frames selected by
TS and present the distribution in Fig. 15. We can observe that the
distribution varies between these two types of actions. For ‘CliffDiving,’ our
sampler tends to select frames from the middle of videos. This is consistent
with the observation that ‘CliffDiving’ videos typically contain noise frames
at the start (e.g., the preparation phase before diving) and the end of the
video (e.g., water splash). On the other hand, for the periodic action
‘BenchPress,’ our sampler also selects frames in a periodic manner, which is
consistent with the regular pattern of this action. To further illustrate
them, we visualize some results of frame selection in TS in Fig. 14. These
findings suggest that our sampler could effectively select frames containing
key actions for each video.

####  4.4.5 Analysis of action alignment

Design of the temporal alignment. For the action alignment, we first analyze
the effectiveness of our temporal coordination module by performing the
temporal transformation and re-arrangement modules separately. In the summary
of results in Table 12, we can see that applying the single Temporal
Transformation provides stable improvements across datasets ($\uparrow$ 1.7
accuracy on SSv2 and $\uparrow$ 3.8 on UCF101). The Temporal Rearrangement
module brings more significant improvements to the baseline performance
compared to the Temporal Transformation module. By applying both modules
sequentially, we can reach the best performance, demonstrating that these two
operations are essential to cope with different misalignment issues (duration
and evolution misalignment) in the temporal dimension.

Setting | UCF101 | SSv2
---|---|---
Baseline (ProtoNet) | 78.7 | 34.2
\+ Temporal Transformation | 82.5 | 35.9
\+ Temporal Re-arrangement | 83.8 | 37.3
\+ Both | 84.2 | 37.9
Table 12: Effectiveness of each module in TC, reported under 5-way 1-shot setting. Solution | UCF101 | HMDB
---|---|---
Enumerate | 84.9 | 59.6
Affine Transformation | 85.6 | 60.4
Mask-based (Our implementation) | 86.2 | 61.1
Table 13: Results of different implementation of SC in our VIM, reported under
5-way 1-shot setting.

Different designs for spatial alignment Furthermore, to reveal the
effectiveness of spatial coordination (SC) module design, we compare our
implementation with other alternative designs:

  * (1)

Enumerate: it enumerates through all possible integer offsets in
$\mathbb{Z}^{2}$, and straightly indexes the intersection area in feature maps
by $x,y$ coordinates. The distance between support and query features is
calculated within this intersection area. The offset leading to the minimum
distance is considered optimal. This exhaustive design can be regarded as a
simple baseline.

  * (2)

Affine Transformation: it generates a 2D grid (i.e., parameters that describe
an affine transformation) according to our predicted offset. Then it performs
corresponding affine transformation over the entire 2D feature map to obtain
features in the intersection area with re-sampling. (similar to STN STN2015 ).

We report their results on two appearance-dominated datasets: UCF101 and HMDB
in Table 13. As we can see, our implementation outperforms the simple
enumeration method by a great margin, which also demonstrates the ability of
our offset predictor in SC. Compared to the affine transformation
implementation, our alignment design is definitely more straightforward and
computationally tractable, with slightly better performance.

Per-category improvement. The improvement on the SSv2 dataset for each
category using the proposed alignment model is presented in Fig. 16. Note that
here we remove the adaptive sampling in VIM to analyze the gain from the
alignment independently. What stands out in this figure is that the
performance increases by a large margin for all categories. Moreover, some
categories’ accuracy (_e.g._ “pouring sth out of sth”, “approaching sth”,
“poking a stack of sth”) rose sharply ($>20\%$ improvement) using our
alignment. Interestingly, these action classes are also the ones that are most
vulnerable to the misalignment problem. This further proves the effectiveness
of our temporal alignment.

Figure 16: Class-specific improvement using our alignment model compared to the ProtoNet baseline in SSv2. Here we remove the adaptive sampling in VIM to analyze the gain from the action alignment independently. | Temporal first (default) | Spatial First
---|---|---
UCF | 86.2 | 82.9 ($\downarrow$3.3)
HMDB | 61.1 | 58.7 ($\downarrow$2.4)
Kinetics | 73.8 | 69.6 ($\downarrow$4.2)
SSv2 | 40.8 | 38.5 ($\downarrow$2.3)
Table 14: The results (1-shot) under different orders of alignment in VIM.

Temporal or spatial alignment first? We further investigate the effect of
alignment order by exchanging the order of the spatial coordination and
temporal coordination modules in VIM. The results are listed in Table 14,
which suggests that the performance is significantly affected by the order.
Notably, the VIM achieves the highest performance across all datasets when
temporal alignment is performed first, followed by spatial alignment.
Meanwhile, we also observe a sharp decline in performance on UCF and Kinetics
when spatial alignment is conducted first. This decline can be attributed to
the spatial coordination module’s reliance on temporally consistent frame
features to predict accurate spatial offsets. Based on these findings, we can
conclude that a preliminary temporal alignment is essential for successful
spatial alignment in VIM.

Comparison to sequence alignment methods. To validate the effectiveness and
superiority of our proposed alignment approach, we replace the action
alignment procedure in VIM with other popular sequence alignment algorithms:

  * •

Soft-DTW softdtw . It enables the Dynamic Time Warping (DTW) algorithm to be
differentiable for end-to-end training in neural networks.

  * •

Drop-DTW drop-dtw . To cope with the presence of outliers that can be
arbitrarily interspersed in the sequences, it performs alignment while
automatically dropping the outlier elements from the matching process.

  * •

VAVA align_wild . It learns to align sequential actions in the wild that
involve diverse temporal variations (e.g., background frames) by enforcing
temporal priors on the optimal transport matrix.

We present the results of replacing our alignment model with the above methods
in Table 15. Both of them can also relieve the misalignment issue and improve
performance. VAVA surpasses the Soft-DTW and Drop-DTW in most cases thanks to
its ability to handle noisy frames (e.g., background frames) during alignment.
It’s worth noting that these algorithms only align actions in the temporal
dimension. In contrast, our framework performs action alignment in both
temporal and spatial dimensions, resulting in the best overall performance
compared to the other methods. Additionally, utilizing
$\mathcal{L}_{\text{inter}}$ can further improve our performance, emphasizing
the importance of explicit guidance for alignment.

Action alignment in VIM | HMDB | UCF | Kinetics | SSv2
---|---|---|---|---
Soft-DTW softdtw  | 58.0 | 81.6 | 73.0 | 38.2
Drop-DTW drop-dtw  | 57.2 | 82.5 | 72.6 | 38.4
VAVA align_wild  | 58.9 | 83.5 | 73.2 | 37.8
Ours (w/o $\mathcal{L}_{\text{inter}}$) | 60.6 | 84.6 | 73.6 | 39.0
Ours (with $\mathcal{L}_{\text{inter}}$) | 61.1 | 86.2 | 73.8 | 40.8

Table 15: Results of replacing our proposed action alignment in VIM with other
alignment algorithms.  Figure 17: The alignment cost with different operations
on input videos. ‘SA only’ indicates that we only adopt spatial amplifier
only. ‘TS’ means we only perform temporal selector. Figure 18: Density of
saliency maps in spatial amplifier. A higher density indicates reduced
amplification.

####  4.4.6 Interaction between the sampler and alignment

Impact on alignment cost. We first delve into the effect of adaptive sampling
on action alignment. To quantify the alignment process, we calculate the
alignment cost for each input video as follows:

$\displaystyle\mathcal{C}_{\text{align}}=\underbrace{(|1-a|+|b|)+\lVert\mathbf{M}_{e}-I\rVert_{2}}_{\text{temporal
cost}}+\underbrace{\lVert O\rVert}_{\text{spatial cost}}$ (35)

where $(a,b)$ are the affine parameters for linear temporal transformation,
$T$ is the number of frames, $\mathbf{M}_{e}\in\mathbb{R}^{T\times T}$ denotes
the normalized evolution correlation matrix calculated in temporal re-
arrangement (Eq. 17), $I\in\mathbb{R}^{T\times T}$ is an identity matrix
(indicates two videos are frame-wise consistent), $O\in\mathbb{R}^{T\times 2}$
indicates the predicted spatial offset in Eq. 19 (here we normalize values in
$O$ into [-1,1]). In this context, a higher cost indicates a larger degree of
misalignment in the input video sequences. It is worth noting that the
alignment cost is determined based on the estimated parameters of our
alignment model, rather than an exact measurement. Our results are presented
with the mean value of $\mathcal{C}_{\text{align}}$, both with and without
adaptive video sampling, and are summarized in Figure 17. The findings
demonstrate that incorporating the sampler significantly reduces the overall
alignment cost of our action alignment model. Furthermore, it is observed that
the temporal alignment cost slightly increases when we only employ the spatial
amplifier.

Impact on sampling strategy. We also investigate how the feature-level
alignment operation impacts the behavior of the video sampler. To quantify the
behavior of our sampler, we compute the density (values between 0$\sim$1) of
generated 2D saliency map $\mathbf{M}_{s}$ (Eq. 8). In our spatial amplifier,
a higher density corresponds to reduced amplification. For example, a density
of 1 implies equal importance for all pixels, and therefore no local region
will be amplified. As seen in Fig. 18, there is a slight increase in density
when the alignment operation is incorporated at the feature level. This
suggests that the presence of spatial alignment allows for a less optimal
solution for amplification, as the critical regions will be further aligned at
the feature level. On the other hand, the sampler is required to provide more
precise and concentrated amplification (i.e., effectively highlighting the
actor region) as input to subsequent networks when no alignment of spatial
features is present. Consequently, the existence of the alignment can
alleviate the training difficulty faced by the video sampler.

Balance between intra- and inter-video information. As presented in Eq. 34,
the coefficients of $\mathcal{L}_{\text{intra}}$ and
$\mathcal{L}_{\text{inter}}$ control the preference of VIM for learning intra-
and inter-video information during training, i.e., balance the optimization of
the sampler and alignment. To evaluate the impact of these weights, we perform
experiments using different ratios of $\mu_{1},\mu_{2}$. Specifically, we set
the values for $\mu_{1},\mu_{2}$ to
$\\{1,0\\},\\{1,0.5\\},\\{0.5,1\\},\\{1,1\\}$, and $\\{0,1\\}$, and train our
VIM on each dataset using these pairs of coefficients. The results are
presented in Table 16. Disabling either loss term (i.e., setting its
coefficient to 0) leads to a rapid drop in performance. When treating them
equally during training, we can already reach an ideal performance. However,
scaling the ratio of $\mathcal{L}_{\text{inter}}$ and
$\mathcal{L}_{\text{intra}}$ to 2 (i.e., $\mu_{1}=0.5,\mu_{2}=1$) further
boosts the performance on both UCF101 and SSv2 datasets, which demonstrates
that the inter-video information may be more valuable for the final
recognition in our VIM.

Coefficient of $\mathcal{L}_{\text{intra}}$ and $\mathcal{L}_{\text{inter}}$ | UCF | SSv2
---|---|---
$\mu_{1}=1,~{}\mu_{2}=0$ | 84.6 | 39.0
$\mu_{1}=0,~{}\mu_{2}=1$ | 83.5 | 40.3
$\mu_{1}=1,~{}\mu_{2}=0.5$ | 85.1 | 40.1
$\mu_{1}=0.5,~{}\mu_{2}=1$ | 86.2 | 40.8
$\mu_{1}=1,~{}\mu_{2}=1$ | 85.8 | 40.5
Table 16: The results of VIM under different coefficients of
$\mathcal{L}_{\text{intra}}$ and $\mathcal{L}_{\text{inter}}$.

Setting | UCF | HMDB | Kinetics | SSv2
---|---|---|---|---
VIM | 86.2 | 61.6 | 73.8 | 40.8
Remove sampler during testing | 84.7 | 58.8 | 71.9 | 37.7
Remove alignment during testing | 82.6 | 58.1 | 71.1 | 34.3
Individual sampler | 83.4 | 59.6 | 72.1 | 36.8
Individual alignment | 85.4 | 60.4 | 73.0 | 39.6

Table 17: Results of test-time isolation of VIM.

Test-time isolation. We further conducted experiments to analyze the
interdependence between the sampler and alignment modules in our VIM. We
removed either the sampler or alignment module during test time while keeping
both during training, a process we call “test-time isolation”. The results are
presented in Table 17, where we observe a significant drop in the performance
of VIM when either module was removed during testing. Most importantly, the
performance of the test-time isolated models is also lower than that of
individually trained sampler or alignment models (refer to Table 4). These
findings indicate that optimizing the sampler and alignment jointly in our VIM
does encourage them to work together and be deeply coupled, rather than a
simple incremental stack.

Models | Setting | UCF101 | SSv2
---|---|---|---
Number | Resolution
ProtoNet | 8 | 224$\times$224 | 78.7 | 34.2
8 | 448$\times$448 | 79.9 | 34.7
20 | 224$\times$224 | 80.8 | 35.6
20 | 448$\times$448 | 82.1 | 36.3
\+ Our sampler | 8 | 224$\times$224 | 83.4 | 37.8
Table 18: Comparison with the estimated upper bound of intra-video
information. Results are reported under 1-shot setting.

Model | Setting | UCF | SSv2
---|---|---|---
ProtoNet | Baseline | 78.7 | 34.2
\+ Temporal Exhaustive | 82.3 | 36.5
\+ Spatial Exhaustive | 82.7 | 35.3
\+ Both | 83.9 | 37.1
\+ Our Temporal Coordination | 84.2 | 37.9
\+ Our Spatial Coordination | 83.3 | 36.0
\+ Both (Our complete alignment) | 85.4 | 39.6

Table 19: Comparison with the estimated upper bound of inter-video
information. Results are reported under 1-shot setting.

Figure 19: Results of the limit test. “standard” indicates the standard few-
shot data split.

Figure 20: Results of many-way 1-shot setting. The number of ways ranges from
5 to 10.

####  4.4.7 Probe the upper bound of video information

Intra-video information. As previously discussed, increasing the number and
resolution of input frames can directly improve the coverage and utilization
of video data for few-shot learners. To further investigate the effectiveness
of our sampler, we follow this naive solution to estimate a rough upper bound
of intra-video information at the data level. Results are summarized in Table
18. Although this approach incurs significant GPU memory and computational
costs, we can observe consistent improvements by increasing the frame number
and resolution. Specifically, the performance under 5-way 1-shot with $T=20$
and $H=W=448$ (the max values we can afford for training) can be considered an
upper bound of intra-video information at the data level. However, when
integrating the baseline with our proposed sampler, its performance can
surpass the upper bound by a significant margin without enlarging the frame
number or resolution. This also demonstrates that this naive solution does
work but is not ideal compared to our adaptive sampler.

Inter-video information. The rough upper bound of inter-video information can
be estimated in an exhaustive alignment manner. As for the temporal dimension,
we imitate TRX trx2021  to construct exhaustive representations of
pairs/triplets of frames following the temporal order. The metric learning
between two video features is conducted over all possible representations, and
we calculate a mean value as the final distance. As for the spatial dimension,
we adopt the enumerate design (as described in the 2nd paragraph of Sec.
4.4.5) to estimate the upper bound results. Moreover, we add more non-integer
offsets by interpolating between the possible integer offsets to cover more
candidates. In the same way, after calculating metric distances with all
enumerated offsets, we regard the minimum distance as the final result. The
results of the above exhaustive alignment and our alignment model are
summarized in Table 19. Exhaustive alignment provides stable improvements
across datasets, especially on UCF101, as seen in our results. However, our
proposed alignment still outperforms it in all cases, demonstrating that our
alignment model can find good alignments using a limited amount of training
data. Although the improvements provided by spatial exhaustive alignment are
comparable to our spatial consistency (SC) method, our SC is more flexible and
learnable.

####  4.4.8 Limit test with extremely fewer data

To further validate the effectiveness of our VIM, we conducted a limit test
using an extremely limited amount of video data for training. In standard few-
shot data splits, each action category typically contains approximately
100-130 samples for training. In the limit test, we reduced this number
further to 50 and 20 to analyze the performance of our VIM and other methods
under extremely challenging circumstances. The results are summarized in Fig.
20. Our VIM still delivered significant improvements over the baseline in
these extreme circumstances. Although the HyRSM HyRSM2022  performs comparably
to our VIM in Kinetics and SSv2 datasets under standard settings, our VIM
outperforms it by a remarkable margin when training with limited data. These
findings indicate that our VIM is more robust and effective than existing
methods when faced with extremely limited training data.

####  4.4.9 Many-way few-shot

We have extended the conventional 5-way few-shot setting, which consists of 5
categories in each episode, to a more adaptable many-way few-shot setting. In
Fig. 20, we present the 1-shot results from 6-way to 10-way. It is evident
that as the number of ways increases, it becomes more challenging for the
classifier to distinguish among more action categories, resulting in a decline
in performance. Nevertheless, our proposed VIM consistently surpasses current
state-of-the-art methods such as HyRSM HyRSM2022  and OTAM otam2020 . These
results highlight the superiority of our VIM in handling diverse scenarios,
enabling it to generalize effectively.

| Training
---
domain
| Test
---
domain
ProtoNet | OTAM otam2020  | HyRSM HyRSM2022  | VIM
UCF | UCF | 78.7 | 79.9 | 83.9 | 86.2
HMDB | HMDB | 54.2 | 54.5 | 60.3 | 61.1
SSv2 | SSv2 | 34.2 | 36.4 | 40.6 | 40.8
Kinetics | UCF | 78.5 | 79.7 | 83.1 | 83.5
Kinetics | HMDB | 53.9 | 54.0 | 56.7 | 58.1
Kinetics | SSv2 | 31.6 | 30.6 | 32.0 | 32.4

Figure 21: The ability of domain generalization. Results are reported under
1-shot setting.

Model | Params | Inference time | Performance
---|---|---|---
ProtoNet | 25.8 M | 0.67 s/task | 78.7
OTAM otam2020  | 25.8 M | 1.29 s/task | 79.9
TRX trx2021  | 47.1 M | 0.92 s/task | 78.6
HyRSM HyRSM2022  | 65.6 M | 0.89 s/task | 83.9
VIM (Ours) | 58.2 M | 0.85 s/task | 86.2
Figure 22: Complexity analysis. The results are reported under 5-way 1-shot
setting on UCF101 dataset. Inference time is tested on a single NVIDIA V100
GPU.

Figure 23: Visualization results of sampler in our VIM in three sample episode
tasks from UCF and SSv2 datasets. The 1st row: frames selected by Temporal
Selector (TS), the 2nd row: saliency maps, the 3rd row: frames amplified by
Spatial Amplifier (SA). 2-way 1-shot is illustrated here.  Figure 24:
Visualization results of action alignment on SSv2. Action duration (highlight
with blue boxes) is obtained by predicted warp parameters in Temporal
Coordination (TC). Action-specific spatial region (highlight with red box) is
located with predicted offset in Spatial Coordination (SC).

(a)

(b)

Figure 25: Visualization results of the complete VIM on selected videos on
UCF101 and SSv2 datasets. Selected frames are indicated by green bar, while
other results are presented in the same way as Fig. 23 and Fig. 24.

####  4.4.10 Complexity analysis

We further analyze the complexity and overhead of our VIM. As shown in Fig.
22, although our VIM introduces additional parameters (32.4M) to the baseline
(ProtoNet), our inference speed is still fast compared to current state-of-
the-art methods. In particular, OTAM otam2020  incurs heavy time costs due to
its dynamic programming operation. TRX trx2021  requires a temporally
exhaustive feature comparison, leading to a relatively slow inference speed.
HyRSM HyRSM2022  has the largest number of parameters as it includes many
high-dimensional linear transformation layers. In comparison, our VIM model is
faster and achieves the best performance.

####  4.4.11 Ability of domain generalization

To evaluate the domain generalization ability of VIM, we investigated whether
few-shot learners trained on a single dataset could be generalized to other
domains without additional fine-tuning. The results are summarized in Fig. 22.
They show that our Kinetics pre-trained VIM can generalize well to other
domains and achieve ideal performance, particularly on UCF and HMDB datasets.
Moreover, we observed that this domain transfer paradigm could achieve
comparable performance to standard training for various methods in the UCF and
HMDB datasets. This can be attributed to the similarity in domain knowledge
between the Kinetics dataset and UCF and HMDB. In contrast, due to the
significant domain gap between Kinetics and SSv2, the improvement from
Kinetics to SSv2 is limited, and its performance is significantly behind that
of standard training. However, our VIM still provides the largest improvement
under this circumstance compared to other methods.

####  4.4.12 Visualization results

Video sampler. We first visualize the temporal-spatial sampling results for
videos in Fig. 23. We observe that TS can select frames containing crucial and
complete action processes. Moreover, our generated salience map complements by
indicating the important spatial regions for each frame. Accordingly, as the
results of SA show, these regions with higher saliency are well amplified and
emphasized, which in turn, makes them easier to be recognized. Meanwhile, the
amplified images still maintain relatively complete information of the
original ones. In summary, the visualization tellingly depicts the
effectiveness of our sampling strategy in temporal and spatial dimensions.

Action alignment. We further visualize the alignment results to illustrate the
effectiveness of our proposed method, which are presented in Fig. 24. It can
be observed that there exists a clear duration and spatial-temporal evolution
misalignment between the support and query videos. It’s clear that the
duration is well-aligned by the TC, which filters the insignificant background
frame noise. Besides, the spatial regions coordinated by SC focus on the
common action-specific part between paired frames. For example, the region of
hand (row 3 col 2) and the action-specific object ‘cup’ (row 3 col 1) can be
located precisely, which leads to a well-aligned spatial evolution for videos
before being compared.

Complete VIM. Finally, we visualize the results of complete VIM in Fig. 25. It
can be seen that the video sampler could preserve and emphasize the
informative parts of videos, then actions in paired videos are further aligned
in both spatial and temporal dimensions. This observation proves that adaptive
sampling and action alignment steps are complementary to each other, which
works hand-in-hand to maximize the intra- and inter-video information in our
VIM framework.

##  5 Conclusion

This paper presents a novel framework called Video Information Maximization
(VIM) for few-shot action recognition. VIM aims to maximize both intra- and
inter-video information by utilizing adaptive sampling and action alignment,
respectively. Adaptive sampling is achieved through a video sampler that
selects important frames and amplifies critical spatial regions based on the
task at hand. It is able to preserve and emphasize informative spatiotemporal
parts of video clips, while eliminating interference in the input data level.
The action alignment performs temporal and spatial alignment sequentially at
the feature level, resulting in a more precise measurement of inter-video
similarity. To further optimize VIM, the framework incorporates auxiliary loss
terms based on mutual information measurement during training, which provide
concrete guidance for intra- and inter-video information learning.
Experimental results demonstrate that VIM effectively improves few-shot
recognition performance and achieves state-of-the-art results in most cases.

## References

  * (1) Bachman, P., Hjelm, R.D., Buchwalter, W.: Learning representations by maximizing mutual information across views.  Advances in neural information processing systems 32 (2019)
  * (2) Belghazi, M.I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Courville, A., Hjelm, D.: Mutual information neural estimation.  In: International conference on machine learning, pp. 531–540. PMLR (2018)
  * (3) Beraha, M., Metelli, A.M., Papini, M., Tirinzoni, A., Restelli, M.: Feature selection via mutual information: New theoretical insights.  In: 2019 international joint conference on neural networks (IJCNN), pp. 1–9. IEEE (2019)
  * (4) Berthet, Q., Blondel, M., Teboul, O., Cuturi, M., Vert, J.P., Bach, F.: Learning with differentiable pertubed optimizers.  Advances in neural information processing systems 33, 9508–9519 (2020)
  * (5) Bishay, M., Zoumpourlis, G., Patras, I.: Tarn: Temporal attentive relation network for few-shot and zero-shot action recognition.  arXiv preprint arXiv:1907.09021 (2019)
  * (6) Caba Heilbron, F., Escorcia, V., Ghanem, B., Carlos Niebles, J.: Activitynet: A large-scale video benchmark for human activity understanding.  In: Proceedings of the ieee conference on computer vision and pattern recognition, pp. 961–970 (2015)
  * (7) Cao, K., Ji, J., Cao, Z., Chang, C.Y., Niebles, J.C.: Few-shot video classification via temporal alignment.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10618–10627 (2020)
  * (8) Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset.  In: proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299–6308 (2017)
  * (9) Cordonnier, J.B., Mahendran, A., Dosovitskiy, A., Weissenborn, D., Uszkoreit, J., Unterthiner, T.: Differentiable patch selection for image recognition.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2351–2360 (2021)
  * (10) Cubuk, E.D., Zoph, B., Mane, D., Vasudevan, V., Le, Q.V.: Autoaugment: Learning augmentation policies from data.  arXiv preprint arXiv:1805.09501 (2018)
  * (11) Cuturi, M., Blondel, M.: Soft-dtw: a differentiable loss function for time-series.  In: International conference on machine learning, pp. 894–903. PMLR (2017)
  * (12) Devroye, L.: Sample-based non-uniform random variate generation.  In: Proceedings of the 18th conference on Winter simulation, pp. 260–265 (1986)
  * (13) Dvornik, M., Hadji, I., Derpanis, K.G., Garg, A., Jepson, A.: Drop-dtw: Aligning common signal between sequences while dropping outliers.  Advances in Neural Information Processing Systems 34, 13782–13793 (2021)
  * (14) Edwards, H., Storkey, A.: Towards a neural statistician.  arXiv preprint arXiv:1606.02185 (2016)
  * (15) Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks.  In: International conference on machine learning, pp. 1126–1135. PMLR (2017)
  * (16) Fu, J., Zheng, H., Mei, T.: Look closer to see better: Recurrent attention convolutional neural network for fine-grained image recognition.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4438–4446 (2017)
  * (17) Fu, Y., Zhang, L., Wang, J., Fu, Y., Jiang, Y.G.: Depth guided adaptive meta-fusion network for few-shot video recognition.  In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 1142–1151 (2020)
  * (18) Ghodrati, A., Bejnordi, B.E., Habibian, A.: Frameexit: Conditional early exiting for efficient video recognition.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15608–15618 (2021)
  * (19) Goyal, R., Ebrahimi Kahou, S., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., et al.: The" something something" video database for learning and evaluating visual common sense.  In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5842–5850 (2017)
  * (20) Guo, B., Nixon, M.S.: Gait feature subset selection by mutual information.  IEEE Transactions on Systems, MAN, and Cybernetics-part a: Systems and Humans 39(1), 36–46 (2008)
  * (21) Hariharan, B., Girshick, R.: Low-shot visual recognition by shrinking and hallucinating features.  In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3018–3027 (2017)
  * (22) He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778 (2016)
  * (23) Hjelm, R.D., Fedorov, A., Lavoie-Marchildon, S., Grewal, K., Bachman, P., Trischler, A., Bengio, Y.: Learning deep representations by mutual information estimation and maximization.  arXiv preprint arXiv:1808.06670 (2018)
  * (24) Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K.: Spatial transformer networks.  arXiv preprint arXiv:1506.02025 (2015)
  * (25) Kingma, D.P., Welling, M.: Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114 (2013)
  * (26) Kong, Y., Fu, Y.: Human action recognition and prediction: A survey.  International Journal of Computer Vision 130(5), 1366–1401 (2022)
  * (27) Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., Serre, T.: Hmdb: a large video database for human motion recognition.  In: 2011 International conference on computer vision, pp. 2556–2563. IEEE (2011)
  * (28) Kumar Dwivedi, S., Gupta, V., Mitra, R., Ahmed, S., Jain, A.: Protogan: Towards few shot learning for action recognition.  In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 0–0 (2019)
  * (29) Li, H., Dong, W., Mei, X., Ma, C., Huang, F., Hu, B.G.: Lgm-net: Learning to generate matching networks for few-shot learning.  In: International conference on machine learning, pp. 3825–3834. PMLR (2019)
  * (30) Li, S., Liu, H., Qian, R., Li, Y., See, J., Fei, M., Yu, X., Lin, W.: Ta2n: Two-stage action alignment network for few-shot action recognition.  In: Proceedings of the AAAI Conference on Artificial Intelligence, 2, pp. 1404–1411 (2022)
  * (31) Lin, W., He, X., Dai, W., See, J., Shinde, T., Xiong, H., Duan, L.: Key-point sequence lossless compression for intelligent video analysis.  IEEE MultiMedia 27(3), 12–22 (2020)
  * (32) Lin, W., Liu, H., Liu, S., Li, Y., Qian, R., Wang, T., Xu, N., Xiong, H., Qi, G.J., Sebe, N.: Human in events: A large-scale benchmark for human-centric video analysis in complex events.  arXiv preprint arXiv:2005.04490 (2020)
  * (33) Liu, H., Li, J., Li, D., See, J., Lin, W.: Learning scale-consistent attention part network for fine-grained image recognition.  IEEE Transactions on Multimedia (2021)
  * (34) Liu, H., Lv, W., See, J., Lin, W.: Task-adaptive spatial-temporal video sampler for few-shot action recognition.  In: Proceedings of the 30th ACM International Conference on Multimedia, pp. 6230–6240 (2022)
  * (35) Liu, W., Tekin, B., Coskun, H., Vineet, V., Fua, P., Pollefeys, M.: Learning to align sequential actions in the wild.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2181–2191 (2022)
  * (36) Ma, N., Zhang, X., Zheng, H.T., Sun, J.: Shufflenet v2: Practical guidelines for efficient cnn architecture design.  In: Proceedings of the European conference on computer vision (ECCV), pp. 116–131 (2018)
  * (37) Meng, Y., Lin, C.C., Panda, R., Sattigeri, P., Karlinsky, L., Oliva, A., Saenko, K., Feris, R.: Ar-net: Adaptive frame resolution for efficient action recognition.  In: European Conference on Computer Vision, pp. 86–104. Springer (2020)
  * (38) Nguyen, K.D., Tran, Q.H., Nguyen, K., Hua, B.S., Nguyen, R.: Inductive and transductive few-shot video classification via appearance and temporal alignments.  In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XX, pp. 471–487. Springer (2022)
  * (39) Obeso, A.M., Benois-Pineau, J., Vázquez, M.S.G., Acosta, A.Á.R.: Visual vs internal attention mechanisms in deep neural networks for image classification and object detection.  Pattern Recognition 123, 108411 (2022)
  * (40) Patravali, J., Mittal, G., Yu, Y., Li, F., Chen, M.: Unsupervised few-shot action recognition via action-appearance aligned meta-adaptation.  In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8484–8494 (2021)
  * (41) Perrett, T., Masullo, A., Burghardt, T., Mirmehdi, M., Damen, D.: Temporal-relational crosstransformers for few-shot action recognition.  arXiv preprint arXiv:2101.06184 (2021)
  * (42) Recasens, A., Kellnhofer, P., Stent, S., Matusik, W., Torralba, A.: Learning to zoom: a saliency-based sampling layer for neural networks.  In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 51–66 (2018)
  * (43) Snell, J., Swersky, K., Zemel, R.S.: Prototypical networks for few-shot learning.  arXiv preprint arXiv:1703.05175 (2017)
  * (44) Soomro, K., Zamir, A.R., Shah, M.: Ucf101: A dataset of 101 human actions classes from videos in the wild.  arXiv preprint arXiv:1212.0402 (2012)
  * (45) Sung, F., Yang, Y., Zhang, L., Xiang, T., Torr, P.H., Hospedales, T.M.: Learning to compare: Relation network for few-shot learning.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1199–1208 (2018)
  * (46) Tian, X., Zhang, Z., Lin, S., Qu, Y., Xie, Y., Ma, L.: Farewell to mutual information: Variational distillation for cross-modal person re-identification.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1522–1531 (2021)
  * (47) Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3d convolutional networks.  In: Proceedings of the IEEE international conference on computer vision, pp. 4489–4497 (2015)
  * (48) Tschannen, M., Djolonga, J., Rubenstein, P.K., Gelly, S., Lucic, M.: On mutual information maximization for representation learning.  arXiv preprint arXiv:1907.13625 (2019)
  * (49) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need.  arXiv preprint arXiv:1706.03762 (2017)
  * (50) Vinyals, O., Blundell, C., Lillicrap, T., Kavukcuoglu, K., Wierstra, D.: Matching networks for one shot learning.  arXiv preprint arXiv:1606.04080 (2016)
  * (51) Wang, L., Xiong, Y., Wang, Z., Qiao, Y., Lin, D., Tang, X., Van Gool, L.: Temporal segment networks: Towards good practices for deep action recognition.  In: European conference on computer vision, pp. 20–36. Springer (2016)
  * (52) Wang, X., Zhang, S., Qing, Z., Tang, M., Zuo, Z., Gao, C., Jin, R., Sang, N.: Hybrid relation guided set matching for few-shot action recognition.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19948–19957 (2022)
  * (53) Wang, Y., Chen, Z., Jiang, H., Song, S., Han, Y., Huang, G.: Adaptive focus for efficient video recognition.  In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16249–16258 (2021)
  * (54) Wang, Y., Lv, K., Huang, R., Song, S., Yang, L., Huang, G.: Glance and focus: a dynamic approach to reducing spatial redundancy in image classification.  Advances in Neural Information Processing Systems 33, 2432–2444 (2020)
  * (55) Wang, Y., Yue, Y., Lin, Y., Jiang, H., Lai, Z., Kulikov, V., Orlov, N., Shi, H., Huang, G.: Adafocus v2: End-to-end training of spatial dynamic networks for video recognition.  arXiv preprint arXiv:2112.14238 (2021)
  * (56) Wang, Y.X., Girshick, R., Hebert, M., Hariharan, B.: Low-shot learning from imaginary data.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7278–7286 (2018)
  * (57) Wu, W., He, D., Tan, X., Chen, S., Wen, S.: Multi-agent reinforcement learning based frame sampling for effective untrimmed video recognition.  In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6222–6231 (2019)
  * (58) Wu, Z., Xiong, C., Ma, C.Y., Socher, R., Davis, L.S.: Adaframe: Adaptive frame selection for fast video recognition.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1278–1287 (2019)
  * (59) Zhang, H., Zhang, L., Qi, X., Li, H., Torr, P.H., Koniusz, P.: Few-shot action recognition with permutation-invariant attention.  In: Proceedings of the European Conference on Computer Vision (ECCV). Springer (2020)
  * (60) Zhang, S., Zhou, J., He, X.: Learning implicit temporal alignment for few-shot video classification.  arXiv preprint arXiv:2105.04823 (2021)
  * (61) Zhang, Y., Ding, L., Li, Y., Lin, W., Zhao, M., Yu, X., Zhan, Y.: A regional distance regression network for monocular object distance estimation.  Journal of Visual Communication and Image Representation 79, 103224 (2021)
  * (62) Zheng, H., Fu, J., Zha, Z.J., Luo, J.: Looking for the devil in the details: Learning trilinear attention sampling network for fine-grained image recognition.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5012–5021 (2019)
  * (63) Zheng, S., Chen, S., Jin, Q.: Few-shot action recognition with hierarchical matching and contrastive learning.  In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part IV, pp. 297–313. Springer (2022)
  * (64) Zhi, Y., Tong, Z., Wang, L., Wu, G.: Mgsampler: An explainable sampling strategy for video action recognition.  In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1513–1522 (2021)
  * (65) Zhu, L., Yang, Y.: Compound memory networks for few-shot video classification.  In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 751–766 (2018)
  * (66) Zhu, L., Yang, Y.: Label independent memory for semi-supervised few-shot video classification.  IEEE Transactions on Pattern Analysis and Machine Intelligence 44(1), 273–285 (2020)