# Few Shot Activity Recognition Using Variational Inference

Neeraj Kumar1111Contact Author Siddhansh Narang2
1Indian Institute of Technology, Delhi, India
2RAeS (Royal Aeronautical Society), UK
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

There has been a remarkable progress in learning a model which could recognize
novel classes with only a few labeled examples in the last few years. Few-shot
learning (FSL) for action recognition is a challenging task of recognizing
novel action categories which are represented by few instances in the training
data.

We propose a novel variational inference based architectural framework (HF-AR)
for few shot activity recognition. Our framework leverages volume-preserving
Householder Flow to learn a flexible posterior distribution of the novel
classes. This results in better performance as compared to state-of-the-art
few shot approaches for human activity recognition. approach consists of base
model and an adapter model. Our architecture consists of a base model and an
adapter model. The base model is trained on seen classes and it computes an
embedding that represent the spatial an temporal insights extracted from the
input video, e.g. combination of Resnet-152 and LSTM based encoder-decoder
model. The adapter model applies a series of Householder transformations to
compute a flexible posterior distribution that lends higher accuracy in the
few shot approach.

Extensive experiments on three well-known datasets: UCF101, HMDB51 and
Something-Something-V2, demonstrate similar or better performance on 1-shot
and 5-shot classification as compared to state-of-the-art few shot approaches
that use only RGB frame sequence as input. To the best of our knowledge, we
are the first to explore variational inference along with householder
transformations to capture the full rank covariance matrix of posterior
distribution, for few shot learning in activity recognition.

##  1 Introduction

Action recognition in videos has received increasing attention from the deep
learning communityWang et al. (2011); Simonyan and Zisserman (2014); Tran et
al. (2015); Carreira and Zisserman (2017) in recent years, due to its great
potential value in industry and real-world applications. Convolutional Neural
Network (CNN) based methods (Carreira and Zisserman, 2017; Tran et al., 2015)
have achieved tremendous success in recognizing actions from videos in the
supervised learning paradigm.

However, the performance of these methods deteriorates drastically Xu et al.
(2018) when recognizing action classes that are not adequately represented in
the training data. This problem limits the deployment of these methods in real
world applications where the number of action classes to be recognized
increases rapidly with use cases. In certain cases, the number of samples for
novel classes are quite few for even traditional data augmentation techniques
Krizhevsky et al. (2012) to work. Therefore, systems with more advanced
learning paradigm like Few Shot Learning Hariharan and Girshick (2016), which
learns to recognize novel classes from only a few examples (few-shots), have
come into prominence.

Videos are more complex than images as one requires the complete information
of the temporal aspects in recognizing some specific actions, such as opening
the door. In the previous literature of video classification, 3D convolution
and optical flow are two of the most popular methods to model temporal
relations. The direct output of neural network encoders is always a temporal
sequence of deep encoded features. Loss of information is even more severe for
few-shot learning. It is hard to learn the local temporal patterns which are
useful for few-shot classification with limited amount of data. Utilizing the
long-term temporal ordering information, which is often neglected in previous
works on video classification, might potentially help with few-shot learning.
For example, if the model could verify that there is an act of playing long
jump before a close-up view of a jumping person, the model will then become
quite confident about predicting the class of this query video to be playing
long jump, rather than some other potential predictions like jumping exercises
or warm-up activities. In addition, for two videos in the same class, even
though they both contain the act of jumping, the exact temporal duration of
each atomic step can vary dramatically. These non-linear temporal variations
in videos pose a significant challenge to few-shot video activity
classification.

Prior research on few-shot learning for action recognition (Mishra et al.,
2018b; Guo et al., 2018; Zhu and Yang, 2018; Dwivedi et al., 2019; Zhang et
al., 2020; Cao et al., 2020) leverage a variety of approaches, but either
suffer from high computational complexity or are not able to deliver
sufficient accuracy on datasets such as Something-Something-V2. Mishra et al.
(2018b), projects the the action class to visual space using word2vec, while
Guo et al. (2018) learns a graph based representation to classify the actions
and Zhu and Yang (2018) uses multi-saliency based approach to learn the video
representation. Dwivedi et al. (2019) leverages a class-prototype vector with
conditional GAN based approach to recognize the activity, while Zhang et al.
(2020) uses C3D based features along with spatial and temporal network to
detect unseen classes. Recently, Cao et al. (2020) used a temporal alignment
network to generate the score used for few shot prediction. In an attempt to
improve accuracy, we explore a novel approach using variational inference (HF-
AR: Householder Flow Tomczak and Welling (2016) for Activity Recognition) with
normalizing flows and study the performance in detail on multiple well-known
datasets. We demonstrate close or superior performance using a two stage
variational inference architecture with a base and adapter model. The first
stage consists of base model having pretrained Resnet-152He et al. (2016) and
stacked LSTMHochreiter and Schmidhuber (1997) layers to capture the spatial
and temporal information and generate a relevant video embedding. The second
stage is an adapter model that uses a series of householder transformations.

Specifically, we make the following contributions:

  * •

We propose an architectural framework (HF-AR) that uses a base and an adapter
model. The adapter model uses variational inference with volume preserving
Householder FlowTomczak and Welling (2016).

  * •

This approach learns a full rank covariance matrix of the posterior
distribution of datasets having novel classes with few samples. This flexible
posterior distribution helps in delivering higher accuracy in few shot video
classification.

  * •

Detailed experimental analysis on three datasets namely: UCF101, HMDB51 and
Something-Something-V2 demonstrates superior results for 5-shot classification
as compared to state-of-the-art few shot approaches.

  * •

Ablation study on series of householder transformation demonstrates that the
length of the householder transformation, $T$, depends on $k$ in the k-shot
setting . Higher the value of $k$, more is the length required, $T$ to learn
the posterior distribution to prevent over-fitting. .

##  2 Related Work

#### Few Shot Learning

Few shot learning in image classification is predominantly based on meta-
learning (Ren et al., 2018; Zhang et al., 2018; Finn et al., 2017). Under the
umbrella of meta-learning, various approaches like metric learning Snell et
al. (2017); Vinyals et al. (2016); Sung et al. (2017), learning optimization
Ravi and Larochelle (2017) and learning to initialize and fine tune Finn et
al. (2017) are proposed.

Metric learning methods learn the similarity between images Vinyals et al.
(2016) to classify samples of novel classes at inference based on nearest
neighbors with labelled samples. Convolutional Siamese Net Koch (2015) employs
a unique structure to naturally rank similarity between inputs. Once a network
has been tuned, it can then capitalize on powerful discriminative features to
generalize the predictive power of the network not just to new data, but to
entirely new classes from unknown distributions. Learning optimization
involves memory based networks like LSTM Xu et al. (2017) or memory augmented
networks Mishra et al. (2018c) which generate updates for the classifier
rather than using gradients. The meta-learner also learns a task-common weight
initialization which captures shared knowledge across tasks. Learning to
initialize and fine-tune aims to make minimal changes in the network when
adapting to the new task of classifying novel classes with few examples. Graph
Neural Networks (Garcia and Bruna, 2018; Kim et al., 2019; Gidaris and
Komodakis, 2019) construct a weighted graph to represent all the data and
measure the similarity between data.

Other methods use data augmentation, which learns to augment labeled data in
unseen classes for supervised training (Hariharan and Girshick, 2017; Wang et
al., 2018). In this paper, we have propose a novel two-stage architecture, HF-
AR, with a base and adapter model, in which the base model learns the
knowledge (via embedding) from seen classes. This is then used in the adapter
model that uses variational inference with a series of householder
transformations, in the fine tuning phase to classify the novel classes.

#### Few shot Action Recognition:

In contrast to few shot learning in images, few shot approaches in video
classification have gotten less attention. In Mishra et al. (2018b) each class
attribute is projected to the visual space. In the visual space each class is
represented by a Gaussian distribution. Our approach uses pre-trained models
and variational inference based architecture to generate embedding to predict
the novel classes. Neural graph matching network Guo et al. (2018) used graph
generator to leverage the structure of 3D data through a graphical
representation. Xu et al. (2018) combines dilated temporal convolution and
densely layer connection to perform video action recognition.

Compound Memory Networks Zhu and Yang (2018) introduces a multi-saliency
embedding algorithm to encode video into a fixed-size matrix representation.
Then, they propose a compound memory network (CMN) to compress and store the
representation and classify videos by matching and ranking. Our method uses
pretrained Resnet-152 and LSTM layers to capture the spatial and temporal
information to be used at fine tuning stage using variational inference.
Attribute-based feature generation for unseen classes from GAN by using Fisher
vector representation was explored in zero-shot learning in Zhang and Peng
(2018). It proposes multi-level semantic inference to boost video feature
synthesis, which captures the discriminative information implied in the joint
visual-semantic distribution via feature-level and label-level semantic
inference along with matching-aware Mutual Information Correlation to overcome
information degradation issue. This captures seen-to-unseen correlation in
matched and mismatched visual-semantic pairs by mutual information, providing
the zero-shot synthesis procedure with robust guidance signals. Our approach,
HF-AR, uses only RGB frames as input and captures the posterior distribution
of novel classes, with few samples, using variational inference.

ProtoGAN Dwivedi et al. (2019) proposes a GAN model which conditions CGAN on a
class-prototype vector which is learnt through Class Prototype Transfer
Network (CPTN) to synthesize additional video features for the action
recognition classifier. Action Relation Network Zhang et al. (2020) contains
C3D-based encoder, relation network and spatial and temporal attention units
which refine the aggregation step. Cao et al. (2020) uses Temporal Alignment
Module (TAM), a novel few-shot learning framework that can learn to classify a
previous unseen video. It computes temporal alignment score for each potential
query support pair by averaging per-frame distances along a temporal alignment
path, which enforces the score used to make prediction to preserve temporal
ordering. Our approach (HF-AR) uses pretrained Resnet-152 He et al. (2016) and
stacked LSTM Hochreiter and Schmidhuber (1997) to capture the spatial and
temporal information during pre-training on seen classes. During the fine
tuning stage on novel classes, we use a series of householder transformations
to capture the full rank covariance matrix based distribution of the dataset.

##  3 Mathematical background

###  3.1 Variational Autoencoder

Let us consider some dataset $X=\\{x^{(i)}\\}_{i=1}^{N}$ consisting of $N$
i.i.d. samples of some continuous or discrete variable $x$. We assume that the
data are generated by some random process, involving an unobserved continuous
random variable $z$. The process consists of two steps: (1) a value $z^{(i)}$
is generated from some prior distribution $p(z)$; (2) a value $x^{(i)}$ is
generated from some conditional distribution $p(x|z)$. Given $N$ data points
$\mathbf{X}=\\{\mathbf{x}_{1},\ldots,\mathbf{x}_{N}\\}$ we typically aim at
maximizing the marginal log-likelihood:

$\ln p(\mathbf{X})=\sum_{i=1}^{N}\ln p(\mathbf{x}_{i}),$ (1)

with respect to parameters. This task could be troublesome because the
integral of the marginal likelihood $p(x)=\int p(z)p(x|z)\,dz$ is intractable
(so we cannot evaluate or differentiate the marginal likelihood). These
intractabilities are quite common and appear in cases of moderately
complicated likelihood functions $p(x|z)$, e.g. a neural network with a
nonlinear hidden layer. To overcome this issue one can introduce an inference
model (an encoder) $q(\mathbf{z}|\mathbf{x})$ and optimize the variational
lower bound:

$\ln p(\mathbf{x})\geq\mathbb{E}_{q(\mathbf{z}|\mathbf{x})}[\ln
p(\mathbf{x}|\mathbf{z})]-\mathrm{KL}\big{(}q(\mathbf{z}|\mathbf{x})||p(\mathbf{z})\big{)},$
(2)

where $p(\mathbf{x}|\mathbf{z})$ is called a decoder and
$p(\mathbf{z})=\mathcal{N}(\mathbf{z}|\mathbf{0},\mathbf{I})$ is the prior.
There are various ways of optimizing this lower bound but for continuous
$\mathbf{z}$ this could be done efficiently through a re-parameterization of
$q(\mathbf{z}|\mathbf{x})$ (Kingma and Welling, 2014; Rezende et al., 2014).
Then the architecture is called a variational auto-encoder (VAE).

Typically, a diagonal covariance matrix of the encoder is assumed, i.e.,
$q(\mathbf{z}|\mathbf{x})=\mathcal{N}\big{(}\mathbf{z}|\boldsymbol{\mu}(\mathbf{x}),\mathrm{diag}(\boldsymbol{\sigma}^{2}(\mathbf{x}))\big{)}$,
where $\boldsymbol{\mu}(\mathbf{x})$ and $\boldsymbol{\sigma}^{2}(\mathbf{x})$
are parameterized by the NN. However, this assumption can be insufficient and
not flexible enough to match the true posterior.

###  3.2 Householder Flow

A normalizing flow, formulated by (Tabak and Turner, 2013; Tabak and Vanden-
Eijnden, 2010; Rezende and Mohamed, 2015) is a framework for designing
flexible posterior distribution by starting with an initial random variable
with a simple distribution for generating $\mathbf{z}^{(0)}$ and then applying
a series of invertible transformations $\mathbf{f}^{(t)}$, for $t=1,\ldots,T$.
Due to this , the last random variable $\mathbf{z}^{(T)}$ has more richer
distribution. The aim of is to optimize the following objective function once
we choose the transformations $\mathbf{f}^{(t)}$ for which we can compute the
jacobian determinant.

$\ln p(\mathbf{x})\geq\mathbb{E}_{q(\mathbf{z}^{(0)}|\mathbf{x})}\Big{[}\ln
p(\mathbf{x}|\mathbf{z}^{(T)})+\sum_{t=1}^{T}\ln\Big{|}\mathrm{det}\frac{\partial\mathbf{f}^{(t)}}{\partial\mathbf{z}^{(t-1)}}\Big{|}\Big{]}\\\
-\mathrm{KL}\big{(}q(\mathbf{z}^{(0)}|\mathbf{x})||p(\mathbf{z}^{(T)})\big{)}.$
(3)

We have used the volume preserving flows that apply series of Householder
transformations that we refer to as the Householder flow. The volume
preserving flows provide the series of transformation such that the jacobian
determinant equals one. This helps in reducing the computation complexity
while successfully providing a flexible posterior distribution.

#### Motivation:

Any full-covariance matrix $\boldsymbol{\Sigma}$ can be represented by the
eigenvalue decomposition using eigenvectors and eigenvalues:

$\boldsymbol{\Sigma}=\mathbf{U}\mathbf{D}\mathbf{U}^{\top},$ (4)

where $\mathbf{U}$ is an orthogonal matrix with eigenvectors in columns,
$\mathbf{D}$ is a diagonal matrix with eigenvalues. The goal is to model the
matrix $\mathbf{U}$ to obtain a full-covariance matrix. This process requires
a linear transformation of a random variable using an orthogonal matrix
$\mathbf{U}$. Since the absolute value of the Jacobian determinant of an
orthogonal matrix is $1$, for $\mathbf{z}^{(1)}=\mathbf{U}\mathbf{z}^{(0)}$
one gets
$\mathbf{z}^{(1)}\sim\mathcal{N}(\mathbf{U}\boldsymbol{\mu},\mathbf{U}\
\mathrm{diag}(\boldsymbol{\sigma}^{2})\ \mathbf{U}^{\top})$. If
$\mathrm{diag}(\boldsymbol{\sigma}^{2})$ coincides with true $\mathbf{D}$,
then it would be possible to resemble the true full-covariance function.

Generally, the task of modelling an orthogonal matrix is rather non-trivial.
However, first we notice that any orthogonal matrix can be represented in the
following form (Bischof and Sun, 1996; Sun and Bischof, 1995):

######  Theorem 1.

_(The Basis-Kernel Representation of Orthogonal Matrices)_
For any $M\times M$ orthogonal matrix $\mathbf{U}$ there exist a full-rank
$M\times K$ matrix $\mathbf{Y}$ (the basis) and a nonsingular (triangular)
$K\times K$ matrix $\mathbf{S}$ (the kernel), $K\leq M$, such that:

$\mathbf{U}=\mathbf{I}-\mathbf{Y}\mathbf{S}\mathbf{Y}^{\top}.$ (5)

The value $K$ is called the degree of the orthogonal matrix. Further, it can
be shown that any orthogonal matrix of degree $K$ can be expressed using the
product of Householder transformations (Bischof and Sun, 1996; Sun and
Bischof, 1995), namely:

######  Theorem 2.

Any orthogonal matrix with the basis acting on the $K$-dimensional subspace
can be expressed as a product of exactly $K$ Householder transformations:

$\mathbf{U}=\mathbf{H}_{K}\mathbf{H}_{K-1}\cdots\mathbf{H}_{1},$ (6)

where $\mathbf{H}_{k}=\mathbf{I}-\mathbf{S}_{kk}\mathbf{Y}_{\cdot
k}(\mathbf{Y}_{\cdot k})^{\top}$, for $k=1,\ldots,K$.

Theoretically, Theorem 2 shows that we can model any orthogonal matrix using
$K$ Householder transformations.

Figure 1: a. Figure shows the general architecture of VAE+HF b. Figure shows
the series of housefolder transformationTomczak and Welling (2016)

#### Definition

For a given vector $\mathbf{z}^{(t-1)}$ the reflection hyperplane can be
defined by a vector (a Householder vector) $\mathbf{v}_{t}\in\mathbb{R}^{M}$
that is orthogonal to the hyperplane, and the reflection of this point about
the hyperplane is (Householder, 1958):

$\displaystyle\mathbf{z}^{(t)}$
$\displaystyle=\Big{(}\mathbf{I}-2\frac{\mathbf{v}_{t}\mathbf{v}_{t}^{\top}}{||\mathbf{v}_{t}||^{2}}\Big{)}\mathbf{z}^{(t-1)}$
(7) $\displaystyle=\mathbf{H}_{t}\mathbf{z}^{(t-1)},$ (8)

where
$\mathbf{H}_{t}=\mathbf{I}-2\frac{\mathbf{v}_{t}\mathbf{v}_{t}^{\top}}{||\mathbf{v}_{t}||^{2}}$
is called the Householder matrix.

The series of such transformation results in
$\ln\Big{|}\mathrm{det}\frac{\partial\mathbf{H}_{t}\mathbf{z}^{(t-1)}}{\partial\mathbf{z}^{(t-1)}}\Big{|}=0$,
for $t=1,\ldots,T$. We start with the simple diagonal covariance matrix for
$\mathbf{z}^{(0)}$ and after series of householder transformation results in
vectors $\mathbf{v}_{t}$, $t=1,\ldots,T$, which leads to learn full covariance
matrix for more flexible posterior distribution. Figure 1 shows the idea of
householder flow where where $T$ operations are required to generate flexible
posterior distribution with approximated full covariance matrix.

##  4 Architecture

Our proposed architecture (HF-AR) consists of two stages: base model and
adapter model (Fig. 2). The first stage (Fig. 3) is an encoder-decoder based
architecture to generate video embeddings on seen classes. The second stage is
an adapter model which takes as input the embedding generated by the base
model and applies a series of householder transformations to finally predict
the novel classes in a few shot fashion.

Figure 2: 2-stage Proposed Architecture (HF-AR): Base and Adapter model. Base
model with RGB frames as input and Adapter model with embeddings as input and
applies a series of householder transformations. Figure 3: Base Model: takes
video (RGB) frames as input. These are fed to pretrained Resenet-152 in the
encoder followed by stacked LSTM layers in the decoder.

###  4.1 Notation

Let $x$, $\bar{x}$, $y$ represent real video frames, video embeddings, class
labels respectively. Let $\mathcal{S}$ = $\\{x^{s}$,
$y^{s}|x^{s}\in\mathcal{X}^{s}$, $y^{s}\in\mathcal{Y}^{s}\\}$ be the training
set for seen classes where $x^{s}$ denotes the video frames , $y^{s}$ denotes
the class labels in $\mathcal{Y}^{s}=\\{y^{s}_{1},\ldots,y^{s}_{S}\\}$ with
$S$ seen classes. The video embeddings $\bar{x}$ are calculated from the base
model with $x^{s}$ video frames as an input. Additionally,
$\mathcal{N}=\\{x^{n},y^{n})|{x^{n}}\in\mathcal{X}^{n},y^{n}\in\mathcal{Y}^{n}\\}$
is available during adapter training from novel categories where $y^{n}$ is a
class from a disjoint label set
$\mathcal{Y}^{n}=\\{y^{n}_{1},\ldots,y^{n}_{N}\\}$ of $N$ novel labels. During
the adaptor training , the video embeddings $\bar{x}$ are fed to the
pretrained base model with multiple householder transformations to predict the
novel classes in few shot setting.

###  4.2 Base Model

The base model has encoder$-$decoder architecture in which the encoder takes
video frames($x^{s}$) and feed it to the pretrained Resnet-152 He et al.
(2016) model followed by the $3$ linear layers to generate the $512$
dimensional video embedding($\bar{x^{s}}$). The decoder takes the embedding
and send it to the stacked LSTM Hochreiter and Schmidhuber (1997) layer
followed by linear layers to predict the seen classes. When the number of
samples in the seen classes($y^{s}$) are high during the training, the encoder
generates effective embeddings which helps the decoder to predict the seen
classes with high accuracy. Pretrained Resnet-152 helps in exploiting the
higher level features in the frames which compresses the information into
small sized embeddings. The stacked LSTM helps in extracting the temporal
aspects of the given input video which helps the base model to generate better
embeddings.

###  4.3 Adapter Model

The adapter model takes the video embedding as input and applies a series of
volume preserving householder transformations to predict the novel classes.
The householder transformation help the model to learn a more flexible
posterior distribution of dataset with novel classes in few shot setting, with
full rank covariance matrix .

###  4.4 Losses

The optimisation function for variational inference is given in Equation 9:

$\ln p(\mathbf{x})\geq\mathbb{E}_{q(\mathbf{z}^{(0)}|\mathbf{x})}\Big{[}\ln
p(\mathbf{x}|\mathbf{z}^{(T)})+\sum_{t=1}^{T}\ln\Big{|}\mathrm{det}\frac{\partial\mathbf{f}^{(t)}}{\partial\mathbf{z}^{(t-1)}}\Big{|}\Big{]}\\\
-\beta\cdot\mathrm{KL}\big{(}q(\mathbf{z}^{(0)}|\mathbf{x})||p(\mathbf{z}^{(T)})\big{)}.$
(9)

#### Cross Entropy loss

In multi class classification, the class label($y^{s}$) follows the
multinomial distribution given in Equation 10. So the first part of Equation 9
i.e. $\ln p(\mathbf{x}|\mathbf{z}^{(T)})$ is the cross entropy loss in our set
up. The cross entropy loss(CE) is given by Equation 11 .

$\sum_{j=1}^{N}y^{(j)}=1$ (10)

$CE(\hat{y},y)=\sum_{j=1}^{N}y^{(j)}\log\hat{y}^{(j)}$ (11)

where $y^{(j)}$ is the true label i.e $0$ or $1$ and $\hat{y}^{(j)}$ is the
predicted probability of a class coming from softmax operation.

#### Jacobian Determinant based loss

The second part,
$\ln\Big{|}\mathrm{det}\frac{\partial\mathbf{H}_{t}\mathbf{z}^{(t-1)}}{\partial\mathbf{z}^{(t-1)}}\Big{|}$
becomes zero as we are using the householder flow which is volume preserving
based flow, whose jacobian determinant is 1.

Table 1: Comparison of accuracy between our model (HF-AR) and state-of-the-art approaches on HMDB51, UCF101 and Something-Something V2 datasets Model | HMDB51Kuehne et al. (2011) | UCF101Soomro et al. (2012) | Something-SomethingV2 Goyal et al. (2017)
---|---|---|---
1-shot | 5-shot | 1-shot | 5-shot | 1-shot | 5-shot
GenApp Mishra et al. (2018a) | ${-}$ | ${52.5\pm 3.10}$ | ${-}$ | ${78.6\pm 2.1}$ | $-$ | $-$
ProtoGAN Dwivedi et al. (2019) | $34.7\pm 9.20$ | $54.0\pm 3.90$ | $57.8\pm 3.0$ | $80.2\pm 1.3$ | $-$ | $-$
ARN Zhang et al. (2020) | $\mathbf{44.6\pm 0.9}$ | $59.1\pm 0.8$ | $\mathbf{62.1\pm 1.0}$ | $84.8\pm 0.8$ | $-$ | $-$
TAM Cao et al. (2020) | ${-}$ | ${-}$ | ${-}$ | ${-}$ | $42.8$ | $52.3$
HF-AR | $43.4\pm 0.6$ | $\mathbf{62.2\pm 0.9}$ | $58.6\pm 1.2$ | $\mathbf{86.4\pm 1.6}$ | $\mathbf{43.1\pm 1.7}$ | $\mathbf{55.13\pm 3.1}$

#### Kullback–Leibler(KL) divergence loss

The third part,
$\mathrm{KL}\big{(}q(\mathbf{z}^{(0)}|\mathbf{x})||p(\mathbf{z}^{(T)})\big{)}$
is the KL loss between with $q(\mathbf{z}^{(0)}|\mathbf{x})$ and
$p(\mathbf{z}^{(T)})$ where $\mathbf{z}^{(T)}$ is the last random variable
obtained from householder flow. The KL loss acts as the regularizer which
prevents the model from overfitting and helps in learning a flexible posterior
distribution. $\beta$ value is used to to control the effect of KL loss. The
value of $\beta$ decreases linearly with every epoch.

##  5 Experimental Section

In this work, our task is few-shot video classification, where the objective
is to classify novel classes with only a few examples from the support set.

###  5.1 Datasets

We evaluate our method (HF-AR) on three publicly available datasets for action
recognition. On these datasets, we measure the action recognition performance
with respect to top-1 and top-5 recognition accuracy.

  * •

UCF101Soomro et al. (2012): contains $13320$ videos spanning over $101$
classes. For our experiments, we randomly split the data into $80$ seen and
$21$ novel classes.

  * •

HMDB51Kuehne et al. (2011): contains $6766$ videos spanning over $51$ classes.
For our experiments we randomly split the data into $41$ seen and $10$ novel
classes.

  * •

Something-Something V2 Goyal et al. (2017): We randomly selected $100$ classes
from the whole dataset. For our experiments, we randomly split this data into
$76$ seen and $24$ novel classes.

###  5.2 Implementation Details

#### Pre-Processing Steps

: The videos frames are resized into 224*224 and then normalized before being
fed to the activity recognition model. We have selected $50$ frames of the
video to be used in the proposed model. The training procedure uses the batch
of $150$ during base model training and $4$ during adapter training . We have
used Adam optimizer Kingma and Ba (2015) with the initial learning rate of
$1e-3$ and dynamic adjustment. We have used ReduceLROnPlateau scheduler
Mukherjee et al. (2019) with patience equals to $10$.

###  5.3 Model Details

Base model consists of encoder decoder framework . The encoder uses the
pretrained Resnet 152 which gives $2048$ features for every frame and finally
all the $50$ frames are concatenated to generate the tensor of size
$(batchsize,50,2048)$. Three linear layers are used with channel length of
$(2048\rightarrow 300)$ , $(512\rightarrow 512)$, $(512\rightarrow 512)$. The
$300$ embeddings of every frames are fed into the decoder architecture of the
base model.

The decoder takes the tensor of size $(batchsize,50,512)$ and feeds this to
$3$ stacked LSTM layers. The output of the last layer of LSTM is used as an
input to the following layers. Two linear layers having channel length
$(512\rightarrow 256)$ ,$(256\rightarrow classes)$ are then used to predict
the classes.

The adapter model which is used to predict the classes with few samples uses
the pretrained base model as the source of its input. The pretrained model
gives the output of $256$ features (output of second last layer of base model)
is fed to the householder flow network which predicts the classes. The
householder network uses linear layers to predict the $\mu$ and $\sigma$ and
which generates the vector using the reparametrization trick,
$\mathbf{z}^{(0)}$= $(\mu+\sigma\odot\epsilon)$. The $\mathbf{z}^{(0)}$ goes
into series of householder transformations to generate the final
$\mathbf{z}^{(T)}$ whose feature size is $256$. This goes into linear layer to
predict the novel classes having few samples.

###  5.4 Comparative Analysis

Table 1 presents the comparison of accuracy obtained between our approach (HF-
AR) vs. state-of-the-art approaches on the three datasets with 1-shot and
5-shot classification. For 5-shot classification, HF-AR delivers better
accuracy on all the 3 datasets: $62.2\%$ vs $59.1\%$ (ARN Zhang et al. (2020))
for HMDB51 dataset, $86.4\%$ vs $84.8\%$ (ARN Zhang et al. (2020)) for UCF101
dataset, and $55.13\%$ vs $52.3\%$ (TAM ) for Something-Something V2 dataset.
This improvement is due to flexbile posterior model learned by Householder
Flow in our adapter model. For 1-shot classification, HF-AR delivers slightly
better accuracy for Something-Something V2 dataset: $43.1\%$ vs $42.8\%$ ( Cao
et al. (2020)); slightly lower accuracy for HMDB51 dataset: $43.4\%$ vs
$44.6\%$ (ARN Zhang et al. (2020)); and, lower accuracy for UCF101 dataset:
$58.6\%$ vs $62.1\%$ ( Cao et al. (2020)). For very limited data as in 1-shot
approach, we believe it becomes harder for our variational inference based
approach to attain the right generalization ability.

###  5.5 Ablation Study

#### Effect of length of householder flow

: We examined the behaviour of the length of the series of householder
transformations in the few shot setting. We have done experiments with HMDB51
dataset to study this in detail. Table 2 shows that in 1-shot setting as we
increase the length, the accuracy dips as the model is overfitting. In 5-shot
setting the optimum length increases as compared to 1-shot setting.

Length | 1-shot | 5-shot
---|---|---
T=1 | $35.1\pm 0.6$ | $55.6\pm 0.8$
T=3 | $\mathbf{43.4\pm 0.6}$ | $59.9\pm 0.8$
T=10 | $39.8\pm 0.4$ | $\mathbf{62.2\pm 0.9}$
T=20 | $39.2\pm 0.4$ | $60.8\pm 0.7$
Table 2: Effect of changing the length of householder flow in HF-AR on accuracy with HMDB51 daatset.  Length | 1-shot | 5-shot
---|---|---
3D-CNN Tran et al. (2015) | $45.4\pm 1.6$ | $70.5\pm 1.3$
CompHF | $48.4\pm 2.3$ | $72.2\pm 2.1$
HF-AR | $\mathbf{58.6\pm 1.2}$ | $\mathbf{86.4\pm 1.6}$
Table 3: Effect of various Base Model with UCF101 dataset on accuracy.

#### Effect of various base models

: We studied the effect of various base models in few shot setting with UCF101
dataset. Table 3 shows superior performance of our proposed model with respect
to other base models. 3D-CNN Tran et al. (2015) based base model is not able
to predict high quality video embeddings which leads to lower accuracy on both
seen and unseen classes. However, our proposed method (HF-AR) uses pretrained
ResNet-152 which is able to learn good image features along with LSTM which
captures the temporal aspects. The CompHF base model uses householder flow in
the base model itself to predict the seen classes. It gives lesser accuracy on
seen and unseen classes because the posterior distribution generated from
householder flow fails to capture the more complex multi-modal distribution of
the dataset with larger number of seen classes. This needs further
investigation with more variety of base models.

##  6 Conclusions

We propose a variational inference based architectural framework for few shot
activity recognition. The proposed architecture does not require any optical
flow for input or supervision and provides a unified and efficient method to
predict unseen classes. This method learns the full rank covariance matrix of
the posterior distribution of datasets with novel classes. The idea relies on
the observation that the true full-covariance matrix can be decomposed to the
diagonal matrix with eigenvalues on the diagonal and an orthogonal matrix of
eigenvectors that could be further modeled using a series of Householder
transformations. Experimental study on multiple datasets demonstrates that
variational inference with Householder flows could perform better than other
state of the art approaches for few shot activity recognition. This makes this
approach a very promising direction for further investigation in this and
other related areas in computer vision.

## References

  * Bischof and Sun (1996) C. Bischof and X. Sun.  On orthogonal block elimination.  1996\.
  * Cao et al. (2020) Kaidi Cao, Jingwei Ji, Zhangjie Cao, C. Chang, and Juan Carlos Niebles.  Few-shot video classification via temporal alignment.  2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10615–10624, 2020.
  * Carreira and Zisserman (2017) João Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4724–4733, 2017.
  * Dwivedi et al. (2019) Sai Kumar Dwivedi, Vikram Gupta, Rahul Mitra, Shuaib Ahmed, and Arjun Jain.  Protogan: Towards few shot learning for action recognition.  arXiv preprint, 2019.
  * Finn et al. (2017) Chelsea Finn, P. Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In ICML, 2017.
  * Garcia and Bruna (2018) Victor Garcia and Joan Bruna.  Few-shot learning with graph neural networks.  2018\.
  * Gidaris and Komodakis (2019) Spyros Gidaris and Nikos Komodakis.  Generating classification weights with gnn denoising autoencoders for few-shot learning.  2019\.
  * Goyal et al. (2017) R. Goyal, S. Kahou, Vincent Michalski, Joanna Materzynska, S. Westphal, Heuna Kim, Valentin Haenel, Ingo Fründ, P. Yianilos, Moritz Mueller-Freitag, F. Hoppe, Christian Thurau, I. Bax, and R. Memisevic.  The “something something” video database for learning and evaluating visual common sense.  2017 IEEE International Conference on Computer Vision (ICCV), pages 5843–5851, 2017.
  * Guo et al. (2018) Michelle Guo, Edward Chou, De-An Huang, Shuran Song, Serena Yeung, and Li Fei-Fei.  Neural graph matching networks for fewshot 3d action recognition.  2018\.
  * Hariharan and Girshick (2016) Bharath Hariharan and Ross B. Girshick.  Low-shot visual object recognition.  ArXiv, abs/1606.02819, 2016.
  * Hariharan and Girshick (2017) Bharath Hariharan and Ross B. Girshick.  Low-shot visual recognition by shrinking and hallucinating features.  2017 IEEE International Conference on Computer Vision (ICCV), pages 3037–3046, 2017.
  * He et al. (2016) Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * Hochreiter and Schmidhuber (1997) S. Hochreiter and J. Schmidhuber.  Long short-term memory.  Neural Computation, 9:1735–1780, 1997.
  * Householder (1958) A. Householder.  Unitary triangularization of a nonsymmetric matrix.  J. ACM, 5:339–342, 1958.
  * Kim et al. (2019) Jongmin Kim, Taesup Kim, Sungwoong Kim, and Chang D. Yoo.  Edge-labeling graph neural network for few-shot learning.  2019\.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  CoRR, abs/1412.6980, 2015.
  * Kingma and Welling (2014) Diederik P. Kingma and M. Welling.  Auto-encoding variational bayes.  CoRR, abs/1312.6114, 2014.
  * Koch (2015) Gregory R. Koch.  Siamese neural networks for one-shot image recognition.  2015\.
  * Krizhevsky et al. (2012) A. Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  Imagenet classification with deep convolutional neural networks.  Communications of the ACM, 60:84 – 90, 2012.
  * Kuehne et al. (2011) Hilde Kuehne, Hueihan Jhuang, Estíbaliz Garrote, T. Poggio, and Thomas Serre.  Hmdb: A large video database for human motion recognition.  2011 International Conference on Computer Vision, pages 2556–2563, 2011.
  * Mishra et al. (2018a) Ashish Mishra, V. Verma, M. K. Reddy, Arulkumar Subramaniam, Piyush Rai, and Anurag Mittal.  A generative approach to zero-shot and few-shot action recognition.  2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 372–380, 2018.
  * Mishra et al. (2018b) Ashish Mishra, Vinay Kumar Verma, M Shiva Krishna Reddy, S Arulkumar, Piyush Rai, and Anurag Mittal.  A generative approach to zero-shot and few-shot action recognition.  2018\.
  * Mishra et al. (2018c) Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and P. Abbeel.  A simple neural attentive meta-learner.  In ICLR, 2018.
  * Mukherjee et al. (2019) K. Mukherjee, Alind Khare, and A. Verma.  A simple dynamic learning rate tuning algorithm for automated training of dnns.  ArXiv, abs/1910.11605, 2019.
  * Ravi and Larochelle (2017) S. Ravi and H. Larochelle.  Optimization as a model for few-shot learning.  In ICLR, 2017.
  * Ren et al. (2018) Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel.  Meta-learning for semi-supervised few-shot classification.  CoRR, abs/1803.00676, 2018.
  * Rezende and Mohamed (2015) Danilo Jimenez Rezende and S. Mohamed.  Variational inference with normalizing flows.  In ICML, 2015.
  * Rezende et al. (2014) Danilo Jimenez Rezende, S. Mohamed, and Daan Wierstra.  Stochastic backpropagation and approximate inference in deep generative models.  In ICML, 2014.
  * Simonyan and Zisserman (2014) K. Simonyan and Andrew Zisserman.  Two-stream convolutional networks for action recognition in videos.  In NIPS, 2014.
  * Snell et al. (2017) J. Snell, Kevin Swersky, and R. Zemel.  Prototypical networks for few-shot learning.  In NIPS, 2017.
  * Soomro et al. (2012) K. Soomro, A. Zamir, and M. Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  ArXiv, abs/1212.0402, 2012.
  * Sun and Bischof (1995) X. Sun and C. Bischof.  A basis-kernel representation of orthogonal matrices.  SIAM J. Matrix Anal. Appl., 16:1184–1196, 1995.
  * Sung et al. (2017) Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H. S. Torr, and Timothy M. Hospedales.  Learning to compare: Relation network for few-shot learning.  CoRR, abs/1711.06025, 2017.
  * Tabak and Turner (2013) E. Tabak and C. Turner.  A family of nonparametric density estimation algorithms.  Communications on Pure and Applied Mathematics, 66:145–164, 2013\.
  * Tabak and Vanden-Eijnden (2010) Esteban Tabak and Eric Vanden-Eijnden.  Density estimation by dual ascent of the log-likelihood.  Communications in Mathematical Sciences - COMMUN MATH SCI, 8, 03 2010.
  * Tomczak and Welling (2016) Jakub M. Tomczak and M. Welling.  Improving variational auto-encoders using householder flow.  ArXiv, abs/1611.09630, 2016.
  * Tran et al. (2015) Du Tran, Lubomir D. Bourdev, R. Fergus, L. Torresani, and Manohar Paluri.  Learning spatiotemporal features with 3d convolutional networks.  2015 IEEE International Conference on Computer Vision (ICCV), pages 4489–4497, 2015.
  * Vinyals et al. (2016) Oriol Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, and Daan Wierstra.  Matching networks for one shot learning.  In NIPS, 2016.
  * Wang et al. (2011) Heng Wang, Alexander Kläser, C. Schmid, and C. Liu.  Action recognition by dense trajectories.  CVPR 2011, pages 3169–3176, 2011.
  * Wang et al. (2018) Yu-Xiong Wang, Ross B. Girshick, M. Hebert, and Bharath Hariharan.  Low-shot learning from imaginary data.  2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7278–7286, 2018.
  * Xu et al. (2017) Zhongwen Xu, Linchao Zhu, and Yi Yang.  Few-shot object recognition from machine-labeled web images.  2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5358–5366, 2017.
  * Xu et al. (2018) Baohan Xu, Hao Ye, Yingbin Zheng, Heng Wang, Tianyu Luwang, and Yu-Gang Jiang.  Dense dilated network for few shot action recognition.  In ICMR, 2018.
  * Zhang and Peng (2018) Chenrui Zhang and Y. Peng.  Visual data synthesis via gan for zero-shot video classification.  In IJCAI, 2018.
  * Zhang et al. (2018) R. Zhang, Tong Che, Zoubin Ghahramani, Yoshua Bengio, and Y. Song.  Metagan: An adversarial approach to few-shot learning.  In NeurIPS, 2018.
  * Zhang et al. (2020) Hongguang Zhang, Liyong Zhang, Xiaojuan Qi, Hongdong Li, Philip H. S. Torr, and Piotr Koniusz.  Few-shot action recognition with permutation-invariant attention.  In ECCV, 2020.
  * Zhu and Yang (2018) Linchao Zhu and Y. Yang.  Compound memory networks for few-shot video classification.  In ECCV, 2018.


# Optimizing Neural Network for Computer Vision task in Edge Device

Ranjith M S
Dr. S Parameshwara
Pavan Yadav A
Shriganesh Hegde
Department of Electronics and Communication
The National Institute of Engineering

###### Abstract

The field of computer vision has grown very rapidly in the past few years due
to networks like convolution neural networks and their variants. The memory
required to store the model and computational expense are very high for such a
network limiting it to deploy on the edge device. Many times, applications
rely on the cloud but that makes it hard for working in real-time due to
round-trip delays. We overcome these problems by deploying the neural network
on the edge device itself. The computational expense for edge devices is
reduced by reducing the floating-point precision of the parameters in the
model. After this the memory required for the model decreases and the speed of
the computation increases where the performance of the model is least
affected. This makes an edge device to predict from the neural network all by
itself.

_Keywords_ Embedded System $\cdot$ Edge Computing $\cdot$ Optimization $\cdot$
Deep learning $\cdot$ Computer Vision $\cdot$ Pruning $\cdot$ Quantization
$\cdot$ ARM $\cdot$ convolution neural network

##  1 Introduction

With the rise of Artificial intelligence, it is realized that deep-learning-
based approaches give satisfactory results compared to numerous other state-
of-the-art schemes which are hand-engineered to all the computer vision tasks.
The features learned from Convolutional neural networks outperform other hand-
engineered feature-based methods like SIFT [Mikolajczyk and Schmid (2004)] and
HoG [Dalal and Triggs (2005)] in computer vision tasks like image
classification and object detection.
The availability of large datasets and powerful computation devices made it
possible to train the large and complex neural networks to obtain the desired
performance on many computer vision tasks. The large amount of open-source
pre-trained models trained on large datasets like ImageNet [Challenge ], MS-
COCO [Lin et al. (2014)], SHVN [Netzer et al. (2011)] created a large number
of useful filters especially the features learned from the initial layer helps
in transfer learning a lot. In transfer learning, most of the time only the
last few layers of pre-trained models are modified and trained which counters
the problem of having fewer data to a certain extent.
Currently, a variety of embedded systems are deployed but the usage of neural
networks is limited in edge devices like microcontrollers, Raspberry Pi.
Household devices like Refrigerators, washing machines use a set of logic,
rules for their automatic operations. By optimizing the network trained on a
dataset traditional way of controlling can be replaced by intelligently
monitoring the systems with the power of AI and neural networks [Ranjith M S
and Parameshwara (2020)]. Optimizing the convolutional neural network
architectures like ResNet [ He et al. (2016)], DenseNet [ Huang et al.
(2017)], AlexNet [ Krizhevsky et al. (2012)] which are generally used in
computer vision tasks allows creating many useful applications. for example,
using gestures recognized from the neural network to perform a specific task
based on the requirement, where the task and the gesture recognition are
performed by the edge device itself.

###  1.1 Literature Survey for implementing and training the neural network

Since we use dense layers along with convolutional layers for our task we
prefer He initialization [ He et al. (2015)] as compared with Xavier
Initialization Glorot and Bengio (2010) for initializing the weights while
training the network. But using LSUV initialization [ Mishkin and Matas
(2015)] leads to the learning of very deep nets that produces networks with
test accuracy better or equal to standard methods.
“overfitting" is greatly reduced by randomly omitting half of the feature
detectors on each training case. This prevents complex co-adaptations in which
a feature detector is only helpful in the context of several other specific
feature detectors. Instead, each neuron learns to detect a feature that is
generally helpful for producing the correct answer given the combinatorially
large variety of internal contexts in which it must operate.” [ Hinton et al.
(2012a)].
The optimizers such as Adam [ Kingma and Ba (2014)], RMS [ Hinton et al.
(2012b)] lead for faster convergence. And adaptive optimizers like LARS [ You
et al. (2019)], AdamW [ Loshchilov and Hutter (2017)] lead to better
generalization.
“One of the key elements of super-convergence is training with one learning
rate cycle and a large maximum learning rate” [ Smith and Topin (2019)].
To speed up training we use Batch Normalization [ Ioffe and Szegedy (2015)].

We evaluate the performance of the optimized model for recognizing hand
gestures in raspberry pi. We collect the dataset for our task, in order to
obtain a good dataset, we take images in different lightings, different
cameras, and angles which helps in creating a robust good accurate model. The
neural network is implemented and trained using a device like GPU which has
strong computational power. Then the model is optimized by reducing the
floating-point precision of trained parameters and by pruning the parameters.
The model is then evaluated and deployed on edge device like raspberry pi.

##  2 Experimental Setup

Figure 1: Neural Network Architecture

###  2.1 Preparation of dataset

Dataset is prepared by collecting images for each different class. i.e., for
each unique gesture to be recognized. To obtain a robust model many numbers of
images are captured in different light intensity, angles, background,
distance, camera for every gesture to be recognized. Then images belonging to
a unique class are stored in a separate folder. Different sized datasets of
images are created i.e., captured images are scaled to 64×64, 96×96, 128×128,
256×256 hence each size dataset we will have the images for each class of that
size. This is done to find the optimal size for input images such that the
images with less resolution are trained to the desired accuracy by which we
can reduce the model size and hence computations required for prediction.

####  2.1.1 Steps

  * •

Collect the images for each unique gesture under different lighting
conditions, background, camera, angle and save each gesture image in their
respective folders.

  * •

Create a unique folder for all the required sizes of images (E.g., folder_64,
folder_96 …etc)

  * •

Under each different folder (folders created for the required size dataset)
create the folders for each unique class (gesture) (E.g., one, two, three ….
etc).

  * •

Read the images from the original folder resize them and save accordingly. We
have used OpenCV to read images, resize them and store them accordingly. E.g.
read an image of class ‘one’

  * •

resize it to 64×64, 96×96, 128×128, 256×256 and store them in folder ‘one of
‘folder_64’, ‘folder_96’, ‘folder_128’, ‘folder_256’ respectively

Figure 2: Sample images in dataset

###  2.2 Training the Network

TensorFlow [ Abadi et al. (2016)] is used to create the network and load the
dataset during training. The images are normalized during every iteration
before feeding into the network and a few augmentation’s like horizontal flips
were performed in order to generalize the model well. Dataset is split into
85% for training and 15% for validation. Fig-2 shows the network architecture
for the training image dataset of size 256×256 with 3 channels (RGB). The
normalized input images are 2D-convolved with a kernel size of 3×3 and 32 such
filters are used, and the output is passed through the non-Linearity
activation function ReLU [21]. Then the 2D-max-pooling (kernel size 2×2) is
applied followed by Dropout. Then the output is 2D-convolved with a kernel
size of 3×3 (64 filters) and passing through ReLU activation followed by
2D-max-pooling and Dropout. Then the output is flattened followed by Dropout
and 2 Dense layers having 128 and 5 (as the number of unique gestures is 5)
hidden units respectively. The first hidden layer has the ReLU activation
function and the latter has SoftMax activation.

Table 1: Network structure Layer | Output shape | No. of parameters
---|---|---
Input | (None, 256,256,3) | 0
Conv2D | (None, 254,254,32) | 896
Max-pooling2D | (None, 127,127,32) | 0
Dropout | (None, 127,127,32) | 0
Conv2D | (None, 125,125,64) | 18496
Max-pooling2D | (None, 62,62,64) | 0
Dropout | (None, 62,62,64) | 0
Flatten | (None,246016) | 0
Dropout | (None,246016) | 0
Dense | (None,128) | 31490176
Dense | (None,5) | 645
Total params | 31510213 |

The model is trained using Tesla-K80 GPU which can do up to 8.73 Teraflops
single-precision and up to 2.91 Teraflops double-precision performance with
NVIDIA GPU Boost. The model is set to train for a maximum of 25 epochs
provided the validation loss will not increase continuously for 3 epochs in
that case the training is stopped, and the model weights are saved.
The model is trained using Adam optimizer with a batch size of 32. We used
Categorical cross-entropy as the loss function to train the model. categorical
cross-entropy is given by,

$CE=-\sum_{i}^{c}t_{i}\log(f(s)i)$ (1)

Where f(s)i represents SoftMax function and is given by,

$f(s){i}=\frac{e^{S{i}}}{\sum_{j}^{C}e^{s_{j}}}$ (2)

Where C represents the total number of class and sj are the score inferred by
the neural network for each class in C.

Figure 3: variation of accuracy for 256×256

Figure 4: variation of loss for 256×256

Figure 4 indicates how training and validation loss varied during training in
each epoch and we see that both training and validation loss reach a very low
value by 14 epochs. And the training is stopped after 14 epochs as the
validation loss keeps increasing by a small value. The best weights are saved.

Figure 4 indicates how accuracy varied during training after each epoch.
Training and validation accuracy reach desired values by the end of 14 epochs.

Figure 5: variation of accuracy for 64×64

Figure 6: variation of loss for 64×64

Figure 7: variation of accuracy for 96×96

Figure 8: variation of loss for 96×96

Figure 9: variation of accuracy for 128×128

Figure 10: variation of loss for 128×128

Similar results were obtained when tried with different image sizes and found
that image size with 96x96 works best among all during the test time. Figures
6 \- 10 indicates the variation of accuracy and loss for different input sizes
of the images during training. Since our aim is to fit the model in edge
device memory, it is suitable to choose the model which gives desirable
accuracy keeping image size as low as possible. The model trained for image
size with 96 × 96 was obtained over 98% accuracy on the validation set
indicating the best input size. The model trained with 128×128 and 256×256
images gave very good training accuracy (nearly 100%) but were able to give
only around 92% accuracy on the validation set indicating high variance of the
model. This may be due to fewer data and the usage of large neural networks.

Figure 11: Misclassifications of the network

In many a case one gesture will be similar to the others which usually makes
models to perform badly on them. To reduce such misclassifications higher
weight is assigned for the loss of such classes. Here slightly higher weight
is applied to gestures ‘one’, ‘three’, ‘four’ hence we see a smaller number of
misclassifications (Figure- 11). Else the misclassification for those gesture
will be slightly high.

##  3 Optimization

The trained model able to perform classifications has the parameters with the
floating-point precision of 64-bit occupies more memory while performing the
inference. Since floating-point arithmetic operations need more machine cycles
and the machine cycles required for an operation increases with increase in
floating-point precision. By reducing the floating-point precision machine
cycles required to perform the operation reduces along with the memory. The
reduction in the floating-point precision does not affect the accuracy more
and in rare cases, certain models may gain some accuracy as a result of the
optimization process . Since deep neural networks require more floating-point
operations for its prediction reducing floating-point precision makes it
suitable for edge devices.
Pruning of parameters is done properly using the representative dataset during
the model optimization i.e., reduce parameter count by structured pruning. We
use TensorFlow lite for reducing the floating-point precision of our trained
TensorFlow model which leads to optimization.

Figure 12: Variation of accuracy with optimization

Figure- 12 indicates the variation of accuracy with optimization for models
trained on different image sizes. For a non-optimized models accuracy
indicates the accuracy on validation set (100 examples) and for optimized
models’ accuracy is calculated on test set of 32 examples. We observe that the
model optimized for the Neural network trained on 96×96 images has good
accuracy and performance during inference. Since all the models have similar
accuracy, with models being trained on 96×96 and 64×64 images are slightly
better.

Figure 13: Variation of loss with optimization

Although we have similar accuracy between optimized and non-optimized models
the loss of optimized model is comparatively high. Figure 13 shows the
variation of loss with optimization for neural networks trained on different
image size. The loss for non-optimized models is computed on validation set of
100 examples and loss for optimized models is computed on test set of 32
examples.

Figure 14: Variation of model size with optimization.

Variation of model size with optimization for different input image sizes is
shown in the Figure 14. The size of the model reduces nearly to 33% of initial
size after optimization. We observe that the model with input image shapes of
96×96 and 64×64 after optimization reduces to 15.9MB and 6.5MB respectively
which could be deployed in edge devices.

Table 2: Network structure Image Size | Optimized | Loss | Accuracy | Model Size
---|---|---|---|---
128 x 128 | No | 0.206300 | 0.92370 | 88759408
128 x 128 | Yes | 1.063362 | 0.93750 | 29574240
64 x 64 | No | 0.073400 | 0.97730 | 19553392
64 x 64 | Yes | 0.533179 | 0.96875 | 6505568
96 x 96 | No | 0.089700 | 0.98110 | 47864936
96 x 96 | Yes | 0.566464 | 0.96875 | 15942712
256 x 256 | No | 0.543200 | 0.92800 | 378166376
256 x 256 | Yes | 1.756660 | 0.90625 | 126043192

Table- 2 shows the accuracy and loss of the model before and after the
optimization and the respective model size in bytes. Loss and accuracy for the
optimized model is computed on the test set containing 32 examples and for
non-optimized models they are computed on validation set of 100 examples.

##  4 Conclusion

The optimized models for this use case could be easily deployed on edge device
like raspberry pi. Since the memory required for the models with input image
shape 96×96 and 64×64 is few MB it could be fulfilled by edge devices easily.
In the cases if the edge computing is hard then the aid from neural compute
stick could be obtained for the deployment of the model.

## References

  * Mikolajczyk and Schmid (2004) Krystian Mikolajczyk and Cordelia Schmid.  Scale & affine invariant interest point detectors.  _International journal of computer vision_ , 60(1):63–86, 2004.
  * Dalal and Triggs (2005) Navneet Dalal and Bill Triggs.  Histograms of oriented gradients for human detection.  In _2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05)_ , volume 1, pages 886–893. Ieee, 2005.
  * (3) ImageNet Large Scale Visual Recognition Challenge.  Olga russakovsky, jia deng, hao su, jonathan krause, sanjeev satheesh, sean ma, zhiheng huang, andrej karpathy, aditya khosla, michael bernstein, alexander c. berg, li fei-fei. 2014.  _Computing Research Repository, Vol. abs/1409.0575_.
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , pages 740–755. Springer, 2014.
  * Netzer et al. (2011) Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading digits in natural images with unsupervised feature learning.  2011\.
  * Ranjith M S and Parameshwara (2020) Ranjith M S and S Parameshwara.  Optimizing neural networks for embedded systems.  _IRJET_ , 7(211), 2020.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Huang et al. (2017) Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4700–4708, 2017.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ , 25:1097–1105, 2012.
  * He et al. (2015) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1026–1034, 2015.
  * Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_ , pages 249–256. JMLR Workshop and Conference Proceedings, 2010.
  * Mishkin and Matas (2015) Dmytro Mishkin and Jiri Matas.  All you need is a good init.  _arXiv preprint arXiv:1511.06422_ , 2015.
  * Hinton et al. (2012a) Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov.  Improving neural networks by preventing co-adaptation of feature detectors.  _arXiv preprint arXiv:1207.0580_ , 2012a.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Hinton et al. (2012b) Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky.  Neural networks for machine learning lecture 6a overview of mini-batch gradient descent.  _Cited on_ , 14(8):2, 2012b.
  * You et al. (2019) Yang You, Jonathan Hseu, Chris Ying, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh.  Large-batch training for lstm and beyond.  In _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , pages 1–16, 2019.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Smith and Topin (2019) Leslie N Smith and Nicholay Topin.  Super-convergence: Very fast training of neural networks using large learning rates.  In _Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications_ , volume 11006, page 1100612. International Society for Optics and Photonics, 2019.
  * Ioffe and Szegedy (2015) Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _International conference on machine learning_ , pages 448–456. PMLR, 2015.
  * Abadi et al. (2016) Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al.  Tensorflow: Large-scale machine learning on heterogeneous distributed systems.  _arXiv preprint arXiv:1603.04467_ , 2016.
  * r (22) Model optimization | tensorflow lite.  URL https://www.tensorflow.org/lite/performance/model_optimizationon.  Retrieved from https://www.tensorflow.org/lite/performance/model_optimization.
  * Nair and Hinton (2010) Vinod Nair and Geoffrey E Hinton.  Rectified linear units improve restricted boltzmann machines.  In _Icml_ , 2010.
  * Hunter (2007) John D Hunter.  Matplotlib: A 2d graphics environment.  _Computing in science & engineering_, 9(03):90–95, 2007.
  * Ranjith M S et al. (2021) Ranjith M S and Santhosh, HS and Swamy, MS.  Machine Learning Algorithms for the Detection of Diabetes.  _IRJET_ , 8(27), 2021.
  * (25) Plotly Technologies Inc.  _Collaborative data science_.  Plotly Technologies Inc, Montreal, QC.
  * r (25) Tensorflow lite inference.  URL https://www.tensorflow.org/lite/guide/inference.  Retrieved from https://www.tensorflow.org/lite/guide/inference.
  * r (26) Tensorflow model optimization.  URL https://www.tensorflow.org/model_optimization/guide.  Retrieved from https://www.tensorflow.org/model_optimization/guide.
  * r (27) Tensorflow model optimization toolkit — float16 quantization halves model size — the tensorflow blog.  URL https://blog.tensorflow.org/2019/08/tensorflow-model-optimization-toolkit_5.html.  Retrieved from https://blog.tensorflow.org/2019/08/tensorflow-model-optimization-toolkit_5.html.
  * r (28) Tensorflow lite for microcontrollers.  URL https://www.tensorflow.org/lite/microcontrollers.  Retrieved from https://www.tensorflow.org/lite/microcontrollers.