# Reconciliation of Statistical and Spatial Sparsity For Robust Image and
Image-Set Classification

Hao Cheng, Kim-Hui Yap, and Bihan Wen H. Cheng, K-H. Yap and B. Wen are with
the School of Electrical and Electronic Engineering, Nanyang Technological
University, 639798 Singapore<EMAIL_ADDRESS>(ekhyap,
<EMAIL_ADDRESS>
###### Abstract

Recent image classification algorithms, by learning deep features from large-
scale datasets, have achieved significantly better results comparing to the
classic feature-based approaches. However, there are still various challenges
of image classifications in practice, such as classifying noisy image or
image-set queries, and training deep image classification models over the
limited-scale dataset. Instead of applying generic deep features, the model-
based approaches can be more effective and data-efficient for robust image and
image-set classification tasks, as various image priors are exploited for
modeling the inter- and intra-set data variations while preventing over-
fitting. In this work, we propose a novel Joint Statistical and Spatial Sparse
representation, dubbed J3S, to model the image or image-set data for
classification, by reconciling both their local patch structures and global
Gaussian distribution mapped into Riemannian manifold. To the best of our
knowledge, no work to date utilized both global statistics and local patch
structures jointly via joint sparse representation. We propose to solve the
joint sparse coding problem based on the J3S model, by coupling the local and
global image representations using joint sparsity. The learned J3S models are
used for robust image and image-set classification. Experiments show that the
proposed J3S-based image classification scheme outperforms the popular or
state-of-the-art competing methods over FMD, UIUC, ETH-80 and YTC databases.

###### Index Terms:

Sparse representation, Gaussian model, Riemannian manifold, Dictionary
learning, Visual classification.

##  I Introduction

Image classification is a fundamental problem in image processing and computer
vision. Comparing to classic algorithms based on pre-defined features, recent
image classification schemes applied machine learning techniques to optimize
feature representation directly from the data themselves. More recently, deep
learning approaches for image classification have achieved the state-of-the-
art results on many benchmarking datasets, such as the popular ImageNet [1].
Despite of the promising performance achieved under simple and ideal problem
setups, there are still various challenges when (i) classification based on
queries that contain a set of object variations (i.e., image-set
classification), or (ii) the image data is limited or with relatively low
quality (i.e., weakly supervised classification).

To be specific, while conventional classification tasks process a single image
in each query, image-set classification [2, 3, 4, 5] has recently gained more
attention, in which each query set contains multiple images with strong
correlation (e.g., query object with multiple views, poses or illuminations).
Such type of algorithms are widely applied in applications such as video-based
face classification [2], multi-spectral image classification, etc. Comparing
to the single-image classification algorithms, effective image-set methods
need to additionally exploit the hidden structure among image sets, e.g., the
inter- and intra-set data variations. Furthermore, popular deep features tend
to be generic and incorporate very little prior knowledge by learning from
large-scale, high-quality, and fully annotated training datasets [1]. Such
approaches are ideal with fully supervised learning, but less data-efficient
and robust when training sets are small-scale or corrupted (e.g., noisy).

Recent works on sparse signal modeling have demonstrated their effectiveness
in image representation for various tasks [6, 7, 8, 9, 10, 11, 12]. Comparing
to the deep features, sparse representation is model-based, thus much more
robust to practical challenges such as noise or over-fitting [13]. While many
existing works focused on exploiting image patch-based sparsity, global
statistical properties are typically ignored or failed to be incorporated
jointly in a principled approach. Recent works show that high-order statistics
of image features are critical in classification tasks [14, 15, 16, 17],
leading to better results comparing to many first-order methods.

In this work, we propose a novel Joint Statistical and Spatial Sparse
representation (J3S), i.e., learning the coupled dictionaries for both local
patch features, and the global data Gaussian distribution mapped into
Riemannian manifold. Their dictionary-domain sparse coefficients are
reconciled by solving a sparse coding problem with joint sparsity. We propose
an efficient yet effective alternating minimization algorithm to solve the J3S
sparse coding problem. To the best of our knowledge, no work to date utilized
both global statistics and local patch structures jointly via sparse
representation for image classification. Furthermore, we apply the learned J3S
model for robust image-set and single-image classification applications.
Extensive experimental results on material classification, object recognition
and video-based face recognition tasks are presented, and we demonstrate that
the proposed J3S-based image classification scheme outperforms the popular or
state-of-the-art competing methods.

In short, the contributions of this paper include:

  * •

Learning global statistical and local patch dictionaries for visual
classification task by coupling them with joint sparsity;

  * •

Utilizing principal component analysis (PCA) to reduce the J3S model
complexity while maintaining the effectiveness;

  * •

Investigating the robustness of the proposed model under various conditions
i.e., noisy condition and few-shot setting.

  * •

Achieving the state-of-the-art results on both noisy image and image-set
classification tasks.

The remainder of this article is organized as follows. Section II summarizes
the related work on image or image-set classification problems, including
manifold learning, deep learning and sparse representation. Section III
introduces two kinds of dictionary learning methods based on Gaussian-based
statistical information and patch-based spatial information respectively, the
proposed J3S model and the classification module. Section IV describes the
solution of the proposed J3S model based on alternation minimization and
analyzes the time and space complexity as well as a simple strategy for model
acceleration effectively. Section V demonstrates the performance of the
proposed J3S model for image and image-set tasks over several standard
databases under different conditions such as noise and few-shot. Section VI
concludes with proposals for future work. The preliminary work has appeared in
[18]. 111Significant changes have been made compared to our previous work in
[18]. First, we improve the J3S method by reducing the model complexity with a
simple and efficient approach. Second, we add more description and analysis on
the dictionary learning and classification. Third, we include new experiments
on ablation study to investigate the model convergence and parameter
selection. Furthermore, we conduct an extra experiment on an object
recognition task based on the ETH-80 database to evaluate the generalizability
of the proposed J3S model in different scenarios. Finally, we conduct some
additional popular settings, e.g., noisy condition and few-shot setting are
conducted to validate the performance and robustness of the proposed J3S
model.

##  II Related Work

Image-set classification aims to identify the common class of a multi-image
query. The inherent properties of each query set can be modeled effectively by
popular methods such as manifold learning, deep learning, sparse coding, etc.

Manifold Learning: The classic methods based on Discriminant Canonical
Correlations (DCC) [19] proposed to classify image sets by maximizing the
canonical correlations of within-class sets and minimizing the canonical
correlations of between-class sets. Later on, more subspace methods [20] were
proposed to simplify the geometric structure learning for image sets. However,
these approaches are limited as most image sets lie on a Riemannian manifold
rather than Euclidean subspaces [21, 22], e.g., symmetric positive definite
(SPD) manifold is widely used to represent image sets. To ease the
computation, the Log-Euclidean Riemannian Metric (LERM) framework [23]
proposed to map data from SPD manifold to its tangent Euclidean space.
Besides, Log-Euclidean Manifold Learning (LEML) [22] projects the original SPD
manifold to a lower-dimension discriminative SPD manifold while preserving its
original geometry. More recently, Riemannian Manifold Metric Learning (RMML)
[24] proposed a more generalized metric learning method which can be applied
to multiple manifolds. From a statistical perspective, when modeling image
sets or the multi-channel features via Gaussian distribution, their covariance
matrices for a collection of Gaussian can form a Riemannian manifold of SPD
matrices [21, 25, 26]. Covariance Discriminative Learning (CDL) [21] derived a
Riemannian kernel function to map covariance matrix from manifold space to
Hilbert space, where kernelized linear methods can be used for learning.

Deep Learning: Recently, more works on deep learning have shown its capability
for image-set classification [27, 28, 29]. Deep Reconstruction Model (DRM)
[27] learns a template deep reconstruction model using neural networks and
then uses the minimal reconstruction residual to classify a query set. Multi-
manifold deep learning (MMDML) [28] maps multiple sets of image into a shared
feature subspace to leverage the nonlinear information. More recently, Deep
Match Kernels (DMK) [29] is proposed for image-set classification without
considering specific assumptions on image distribution and geometrical
structures and builds local match kernels to leverage its generic deep
features.

Methods |  | Feature
---
Learning
| Model
---
based
| Manifold
---
Space
| Small-Scale
---
Training
DCC [19] |  | ✓ |  | ✓
AHISD/CHISD [20] | ✓ | ✓ |  | ✓
LEML [22] |  | ✓ | ✓ | ✓
RMML [24] |  | ✓ | ✓ | ✓
CDL [21] |  | ✓ | ✓ | ✓
RSR [30] | ✓ |  | ✓ | ✓
KGDL [31] | ✓ |  | ✓ | ✓
DRM [27] | ✓ | ✓ |  |
MMDML [28] | ✓ | ✓ | ✓ |
DMK [29] | ✓ | ✓ |  |
Proposed | ✓ | ✓ | ✓ | ✓
J3S
TABLE I: Comparison of the key attributes between the proposed J3S method, and
other image-set classification algorithms.

Sparse Representation: Sparse coding based classification represents a query
sample on a dictionary composed of the training samples of all classes, and
then classified by the reconstruction error of each class [32, 33, 34, 35].
Besides, the sparse coefficients can be used as the extracted features for
classification, e.g., linear spatial pyramid matching [36]. Most existing
works focus on sparse coding and dictionary learning on the zero-order
information, i.e., the original feature space, while the first-order and
second-order statistics contain global information and take the correlation of
the data into account. They can be more robust to variations in images and
videos applications, e.g., variations of poses, illumination and occlusions.

Table I summarized the aforementioned related methods, as well as the proposed
J3S method for image-set classification. Furthermore, some recent works also
proposed sparse coding and dictionary learning models on Riemannian manifold
of SPD matrices and Grassmann manifold: Sparse coding on Riemannian manifold
can be converted to a kernel sparse coding problem by deriving valid kernels
for SPD manifold [30, 10] or Grassmann manifold [31]. However, none of the
existing works combined statistical with spatial priors in the sparse
representation. Besides, the robustness of the image-set classification has
been rarely investigated.

##  III Dictionary Construction and Joint Sparse Representation

In this section, we present the J3S model for classification tasks, including
the dictionary construction of statistical and spatial models and joint sparse
coding. Our proposed J3S model can deal well with different types of input
data such as single image and image set.

To obtain the unified feature representations for classifying both a single
image and an image data set, we apply the corresponding data preprocessing
methods. Specifically, for an image set ${\bf{M}}_{i}$ with feature of each
image $\left\\{x_{1},x_{2},...,x_{n}\right\\},x_{j}\in\mathbb{R}^{d}$, we
combine them to construct the image set representation ${\bf{X}}_{i}$
directly; For a single image ${\bf{N}}_{i}$, we employ its deep feature
representation $f({\bf{N}}_{i})\in\mathbb{R}^{w\times h\times c}$ using a pre-
trained CNN extractor as local features to construct
${\bf{X}}_{i}\in\mathbb{R}^{d\times c}$ where $d=w\times h$. Thus, both an
image or image set can be represented in a similar form as
$\mathbf{X}_{i}=\left\\{\bm{x}_{1},\bm{x}_{2},...,\bm{x}_{m_{i}}\right\\}\in\mathbb{R}^{d\times
m_{i}}$, where $d$ is the feature dimension of each image and $m_{i}$ is the
number of image in each image set or the number of channel for a single image.

###  III-A Statistical Dictionary Construction

Based on the Gaussian statistical model, we need to compute the mean vector
$\bm{\mu_{i}}$ and covariance matrix ${\mathbf{C}}_{i}$ in Reproducing Kernel
Hilbert Space (RKHS) for the corresponding Gaussian descriptor
${G_{i}}\left({\bm{\mu_{i}}},{\mathbf{C}}_{i}\right),i=1,2,...,N$. We map
$\mathbf{X}_{i}$ into an RKHS by the mapping function $\eta(\cdot)$ with
Hellinger’s kernel, the mean vector $\bm{\mu_{i}}$ and covariance matrix
${\mathbf{C}}_{i}$ can be computed as:

$\bm{\mu_{i}}=\frac{1}{m_{i}}\sum_{k=1}^{m_{i}}\eta\left(\bm{x}_{k}\right),{\mathbf{C}_{i}}=\frac{1}{m_{i}}\Phi(\mathbf{X}_{i})\mathbf{J}\Phi(\mathbf{X}_{i})^{T}.$
(1)

Here $\Phi(\mathbf{X}_{i})=[\eta(\bm{x}_{1}),...,\eta(\bm{x}_{m_{i}})]$, and
$\mathbf{J}=\mathbf{I}_{d}-\frac{1}{d}\mathbf{1}_{d}\mathbf{1}_{d}^{T}$ is the
centering matrix. However, when the dimension of the original features (i.e.,
$d$) is very high, and the number of samples (i.e., $m_{i}$) is small, such a
Gaussian descriptor ${G_{i}}\left({\bm{\mu_{i}}},{\mathbf{C}}_{i}\right)$ can
not work well. To solve this problem, following [25], we estimate the robust
covariance matrix $\widehat{\mathbf{\Sigma}}_{i}$ by solving a regularized
maximum likelihood estimation problem as:

$\min_{\widehat{\Sigma}_{i}}\log|\widehat{\mathbf{\Sigma}}_{i}|+\operatorname{tr}\left(\widehat{\mathbf{\Sigma}}_{i}^{-1}\widehat{\mathbf{C}}_{i}\right)+\alpha
D_{\mathrm{vN}}(\mathbf{I}_{d},\widehat{\mathbf{\Sigma}}_{i}),$ (2)

where $D_{\mathrm{vN}}$ is the von Neumann matrix divergence [37] of two
matrices and $\alpha\in(0,1)$ is a regularizing parameter. The optimal
solution of problem (2) can be computed as:

$\displaystyle\widehat{\mathbf{\Sigma}}_{i}$
$\displaystyle=\widehat{\mathbf{U}}_{i}\operatorname{diag}\left(\lambda_{k}\right)\widehat{\mathbf{U}}_{i}^{T},$
(3) $\displaystyle\lambda_{k}$
$\displaystyle=\sqrt{\left(\frac{1-\alpha}{2\alpha}\right)^{2}+\frac{\delta_{k}}{\alpha}}-\frac{1-\alpha}{2\alpha}.$

Here $\delta_{k}$ is the diagonal matrix of the singular values in decreasing
order, and $\widehat{\mathbf{U}}_{i}$ is the orthogonal matrix consisting of
the eigenvectors corresponding to the singular values.
$\widehat{\mathbf{U}}_{i}$ and $\delta_{k}$ are computed by the singular value
decomposition (SVD) of the covariance matrix $\mathbf{C}_{i}$ as
$svd(\mathbf{C}_{i})=\widehat{\mathbf{U}}_{i}\operatorname{diag}\left(\delta_{k}\right)\widehat{\mathbf{U}}_{i}^{T}$.

By using the mean vector $\mu_{i}$ and robust covariance matrix
$\widehat{\mathbf{\Sigma}}_{i}$, we can define the embedding symmetric
positive definite matrix ${\bf{P}}_{i}$ as:

${\bf{P}}_{i}=\left[\begin{array}[]{cc}{\widehat{\mathbf{\Sigma}}_{i}+\beta^{2}\bm{\mu_{i}}\bm{\mu_{i}}^{T}}&{\beta\bm{\mu_{i}}}\\\
{\beta\bm{\mu_{i}}^{T}}&{1}\end{array}\right]\in\mathbb{R}^{(d+1)\times(d+1)},$
(4)

where $\beta>0$ is a parameter to balance the orders of magnitude between
them.

Figure 1: The framework of Joint Statistical and Spatial Sparse representation
of image and image set classification.

###  III-B Spatial Dictionary Construction

Similarly, given a sample $\mathbf{X}_{i}$, considering spatial information,
we can learn a patch-based unitary dictionary ${\mathbf{D}}_{i}$ from the
feature map of a single image or gray feature of an image set. For a single
image, we choose the original image or the deep feature as the input to learn
a patch-based unitary dictionary. In contrast, for an image set, we combine
each single image feature to construct a patch-based unitary dictionary to
exploit within the class structure. For image and image set classification,
the objective is to learn a unitary dictionary
$\mathbf{D}_{i}\in\mathbb{R}^{p\times p}$ based on 2D image patches
constructed from sample $\mathbf{X}_{i}$ by solving the following problem with
the synthesis model as:

$\\{\hat{\mathbf{D}}_{i},\hat{\gamma_{k}}\\}=\underset{\mathbf{D}_{i},\\{\gamma_{k}\\}}{\operatorname{argmin}}\sum_{k=1}^{N}\left\|\mathbf{C}_{k}\bm{u}_{i}-\mathbf{D}_{i}\gamma_{k}\right\|_{2}^{2},\text{
s.t. }\mathbf{D}_{i}^{H}\mathbf{D}_{i}=\mathbf{I}_{p},$ (5)

where $\mathbf{C}_{k}\bm{u}_{i}$ is used for extracting patches from
$\mathbf{X}_{i}$, $\bm{u}_{i}$ is the vectorized form of $\mathbf{X}_{i}$, $N$
is the number of total patches and $\mathbf{I}_{p}$ is the identity matrix.

Sparse coding problems under the synthesis model are NP-hard in general, and
even the approximate algorithms are typically expensive [38]. However, since
problem (5) learns the unitary dictionary, it is equivalent to the unitary
transform learning problem [39], i.e., a signal $u$ is approximately
sparsifiable using a learned unitary transform
$\mathbf{W}_{i}\in\mathbb{R}^{m\times p}$, as
$\mathbf{W}_{i}u=\gamma+\mathit{e}$, where $\gamma\in\mathbb{R}^{p}$ is sparse
and $\mathit{e}$ is a small residual in the transform domain. The
corresponding transform learning problem is formulated as

$\hat{\mathbf{W}}_{i}=\underset{\mathbf{W}_{i}}{\operatorname{argmin}}\sum_{k=1}^{N}\left\|\mathbf{W}_{i}\mathbf{C}_{k}\bm{u}_{i}-\gamma_{k}\right\|_{2}^{2}.\text{
s.t. }\mathbf{W}_{i}^{H}\mathbf{W}_{i}=\mathbf{I}_{p}.$ (6)

Based on [39], the two sparsity models can be unified under the unitary
dictionary assumption, i.e., $\mathbf{D}_{i}=\mathbf{W}_{i}^{T}$, and
$\mathbf{D}_{i}^{T}\mathbf{D}_{i}=\mathbf{I}_{p}$.

###### Proposition 1

Under the unitary dictionary assumption, the problems (5) and (6) are
equivalent.

###### Proof 1

Based on the unitary dictionary assumption, we have
$\mathbf{W}_{i}\mathbf{D}_{i}=\mathbf{I}_{p}$ and
$\|\mathbf{W}_{i}\tau\|_{2}=\|\tau\|_{2},\forall\tau.$ Thus, the function in
problem (5) is identical to that in problem (6), i.e.,
$\left\|\mathbf{C}_{k}\bm{u}_{i}-\mathbf{D}_{i}\gamma_{k}\right\|_{2}^{2}=\left\|\mathbf{W}_{i}\mathbf{C}_{k}\bm{u}_{i}-\gamma_{k}\right\|_{2}^{2}$.
Therefore, the problems (5) and (6) become equivalent, and
$\hat{\mathbf{D}}_{i}=\hat{\mathbf{W}}_{i}^{T}$.

Thus, we can obtain the optimal dictionary
$\mathbf{D}_{i}=\hat{\mathbf{W}}_{i}^{T}$ in (5) by solving its equivalent
problem (6) which has an exact and closed-form solution [40], i.e.,
$\hat{\mathbf{W}}_{i}=\mathbf{G}\mathbf{S}^{H}$ where $\mathbf{G},\mathbf{S}$
are computed by the SVD of
$\mathbf{K}\triangleq\sum_{k=1}^{N}\left(\mathbf{C}_{k}\bm{x}_{i}\right)\gamma_{k}^{H}$
as $svd(\mathbf{K})=\mathbf{S}\Sigma\mathbf{G}^{H}$.

Fig. 1 illustrates the framework of J3S, in which a sample
${\hat{\mathbf{X}}}_{j}$ (either an image set or a single image) can be
modelled by a statistical model to obtain the embedding SPD matrix
$\mathbf{P}_{j}$, and simultaneously modelled by a spatial patch-based model
to generate a unitary transform dictionary $\mathbf{D}_{j}$, which are then
used for joint sparse coding.

Figure 2: The framework of classification module of Joint Statistical and
Spatial Sparse representation model.

###  III-C J3S Sparse Coding

To reconcile the two types of dictionaries generated from statistical Gaussian
modeling and spatial patch-wise unitary dictionary learning, we propose the
joint statistical and spatial sparse representation (J3S) model, i.e., for any
query sample, we impose joint sparsity on their statistical and spatial
dictionary-domain coefficients, to maintain the consistency and dependency of
their individual sparse representation. For simplicity, we use the following
symbols to simplify the objective function,

$\begin{split}\begin{aligned}
{\bf{U}_{i}}&=\left[{\varphi\left({{{\mathbf{P}}_{i1}}}\right),\varphi\left({{{\bf{P}}_{i2}}}\right),...,\varphi\left({{{\bf{P}}_{ik}}}\right)}\right],\\\
{\bf{V}_{i}}&=\left[{\phi\left({{{\mathbf{D}}_{i1}}}\right),\phi\left({{{\mathbf{D}}_{i2}}}\right),...,\phi\left({{{\mathbf{D}}_{ik}}}\right)}\right].\end{aligned}\end{split}$

Here, $\varphi\left({{{\bf{P}}_{{i_{1}...k}}}}\right)$ and
$\phi\left({{{\bf{D}}_{{i_{1}...k}}}}\right)$ are the mapping function of
Gaussian and patch-based unitary dictionary corresponding to the $k$ training
samples belonging to the $i$-th class, respectively.

For statistical Gaussian-based dictionary, each embedding matrix
$\mathbf{P}_{i}$ is an SPD matrix which can be viewed as a point on the
corresponding SPD manifold based on Eq (4). Direct vectorization of the SPD
matrix to generate a dictionary will destroy the intrinsic structure, which
may cause information loss. To avoid such loss while measuring the similarity
between two matrices on the SPD manifold, we use a general framework called
LERM [23] to map each of them to its tangent space through matrix logarithm
operation $log(\cdot)$. By using such embedding, we directly measure the
similarity on the tangent space using the Euclidean distance. In this way, we
can get the vectorized form of each matrix as the statistical Gaussian-based
feature for feature representation. To simplify the calculation of SPD matrix
$log(\mathbf{P}_{i})$, here we only extract upper triangle elements to
construct the dictionary, and thus the mapping function can be written as:

$\varphi(\mathbf{P})=triu(log(\mathbf{P})).$ (7)

For patch-based unitary model, $\phi(\cdot)$ is the vectorized mapping
function of a unitary dictionary.

Given the joint statistical and spatial dictionaries, we compute the
coefficient vectors $\bm{\alpha}_{j}$ and $\bm{\gamma}_{j}$ of the $j$-th
query sample with feature representations $\varphi({\hat{\mathbf{P}}}_{j})$
and $\phi({\hat{\mathbf{D}}}_{j})$ by solving the following problem:

$\displaystyle\underset{\\{\bm{\alpha}_{j},\bm{\gamma}_{j}\\}}{\operatorname{min}}\;\theta\left\|\varphi\left({\hat{\mathbf{P}}}_{j}\right)-[{\bf{U}_{1},\bf{U}_{i},...,\bf{U}_{n}}]\bm{\alpha}_{j}\right\|_{2}^{2}$
(8)
$\displaystyle+(1-\theta)\left\|\phi\left({\hat{\mathbf{D}}}_{j}\right)-[{\bf{V}_{1},\bf{V}_{i},...,\bf{V}_{n}}]\bm{\gamma}_{j}\right\|_{2}^{2}$
$\displaystyle+{\lambda_{1}}\left\|\bm{\alpha}_{j}\right\|_{2}^{2}+{\lambda_{2}}\left\|\bm{\gamma}_{j}\right\|_{2}^{2}+{\lambda_{3}}{\left\|{\left[\bm{\alpha}_{j},\bm{\gamma}_{j}\right]}\right\|_{2,1}},$

where $\theta\in(0,1)$ is the weighting parameter defined to balance the scale
of statistical model and patch-based model, and
$\varphi(\cdot),\phi\left(\cdot\right)$ are used to map two kinds of
representations, respectively.
${\left\|{\left[\bm{\alpha}_{j},\bm{\gamma}_{j}\right]}\right\|_{2,1}}$ is the
$\ell_{2,1}\textnormal{-}$norm for row sparse.

By solving the optimization problem in (8), we can get the representation
coefficient vectors $\bm{\alpha}_{j}$ and $\bm{\gamma}_{j}$ that correspond to
the Gaussian and patch-based dictionary models, respectively. With two
coefficient vectors $\bm{\alpha}_{j}$ and $\bm{\gamma}_{j}$, we can compute
the reconstruction loss of the $j$-th query sample using only one particular
sub-dictionaries and corresponded coefficient sub-vectors of the $i$-th class
as:

$\displaystyle{e}_{i}^{j}=\theta\left\|\varphi\left({\hat{\mathbf{P}}}_{j}\right)-\tilde{\mathbf{P}}_{j}^{i}\right\|_{2}^{2}+(1-\theta)\left\|\phi\left({\hat{\mathbf{D}}}_{j}\right)-\tilde{\mathbf{D}}_{j}^{i}\right\|_{2}^{2}$
(9)
$\displaystyle+{\lambda_{1}}\left\|\bm{\alpha}_{j}^{i}\right\|_{2}^{2}+{\lambda_{2}}\left\|\bm{\gamma}_{j}^{i}\right\|_{2}^{2}+{\lambda_{3}}{\left\|{\left[\bm{\alpha}_{j}^{i},\bm{\gamma}_{j}^{i}\right]}\right\|_{2,1}}.$

Here $\bm{\alpha}_{j}^{i}$ and $\bm{\gamma}_{j}^{i}$ are particular
coefficient sub-vectors of the $i$-th class for $J$-th query sample.
$\tilde{\mathbf{P}}_{j}^{i}=\bf{U}_{i}\bm{\alpha_{j}^{i}}$ and
$\tilde{\mathbf{D}}_{j}^{i}=\bf{V}_{i}\bm{\gamma_{j}^{i}}$ are reconstructed
statistical and spatial representations of the $j$-th query sample,
respectively. $\bf{U}_{i}$ and $\bf{V}_{i}$ are two sub-dictionaries of
training samples from the $i$-th class. $\bm{\alpha_{j}^{i}}$ and
$\bm{\gamma_{j}^{i}}$ are the coefficient sub-vectors containing the sparse
code corresponding to the training samples from the $i$-th class. Moreover,
(8) and (9) share similar regularized terms to constrain the overall sparsity
for sparse coding. While for classification, we only keep the reconstruction
loss term, ignoring the influence of the regularized term on it, thus the
reconstruction loss can be rewritten as:

$\tilde{e}_{i}^{j}=\theta\left\|\varphi\left({\hat{\mathbf{P}}}_{j}\right)-\tilde{\mathbf{P}}_{j}^{i}\right\|_{2}^{2}+(1-\theta)\left\|\phi\left({\hat{\mathbf{D}}}_{j}\right)-\tilde{\mathbf{D}}_{j}^{i}\right\|_{2}^{2}.$
(10)

For a visual classification task, the most commonly used algorithm is Nearest
Neighbor (NN), aiming to find the closest labeled sample to the current query
sample according to the pre-defined metric methods and classify the query
sample into the category corresponding to the closest sample. Inspired by the
idea of NN, we assume that features from the same class should be easier to
reconstruct, since their feature representations contain similar embeddings,
while features from different classes will be more difficult and produce
larger reconstruction errors. For $j$-th query sample, we utilize the
reconstruction loss defined in (10) as the measurement for classification and
measure similarity in terms of the overall representation of the whole
category.

$\hat{y}_{j}=\underset{i}{\operatorname{argmin}}\left\\{{\tilde{e}_{i}^{j}}\right\\},\;\;for\;\;i=1,2,...,N.$
(11)

Here $e_{i}$ is the reconstruction error of the $i$-th class computed by (10)
and $\hat{y}_{j}$ is the predicted label of $j$-th query sample.

Fig. 2 shows how the classification module works. Specifically, to classify
the query sample, for each class $i$, we only use labeled samples of the
corresponding category $i$ for joint sparse representation to reconstruct the
query sample. The query data $({\hat{\mathbf{P}}}_{j},{\hat{\mathbf{D}}}_{j})$
can then be classified according to the weighted reconstruction error of each
class.

##  IV Algorithm

We propose a joint sparse representation model for image and image-set
classification tasks. It is obvious that each sub-problem of (8) is convex.
Thus we use alternation minimization to solve the optimization problem.

Update $\bm{\alpha}_{j},\bm{\gamma}_{j}$: The partial derivatives of the
objective function with respect to the $\bm{\alpha}_{j},\bm{\gamma}_{j}$ will
be set to 0.

$\displaystyle\frac{{\partial f}}{\partial\bm{\alpha}_{j}}$
$\displaystyle=\theta\left(-2{{\bf{U}}^{T}}\varphi\left({\hat{\mathbf{P}}}_{j}\right)+2{{\bf{U}}^{T}}{\bf{U}}\bm{\alpha}_{j}\right)$
(12)
$\displaystyle\quad\quad\quad+2{\lambda_{1}}{\bf{I}}\bm{\alpha}_{j}+2{\lambda_{3}}{{\bf{G}}_{j}}\bm{\alpha}_{j}=0.$
$\displaystyle\frac{{\partial f}}{\partial\bm{\gamma}_{j}}$
$\displaystyle=(1-\theta)\left(-2{{\bf{V}}^{T}}\phi\left({\hat{\mathbf{D}}}_{j}\right)+2{{\bf{V}}^{T}}{\bf{V}}\bm{\gamma}_{j}\right)$
(13)
$\displaystyle\quad\quad\quad+2{\lambda_{2}}{\bf{I}}\bm{\gamma}_{j}+2{\lambda_{3}}{{\bf{G}}_{j}}\bm{\gamma}_{j}=0.$

where ${\bf{G}}_{j}$ is a diagonal matrix with the $k$-th diagonal element as
$\frac{1}{{2{{\left\|{\left[{\bm{\alpha}_{j}}_{k},{\bm{\gamma}_{j}}_{k}\right]}\right\|}_{2}}}}$.

Thus we can get the iteration of $\bm{\alpha}_{j},\bm{\gamma}_{j}$ as:

${\bm{\alpha}_{j}}={\left({{{\bf{U}}^{T}}{\bf{U}}+\frac{{{\lambda_{1}}}}{{\theta}}{\bf{I}}+\frac{{{\lambda_{3}}}}{{\theta}}{{\bf{G}}_{j}}}\right)^{-1}}{{\bf{U}}^{T}}\varphi\left({\hat{\mathbf{P}}}_{j}\right).$
(14)
${\bm{\gamma}_{j}}={\left({{{\bf{V}}^{T}}{\bf{V}}+\frac{{{\lambda_{2}}}}{{1-\theta}}{\bf{I}}+\frac{{{\lambda_{3}}}}{{1-\theta}}{{\bf{G}}_{j}}}\right)^{-1}}{{\bf{V}}^{T}}\phi\left({\hat{\mathbf{D}}}_{j}\right).$
(15)

Update ${\mathbf{G}}_{j}$: ${\mathbf{G}}_{j}$ can be updated as follows:

${{{\mathbf{G}}_{j}}_{kk}}=\frac{1}{{2{{\left\|{\left[{\bm{\alpha}_{j}}_{k},{\bm{\gamma}_{j}}_{k}\right]}\right\|}_{2}}+eps}}.$
(16)

where $eps=1e\textnormal{-}16$ is an offset to prevent the unsolvable problem
of Eq (16) in this paper.

###  IV-A Complexity Analysis

We discuss the time and space complexity of our proposed J3S model. Compared
with the steps of the dictionary construction and sparse coding, which require
to construct the dictionaries and learn sparse code jointly, the
classification part is calculated only based on the results obtained in (8),
so we do not consider the impact of the classification part on the complexity
of the algorithm here. As the sub-problems of (8) are all convex and the
objective functions are all lower-bounded, the optimization algorithm can
converge to a local minimum [41]. The time complexity consists of the updating
of $\bm{\alpha}_{j}$, $\bm{\gamma}_{j}$, and ${\mathbf{G}}_{j}$. The
computational complexity of $\bm{\alpha}_{j}$ is $O(N^{2}d_{1}+N^{3})$, and
the computational complexity of $\bm{\gamma}_{j}$ is $O(N^{2}d_{2}+N^{3})$.
Hence, the main time complexity of the proposed algorithm is
$O(s(N^{2}d_{m}+N^{3}))$, where $s$ is the iteration number, $N$ is the number
of training samples, $d_{1}=d(d+1)/2$ and $d_{2}=p\times p$ are dimensions of
two dictionaries, respectively. $d$ is the number of channels or samples and
$w$ is the patch size, and $d_{m}$ is the larger feature dimension of two
dictionaries.

For space complexity, the proposed J3S model needs to save two dictionaries
$\varphi(\mathbf{P})$ and $\phi(\mathbf{D})$ for all samples, a pair of sparse
vectors $\alpha_{j},\gamma_{j}$, and the corresponding matrix $\mathbf{G}$ for
each query sample. The dimensions of statistical dictionary
$\varphi(\mathbf{P})$ and unitary dictionary $\phi(\mathbf{D})$ are equal to
$d_{1}$ and $d_{2}$, respectively.

###  IV-B A simple and effective strategy for Model Acceleration

The time and space complexity of the J3S model depends on the dimension of two
dictionaries. Referring to the process of dictionary construction introduced
earlier, we use the lower triangle form to store matrix information of
$\varphi(\mathbf{P})$ for dimensionality reduction. However, the dimension of
dictionary is still too high when using deep feature with $d=512$ while the
number of training samples $N$ is only hundreds. In this way, $d_{m}\gg N$ and
the time complexity is close to $O(s(N^{2}d_{m}))$, which is not conducive to
the application of the proposed algorithm.

To reduce time and space cost, we use the commonly used principal component
analysis (PCA) method [42] to perform dimensionality reduction operations on
the two dictionaries to eliminate redundant information between different
sizes. For simplicity, we use $\mathbf{S}_{1}\in\mathbb{R}^{N\times
d_{1}},\mathbf{S}_{2}\in\mathbb{R}^{N\times d_{2}}$ to represent the
dictionary of statistical Gaussian model and patch-based model generated from
the whole dataset, respectively as

$\begin{split}\begin{aligned}
{\mathbf{S}_{1}}&=\left[{\varphi\left({{{\mathbf{P}}_{1}}}\right),\varphi\left({{{\mathbf{P}}_{2}}}\right),...,\varphi\left({{{\mathbf{P}}_{N}}}\right)}\right]^{T},\\\
{\mathbf{S}_{2}}&=\left[{\phi\left({{{\mathbf{D}}_{1}}}\right),\phi\left({{{\mathbf{D}}_{2}}}\right),...,\phi\left({{{\mathbf{D}}_{N}}}\right)}\right]^{T}.\end{aligned}\end{split}$

Specifically, we learn a principal component transformation
$\mathit{f_{pca}}:\mathbb{R}^{N\times
d_{m}}\rightarrow\mathbb{R}^{N\times(N-1)}$ to map the data from the original
space to a new low-dimension space. Using the transformation
$\mathit{f_{pca}}$, we get new low-dimensional dictionaries
$\hat{\mathbf{S}}_{1}$ and $\hat{\mathbf{S}}_{2}$ of the statistical and
spatial models as:

${\hat{\mathbf{S}}_{1}}=\mathit{f_{pca}}(\mathbf{S}_{1}),\;{\hat{\mathbf{S}}_{2}}=\mathit{f_{pca}}(\mathbf{S}_{2}).$
(17)

After PCA operation, we store these two matrices for sparse representation
learning. For each iteration, we select the columns corresponding to the
training samples of the $i$-th class to form the dictionaries $\mathbf{U}_{i}$
and $\mathbf{V}_{i}$ to optimize (8). With PCA, we can reduce the dimensions
of two dictionaries $d_{1},d_{2}$ to the same level of $N$, which reduces the
time and space cost, i.e., the time complexity is reduced from
$O(s(N^{2}d_{m}))$ to $O(s(N^{3}))$ with $d_{m}\gg N$. The overall
optimization procedure is formulated as Algorithm 1.

Algorithm 1 Joint statistical and spatial sparse representation.

0: Training data
$\left\\{{{{\bf{X}}_{i}},{{\bf{X}}_{2}},...,{{\bf{X}}_{n}}}\right\\},{\bf{X}}_{i}\in\mathbb{R}^{d\times
m_{i}}$.A query data ${\hat{\mathbf{X}}}_{j}$.

1: Construct two dictionaries $\bf{S}_{1}$ and $\bf{S}_{2}$ of the whole
dataset;

2: Adapt PCA to get the low dimension representation $\hat{\bf{S}}_{1}$ and
$\hat{\bf{S}}_{2}$ of two dictionaries;

3: Initialize ${\bf{G}}_{j}$ as an identity matrix;

4: repeat

5: update $\bm{\alpha}_{j}$ according to Eq (14);

6: update $\bm{\gamma}_{j}$ according to Eq (15);

7: update ${\bf{G}}_{j}$ according to Eq (16);

8: until convergence criterion satisfied.

9: Classify the query data ${\hat{\mathbf{X}}}_{j}$ by (11).

9: The prediction label of the query data ${\hat{\mathbf{X}}}_{j}$.

##  V Experiments

We present experimental results on video-based face recognition, material
classification, and object recognition tasks to demonstrate the effectiveness
of the proposed J3S classification algorithm 222The reproducible
implementations of the J3S algorithms will be made publicly available upon
paper acceptance.. We conduct experiments on four databases: Flickr Material
Database (FMD) [43], UIUC Material Database [44], ETH-80 [3] and YouTube
Celebrities [2]. FMD and UIUC databases are used for image-based
classification, while ETH-80 and YTC databases are used for image set-based
classification. These databases contain samples in different materials, views,
illuminations, and even different modalities. Fig. 3 shows some sample images
with different categories from each database.

Figure 3: Examples of four databases. (a) Flickr Material Database; (b)
YouTube Celebrities Database; (c) UIUC Material Database; (d) ETH-80 Database.

_Flickr Material_ database (FMD) has 10 materials categories of 1000 images in
the wild [43]. Each image is selected from _Flickr.com_ with variations of
illuminations, rotations, and scales. We filter images with the _VGG-VD16_
model pre-trained on the ImageNet database and employ the output of the last
convolution layer as local features with the size of $m_{i}\times 512$.
Following [25], we randomly choose 50 images in each category for gallery and
the other 50 for probes and repeat this experiment ten times.

_UIUC Material_ database contains 216 images of 18 material categories in the
wild [44]. We obtain the deep features of UIUC material by taking the same
measures on FMD. We randomly choose half images in each category for the
gallery and the other half for probes.

_ETH-80_ database contains 80 image sets of 8 object categories [3]. Each
category has 10 sub-object with 41 images of different views. Following [21],
we randomly choose 5 objects as the gallery and the other 5 as probes in each
category. The size of each image is resized to $20\times 20$, and the
intensity feature is used. Thus, each image set can be expressed by the matrix
of $400\times 41$.

_YouTube Celebrities_ (YTC) database contains 1910 video clips of 47 subjects
[2] with different numbers of frames in each video. Following [21, 22], we use
histogram equalization to eliminate light effects in pre-processing step and
randomly select 3 videos per subject for the gallery and 6 videos for probes.
Then, each image is resized to a $20\times 20$ image with the intensity
feature. Thus each video can be expressed by the matrix of $n_{i}\times 400$
where $n_{i}$ is the number of frames in each video.

Methods | ETH-80 | FMD | UIUC | YTC
---|---|---|---|---
AHISD(linear) [20] | 72.50 | 46.72 | 55.37 | 64.65
AHISD(non-linear) [20] | 72.00 | 46.72 | 55.37 | 66.58
CHISD(linear) [20] | 79.75 | 47.52 | 65.09 | 67.24
CHISD(non-linear) [20] | 72.50 | 63.90 | 65.65 | 68.09
MMD [45] | 85.75 | 60.60 | 62.78 | 69.60
MDA [46] | 87.75 | 62.50 | 67.13 | 64.72
SPDML-AIRM [47] | 90.75 | 63.42 | 74.72 | 67.50
SPDML-Stein [47] | 90.75 | 63.80 | 68.24 | 68.10
LEML [22] | 93.50 | 66.60 | 69.17 | 69.85
RMML-SPD [24] | 95.00 | 68.88 | 70.09 | 78.05
RMML-GM [24] | 93.00 | 69.62 | 76.48 | 69.15
CDL-LDA [21] | 94.00 | 76.92 | 78.89 | 70.21
CDL-PLS [21] | 94.00 | 75.36 | 76.39 | 69.94
RSR [30] | 91.50 | 74.92 | 72.59 | 72.77
KGDL [31] | 93.00 | 77.40 | 76.32 | 73.91
DRM [27] | 98.12 | N/A | N/A | 72.55
MMDML[28] | 94.50 | N/A | N/A | 78.5
J3S w/o Spatial Dict. | 95.25 | 81.40 | 83.43 | 82.87
J3S | 96.00 | 82.58 | 84.07 | 83.09
TABLE II: Classification accuracy (in $\%$) over the four selected databases:
AHISD and CHISD are affine subspace based methods; MMD to RMML are nonlinear
manifold based methods; CDL is a Gaussian distribution based method; RSR and
KGDL are based on sparse coding; DRM and MMDML are deep methods. The best
(second best resp.) results are highlighted as Red (Blue resp.).

###  V-A Competing methods

To illustrate the effectiveness of the proposed model, we compare our method
with the following representatives of the subspace, non-linear manifold,
statistical, sparse representation, and deep based methods.

  * •

Affine subspace based methods: AHISD and CHISD [20].

  * •

Nonlinear manifold based methods: MMD[45], MDA [46], SPDML [47], LEML [22],
and RMML [24].

  * •

Gaussian distribution based methods: CDL [21].

  * •

Sparse representation based methods: RSR [30], and KGDL [31].

  * •

Deep based methods: DRM [27], and MMDML [28].

###  V-B Parameter Setting

We apply the implementations of competing methods provided by the authors with
the default settings suggested by the corresponding papers. For MMD, the PCA
percentage is set to $90\%$. For MDA, we set the number of local models,
between-class NN local models, and the subspace dimension the same as [46].
For SPDML, we implement both SPDML-AIRM and SPDML-Stein versions. In both
versions, following [47], ${v_{w}}$ is set as the minimum of the samples in
one class. The new dimension of the low-dimensional manifold and ${v_{b}}$ are
tuned by 5-fold cross-validation. We compare our method with both linear and
non-linear versions of AHISD and CHISD [20], where $98\%$ energy by PCA is
retained in non-linear AHISD and the value of error penalty $C$ in CHISD is
set as same as [20]. For LEML, $\eta$ is tuned from $1e\textnormal{-}3$ to
$1e3$ and the value of $\zeta$ is tuned from $0.1$ to $1$. For RMML, $\lambda$
is set to $0.1$ and $t$ is tuned from $0.2$ to $0.8$. For CDL, the distance
metric is learned with linear discriminant analysis (LDA) and partial least
squares (PLS) in Hilbert space. The reduced feature dimension is set to $c-1$
for LDA, where $c$ is the number of classes. For RSR and KGDL, we use SPAMS as
a sparse solver and set other parameters as suggested in the papers. The
dimension of the subspace of the Grassmann manifold in KGDL is set to 10.

There are four parameters ${\theta}$, ${\lambda_{1}}$, ${\lambda_{2}}$, and
${\lambda_{3}}$ for our proposed J3S method. The weighting parameter
${\theta}$ is defined to balance two sparse representation models and adjusted
based on different scales of databases. For some databases, such as the _UIUC_
database, which contains only a few labeled samples of each class, the
statistical dictionary may be challenging to represent the reliable and
complete information of a class. Thus, we set a small value $\theta=0.1$ to
mitigate the impact of the first term in Eq (8), while we set $\theta=0.6$ for
other databases. All regularization parameters ${\lambda_{1}}$,
${\lambda_{2}}$, and ${\lambda_{3}}$ are all set to $1e\textnormal{-}3$.
Moreover, we implement a common backbone _VGG-VD16_ network pre-trained on the
ImageNet database for feature extractor in this paper. The maximum number of
iterations is set to 50, and we take an early stop when the difference between
loss before and after two iterations is less than $1e\textnormal{-}6$.

###  V-C Image and Image-Set Classification

Table II compares the image classification results using the proposed method,
as well as all selected competing methods. Furthermore, we included two deep
learning methods, DRM [27] and MMDML [28], by quoting the results reported
over _YTC_ database. Note that classic methods randomly choose nine image sets
for each class, where three image sets for training and the rest six for
testing and report the average accuracy of ten times. On the contrary, the
selected deep-based models divide the whole database into five folds with nine
image sets for each class and keep training the model until convergence while
the network input is still based on a single picture. It is clear that our
proposed J3S approach outperforms all competition methods over the _FMD_ ,
_UIUC_ , and _YTC_ databases. For the _ETH-80_ database, our method
outperforms other competition methods except for DRM, which might due to the
way of processing data. Unlike the J3S model, DRM first computes the LBP
features of the training data and generates a subset randomly from the
training samples, which enhances the capability of the deep network. Moreover,
during testing, the learned DRM model is used to reconstruct each image of a
test image-set sample, and a voting strategy is adopted for classification. In
contrast, our J3S model treats all samples in each image set as a
classification object. Tables II also show that using joint two dictionaries
could help integrate multiple information to facilitate classification tasks.

###  V-D Noisy image classification

We simulate _i.i.d._ Gaussian noise with standard deviation $\sigma$ from $5$
to $20$ for all training and testing data on the _UIUC_ and _FMD_ databases to
generate noisy images for classification. Table III and IV show the
classification accuracy of two databases under different noise ratios. The
results show that the proposed J3S method outperforms the competition methods
subject to noise corruption. Also, we can observe a clear downward trend for
classification accuracy from the two tables as the noise level increases.
Simultaneously, our proposed J3S model can still perform better than any other
models in all noise levels. Moreover, we also find that for methods with
supervised dimension reduction, i.e., SPDML-Stein and CDL-LDA, their
performances at a relatively higher noise level $\sigma=20$ are more elevated
than performance at a lower noise level $\sigma=15$ on the _UIUC_ database.
This is partially due to the fact that models can discard the less critical
noise part during the dimensionality reduction process.

Methods | $\sigma=5$ | $\sigma=10$ | $\sigma=15$ | $\sigma=20$
---|---|---|---|---
SPDML-Stein | 66.57 | 67.96 | 64.81 | 65.37
SPDML-AIRM | 74.26 | 73.06 | 71.02 | 70.37
LEML | 69.17 | 69.81 | 67.22 | 66.02
CDL-LDA | 79.63 | 77.96 | 76.85 | 76.94
CDL-PLS | 76.48 | 74.91 | 72.41 | 70.74
J3S | 83.61 | 82.41 | 81.39 | 80.46
TABLE III: Classification accuracy (in $\%$) on noisy data with different noise levels ($\sigma$) of the _UIUC_ database. Methods | $\sigma=5$ | $\sigma=10$ | $\sigma=15$ | $\sigma=20$
---|---|---|---|---
SPDML-Stein | 62.86 | 58.54 | 54.94 | 52.12
SPDML-AIRM | 66.60 | 62.52 | 58.68 | 54.46
LEML | 66.52 | 63.82 | 59.80 | 56.76
CDL-LDA | 76.60 | 74.62 | 71.90 | 70.98
CDL-PLS | 74.24 | 71.68 | 69.02 | 66.44
J3S | 82.04 | 80.10 | 76.05 | 74.46
TABLE IV: Classification accuracy (in $\%$) on noisy data with different noise
levels ($\sigma$) of the _FMD_ database.

###  V-E Ablation Study

####  V-E1 Weight Analysis

As Eq (8) stated, the weighting parameter $\theta$ is used to balance two
dictionary models. We conduct an experiment to investigate the effectiveness
of weighting parameter settings on classification accuracy. Table V shows the
image classification accuracy averaged over the _ETH-80_ , _FMD_ , _UIUC_ ,
and _YTC_ databases, with different values of $\theta$. For _ETH-80_ , _FMD_ ,
and _YTC_ databases, it is obvious that as the weighting parameter of the
statistical model increases from $\theta=0.1$, the classification accuracy
rate has a significantly increase, which is due to the introduction of higher-
order Gaussian information. Compared with the spatial model, the statistical
Gaussian model is more discriminative but still needs the spatial model to
capture the local information. Thus, when the weighting parameter increases to
a certain level (e.g., $\theta$ from 0.5 to 0.7), continuing to increase will
cause the accuracy to fluctuate within a small range. In contrast, we observe
that the classification accuracy on the _UIUC_ database becomes worse when
increasing the weight parameter $\theta$. A potential explanation is that, the
statistical dictionary may be challenging and unreliable to represent the
entire information of a class if only given a few labeled samples such as the
UIUC database. Simultaneously, when the weighting parameter $\theta$
increases, the impact of the statistical term on the loss function becomes
more significant, so the classification accuracy decreases by about $1\%$.

Databases | $\theta=0.1$ | $\theta=0.3$ | $\theta=0.5$ | $\theta=0.7$ | $\theta=0.9$
---|---|---|---|---|---
ETH-80 | 94.00 | 95.00 | 96.00 | 96.00 | 95.00
FMD | 80.58 | 81.86 | 82.50 | 82.36 | 82.50
UIUC | 84.07 | 83.06 | 83.24 | 83.43 | 83.33
YTC | 76.70 | 80.92 | 82.70 | 83.09 | 83.01
TABLE V: Classification accuracy (in $\%$) v.s the weighting parameter
$\theta$.

Moreover, Table VI shows the classification results of the J3S model w/ and
w/o PCA with different values of the weighting parameter $\theta$. We can
observe that our proposed J3S model achieves the best performance with the
same weighting parameter ($\theta=0.1$) under two settings. Meanwhile, we find
that, after PCA dimensionality reduction, the highest accuracy rate has
improved slightly from $83.98\%$ to $84.07\%$ while the algorithm complexity
has decreased, which demonstrates the effectiveness of the J3S model w/ PCA
strategy.

Settings | $\theta=0.1$ | $\theta=0.3$ | $\theta=0.5$ | $\theta=0.7$ | $\theta=0.9$
---|---|---|---|---|---
J3S w/o PCA | 83.98 | 83.33 | 82.50 | 82.36 | 82.50
J3S | 84.07 | 83.06 | 83.24 | 83.43 | 83.33
TABLE VI: J3S model w/ or w/o PCA under different values of the weighting
parameter $\theta$ on the _UIUC_ database. Figure 4: Visualization of
different results based on the sparse representation of joint dictionaries or
two single dictionaries, respectively. Two query samples selected from
different categories are shown in the row on the left, and corresponding
classification results are shown in the right row.

####  V-E2 Feature Selection for Dictionary Construction

To investigate the proper feature for dictionary construction to extract local
information, we try different unitary dictionaries based on deep feature maps,
original gray images, and RGB images, respectively. Table VII shows the
classification accuracy with different feature selections for our proposed J3S
model on clean and noisy image data ($\sigma=20$) of the _UIUC_ database. We
can observe that without a unitary dictionary, the model performs worse than
other settings under the noisy condition, which can explain the role of the
spatial module on the robustness of the model from one side. Meanwhile, we
observe that the accuracy of methods based on a unitary dictionary of gray or
RGB image drops smaller than the unitary dictionary based on a deep feature
map under the noise condition. Since noisy images are fed into a deep CNN
structure pre-trained on clean data, it is more difficult to distinguish the
noise portion than shallow image features.

Settings | Acc (Clean) | Acc (Noise)
---|---|---
w/o unitary dict. | 83.43 | 80.00
w/ Deep feature based unitary dict. | 84.07 | 80.46
w/ Gray image based unitary dict. | 83.15 | 80.12
w/ RGB image based unitary dict. | 80.37 | 80.18
TABLE VII: Classification accuracy (in $\%$) on clean and noisy
($\sigma\\!=\\!20$) data of the _UIUC_ database.

####  V-E3 Convergence Analysis

According to Eq (8), it is easily proved that each part of the objective
function is convex. With alternation minimization, this optimization problem
can be divided into two convex problems and solved easily. Fig. 5 shows that
the J3S model can converge within a few iterations with different
regularization parameters. Meanwhile, we can observe that after only one
iteration, the J3S model will reduce the value of the loss function to near
convergence. Moreover, compared Fig. 5$(a)$ with Fig. 5$(b)$, we find that
increasing regularization parameters $\lambda_{1}$ and $\lambda_{2}$ for the
J3S model will make the convergence more stable but require more iterations to
converge fully, i.e., small regularization parameters need more iterations to
converge just like $28$ in Fig. 5$(b)$ than $4$ in Fig. 5$(a)$. A potential
explanation is that the regularization parameters $\lambda_{1}$ and
$\lambda_{2}$ of two sparse models are adopted to control the stringency of
sparse vectors $\alpha_{j}$ and $\gamma_{j}$ for a query sample with index
$j$. The constraint of the regular terms on the J3S model is proportional to
the scale of the corresponding regularization parameters.

(a) $\lambda_{1}=\lambda_{2}=1e\textnormal{-}3,\lambda_{3}=0.1$. (b)
$\lambda_{1}=\lambda_{2}=\lambda_{3}=0.1$.

Figure 5: Convergence curves with different settings of three regularization parameters $\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$ on the _UIUC_ database. We only show the loss value of 30 iterations on the basis of ensuring the convergence of the model. Methods | $K=1$ | $K=2$ | $K=3$ | $K=4$ | $K=5$ | $K=6$
---|---|---|---|---|---|---
SPDML-Stein | 42.41 | 53.70 | 60.56 | 64.81 | 66.57 | 68.24
SPDML-AIRM | 48.06 | 62.13 | 68.24 | 70.83 | 72.69 | 74.72
LEML | N/A | 53.43 | 61.76 | 65.09 | 67.59 | 69.17
CDL-LDA | 26.48 | 38.89 | 47.22 | 51.56 | 65.19 | 78.89
CDL-PLS | 55.19 | 67.96 | 72.64 | 74.56 | 75.56 | 76.39
CNN+Mean+SVM | 59.60 | 70.44 | 75.56 | 77.78 | 78.70 | 81.67
CNN+Gau+SVM | 61.61 | 72.22 | 77.40 | 79.58 | 81.90 | 84.01
J3S | 61.94 | 75.56 | 78.89 | 80.83 | 82.41 | 84.07
TABLE VIII: Classification accuracy (in $\%$) with only $K$ labeled training
samples of each class on the _UIUC_ database.

###  V-F Few-shot Classification

We consider a popular and challenging setting, i.e., the few-shot setting for
image classification tasks, to investigate the robustness of the proposed J3S
model with only a few supervised information. For few-shot learning, the whole
dataset is divided into two non-overlapping label sets, i.e., training set and
testing set. Following the meta-learning strategy [48], most existing few-shot
methods construct $N$-way $K$-shot tasks on the testing set to evaluate the
generalized performance of proposed models trained on the training set. Here
$N$ and $K$ are the number of class and labeled samples, respectively, and $N$
and $K$ are often small values.

Typical few-shot methods follow a methodology that learns the models only from
the training set and tests the classification accuracy on the testing set.
Unlike this learning strategy, based on representative learning, the proposed
J3S model solves the classification task by utilizing the reconstruction loss
computed by two coefficient vectors learned from both labeled training samples
and each query sample. A little different from the typical few-shot setting,
we set the value of $N$ to the total number of categories instead of a
commonly used fixed value $5$. In contrast, we still set $K$ to a small
number, identical to the few-shot setting.

Table VIII shows classification accuracy on the _UIUC_ database with different
numbers of the labeled samples in each class for dictionary learning. As
mentioned before, the _UIUC_ database only has $6$ training samples for each
class in total, so the general classification setting with all training
samples satisfies one kind of few-shot setting with $K=6$. From table VIII, we
can observe that our proposed J3S model outperforms the other methods in all
few-shot settings, i.e., $K$ from 1 to 6. Note that almost all Gaussian-based
models, i.e., CDL-LDA, CDL-PLS, two Gaussian-based CNN models, and our
proposed J3S model perform well even when using only a few training samples
for each class. The result demonstrates that the global statistical model can
implement rich information of a class, enhancing the classification capability
in few-shot tasks. Meanwhile, we observe that our J3S model and the method
CNN+Gau+SVM perform better than the other two Gaussian-based methods CDL-LDA
and CDL-PLS. A potential explanation is that, CDL-LDA and CDL-PLS learned only
based on the covariance matrix while the J3S model and CNN+Gau+SVM method
jointly implement first-order and second-order information, which leads to
better accuracy. Moreover, CDL-LDA performs poorly when $K=1$ because for the
1-shot setting, the feature dimension $\mathbf{D}$ is much larger than the
number of training samples $m$ (here $m\\!=\\!18\times K\\!=\\!18$). After
feature projection (so-called dimensionality reduction), the LDA-based model
cannot maintain the difference among neighbors and keep the within-class
variance to a minimum value for Nearest Neighbor classification. In comparison
to LDA, PLS has proven to be helpful in this situation as it is not limited by
the low discrimination dimensions.

Additionally, as the supervised information decreases, i.e., $K$ is selected
from $6$ to $2$, the gap between our proposed J3S model and the method
CNN+Gau+SVM becomes larger. This might due to the effectiveness of spatial
information when the statistical model cannot provide sufficient information
for classification. However, when $K=1$, the difference among all methods will
become minor because only one support sample of each category can be utilized
for learning, which results in insufficient information for classification,
and the model is vulnerable to bias.

##  VI Conclusion

In this paper, we proposed a novel J3S model for robust image and image-set
classification. Gaussian distribution is used to keep high-order image
statistical information, while patch-based sparse representation is used to
capture image local structure.  A simple and effective dimensionality
reduction operation by PCA is utilized to reduce the algorithm complexity. We
conducted experiments over four popular databases for clean and noisy image
classification tasks. Moreover, we conducted parameter sensitivity analysis
and tested the robustness of the algorithm under the popular few-shot setting.
Results show that our proposed method achieves superior performance compared
to a variety of algorithms under several settings.

## References

  * [1] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. F., “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [2] M. Kim, S. Kumar, V. Pavlovic, and H. Rowley, “Face tracking and recognition with visual constraints in real-world videos,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2008, pp. 1–8.
  * [3] B. Leibe and B. Schiele, “Analyzing appearance and contour based methods for object categorization,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , vol. 2, 2003, pp. 402–409.
  * [4] S. Gao, Z. Zeng, K. Jia, T.-H. Chan, and J. Tang, “Patch-set-based representation for alignment-free image set classification,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 26, no. 9, pp. 1646–1658, 2015.
  * [5] R. Wang, X.-J. Wu, and J. Kittler, “Graph embedding multi-kernel metric learning for image set classification with grassmann manifold-valued features,” _IEEE Transactions on Multimedia_ , 2020.
  * [6] Z. Zhu, F. Guo, H. Yu, and C. Chen, “Fast single image super-resolution via self-example learning and sparse representation,” _IEEE Transactions on Multimedia_ , vol. 16, no. 8, pp. 2178–2190, 2014.
  * [7] S. Gao, L.-T. Chia, I. W.-H. Tsang, and Z. Ren, “Concurrent single-label image classification and annotation via efficient multi-layer group sparse coding,” _IEEE Transactions on multimedia_ , vol. 16, no. 3, pp. 762–771, 2014.
  * [8] Q. Feng and Y. Zhou, “Kernel combined sparse representation for disease recognition,” _IEEE Transactions on Multimedia_ , vol. 18, no. 10, pp. 1956–1968, 2016.
  * [9] L. Li, D. Wu, J. Wu, H. Li, W. Lin, and A. C. Kot, “Image sharpness assessment by sparse representation,” _IEEE Transactions on Multimedia_ , vol. 18, no. 6, pp. 1085–1097, 2016.
  * [10] A. Cherian and S. Sra, “Riemannian dictionary learning and sparse coding for positive definite matrices,” _IEEE transactions on neural networks and learning systems_ , vol. 28, no. 12, pp. 2859–2871, 2016.
  * [11] B. Wang, Y. Hu, J. Gao, Y. Sun, F. Ju, and B. Yin, “Learning adaptive neighborhood graph on grassmann manifolds for video/image-set subspace clustering,” _IEEE Transactions on Multimedia_ , vol. 23, pp. 216–227, 2020\.
  * [12] P. Jing, Y. Shang, L. Nie, Y. Su, J. Liu, and M. Wang, “Learning low-rank sparse representations with robust relationship inference for image memorability prediction,” _IEEE Transactions on Multimedia_ , 2020.
  * [13] B. Wen, S. Ravishankar, and Y. Bresler, “Frist—flipping and rotation invariant sparsifying transform learning and applications,” _Inverse Problems_ , vol. 33, no. 7, p. 074007, 2017.
  * [14] T. Kobayashi, “Dirichlet-based histogram feature transform for image classification,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2014, pp. 3278–3285.
  * [15] P. Li, X. Lu, and Q. Wang, “From dictionary of visual words to subspaces: Locality-constrained affine subspace coding,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 2348–2357.
  * [16] P. Li, J. Xie, Q. Wang, and W. Zuo, “Is second-order information helpful for large-scale visual recognition?” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2070–2078.
  * [17] T. T. Nguyen, T. P. Nguyen, and F. Bouchara, “Prominent local representation for dynamic textures based on high-order gaussian-gradients,” _IEEE Transactions on Multimedia_ , 2020.
  * [18] H. Cheng and B. Wen, “Joint statistical and spatial sparse representation for robust image and image-set classification,” in _2020 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2020, pp. 2411–2415.
  * [19] T. Kim, J. Kittler, and R. Cipolla, “Discriminative learning and recognition of image set classes using canonical correlations,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 29, no. 6, 2007.
  * [20] H. Cevikalp and B. Triggs, “Face recognition based on image sets,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2010, pp. 2567–2573.
  * [21] R. Wang, H. Guo, L. S. Davis, and Q. Dai, “Covariance discriminative learning: A natural and efficient approach to image set classification,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2012, pp. 2496–2503.
  * [22] Z. Huang, R. Wang, S. Shan, X. Li, and X. Chen, “Log-euclidean metric learning on symmetric positive definite manifold with application to image set classification.” in _International Conference on Machine Learning_ , 2015, pp. 720–729.
  * [23] V. Arsigny, P. Fillard, X. Pennec, and N. Ayache, “Geometric means in a novel vector space structure on symmetric positive-definite matrices,” _SIAM journal on matrix analysis and applications_ , vol. 29, no. 1, pp. 328–347, 2007\.
  * [24] P. Zhu, H. Cheng, Q. Hu, Q. Wang, and C. Zhang, “Towards generalized and efficient metric learning on riemannian manifold.” in _IJCAI_ , 2018, pp. 3235–3241.
  * [25] Q. Wang, P. Li, W. Zuo, and L. Zhang, “RAID-G: Robust estimation of approximate infinite dimensional Gaussian with application to material recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 4433–4441.
  * [26] Q. Wang, P. Li, and L. Zhang, “G2DeNet: Global Gaussian distribution embedding network and its application to visual recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2730–2739.
  * [27] M. Hayat, M. Bennamoun, and S. An, “Deep reconstruction models for image set classification,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 37, no. 4, pp. 713–727, 2014.
  * [28] J. Lu, G. Wang, W. Deng, P. Moulin, and J. Zhou, “Multi-manifold deep metric learning for image set classification,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2015.
  * [29] H. Sun, X. Zhen, Y. Zheng, G. Yang, Y. Yin, and S. Li, “Learning deep match kernels for image-set classification,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 3307–3316.
  * [30] M. Harandi, C. Sanderson, R. Hartley, and B. Lovell, “Sparse coding and dictionary learning for symmetric positive definite matrices: A kernel approach,” in _European Conference on Computer Vision_ , 2012, pp. 216–229.
  * [31] M. Harandi, C. Sanderson, C. Shen, and B. Lovell, “Dictionary learning and sparse coding on Grassmann manifolds: An extrinsic solution,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2013, pp. 3120–3127.
  * [32] J. Wright, A. Yang, A. Ganesh, S. Sastry, and Y. Ma, “Robust face recognition via sparse representation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 31, no. 2, pp. 210–227, 2009.
  * [33] L.-W. Kang, C.-Y. Hsu, H.-W. Chen, C.-S. Lu, C.-Y. Lin, and S.-C. Pei, “Feature-based sparse representation for image similarity assessment,” _IEEE Transactions on multimedia_ , vol. 13, no. 5, pp. 1019–1030, 2011.
  * [34] B. Wen, S. Ravishankar, and Y. Bresler, “Structured overcomplete sparsifying transform learning with convergence guarantees and applications,” _International Journal of Computer Vision_ , vol. 114, no. 2-3, pp. 137–167, 2015.
  * [35] L. Wang, S. WANG, D. Kong, B. Yin _et al._ , “Hardness-aware dictionary learning: Boosting dictionary for recognition,” _IEEE Transactions on Multimedia_ , 2020.
  * [36] J. Yang, K. Yu, Y. Gong, and T. Huang, “Linear spatial pyramid matching using sparse coding for image classification,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2009, pp. 1794–1801.
  * [37] B. Kulis, M. A. Sustik, and I. S. Dhillon, “Low-rank kernel learning with bregman matrix divergences.” _Journal of Machine Learning Research_ , vol. 10, no. 2, 2009.
  * [38] Y. C. Pati, R. Rezaiifar, and P. S. Krishnaprasad, “Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition,” in _Proceedings of 27th Asilomar conference on signals, systems and computers_. IEEE, 1993, pp. 40–44.
  * [39] B. Wen, Y. Li, Y. Li, and Y. Bresler, “A set-theoretic study of the relationships of image models and priors for restoration problems,” _arXiv preprint arXiv:2003.12985_ , 2020.
  * [40] S. Ravishankar and Y. Bresler, “Sparsifying transform learning with efficient optimal updates and convergence guarantees,” _IEEE Transactions on Signal Processing_ , vol. 63, no. 9, pp. 2389–2404, 2015.
  * [41] U. Niesen, D. Shah, and G. Wornell, “Adaptive alternating minimization algorithms,” _IEEE Transactions on Information Theory_ , vol. 55, no. 3, pp. 1423–1429, 2009.
  * [42] A. M. Martinez and A. C. Kak, “Pca versus lda,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 23, no. 2, pp. 228–233, 2001\.
  * [43] L. Sharan, R. Rosenholtz, and E. H. Adelson, “Material perception: What can you see in a brief glance?” _Journal of Vision_ , vol. 9, no. 8, pp. 784–784, 2009.
  * [44] Z. Liao, J. Rock, Y. Wang, and D. Forsyth, “Non-parametric filtering for geometric detail extraction and material representation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , June 2013.
  * [45] R. Wang, S. Shan, X. Chen, and W. Gao, “Manifold-manifold distance with application to face recognition based on image set,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2008, pp. 1–8.
  * [46] R. Wang and X. Chen, “Manifold discriminant analysis,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2009, pp. 429–436.
  * [47] M. Harandi, M. Salzmann, and R. Hartley, “From manifold to manifold: Geometry-aware dimensionality reduction for SPD matrices,” in _European Conference on Computer Vision_ , 2014, pp. 17–32.
  * [48] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra _et al._ , “Matching networks for one shot learning,” in _Advances in neural information processing systems_ , 2016, pp. 3630–3638.