# Challenges and Opportunities in 3D Content Generation

Ke Zhao, Andreas Larsen<EMAIL_ADDRESS>

###### Abstract

This paper explores the burgeoning field of 3D content generation within the
landscape of Artificial Intelligence Generated Content (AIGC) and large-scale
models. It investigates innovative methods like Text-to-3D and Image-to-3D,
which translate text or images into 3D objects, reshaping our understanding of
virtual and real-world simulations. Despite significant advancements in text
and image generation, automatic 3D content generation remains nascent. This
paper emphasizes the urgency for further research in this area. By leveraging
pre-trained diffusion models, which have demonstrated prowess in high-fidelity
image generation, this paper aims to summary 3D content creation, addressing
challenges such as data scarcity and computational resource limitations.
Additionally, this paper discusses the challenges and proposes solutions for
using pre-trained diffusion models in 3D content generation. By synthesizing
relevant research and outlining future directions, this study contributes to
advancing the field of 3D content generation amidst the proliferation of
large-scale AIGC models.

###### Index Terms:

3D generation, Diffusion models

##  I Introduction

In the wave of Artificial Intelligence Generated Content (AIGC) and large-
scale models [1, 2, 3], creating three-dimensional (3D) content has become an
exciting research area, involving various innovative methods like Text-to-3D
[4, 5, 6, 7, 8] and Image-to-3D [9, 10, 11, 12], translating text or images
into 3D objects. These advancements are reshaping our understanding of
building virtual worlds and simulating real ones.

In recent years, with the continuous improvement of deep learning technology,
significant achievements have been made in text and image generation within
the AIGC field. Text generation techniques have been widely applied in
language translation, question answering systems, and other language-related
tasks, while image generation techniques can create visual images based on
textual descriptions. However, compared to text and image generation,
automatic generation of 3D content is still in its early stages and has not
achieved the same level of breakthrough.

Given this, further research and development in 3D content generation
technology are urgently needed. 3D content finds broad applications in fields
such as filmmaking, architectural design, virtual reality (VR), and augmented
reality (AR). Currently, the mainstream production of 3D content still relies
on professional 3D modelers, limiting production efficiency and raising
industry entry barriers. Therefore, utilizing artificial intelligence
technology to automate 3D content generation can significantly enhance
production efficiency, reduce industry barriers, and further drive related
industries’ development.

In the field of 3D content generation, the application of zero-shot learning
methods is relatively limited, mainly due to the smaller scale and diversity
of 3D data compared to 2D image data. Traditional 3D content generation
methods typically rely on specialized 3D datasets to train generative models.
While these methods can generate 3D objects with consistent geometric
structures, they are limited by the lack of large 3D datasets, the absence of
efficient 3D content generation architectures, and the enormous computational
resources required to train these models. These factors collectively make it
challenging for traditional 3D content generation methods to make significant
progress in the short term. Therefore, this study will focus on how to use
pre-trained diffusion models to guide 3D content generation.

As a powerful generative model, diffusion models have demonstrated their
capabilities in training on billions of image-text pairs, particularly showing
the ability to high-fidelity images generation and editing [13, 14, 15, 16,
17, 18, 19, 20]. Applying pre-trained diffusion models to 3D content
generation can significantly reduce the demand for computational resources and
the reliance on 3D datasets, thereby enhancing the feasibility and efficiency
of 3D content generation. This paper will delve into and analyze methods for
3D content generation, with a special focus on the prior knowledge of
diffusion models and the representation of 3D data. 3D content generation
tasks are mainly divided into two categories: Text-to-3D and Image-to-3D. To
comprehensively evaluate the advantages and limitations of different methods,
this study will conduct a horizontal comparison of these models in terms of
efficiency and quality. Additionally, this paper will discuss the challenges
faced when using pre-trained diffusion models for 3D content generation and
propose potential solutions.

To further expand the depth and breadth of this research, it is necessary to
first review the technical background and theoretical foundation of 3D content
generation, including traditional methods of 3D modeling, basic concepts of 3D
graphics, and the latest developments in the AIGC field. We will then detail
the working principles of diffusion models and how they are trained to handle
image-text pairs, as well as discuss how these models can adapt to 3D content
generation tasks. This paper will subsequently describe in detail the specific
implementation methods of Text-to-3D and Image-to-3D, including data
preprocessing, networks, loss functions, etc. Faced with challenges in 3D
content generation, such as data scarcity, computational resource limitations,
and the diversity of generated content, we will discuss their feasibility and
potential impact.

The contributions of this study can be summarized as follows:

  * •

This paper summarizes relevant research on 3D content generation and analyzes
the specific implementation of text-to-3D and image-to-3D, discussing the
challenges faced when using diffusion models for 3D content generation.

  * •

This paper outlines potential future research directions in the field of 3D
content generation under the backdrop of large-scale AIGC models.

##  II Related work

###  II-A 3D Representation

3D Representation is a crucial aspect of computer graphics, enabling the
digital visualization and manipulation of three-dimensional objects and
scenes. Different methods of 3D representation offer varying degrees of
detail, efficiency, and application suitability. Here, we delve into several
key types of 3D representations, including Explicit Representations, Point
Clouds, and Voxels.

##### Explicit Representations

Explicit representations define 3D objects with precise mathematical
descriptions. This category includes mesh-based methods, which represent
surfaces using vertices, edges, and faces. Meshes are widely used due to their
flexibility and efficiency in rendering.

##### Meshes

A mesh is composed of polygons, typically triangles, which are connected by
their edges and vertices to form the surface of a 3D object. Meshes are
favored in applications requiring detailed and smooth surfaces, such as video
games, animations, and simulations. They allow for efficient rendering [21,
22] and easy manipulation but can become complex when representing highly
detailed surfaces.

##### Point Clouds

Point clouds [23, 24, 25, 25] represent 3D shapes as a collection of discrete
points in space. Each point has a specific position (x, y, z coordinates) and
often includes additional attributes like color or normal vectors. Point
clouds are particularly useful in scanning and reconstruction applications, as
they can directly capture the surface geometry of objects from real-world
scans. They are used in fields like autonomous driving, where LiDAR sensors
generate point cloud data to represent the environment. However, point clouds
can be sparse and may require significant processing to convert into other
forms of representation, like meshes or voxel grids, for further use. The lack
of explicit connectivity between points can also complicate certain operations
like surface reconstruction and rendering.

##### Voxels

Voxels [26, 27, 28, 29, 30], or volumetric pixels, are the 3D equivalent of
pixels in a 2D image. They divide the 3D space into a regular grid of small
cubes, each storing information about the material present at that location.
Voxel grids are highly suitable for applications requiring a uniform
representation of space, such as medical imaging (CT and MRI scans),
geological modeling, and some types of simulations (e.g., fluid dynamics).
They are excellent for representing volumetric data and can easily handle
complex topologies and internal structures of object. The primary challenge
with voxels is their memory consumption, as high-resolution voxel grids can
become extremely large. This makes them less efficient for detailed surface
representations compared to meshes. However, advancements in sparse voxel
representations and efficient storage techniques are mitigating these issues.

###  II-B Implicit Representations

##### Neural Radiance Fields

Neural Radiance Fields (NeRF) [31, 32, 33, 34, 35, 36, 37] is a groundbreaking
technique in computer vision that synthesizes novel views of complex 3D scenes
from a sparse set of 2D images. NeRF represents a scene using a continuous 5D
function that maps spatial coordinates and viewing directions to color and
density values. This enables the creation of photorealistic images from new
viewpoints by interpolating between the input images. NeRF employs a fully-
connected neural network to optimize a volumetric scene function, which,
through volume rendering techniques, accumulates colors and densities to form
a 2D image. The process involves marching rays through the scene, sampling 3D
points, and predicting color and density values for each point using the
neural network. This allows for high-fidelity reconstructions of scenes with
intricate details and realistic lighting effects. Despite its impressive
capabilities, the classic NeRF approach has some limitations, such as long
training times and the requirement to train a new model for each scene. To
address these challenges, several extensions and improvements have been
proposed. For instance, InstantNeRF [38] utilizes a multiresolution hash table
to accelerate training, while PixelNeRF [39] leverages convolutional neural
networks to enable view synthesis from a single image without explicit 3D
supervision.

##### 3D Gaussian Splatting

3D Gaussian Splatting [40, 41, 42, 43, 44] is an advanced rasterization
technique used for real-time rendering of photorealistic scenes. It begins by
estimating a point cloud from a set of 2D images using a method called
Structure from Motion (SfM). Each point in this cloud is then converted into a
3D Gaussian, characterized by its position, color, transparency, and shape.
These Gaussians are then rasterized to create high-quality images that can be
rendered in real-time. The main advantage of 3D Gaussian Splatting over
traditional methods like Neural Radiance Fields (NeRF) and photogrammetry lies
in its efficiency and real-time capabilities. Unlike NeRF, which can be
computationally intensive and slow to render, Gaussian Splatting offers faster
rendering speeds while maintaining high visual quality. This makes it
particularly suitable for applications requiring real-time interactions, such
as virtual reality (VR) and augmented reality (AR). The training process for
3D Gaussian Splatting involves rasterizing the Gaussians, calculating the loss
based on the difference between the rasterized image and the ground truth, and
adjusting the Gaussian parameters to minimize this loss. This process allows
for detailed and accurate representation of complex scenes with high
efficiency. Despite its advantages, 3D Gaussian Splatting does require
significant VRAM and is not yet fully compatible with existing rendering
pipelines. However, its potential applications are vast, ranging from
enhancing virtual property tours and urban planning to creating photorealistic
avatars for telepresence in VR environments.

###  II-C Diffusion models

Diffusion models [45, 46, 47, 48, 49] are based on a Markov chain that
transitions data points from a clear state to a diffused state and vice versa.
The core idea is to learn the reverse process of this Markov chain, which
allows for sampling from complex distributions.

##### Forward Process (Diffusion)

The forward process can be thought of as a Markov chain with transition
probabilities $p(x_{t-1}|x_{t})$, where $x_{t}$ is the data at step $t$ and
$x_{t-1}$ is the data at step $t-1$. The transition is typically defined by
adding Gaussian noise:

$x_{t-1}=f(x_{t})+\epsilon_{t}$ (1)

where $f(x_{t})$ is the reverse process (denoising), and $\epsilon_{t}$ is
Gaussian noise with variance $\sigma_{t}^{2}$. The variance $\sigma_{t}$ is
typically chosen from a predefined schedule.

##### Reverse Process (Denoising)

The reverse process involves learning a model $f_{\theta}(x_{t},t)$ that,
given a noisy data point $x_{t}$ and the step $t$, predicts the previous data
point $x_{t-1}$. This is achieved by training the model to minimize the
following loss function:

$L(\theta)=\mathbb{E}_{x_{0}\sim
p(x_{0})}\left[\|x_{0}-f_{\theta}(x_{T},T)\|_{2}^{2}\right]$ (2)

where $x_{0}$ is the original data point, $x_{T}$ is the fully diffused data
point after $T$ steps, and $p(x_{0})$ is the data distribution.

##### Training Objective

The training objective can also be formulated in terms of the log-likelihood
of the data under the diffusion process. The joint distribution of the data
and the noise is given by:

$p(x_{0},x_{1},...,x_{T})=p(x_{0})\prod_{t=1}^{T}p(x_{t-1}|x_{t})$ (3)

The log-likelihood of $x_{0}$ can be written as:

$\log p(x_{0})=\log\left(\prod_{t=1}^{T}p(x_{t-1}|x_{t})\right)+\log p(x_{0})$
(4)

By introducing the reverse process, the log-likelihood can be maximized by
minimizing the negative log-likelihood:

$-\log p(x_{0})=-\log\left(\prod_{t=1}^{T}p(x_{t-1}|x_{t})\right)-\log
p(x_{0})+C$ (5)

where $C$ is a constant term that does not depend on $\theta$.

##### Sampling

For sampling, the model is used in reverse, starting from a noise
distribution:

$x_{t-1}=f_{\theta}(x_{t},t)$ (6)

By iteratively applying $f_{\theta}$ and decreasing $t$, one can generate new
samples that approximate the original data distribution $p(x_{0})$.

In fact, the variance schedule $\sigma_{t}$ can be chosen in various ways,
such as a linear or exponential schedule. The denoising model $f_{\theta}$ can
be any neural network architecture, often a deep convolutional network. The
training process involves optimizing $\theta$ to minimize the reconstruction
loss or maximize the log-likelihood.

In this section, we provides a more in-depth look at the diffusion model’s
principles and the mathematical framework behind it. The actual implementation
may involve additional complexities and optimizations.

##  III 3D Content generation

Diffusion models have significant advantages in generating 3D content,
especially when combined with NeRF (Neural Radiance Fields) and 3D Gaussian
Splatting technologies. NeRF represents scenes as continuous volumetric
fields, enabling the generation of realistic 3D scene renderings. Diffusion
models, on the other hand, provide an effective method for generating high-
quality 3D models from noise. By combining these two technologies, the details
and realism of 3D assets can be further enhanced. The 3D Gaussian Splatting
technology is a method of representing 3D data as a set of Gaussian
distributions, which helps maintain the structural integrity of the 3D models
during the diffusion process. By integrating diffusion models with these
technologies, researchers can explore new methods of generating 3D assets that
not only improve generation efficiency but also provide greater flexibility
and control while preserving details.

Figure 1: DreamFusion [50] utilizes a pretrained text-to-image diffusion model
to generate realistic 3D models from text prompts.. Figure 2: We demonstrate
DreamFusion’s capability to translate text into 3D assets. Figure 3: We show
the pipeline of Dreamfusion.

###  III-A Text-to-3D

Text-to-3D [51, 50, 6, 52, 53, 54, 55, 8, 56, 57] is an emerging
interdisciplinary field that focuses on converting textual descriptions into
three-dimensional models, such as shown in Fig. 1, 2 and 3.. This technology
has a wide range of applications, including virtual reality, gaming, and
design. The goal is to create a system that can interpret natural language and
generate a 3D representation that aligns with the textual description.

Data Representation: The core of text-to-3D methods lies in how they represent
and process data. There are primarily two approaches:

Volumetric Representation: Early methods used voxel grids to represent 3D
shapes. This approach allows for direct 3D shape manipulation but can be
computationally intensive and storage-heavy. Surface Representation: More
recent methods focus on mesh surfaces, which are more memory-efficient and
align better with human perception of 3D shapes. Neural Networks and Learning:
The evolution of neural networks has been pivotal in advancing text-to-3D
conversion. Key developments include:

Convolutional Neural Networks (CNNs): Used for feature extraction from images,
which can be used to guide the generation process. Recurrent Neural Networks
(RNNs): Effective for sequential data processing, such as sentences in natural
language. Transformers: State-of-the-art models that excel at handling
sequential data and have been adapted for text-to-3D tasks. Generative Models:
Generative models are at the heart of text-to-3D systems. They include:

Generative Adversarial Networks (GANs) [58, 59]: Known for their ability to
generate realistic images, some adaptations are used to generate 3D shapes.
Variational Autoencoders (VAEs) [60, 61, 62]: Used to learn a low-dimensional
representation of the data, which can then be decoded into 3D shapes.
Diffusion Models: A newer approach that has shown promise in generating high-
quality 3D shapes from text. Attention Mechanisms: Attention mechanisms [63]
have proven to be highly effective in text-to-3D tasks by allowing the model
to focus on different parts of the input text and generate corresponding 3D
features.

Depth priors: Depth estimation [64, 65, 66, 67] provides a prior for depth
information, supervising subsequent model generation. The entire process
begins with initializing a 3D model, followed by rendering images with added
Gaussian noise from random angles. Subsequently, the diffusion model [47] is
utilized to optimize the 3D model through backpropagation using SDS loss and a
series of reference image losses.

Loss Functions: Custom loss functions are crucial for training models to
generate accurate 3D shapes. Common loss functions include:

Perceptual Loss: Measures the difference between the generated shape and a
target shape based on perceptual features. Chamfer Distance: A metric that
measures the similarity between two point clouds, often used to compare the
generated mesh with a reference mesh. Earth Mover’s Distance (EMD): Used to
measure the distribution difference between two sets of points, which can be
applied to 3D shapes. Evaluation Metrics: Evaluating the quality of generated
3D models is critical. Metrics used include:

Fidelity to Description: How well the generated model matches the textual
description. Visual Quality: The realism and detail of the generated 3D
shapes. Diversity: The range of shapes that can be generated from different
textual inputs. Applications: The applications of text-to-3D are vast and
include:

Content Creation: Automatically generating 3D content for games, films, or
virtual environments. Design Assistance: Assisting designers by quickly
converting textual ideas into visual 3D concepts. Educational Tools: Teaching
3D modeling concepts through natural language interactions. Challenges:
Despite the progress, text-to-3D faces several challenges:

Ambiguity in Language: Textual descriptions can be ambiguous, making it
difficult to generate a single, accurate 3D model. Complexity of 3D Shapes:
Capturing the full complexity of 3D shapes from text is a complex task.
Computational Resources: High-quality 3D generation can be computationally
expensive.

###  III-B Image-to-3D

Recent research has been focusing on exploring image-to-3D techniques [68, 6,
69, 70, 10, 71, 72] to describe scene details and appearance more intuitively.
These methods aim to reconstruct outstanding and high-fidelity 3D models from
given images, serving as the primary means to depict the visual effects of
scenes, which are more detailed than language. Similar to text-to-3D methods,
several image-to-3D methods utilize volume representations like NeRF to
naturally introduce multi-view consistency. For example, NeuralLift-360[73]
utilizes monocular depth and CLIP-guided diffusion priors to respectively
regulate geometry and appearance optimization, thereby elevating a single
image to a 3D scene represented by NeRF. RealFusion[72] and NeRDi[74] extract
text embeddings via text inversion to fine-tune pre-trained image diffusion
models and combine score distillation loss to optimize volume representations.
Magic123[7] introduces the 3D prior of the pre-trained viewpoint-conditioned
diffusion model Zero-1-to-3[10](as shown in Fig. 4 and 5.) in a coarse-to-fine
framework through two optimization stages to generate texture meshes matching
specified images. Make-it-3D[54] enhances texture and geometric structure in
the refinement stage of a two-stage method to produce high-quality textured
point clouds. Subsequent works[71, 75] have been continuously improving upon
previous results. Recently, 3D Gaussian Splatter (3DGS)[76] has emerged as a
promising modeling and real-time rendering technique. Based on 3DGS,
DreamGaussian[42] proposes an efficient two-stage framework for text-driven
and image-driven 3D generation. In the first stage, DreamGaussian utilizes SDS
loss (i.e., 2D diffusion prior and CLIP-guided diffusion prior) to generate
the target object represented by 3D Gaussians. Subsequently, DreamGaussian
[42] extracts textured meshes from the optimized 3D Gaussians by querying
local densities and refines textures in UV space. To evaluate viewpoint
synthesis quality, CLIP similarity, PSNR, and LPIPS are commonly used as
metrics.

PSNR: PSNR is a widely used standard for quantifying image reconstruction or
image compression quality. It measures the pixel-level difference between the
original image and the compressed or reconstructed image. PSNR is computed
based on the Mean Squared Error (MSE) between two images. Generally, a higher
PSNR value indicates a closer quality match between the reconstructed image
and the original image. It primarily assesses pixel-level similarity between
the reconstructed or compressed image and the original image, but may not
always align with human perceptual differences.

LPIPS: LPIPS is a more modern deep learning-based metric used to evaluate the
perceptual quality and similarity of images. LPIPS computes similarity by
comparing activations of deep neural networks when processing two images. This
approach aims to be closer to the human visual perception system. LPIPS is
used to assess perceptual similarity of images, especially in cases where
pixel-level metrics might not capture all aspects of human perception.

CLIP-Similarity: CLIP-Similarity is a metric used to evaluate semantic
similarity of images based on features extracted by the CLIP model. Unlike
traditional image similarity metrics that focus on pixel-level details, CLIP-
Similarity measures the semantic or contextual similarity between two images.
It is particularly useful when evaluating criteria beyond purely visual or
pixel-level accuracy and involves contextual and conceptual consistency.

Figure 4: We show the pipeline of method Zero-1-to-3 [10]. Figure 5:
Zero-1-to-3 provides a visual template for users to easily create desired 3D
assets according to their specific needs.

##  IV Applications

This section explores the practical application of 3D generation technologies,
highlighting their impact on different domains and their potential to
revolutionize traditional workflows.

####  IV-1 Film and Entertainment Industry

One of the most prominent applications of 3D generation is in the film and
entertainment industry. Traditional methods of creating 3D assets for movies
and visual effects often involve manual labor and extensive time investment.
However, with advancements in AI-driven 3D generation, filmmakers can now
streamline the production process by automating the creation of complex 3D
models, characters, and environments. This not only accelerates the production
timeline but also enables filmmakers to unleash their creativity without being
constrained by technical limitations.

####  IV-2 Architectural Design and Visualization

Architects and designers rely heavily on 3D visualization to convey their
concepts to clients and stakeholders. AI-powered 3D generation tools offer
architects the ability to quickly generate realistic architectural models from
simple sketches or textual descriptions. By automating the tedious task of 3D
modeling, architects can focus more on design iterations and exploring
creative ideas. Additionally, these tools facilitate better communication
between designers and clients, leading to more efficient decision-making
processes.

###  IV-A Virtual Reality (VR) and Augmented Reality (AR)

VR and AR technologies have gained significant traction in recent years,
offering immersive experiences in gaming, education, training, and simulation.
AI-driven 3D generation plays a crucial role in enhancing the realism and
interactivity of VR and AR environments. By dynamically generating 3D assets
based on user interactions or environmental conditions, developers can create
more dynamic and responsive virtual worlds. This not only enhances user
engagement but also opens up new possibilities for interactive storytelling
and experiential learning.

###  IV-B Product Design and E-Commerce

In the realm of product design and e-commerce, 3D generation technologies are
revolutionizing the way products are conceptualized, prototyped, and marketed.
By automatically generating 3D models of products from 2D images or textual
descriptions, designers and retailers can showcase their products in a more
immersive and interactive manner. This enables consumers to visualize products
more accurately before making purchasing decisions, leading to higher customer
satisfaction and reduced product return rates.

###  IV-C Medical Imaging and Simulation

In the field of medicine, AI-driven 3D generation has the potential to
revolutionize medical imaging and simulation. By automatically converting
medical scans such as MRI or CT images into detailed 3D models, healthcare
professionals can better visualize and analyze complex anatomical structures.
This facilitates more accurate diagnosis, treatment planning, and surgical
simulations, ultimately leading to improved patient outcomes and reduced
healthcare costs.

The applications of AI-driven 3D generation are vast and diverse, spanning
across industries such as film and entertainment, architecture, virtual
reality, product design, and healthcare. As technology continues to advance,
we can expect these applications to become even more widespread, transforming
the way we create, interact with, and experience 3D content in the digital
age.

##  V Conclusion

In conclusion, this paper has explored the emerging field of 3D content
generation in the context of Artificial Intelligence Generated Content (AIGC)
and large-scale models. While significant progress has been made, challenges
such as data scarcity and computational limitations persist. However,
leveraging pre-trained diffusion models shows promise in enhancing the
efficiency of 3D content creation. By analyzing various methods and discussing
challenges and potential solutions, this research provides insights into the
current state and future directions of 3D content generation. Moving forward,
further research is needed to overcome existing limitations and fully unleash
the potential of AI in this domain.

## References

  * [1] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 10 684–10 695.
  * [2] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans _et al._ , “Photorealistic text-to-image diffusion models with deep language understanding,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 36 479–36 494, 2022.
  * [3] A. Sanghi, H. Chu, J. G. Lambourne, Y. Wang, C.-Y. Cheng, M. Fumero, and K. R. Malekshan, “Clip-forge: Towards zero-shot text-to-shape generation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 18 603–18 613.
  * [4] A. Nichol, H. Jun, P. Dhariwal, P. Mishkin, and M. Chen, “Point-e: A system for generating 3d point clouds from complex prompts,” _arXiv preprint arXiv:2212.08751_ , 2022.
  * [5] H. Jun and A. Nichol, “Shap-e: Generating conditional 3d implicit functions,” _arXiv preprint arXiv:2305.02463_ , 2023.
  * [6] C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M.-Y. Liu, and T.-Y. Lin, “Magic3d: High-resolution text-to-3d content creation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 300–309.
  * [7] G. Qian, J. Mai, A. Hamdi, J. Ren, A. Siarohin, B. Li, H.-Y. Lee, I. Skorokhodov, P. Wonka, S. Tulyakov _et al._ , “Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors,” _arXiv preprint arXiv:2306.17843_ , 2023.
  * [8] Z. Wang, C. Lu, Y. Wang, F. Bao, C. Li, H. Su, and J. Zhu, “Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation,” _arXiv preprint arXiv:2305.16213_ , 2023.
  * [9] R. Shi, H. Chen, Z. Zhang, M. Liu, C. Xu, X. Wei, L. Chen, C. Zeng, and H. Su, “Zero123++: a single image to consistent multi-view diffusion base model,” _arXiv preprint arXiv:2310.15110_ , 2023.
  * [10] R. Liu, R. Wu, B. Van Hoorick, P. Tokmakov, S. Zakharov, and C. Vondrick, “Zero-1-to-3: Zero-shot one image to 3d object,” _arXiv preprint arXiv:2303.11328_ , 2023.
  * [11] A. Jain, B. Mildenhall, J. T. Barron, P. Abbeel, and B. Poole, “Zero-shot text-guided object generation with dream fields,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 867–876.
  * [12] A. Raj, S. Kaza, B. Poole, M. Niemeyer, N. Ruiz, B. Mildenhall, S. Zada, K. Aberman, M. Rubinstein, J. Barron _et al._ , “Dreambooth3d: Subject-driven text-to-3d generation,” _arXiv preprint arXiv:2303.13508_ , 2023.
  * [13] P. Li, Q. Huang, Y. Ding, and Z. Li, “Layerdiffusion: Layered controlled image editing with diffusion models,” in _SIGGRAPH Asia 2023 Technical Communications_ , 2023, pp. 1–4.
  * [14] A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or, “Prompt-to-prompt image editing with cross attention control,” _arXiv preprint arXiv:2208.01626_ , 2022.
  * [15] N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman, “Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 22 500–22 510.
  * [16] P. Li, Q. Nie, Y. Chen, X. Jiang, K. Wu, Y. Lin, Y. Liu, J. Peng, C. Wang, and F. Zheng, “Tuning-free image customization with image and text guidance,” _arXiv preprint arXiv:2403.12658_ , 2024.
  * [17] T. Brooks, A. Holynski, and A. A. Efros, “Instructpix2pix: Learning to follow image editing instructions,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 18 392–18 402.
  * [18] G. Couairon, J. Verbeek, H. Schwenk, and M. Cord, “Diffedit: Diffusion-based semantic image editing with mask guidance,” _arXiv preprint arXiv:2210.11427_ , 2022.
  * [19] O. Gafni, A. Polyak, O. Ashual, S. Sheynin, D. Parikh, and Y. Taigman, “Make-a-scene: Scene-based text-to-image generation with human priors,” in _European Conference on Computer Vision_. Springer, 2022, pp. 89–106.
  * [20] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, “Zero-shot text-to-image generation,” in _International conference on machine learning_. Pmlr, 2021, pp. 8821–8831.
  * [21] M. Botsch, L. Kobbelt, M. Pauly, P. Alliez, and B. Lévy, _Polygon mesh processing_. CRC press, 2010.
  * [22] L. A. Shirman and C. H. Sequin, “Local surface interpolation with bézier patches,” _Computer Aided Geometric Design_ , vol. 4, no. 4, pp. 279–295, 1987.
  * [23] K.-A. Aliev, A. Sevastopolsky, M. Kolos, D. Ulyanov, and V. Lempitsky, “Neural point-based graphics,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16_. Springer, 2020, pp. 696–712.
  * [24] O. Wiles, G. Gkioxari, R. Szeliski, and J. Johnson, “Synsin: End-to-end view synthesis from a single image,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 7467–7477.
  * [25] G. Kopanas, J. Philip, T. Leimkühler, and G. Drettakis, “Point-based neural rendering with per-view optimization,” in _Computer Graphics Forum_ , vol. 40, no. 4. Wiley Online Library, 2021, pp. 29–43.
  * [26] L. Liu, J. Gu, K. Zaw Lin, T.-S. Chua, and C. Theobalt, “Neural sparse voxel fields,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 15 651–15 663, 2020.
  * [27] S. Fridovich-Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa, “Plenoxels: Radiance fields without neural networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 5501–5510.
  * [28] K. Schwarz, A. Sauer, M. Niemeyer, Y. Liao, and A. Geiger, “Voxgraf: Fast 3d-aware image synthesis with sparse voxel grids,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 33 999–34 011, 2022.
  * [29] C. Sun, M. Sun, and H.-T. Chen, “Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 5459–5469.
  * [30] D. Maturana and S. Scherer, “Voxnet: A 3d convolutional neural network for real-time object recognition,” in _2015 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2015, pp. 922–928.
  * [31] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” in _European conference on computer vision_. Springer, 2020, pp. 405–421.
  * [32] M. Niemeyer and A. Geiger, “GIRAFFE: Representing scenes as compositional generative neural feature fields,” in _CVPR_ , 2021.
  * [33] J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan, “Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 5855–5864.
  * [34] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman, “Mip-nerf 360: Unbounded anti-aliased neural radiance fields,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 5470–5479.
  * [35] D. Verbin, P. Hedman, B. Mildenhall, T. Zickler, J. T. Barron, and P. P. Srinivasan, “Ref-nerf: Structured view-dependent appearance for neural radiance fields,” in _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2022, pp. 5481–5490.
  * [36] Z. Li, Q. Wang, F. Cole, R. Tucker, and N. Snavely, “Dynibar: Neural dynamic image-based rendering,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 4273–4284.
  * [37] A. Chen, Z. Xu, F. Zhao, X. Zhang, F. Xiang, J. Yu, and H. Su, “MVSNeRF: Fast generalizable radiance field reconstruction from multi-view stereo,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 14 124–14 133.
  * [38] T. Müller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” _ACM Transactions on Graphics (ToG)_ , vol. 41, no. 4, pp. 1–15, 2022.
  * [39] A. Yu, V. Ye, M. Tancik, and A. Kanazawa, “pixelnerf: Neural radiance fields from one or few images,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 4578–4587.
  * [40] R. Abdal, W. Yifan, Z. Shi, Y. Xu, R. Po, Z. Kuang, Q. Chen, D.-Y. Yeung, and G. Wetzstein, “Gaussian shell maps for efficient 3d human generation,” _arXiv preprint arXiv:2311.17857_ , 2023.
  * [41] Z. Chen, F. Wang, and H. Liu, “Text-to-3d using gaussian splatting,” _arXiv preprint arXiv:2309.16585_ , 2023.
  * [42] J. Tang, J. Ren, H. Zhou, Z. Liu, and G. Zeng, “Dreamgaussian: Generative gaussian splatting for efficient 3d content creation,” _arXiv preprint arXiv:2309.16653_ , 2023.
  * [43] X. Liu, X. Zhan, J. Tang, Y. Shan, G. Zeng, D. Lin, X. Liu, and Z. Liu, “Humangaussian: Text-driven 3d human generation with gaussian splatting,” _arXiv preprint arXiv:2311.17061_ , 2023.
  * [44] T. Yi, J. Fang, G. Wu, L. Xie, X. Zhang, W. Liu, Q. Tian, and X. Wang, “Gaussiandreamer: Fast generation from text to 3d gaussian splatting with point cloud priors,” _arXiv preprint arXiv:2310.08529_ , 2023.
  * [45] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” _Advances in neural information processing systems_ , vol. 33, pp. 6840–6851, 2020.
  * [46] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _CVPR_ , 2022.
  * [47] Y. Bai, Y. Fan, X. Wang, Y. Zhang, J. Sun, C. Yuan, and Y. Shan, “High-fidelity facial avatar reconstruction from monocular video with generative priors,” in _CVPR_ , 2023.
  * [48] S. Luo and W. Hu, “Diffusion probabilistic models for 3d point cloud generation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 2837–2845.
  * [49] R. Cai, G. Yang, H. Averbuch-Elor, Z. Hao, S. Belongie, N. Snavely, and B. Hariharan, “Learning gradient fields for shape generation,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16_. Springer, 2020, pp. 364–381.
  * [50] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall, “DreamFusion: Text-to-3d using 2d diffusion,” in _Int. Conf. Learn. Represent._ , 2023.
  * [51] P. Li, C. Tang, Q. Huang, and Z. Li, “Art3d: 3d gaussian splatting for text-guided artistic scenes generation,” _arXiv:2405.10508_ , 2024.
  * [52] P. Li and B. Li, “Generating daylight-driven architectural design via diffusion models,” _arXiv preprint arXiv:2404.13353_ , 2024.
  * [53] E. R. Chan, K. Nagano, M. A. Chan, A. W. Bergman, J. J. Park, A. Levy, M. Aittala, S. De Mello, T. Karras, and G. Wetzstein, “Generative novel view synthesis with 3d-aware diffusion models,” _arXiv preprint arXiv:2304.02602_ , 2023.
  * [54] J. Tang, T. Wang, B. Zhang, T. Zhang, R. Yi, L. Ma, and D. Chen, “Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior,” _arXiv preprint arXiv:2303.14184_ , 2023.
  * [55] T. Shen, J. Gao, K. Yin, M.-Y. Liu, and S. Fidler, “Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 6087–6101, 2021.
  * [56] W. Li, R. Chen, X. Chen, and P. Tan, “Sweetdreamer: Aligning geometric priors in 2d diffusion for consistent text-to-3d,” _arXiv preprint arXiv:2310.02596_ , 2023.
  * [57] G. Metzer, E. Richardson, O. Patashnik, R. Giryes, and D. Cohen-Or, “Latent-nerf for shape-guided generation of 3d shapes and textures,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 12 663–12 673.
  * [58] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” _Advances in neural information processing systems_ , vol. 27, 2014.
  * [59] R. Abdal, Y. Qin, and P. Wonka, “Image2stylegan: How to embed images into the stylegan latent space?” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 4432–4441.
  * [60] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” _arXiv preprint arXiv:1312.6114_ , 2013.
  * [61] M. J. Kusner, B. Paige, and J. M. Hernández-Lobato, “Grammar variational autoencoder,” in _International conference on machine learning_. PMLR, 2017, pp. 1945–1954.
  * [62] Y. Pu, Z. Gan, R. Henao, X. Yuan, C. Li, A. Stevens, and L. Carin, “Variational autoencoder for deep learning of images, labels and captions,” _Advances in neural information processing systems_ , vol. 29, 2016.
  * [63] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [64] P. Li and Z. Li, “Efficient temporal denoising for improved depth map applications,” in _Proc. Int. Conf. Learn. Representations, Tiny papers_ , 2023.
  * [65] R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun, “Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 44, no. 3, pp. 1623–1637, 2020.
  * [66] S. F. Bhat, I. Alhashim, and P. Wonka, “Adabins: Depth estimation using adaptive bins,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 4009–4018.
  * [67] V. Guizilini, R. Ambrus, S. Pillai, A. Raventos, and A. Gaidon, “3d packing for self-supervised monocular depth estimation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 2485–2494.
  * [68] K. Jiang, S.-Y. Chen, F.-L. Liu, H. Fu, and L. Gao, “NeRFFaceEditing: Disentangled face editing in neural radiance fields,” in _SIGGRAPH Asia Conference Papers_ , 2022.
  * [69] Z. Liu, Y. Feng, M. J. Black, D. Nowrouzezahrai, L. Paull, and W. Liu, “Meshdiffusion: Score-based generative 3d mesh modeling,” _arXiv preprint arXiv:2303.08133_ , 2023.
  * [70] P. Li, B. Li, and Z. Li, “Sketch-to-Architecture: Generative AI-aided Architectural Design,” in _Pacific Graphics Short Papers and Posters_. The Eurographics Association, 2023.
  * [71] J. Sun, B. Zhang, R. Shao, L. Wang, W. Liu, Z. Xie, and Y. Liu, “Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior,” _arXiv preprint arXiv:2310.16818_ , 2023.
  * [72] L. Melas-Kyriazi, I. Laina, C. Rupprecht, and A. Vedaldi, “Realfusion: 360deg reconstruction of any object from a single image,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 8446–8455.
  * [73] D. Xu, Y. Jiang, P. Wang, Z. Fan, Y. Wang, and Z. Wang, “Neurallift-360: Lifting an in-the-wild 2d photo to a 3d object with 360deg views,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 4479–4489.
  * [74] C. Deng, C. Jiang, C. R. Qi, X. Yan, Y. Zhou, L. Guibas, D. Anguelov _et al._ , “Nerdi: Single-view nerf synthesis with language-guided diffusion as general image priors,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 20 637–20 647.
  * [75] W. Yu, L. Yuan, Y.-P. Cao, X. Gao, X. Li, L. Quan, Y. Shan, and Y. Tian, “Hifi-123: Towards high-fidelity one image to 3d content generation,” _arXiv preprint arXiv:2310.06744_ , 2023.
  * [76] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering,” _ACM Transactions on Graphics (ToG)_ , vol. 42, no. 4, pp. 1–14, 2023.


# Collaborative Representation for SPD Matrices with Application to Image-Set
Classification

Li Chu, Rui Wang, and Xiao-Jun Wu L. Chu, R. Wang, and X.-J. Wu (Corresponding
author) are with the School of Artificial Intelligence and Computer Science,
Jiangnan University, Wuxi 214122, China. L. Chu, R. Wang, and X.-J. Wu are
also with Jiangsu Provincial Engineering Laboratory of Pattern Recognition and
Computational Intelligence, Jiangnan University e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>xiaojun_wu_jnu@163.com).

###### Abstract

Collaborative representation-based classification (CRC) has demonstrated
remarkable progress in the past few years because of its closed-form
analytical solutions. However, the existing CRC methods are incapable of
processing the nonlinear variational information directly. Recent advances
illustrate that how to effectively model these nonlinear variational
information and learn invariant representations is an open challenge in the
community of computer vision and pattern recognition To this end, we try to
design a new algorithm to handle this problem. Firstly, the second-order
statistic, i.e., covariance matrix is applied to model the original image
sets. Due to the space formed by a set of nonsingular covariance matrices is a
well-known Symmetric Positive Definite (SPD) manifold, generalising the
Euclidean collaborative representation to the SPD manifold is not an easy
task. Then, we devise two strategies to cope with this issue. One attempts to
embed the SPD manifold-valued data representations into an associated tangent
space via the matrix logarithm map. Another is to embed them into a
Reproducing Kernel Hilbert Space (RKHS) by utilizing the Riemannian kernel
function. After these two treatments, CRC is applicable to the SPD manifold-
valued features. The evaluations on four banchmarking datasets justify its
effectiveness.

###### Index Terms:

Collaborative represenration based classification, Symmetric positive definite
matrix, Tangent space, Riemannian kernel space, Image set classification.

##  I Introduction

Sparse representation based classification (SRC) has recently attracted
increasing attention in the domain of digital image processing and pattern
recognition due to its robustness [1, 2, 3, 4, 5, 6, 7, 8, 9]. It finds a
sparse representation of the testing sample over the dictionary with sparsity
constraints computed by minimizing the $L_{1}$-norm based reconstruction
error. Based on this, many papers explore innovative applications or design
more discriminative dictionary learning models to further improve the visual
classification performance [2, 10, 11]. Recently, some works have shown that
collaborative representation is more efficient than $L_{1}$-norm based
sparsity constraint. To this end, Zhang et al.[12] propose a collaborative
representation based classification (CRC) method using the $L_{2}$-norm based
regularization, and the experimental results show that CRC is able to achieve
competitive classification performance but with lower computational complexity
than SRC. Hence, CRC has attracted much attention due to its effectiveness
[13, 14, 15].

However, the existing collaborative representation methods are mainly designed
for Euclidean visual data, which can not encode the nonlinear variational
information between image sets efficiently. Recently, image set classification
has been proven to be an area of increasing vitality in the computer vision
and pattern recognition community [16, 17, 18, 19, 20, 21, 22, 23, 24, 25].
Compared to single-shot image-based classification [26, 27, 28, 29, 30, 31],
each image set generally contains a large number of images that belong to the
same class and cover large variations such as pose changes, illumination
differences and partial occlusions. When referring to the mathematical models
of image set data, covariance matrix [32, 33, 34], linear subspace [35, 36,
37] and Gaussian distribution[38] are three well-known representation methods,
among which, covariance matrices provide a natural data representation by
computing the second-order statistic of a group of samples. Thus, we select it
to describe each image set in this paper. According to the previous works [16,
22, 32], we know that the space formed by a family of covariance matrices with
the same dimentionality is a non-linear Riemannian manifold. Specifically, it
is an SPD manifold. As a consequence, the Euclidean learning approaches can
not be applied to the SPD manifold-valued data directly. To handle this
problem, some distance metrics have been developed on the Riemannian manifold
for similarity measurement, such as Log-Euclidean Metric (LEM) [39] and
Affine-Invariant Riemannian Metric (AIRM) [40]. By making use of these
Riemannian metrices, the Euclidean computations can be generalized to the non-
Euclidean data representations according to the following tactics.

Inspired by the superiority of kernel representation [41], the first type of
tactic is to embed the original Riemannian manifold into a Reproducing Kernel
Hilbert Space (RKHS) via Riemannian kernel functions [32, 35, 42, 43].
Although these methods can achieved favourable performance on several visual
classification tasks, the Riemannian geometrical structure of the original set
data may be distorted in the process of RKHS transformation. To solve this
issue, some algorithms have been put forward to jointly perform similarity
metric learning and embedding mapping learning on the original Riemannian
manifolds directly. Accordingly, a lower-dimensional and more discriminative
subspace can be generated for image set classification. Compared with the
aforementioned strategy, this type of approach takes the Riemannian geometry
of the original data manifold into fully consideration in the dimensionality
reduction process, thus may lead to impressive classification performance. As
a matter of fact, the methods mentioned above just model each given image set
from a single geometrical perspective, which may lose some useful structural
information for visual data. As a countermeasure, Wang et al.[23] and Huang et
al. [44] utilize the metric learning method to learn an effective metric space
for the heterogeneous features extracted from the multiple Riemannian
manifold-valued descriptors.

In parallel with the above developments, sparse representation methods have
received widespread attention in the context of Riemannian manifold due to its
simplicity and striking performance [45, 46, 42, 47, 48]. Obviously, these
methods make use of sparse representation to perform representation learning,
and $L_{1}$-norm based sparsity usually leads to higher computational cost.
Since the efficient $L_{2}$-norm based collaborative representation has
already shown its discriminatory power and higher computational efficiency, it
is valuable to study whether CRC can also perform well on the SPD manifold.
This study is planned for proposing the first attempt in this domain. With
this objective, we present a collaborative representation-based image set
classification in the scenario of SPD manifold in this paper. Fig. 1 indicates
the basic idea of traditional Euclidean-based CRC and our approach. Compared
with the Euclidean-based CRC (the top line of Fig. 1), we use the covariance
matrix to model each image set. Generally speaking, the space formed by a set
of covariance matrices is not a vector space but instead adhering to an SPD
manifold. In order to make the collaborative representation applicable to the
SPD manifold, we offer two strategies. The first strategy is to embed the
original SPD manifold into an associated tangent space via matrix logarithm
map (we use Log_CRC to represent this method). Another strategy is to exploit
the Riemannian kernel function to map the original SPD manifold into RKHS (we
call this method LogEK_CRC). In these two cases, the collaborative
representation can be applied to conduct further feature learning and
classification. The evaluations on four different benchmarking datasets verify
the effectiveness of the proposed method.

Figure 1: Overview of the Euclidean-based CRC and our method. Euclidean-based
CRC: Each image set is described as a vector feature. Our method: Each image
set is represented by a covariance matrix which resides on the SPD manifold.
To make collaborative representation work for the manifold-value data, we
provide two strategies, one is to embed the SPD manifold into a tangent space
via matrix logarithm map(Strategy 1 in the figure). Another transforms the SPD
manifold to RKHS by exploiting Riemannian kernel function(Strategy 2 in the
figure).

##  II Related Works

The existing Riemannian manifold-based image set classification methods can be
grouped into four categories, such as Riemannian kernel-based discriminative
learning methods [32, 35, 42, 43, 37], Riemannian manifold dimensionality
reduction methods [16, 33, 34, 36], multiple statistics fusion-based methods
[23, 44], and Riemannian manifold-based sparse representation methods [45, 46,
42, 47, 48]. The working mechanism of the first type of approach is to embed
the original Riemannian manifold into a Reproducing Kernel Hilbert Space
(RKHS) via Riemannian kernel functions [32, 35, 42, 43]. Therein, Wang et al.
[32] derive a Log-Euclidean Metric-based kernel function to transform the
original SPD manifold into RKHS. Similarly, Hamm et al. [35] embed the
original Grassmann manifold into RKHS via the projection kernel function.
Besides, both [32] and [35] utilize the kernel discriminant analysis algorithm
to perform discriminant subspace learning. In order to improve the description
ability of the Riemannian kernel-based approach, Wang et al. [gemkml] first
builds a lightweight feature extraction network on the original Grassmann
manfiold for the sake of learning fine-grained geometric features, Then, the
projection kernel function is applied to map the generated data
representations into the kernel space for the subsequent metric learning
process. However, one shortcoming of this type of approach is that the
Riemannian geometrical structure of the original set data may be changed in
the feature transformation process.

To solve this problem, a slice of algorithms have been proposed to jointly
perform similarity metric learning and embedding mapping learning on the
original Riemannian manifolds directly. Wherein, Harandi et al. [34] propose
to model the orthonormal mapping from the original high-dimensional SPD
manifold to a lower-dimensional one with a metric learning framework.
Similarly, Huang et al. [36] construct a metric learning framework on the
original Grassmann manifold for the purpose of directly yielding a lower-
dimensional and more appropriate Grassmann manifold for classification.
Differently, Huang et al. [16] propose a novel Log-Euclidean metric learning
framework on the SPD manifold tangent space, which can not only produce a
desirable new SPD manifold, but also demonstrate higher computational
efficiency. As a matter of fact, the aforementioned approaches just model each
given image set from a single geometrical perspective, which may lose some
useful structural information for visual classification. Therefore, Wang et
al. [23] and Huang et al. [44] propose to model each given image set with
different visual statistics simultaneously. Since different statistics reside
in different topological spaces, the metric learning framework is desinged to
fuse these hybrid features into a unified subspace for improved
classification.

In parallel with the above developments, the Riemannian manifold-based sparse
representation methods have obtained widespread attention due to its
simplicity and efficiency in representation learning [45, 46, 42, 47, 48].
Therein, Kai et al.[45] propose to embed the original SPD manifold-valued data
into an associated tangent space via matrix logarithm map. In this case, the
existing sparse representation methods apply. Different from [45], Harandi et
al.[42] utilize the Stein kernel to embed the original SPD manifold-valued
features into RKHS. Consequently, the sparse representation also can be
carried out in it. Similarly, Li et al.[47] and Harandi et al.[48] try to
generalise the sparse representation to the Riemannian manifolds by exploiting
the Log-Euclidean kernel and Jeffrey kernel, respectively.

##  III Background Theory

Before introducing our method, we briefly review the covariance matrix,
tangent space, and Riemannian kernel function, respectively.

###  III-A Covariance matrix

Let $S_{i}=\left\\{s_{1},s_{2},...,s_{n_{i}}\right\\}$ be an image set with
$n_{i}$ images, where $s_{i}\in\textup{R}^{d}$ is the $i-$th vectorized
instance. We model the image set $S_{i}$ as the $d\times d$ covariance
matrix[32],[33],[34]:

$X=\frac{1}{n-1}\sum_{i=1}^{n}\left(s_{i}-u_{S_{i}}\right)\left(s_{i}-u_{S_{i}}\right)^{T}$
(1)

where $u_{S_{i}}$ is the mean of $S_{i}$. In order to prevent covariance
matrix from being singular, we add a small perturbation value to $X$ :
$\hat{X}\Leftarrow\mathop{X}+\lambda I$, where $\lambda$ is set to
$10^{-3}\times trace\left(C\right)$ and $I$ is the identity matrix[32].

###  III-B Tangent space and Riemannian kernel function

The covariance matrix $X$ computed by Eq.(1) does not lie in a vector space
but instead adhering to a nonlinear Riemannian manifold, i.e., SPD manifold
$Sym_{\textup{d}}^{+}$. As a consequence, the Euclidean computations are
inapplicable. Arsigny et al.[49] propose to use the matrix logarithm map to
transform the manifold of covariance matrices into a vector space of symmetric
matrices. Let $X=U\Sigma U^{T}$ be the eigen-decomposition of covariance
matrix $X$, where $\Sigma$ is the diagonal matrix of eigenvalues, $U$ is the
corresponding eigenvectors matrix. Then, the matrix logarithm map can be
expressed as

$\textup{log}\left(X\right)=U\textup{log}\left(\Sigma\right)U^{T}$ (2)

where $\textup{log}\left(\Sigma\right)$ is the diagonal matrix of the
eigenvalue logarithms.

When it comes to similarity measurement on SPD manifold, Log-Euclidean
Metric(LEM)[39] is a widely used distance measure, which corresponds to an
Euclidean metric in the logarithmic domain. The distance between two points
$X_{i},X_{j}$ on $Sym_{\textup{d}}^{+}$ is computed by LEM as:

$d_{LEM}\left(X_{i},X_{j}\right)=\left\|\textup{log}\left(X_{i}\right)-\textup{log}\left(X_{j}\right)\right\|_{\textup{F}}$
(3)

where $\left\|\cdot\right\|_{\textup{F}}$ means the matrix Frobenius norm.
Under this metric, a Riemannian kernel[47] can be formulated as:

$k\left(X_{i},X_{j}\right)=\textup{exp}\left(-\beta\left\|\textup{log}\left(X_{i}\right)-\textup{log}\left(X_{j}\right)\right\|_{\textup{F}}^{2}\right)$
(4)

the validity of which has been confirmed in[47].

##  IV Collaborative representation for SPD Matrices

This section minutely presents the proposed collaborative representation
method for SPD matrices. Specifically, Section A discusses the strategic
details of mapping the SPD manifold into an associated tangent space, and we
introduce the collaborative representation in RKHS in Section B. Let
$\widetilde{S}=\left\\{S_{1},S_{2},...,S_{N}\right\\}$ be the gallery composed
by $N$ image sets, with each set expressed as:
$S_{i}=\left\\{s_{1},s_{2},...,s_{n_{i}}\right\\}\in\textup{R}^{d\times
n_{i}}$, where $i=1\to N$, and $n_{i}$ means the number of images in the
$i-$th image set, $S_{i}$ belongs to category $c_{i}$. From the previous
studies[32],[33],[34], we know that each image set can be represented by a
covariance matrix in Eq.(1). Then, we use
$\widetilde{X}=\left\\{X_{1},X_{2},...,X_{N}\right\\}$ to denote the computed
SPD manifold-valued feature representations of the gallery $\widetilde{S}$.

###  IV-A Collaborative representation in tangent space(Log_CRC)

Collaborative representation has been proven to have a good performance for
Euclidean data. But, it can not be applied to the SPD manifold-valued features
directly. To address this problem, we embed each sample of $\widetilde{X}$
into the tangent space by using the logarithm map
$\Psi_{\textup{log}}:Sym_{\textup{d}}^{+}\rightarrow T$, where $T$ means the
associated tangent space of $\widetilde{X}$. And the points in the tangent
space are expressed as
$L_{X}=\left\\{L_{1},L_{2},......,L_{N}\right\\},L_{i}=\textup{log}(X_{i})\in\textup{R}^{d\times
d}$. To facilitate the subsequent computations, we further convert each
tangent map into a vector. $l_{X}=\left\\{l_{1},l_{2},......,l_{N}\right\\}$
is used to denote the vectorized matrix of $L_{X}$ and
$l_{i}=vec(L_{i})\in\textup{R}^{d^{2}}$. Similarly, we can use the same
procedure to obtain the vectorized query set. For each query data $y$, it can
be collaboratively represented as $l_{X}$ and classified by measuring which
class leads to the minimum reconstruction error. The representation vector $w$
is obtained by solving the following objective function:

$w=\textup{min}\left\\{\left\|y-l_{X}w\right\|_{2}^{2}+\lambda_{1}\left\|w\right\|_{2}\right\\}$
(5)

where $\lambda_{1}$ is the regularization parameter.

The solution to Eq.5 can be easily derived as:

$w=\left(l_{X}^{T}l_{X}+\lambda_{1}I\right)^{-1}l_{X}^{T}y$ (6)

where $I$ is an identity matrix. The reconstruction error between the query
sample $y$ and the gallery with respect to class $c$ is

$Err_{c}=\left\|y-l_{X}^{c}w_{c}\right\|_{2}/\left\|w_{c}\right\|_{2}$ (7)

where $l_{X}^{c}$ denotes the data collected by $l_{X}$ from the $c-$th class
and $w_{c}$ is the corresponding collaborative representation coefficients
with respect to class $c$, $y$ is assigned to the class with the smallest
reconstruction error

$label=\underset{c}{\textup{min}}\left(Err_{c}\right),c=1,2,...k$ (8)

where $k$ indicates the number of classes.

Algorithm: The proposed method
---
Input:
$\bullet$ The gallery: $\widetilde{X}=\left\\{X_{1},X_{2},...,X_{N}\right\\}$,
$X_{i}\in Sym_{\textup{d}}^{+}$
$\bullet$ The regularization parameter: $\lambda_{1}$, $\lambda_{2}$
Output:
$\bullet$ Class label
Log_CRC method:
1: Get the tangent space data
$L_{X}=\left\\{L_{1},L_{2},......,L_{N}\right\\}$ by logarithm map in
Eq.$(\ref{log2})$.
2: Obtain the vectorized data in the tangent space:
$l_{X}=\left\\{l_{1},l_{2},......,l_{N}\right\\}$.
3: Compute the collaborative representation coefficient for each vectorized
test point $y$ via:
$w=\left(l_{X}^{T}l_{X}+\lambda_{1}I\right)^{-1}l_{X}^{T}y$.
4: Compute the reconstruction error of $y$ and the data for class $c$ via
$Err_{c}=\left\|y-l_{X}^{c}w_{c}\right\|_{2}/\left\|w_{c}\right\|_{2}$.
5: Get the class label by
$\underset{c}{\textup{min}}\left(Err_{c}\right),c=1,2,...k$.
LogEK_CRC method:
1: Compute the $K_{\widetilde{X}\widetilde{X}}$, $K_{\widetilde{X}Y}$ and
$K_{YY}$ by Riemannian kernel function in Eq.(4).
2: Perform SVD on $K_{\widetilde{X}\widetilde{X}}=U\Sigma U^{T}$.
2: Compute $\Psi=\left(U\Sigma^{1/2}\right)^{T}$ in Eq.(11) and then get
$\Phi=\left(\Psi^{T}\right)^{-1}K_{\widetilde{X}Y}$.
4: Get the collaborative representation coefficient for each query point $Y\in
Sym_{\textup{d}}^{+}$ via:
$w=\left(\Psi^{T}\Psi+\lambda_{2}I\right)^{-1}\Psi^{T}\Phi$.
5: Compute
$\underset{c}{\textup{min}}\left\\{\left\|\Phi-\Psi_{c}w_{c}\right\|_{2}/\left\|w_{c}\right\|_{2}\right\\}$
to get the class label of $Y$.

###  IV-B Collaborative representation in RKHS(LogEK_CRC)

To make the Euclidean collaborative representation applicable to the SPD
manifold-valued data, another strategy is to transform the SPD manifold to
RKHS by employing the Riemannian kernel function specified in Eq. 4. This
mapping process is expressed as: $\o:Sym_{\textup{d}}^{+}\rightarrow H$. We
use
$\o(\widetilde{X})=\left\\{\phi(X_{1}),\phi(X_{2}),......,\phi(X_{N})\right\\}$
to represent the new feature representation of $\widetilde{X}$ in Riemannian
kernel space $H$. For each query point $Y$ on $Sym_{\textup{d}}^{+}$, the
Riemannian kernel collaborative representation[50] in the RKHS is formulated
as:

$\begin{split}\leavevmode\resizebox{305.33788pt}{}{$w=\textup{min}\left\\{\left\|\o\left(Y\right)-\o(\widetilde{X})w\right\|_{2}^{2}+\lambda_{2}\left\|w\right\|_{2}^{2}\right\\}$}\end{split}$
(9)

where $\lambda_{2}$ is the regularization parameter. According to Mercer
theorem[51], the inner product between two points in kernel space can be
represented by their kernel function
$k(X_{i},X_{j})=\left\langle\o\left(X_{i}\right),\o\left(X_{j}\right)\right\rangle$.
With the Riemannian kernel defined in Eq.$(\ref{LE_K})$, Eq.$(\ref{KCRC})$ is
expanded as

$\begin{split}\leavevmode\resizebox{328.82706pt}{}{$K_{YY}-2w^{T}K_{\widetilde{X}Y}+w^{T}K_{\widetilde{X}\widetilde{X}}w+\lambda_{2}\left\|w\right\|_{2}^{2}$}\end{split}$
(10)

where, $K_{\widetilde{X}\widetilde{X}}$ is a $N\times N$ matrix, each element
of the matrix describes the similarity between data in the gallery, $i.e$,
$k(X_{i},X_{j})$, where $i=1,2,...N$ and $j=1,2,...N$. $K_{\widetilde{X}Y}$ is
a $N\times 1$ vector which consists of $k(X_{i},Y),i=1,2,...N$.
Inspired by [52], in order to further simplify Eq.$(\ref{KCRC2})$, the matrix
$K_{\widetilde{X}\widetilde{X}}$ is rewritten through Singular Value
Decomposition (SVD)

$\begin{split}\leavevmode\resizebox{328.82706pt}{}{$K_{\widetilde{X}\widetilde{X}}=U\Sigma
U^{T}=U\Sigma^{1/2}\left(\Sigma^{1/2}\right)^{T}U^{T}=\Psi^{T}\Psi$}\end{split}$
(11)

where $\Sigma$ is the diagonal matrix of eigenvalues, $U$ is the corresponding
eigenvectors matrix. $\Psi=\left(U\Sigma^{1/2}\right)^{T}$ is the approximate
representation of $\widetilde{X}$ in Riemannian kernel space, i.e.,
$\o(\widetilde{X})$. Further, $K_{\widetilde{X}Y}$ can be rewriten as

$K_{\widetilde{X}Y}=\Psi^{T}\left(\Psi^{T}\right)^{-1}K_{\widetilde{X}Y}=\Psi^{T}\Phi$

(12)

where $\Phi=\left(\Psi^{T}\right)^{-1}K_{\widetilde{X}Y}$ is the approximate
representation of $Y$ in Riemannian kernel space, i.e., $\o(Y)$. Similarly,
$K_{YY}=\Phi^{T}\Phi$. Thus, we get

$\begin{split}&\Phi^{T}\Phi-2w^{T}\Psi^{T}\Phi+w^{T}\Psi^{T}\Psi
w+\lambda_{2}\left\|w\right\|_{2}^{2}\\\ &=\left\|\Phi-\Psi
w\right\|_{2}^{2}+\lambda_{2}\left\|w\right\|_{2}^{2}\end{split}$ (13)

$w$ can be derived as

$w=\left(\Psi^{T}\Psi+\lambda_{2}I\right)^{-1}\Psi^{T}\Phi$ (14)

then we obtain the label of $Y$

$\begin{split}label&=\underset{c}{\textup{min}}\left\\{\left\|\phi(Y)-\phi(\widetilde{X})_{c}w_{c}\right\|_{2}/\left\|w_{c}\right\|_{2}\right\\}\\\
&=\underset{c}{\textup{min}}\left\\{\left\|\Phi-\Psi_{c}w_{c}\right\|_{2}/\left\|w_{c}\right\|_{2}\right\\}\\\
&c=1,2,...,k\end{split}$ (15)

where $\Psi_{c}$ is the data associated with the $c-th$ class, $w_{c}$ is the
collaborative representation coefficients with respect to class $c$, $k$
indicates the number of classes.

###  IV-C Relation With the Previous Works

The propsoed approach is related to two previous works [53, 54]. In this part,
we summarize some essential differences between our method and those
introduced in [53] and [54] in the following two paragraphs.

Relation With [53]. In fact, both the proposed algorithm and [53] try to
generalise the Euclidean collaborative representation to the Riemannian
manifold-valued feature representations for the sake of improving the image
set classification performnce on some challenging visual scenarions. However,
there exist some fundamental differences among them: 1) for feature
representation, the propsoed approach is executed in the scenario of SPD
manifiold, while [53] is carrried out in the context of Grassmann manifold; 2)
based on the generated Grassmann manifold-valued features, [53] constructs the
$P+V$ model to encode the inter-class similarity information and the intra-
class variational information of the data, while the propsoed approach does
not consider this explicitly; 3) with the constructed $P+V$ model, [53]
implements the collaborative representation framework with prototype learning
in the space spanned by a set of symmetric matrices. As a result, the
dictionary learning problem could be converted into standard Euclidean
problem. Different from [53], our method implements it in two ways. The first
is to map the original SPD manifold into an associated tangent space, and the
second is to embed it into the RKHS. Since both the tangent space and the RKHS
conform to the Euclidean geometry, the collaborative representation applies.

Relation With [54]. Actually, both the proposed algorithm and [54] devote to
extend the traditional single image-based collaborative representation to the
image set-based one to inject new vigour for image set classification. But,
they demonstrate different representation learning mechanisms for this
problem. Specifically, for similarity measurement between the query set and
each gallery set, [54] model the query set and the whole training dictionary
as a convex hull to define the collaborative representation-based set-to-sets
distance for classification. Hence, [54] has two characteristics: 1) the
distinctiveness of the images in the query set has been exploited; 2) the
feature learning and classification process is implemented in the Euclidean
space. Different from [54], our approach first model the original training and
test image sets onto the SPD manifold for the purpose of mining the nonlinear
geometrical structural information of the data. In order to perform similarity
measurement and classification, we offer two tactics. One is to map the SPD
manifold-valued features into the tangent space, another is to transform them
into RKHS. In these two scenarios, the Euclidean collaborative representation
framework can be naturally applied. However, the proposed method does not
consider the distinctiveness of the instances in the query set in the
representation learning process. The aforementioned differences also leads to
the different objective functions among them. Besides, the propsoed approach
is evaluated on four different visual classification tasks, while [54] is
limited to face recognition task.

##  V Experiments And Analysis

We test the classification performance of our method on four typical visual
tasks: object categorization, face recognition, virus cell classification and
dynamic scene classification. For the task of set-based object categorization,
we use the ETH-80 dataset[55]. The widely used YouTube Celebrities (YTC)
dataset [32] is applied to the face recognition task. We utilize the MDSD
dataset[56] for the task of video-based dynamic scene classification and Virus
[57] for virus cell classification task.

TABLE I:  Average recognition rates and standard deviations of different
menthods on ETH-80 [55], Virus [57] , MDSD [56] and YTC[32] datasets.

Method |  ETH-80[55] |  Virus [57] |  MDSD [56] |  YTC[32]
---|---|---|---|---
CDL[32] |  93.75$\pm$3.43 |  48.33$\pm$3.60 |  30.51$\pm$2.82 |  68.72$\pm$2.96
GDA[35] |  93.25$\pm$4.80 |  47.00$\pm$2.49 |  30.51$\pm$7.78 |  65.78$\pm$3.34
PML[36] |  90.00$\pm$3.53 |  47.33$\pm$3.43 |  29.32$\pm$4.66 |  67.62$\pm$3.32
LEML[16] |  92.25$\pm$3.19 |  55.67$\pm$9.94 |  29.74$\pm$3.89 |  69.04$\pm$3.84
MMML[23] |  95.00$\pm$1.89 |  51.13$\pm$7.67 |  31.95$\pm$6.26 |  76.70$\pm$2.81
SPDML-AIRM[33] |  91.65$\pm$3.34 |  52.33$\pm$8.42 |  21.21$\pm$5.06 |  64.66$\pm$2.92
SPDML-Stein[33] |  90.83$\pm$3.62 |  51.60$\pm$8.10 |  21.79$\pm$4.88 |  61.87$\pm$3.32
Frob_SRC[46] |  93.25$\pm$4.43 |  54.75$\pm$7.83 |  23.75$\pm$5.67 |  63.16$\pm$2.72
LogEK_SRC[47] |  94.88$\pm$4.17 |  56.33$\pm$6.37 |  27.95$\pm$4.94 |  72.55$\pm$2.77
Log_SRC[45] |  95.12$\pm$4.17 |  55.67$\pm$7.04 |  34.83$\pm$5.70 |  74.68$\pm$2.71
LogEK_CRC |  96.50$\pm$3.76 |  61.67$\pm$3.93 |  35.64$\pm$4.09 |  78.87$\pm$2.44
Log_CRC |  97.50$\pm$1.67 |  64.67$\pm$5.02 |  36.15$\pm$5.19 |  78.90$\pm$2.53

###  V-A Datasets and settings

The ETH-80 dataset consists of 8 categories, such as apples, cows, cups, dogs,
horses, pears, tomatoes and cars, with each category has 10 image sets, and
each image set contains 41 images of different perspectives.We randomly select
five from each category for traning and the remaining five for testing. The
top line of Fig. 2 shows some examples of this dataset.

The Virus dataset is composed of 15 categories, each of which consists of 5
image sets. There are 20 different images in each set. For this dataset, 3
image sets are randomly chosen for training and the rest 2 image sets are used
for testing. Some examples of Virus dataset are presented in the second line
of Fig. 2.

The MDSD dataset contains 13 different categories of dynamic scenes, with each
class consists of 10 video sequences collected in an unconstrained setting. We
randomly select 7 videos for training and the rest for testing in each class.
As presented in the third line of Fig. 2, there are some sample images of this
dataset.

For the YouTube Celebrities dataset, there are 1910 video clips of 47 subjects
involved in it. Each clip consists of hundreds of face frames. We randomly
choose 9 video clips in each subject with 3 for training and 6 for testing.
Some sample face frames of this dataset are shown in the bottom line of Fig.
2.

Figure 2: Examples in four datasets. Top line: ETH-80. The second line: Virus.
The third line: MDSD. Bottom line: YTC.

To keep consistent with the previous works[32] [23] [16] [45] [47], we conduct
ten-fold cross-validation experiments, i.e., repeat the randomly selecting
process of the gallery/probes ten times. Then, the average classification
scores are reported for each method on the four used datasets. Besides, we
resize each given image into a $20\times 20$ grayscale one.

Figure 3: The effect of regularization parameter $\lambda_{1}$ on accuracy for
Log_CRC method.  Figure 4: The effect of regularization parameter
$\lambda_{2}$ on accuracy for LogEK_CRC method.

###  V-B Comparative methods

In order to study the effectiveness of the proposed method, we compare the
proposed method with some representative image set classification methods
including: Covariance Discriminant Learning(CDL)[32] , Grassmann Discriminant
Analysis (GDA)[35], Projection Metric Learning (PML)[36], Log-Euclidean Metric
Learning (LEML)[16], Multiple Manifolds Metric Learning(MMML)[23], SPD
Manifold Leaning (SPDML-AIRM, SPDML-Stein)[33], Generalized Dictionary
Learning and Sparse Coding using Frobenius Norm(Frob_SRC)[46], Logarithm
Mapping for Sparse Representation(Log_SRC)[45], and Log-Euclidean Kernels for
Sparse Representation(LogEK_SRC)[47]

We should emphasize that we use the source codes of all the comparative
methods provided by the original authors to conduct experiments on the four
datasets. For CDL, the perturbation is set to $10^{-3}\times
trace\left(X\right)$. For GDA, the number of basis vectors for the subspace
are determined by crossvalidation. In PML, the number of iterations, the
trade-off coefficient $\alpha$ and the target dimensionality $d$ are reported
according to the original work. In LEML, the parameter $\eta$ is tuned in the
range of $\left[0.1,1,10\right]$, $\zeta$ is tuned from $0.1$ to $0.5$. For
SPDML, following the original work[33], $v_{w}$ is fixed as the minimum number
of samples in one class while the dimensionality of the target SPD manifold is
tuned by cross-validation. For LogEK_SRC, we use Riemannian kernel function in
Eq.(4).

###  V-C Result and Analysis

Table I lists the recognition rates of different comparative methods on the
four datasets. Among them, we can intuitively see that our method, especially
Log_CRC, exhibits better performance in terms of recognition scores on the
four datasets. It is also interesting to see that the recognition rates of PML
are lower than that of LEML. The main difference of them is that PML conducts
the projection metric learning on the Grassmann manifold, while LEML learns
the Log-Euclidean Metric on the SPD manifold, which justifies the superiority
of LEM-based SPD manifold dimensionality reduction. The classification
performance comparison between SPDML-Stein and SPDML-AIRM further verifies the
effectiveness of the LEM-based Riemannian metric learning frameworks for image
set classification. Obviously, MMML performs better than other competitors on
the ETH-80, MDSD, and YTC datasets, because of the complementarity of multiple
Riemannian manifolds is qualified to mine more powerful structural information
for improved classification.

Secondly, compared with the sparse representation methods Frob_SRC, Log_SRC,
LogEK_SRC and collaborative representation methods Log_CRC, LogEK_CRC, we can
find that collaborative representation-based classification in tangent space
and Riemannian kernel space for SPD matrix perform better than sparse
representation-based classification. This could demonstrate that the
collaborative representation is more efficient than $L_{1}$-norm based sparse
constraint.

Due to the regularization parameters $\lambda_{1}$ and $\lambda_{2}$ play a
pivotal role in our method, we utilize cross-validation to choose suitable
values for them on the four used datasets. Fig. 3 and Fig. 4 illustrate their
impact on the classification results of Log_CRC and LogEK_CRC. According to
these two figures, the best value of $\lambda_{1}$ on the ETH-80, Virus, MDSD
and YTC datasets are set to 0.01, 0.5, 0.05, and 0.1, respectively. For
$\lambda_{2}$, 0.5, 1, 0.1, and 0.05 are the selected proper values for the
ETH-80, Virus, MDSD, and YTC datasets, respectively.

TABLE II: The average classification scores of the different methods on the
ETH-80, Virus, MDSD, and YTC datasets.

Method | ETH-80 | Virus | MDSD | YTC
---|---|---|---|---
LogEK_CRC |  96.50$\pm$3.76 |  61.67$\pm$3.93 |  35.64$\pm$4.09 |  78.87$\pm$2.44
Log_CRC |  97.50$\pm$1.67 |  64.67$\pm$5.02 |  36.15$\pm$5.19 |  78.90$\pm$2.53
CRC |  91.05$\pm$3.94 |  24.67$\pm$2.81 |  32.05$\pm$6.86 |  63.69$\pm$3.61
SPD_CRC |  95.50$\pm$2.84 |  40.00$\pm$6.85 |  34.62$\pm$5.13 |  64.96$\pm$2.89

TABLE III:  Testing time(s) comparison of five methods on ETH-80 dataset.(For
classifying one image set.)

Method |  ETH-80[55]
---|---
Frob_SRC[46] | 0.212
LogEK_SRC[47] | 0.147
Log_SRC[45] | 0.259
LogEK_CRC | 0.127
Log_CRC | 0.078

###  V-D Ablation Study for Each Component of the Proposed Framework

Although the effectiveness of the proposed approach has been demonstrated, to
validate the contribution of each component of the proposed framework is also
highly appealing. To this end, we make a comparison between LogEK_CRC,
Log_CRC, CRC, and SPD_CRC on the ETH-80, Virus, MDSD, and YTC datasets to
observe their difference in classification performance. Here, ’CRC’ represents
that we directly exploit the Euclidean collaborative representation to perform
feature selection and classification for the original image set samples. For
’SPD_CRC’, it first models the original set data onto the SPD manifold, then
makes use of the Euclidean collaborative representation to conduct feature
learning and classification in this space. The final classification results of
them are listed in Table II. According to this table, it is evident that the
classification accuracies of CRC are significantly lower than that of other
three competitors on the four used datasets. The reason may be that the
original image set samples contain large within-class variational information,
which restrains CRC from mining more pivotal information for classification.
After modeling these data onto the SPD manifold, the classification ability of
SPD_CRC has been further promoted compared to CRC, which confirms the
effectiveness of Riemannian manifold in mining the nonlinear geometrical
structure of the visual data. However, an obvious limitation of SPD_CRC is the
Euclidean collaborative representation-based classification is directly
carried out on the SPD manifold, which inevitably leads to the sub-optimal
solution. From Table II, it is also worth noting that the proposed LogEK_CRC
and Log_CRC outperform SPD_CRC and CRC in terms of classification score on the
four used datasets, visibly. These experimental observations demonstrate the
significance of each module of the proposed framework for image set
classification.

###  V-E Testing Time Comparison

Table III presents the testing time of the proposed methods and several sparse
representation methods on the ETH-80 dataset. The experiments were run on
3.0GHz PC with 4GB RAM and Matlab2016a software. We need to emphasize that the
testing time is computed by classifying one test sample with the whole
gallery. From Table III, it is clear to see that the testing burden of the
proposed Log_CRC and LogEK_CRC has been reduced compared to other competitors.
The reasons are twofold: 1) our method classifies the query set with the
gallery directly instead of classifying by learning a dictionary in SRC
methods; 2) our method has closed-form analytical solutions.

##  VI Conclusion

This paper proposes a collaborative representation-based image set
classification algorithm in the context of SPD manifold. For this approach, we
suggest two ways to implement feature extraction and classification, one is to
map the SPD manifold-valued data representations into the tangent space via
matrix logarithm map, another is to embed them into RKHS using the Riemannian
kernel function. After that, the merits of Euclidean collaborative
representation are qualified to be generalised to the SPD manifold.

Experimental results obtained on four different benchmarking datasets show
that the proposed approach could make a considerable improvement in
classification performance and computational efficiency compared to some
representative image set classification methods. For future work, we plan to
integrate dictionary learning into the proposed framework for further
improving the discriminability of the learned geometric features.

## References

  * [1] Lei Zhang, Meng Yang, and Xiangchu Feng.  Sparse representation or collaborative representation: Which helps face recognition?  In 2011 International conference on computer vision, pages 471–478. IEEE, 2011.
  * [2] Meng Yang, Lei Zhang, Xiangchu Feng, and David Zhang.  Fisher discrimination dictionary learning for sparse representation.  In 2011 International Conference on Computer Vision, pages 543–550. IEEE, 2011.
  * [3] Hui Li, Xiao-Jun Wu, and Tariq Durrani.  Nestfuse: An infrared and visible image fusion architecture based on nest connection and spatial/channel attention models.  IEEE Transactions on Instrumentation and Measurement, 69(12):9645–9656, 2020.
  * [4] Hui Li and Xiao-Jun Wu.  Multi-focus image fusion using dictionary learning and low-rank representation.  International Conference on Image and Graphics, pages 675–686, 2017\.
  * [5] Xiaoqing Luo, Zhancheng Zhang, Baocheng Zhang, and Xiao-Jun Wu.  Image fusion with contextual statistical similarity and nonsubsampled shearlet transform.  IEEE Sensors Journal, 17(6):1760–1771, 2017.
  * [6] C Li, W Yuan, AC Bovik, and X Wu.  No-reference blur index using blur comparisons.  Electronics letters, 47(17):962–963, 2011.
  * [7] Qingbing Sang, Huixin Qi, Xiaojun Wu, Chaofeng Li, and Alan C Bovik.  No-reference image blur index based on singular value curve.  Journal of Visual Communication and Image Representation, 25(7):1625–1630, 2014.
  * [8] Matej Kristan, Aleš Leonardis, Jiří Matas, Michael Felsberg, Roman Pflugfelder, Joni-Kristian Kämäräinen, Martin Danelljan, Luka Čehovin Zajc, Alan Lukežič, Ondrej Drbohlav, et al.  The eighth visual object tracking vot2020 challenge results.  European Conference on Computer Vision, pages 547–601, 2020.
  * [9] Jun Sun, Chao Li, Xiao-Jun Wu, Vasile Palade, and Wei Fang.  An effective method of weld defect detection and classification based on machine vision.  IEEE Transactions on Industrial Informatics, 15(12):6322–6333, 2019\.
  * [10] Lishan Qiao, Songcan Chen, and Xiaoyang Tan.  Sparsity preserving projections with applications to face recognition.  Pattern Recognition, 43(1):331–341, 2010.
  * [11] Li Zhang, Wei-Da Zhou, Pei-Chann Chang, Jing Liu, Zhe Yan, Ting Wang, and Fan-Zhang Li.  Kernel sparse representation-based classifier.  IEEE Transactions on Signal Processing, 60(4):1684–1695, 2012.
  * [12] Zhang Lei, Yang Meng, and Xiangchu Feng.  Sparse representation or collaborative representation: Which helps face recognition?  In International Conference on Computer Vision, 2011.
  * [13] Lei Zhang, Meng Yang, Xiangchu Feng, Yi Ma, and David Zhang.  Collaborative representation based classification for face recognition.  arXiv preprint arXiv:1204.2358, 2012.
  * [14] Wei Li and Qian Du.  Joint within-class collaborative representation for hyperspectral image classification.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 7(6):2200–2208, 2014.
  * [15] Sijia Cai, Lei Zhang, Wangmeng Zuo, and Xiangchu Feng.  A probabilistic collaborative representation based approach for pattern classification.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2950–2959, 2016.
  * [16] Zhiwu Huang, Ruiping Wang, Shiguang Shan, Xianqiu Li, and Xilin Chen.  Log-euclidean metric learning on symmetric positive definite manifold with application to image set classification.  In International conference on machine learning, pages 720–729, 2015.
  * [17] Kai-Xuan Chen and Xiao-Jun Wu.  Component spd matrices: A low-dimensional discriminative data descriptor for image set classification.  Computational Visual Media, 4(3):245–252, 2018.
  * [18] Zifeng Wu, Yongzhen Huang, and Liang Wang.  Learning representative deep features for image set analysis.  IEEE Transactions on Multimedia, 17(11):1960–1968, 2015.
  * [19] Kai-Xuan Chen, Xiao-Jun Wu, Rui Wang, and Josef Kittler.  Riemannian kernel based nyström method for approximate infinite-dimensional covariance descriptors with application to image set classification.  In 2018 24th International Conference on Pattern Recognition (ICPR), pages 651–656. IEEE, 2018.
  * [20] Gong Cheng, Peicheng Zhou, and Junwei Han.  Duplex metric learning for image set classification.  IEEE Transactions on Image Processing, 27(1):281–292, 2017.
  * [21] Rui Wang, Xiao-Jun Wu, and Josef Kittler.  Symnet: A simple symmetric positive definite manifold deep learning method for image set classification.  IEEE Transactions on Neural Networks and Learning Systems, pages 1–15, 2021.
  * [22] Wen Wang, Ruiping Wang, Shiguang Shan, and Xilin Chen.  Discriminative covariance oriented representation learning for face recognition with image sets.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5599–5608, 2017.
  * [23] Rui Wang, Xiao-Jun Wu, Kai-Xuan Chen, and Josef Kittler.  Multiple manifolds metric learning with application to image set classification.  In 2018 24th International Conference on Pattern Recognition (ICPR), pages 627–632. IEEE, 2018.
  * [24] Rui Wang, Xiao-Jun Wu, Kai-Xuan Chen, and Josef Kittler.  Multiple riemannian manifold-valued descriptors based image set classification with multi-kernel metric learning.  IEEE Transactions on Big Data, 2020.
  * [25] Rui Wang, Xiao-Jun Wu, Zhen Liu, and Josef Kittler.  Geometry-aware graph embedding projection metric learning for image set classification.  IEEE Transactions on Cognitive and Developmental Systems, pages 1–1, 2021.
  * [26] Xiao-Jun Wu, Josef Kittler, Jing-Yu Yang, Messer Kieron, and Shitong Wang.  A new direct lda (d-lda) algorithm for feature extraction in face recognition.  Proceedings of the 17th International Conference on Pattern Recognition, 4:545–548, 2004.
  * [27] Y-J Zheng, J-Y Yang, Jian Yang, X-J Wu, and Z Jin.  Nearest neighbour line nonparametric discriminant analysis for feature extraction.  Electronics Letters, 42(12):679–680, 2006.
  * [28] Su-Gen Chen and Xiao-Jun Wu.  A new fuzzy twin support vector machine for pattern classification.  International Journal of Machine Learning and Cybernetics, 9(9):1553–1564, 2018.
  * [29] Guan-Lin Shen and Xiao-Jun Wu.  Content based image retrieval by combining color, texture and centrist.  2013 Constantinides International Workshop on Signal Processing, pages 16–16, 2013.
  * [30] Xiaoqing Luo, Zhancheng Zhang, and Xiaojun Wu.  A novel algorithm of remote sensing image fusion based on shift-invariant shearlet transform and regional selection.  AEU-International Journal of Electronics and Communications, 70(2):186–197, 2016.
  * [31] Zhen-Hua Feng, Josef Kittler, Muhammad Awais, Patrik Huber, and Xiao-Jun Wu.  Face detection, bounding box aggregation and pose estimation for robust facial landmark localisation in the wild.  Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 160–169, 2017.
  * [32] Ruiping Wang, Huimin Guo, Larry S Davis, and Qionghai Dai.  Covariance discriminative learning: A natural and efficient approach to image set classification.  In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 2496–2503. IEEE, 2012.
  * [33] Mehrtash T Harandi, Mathieu Salzmann, and Richard Hartley.  From manifold to manifold: Geometry-aware dimensionality reduction for spd matrices.  In European conference on computer vision, pages 17–32. Springer, 2014.
  * [34] Mehrtash Harandi, Mathieu Salzmann, and Richard Hartley.  Dimensionality reduction on spd manifolds: The emergence of geometry-aware methods.  IEEE transactions on pattern analysis and machine intelligence, 40(1):48–62, 2018.
  * [35] Jihun Hamm and Daniel D Lee.  Grassmann discriminant analysis: a unifying view on subspace-based learning.  In Proceedings of the 25th international conference on Machine learning, pages 376–383. ACM, 2008.
  * [36] Zhiwu Huang, Ruiping Wang, Shiguang Shan, and Xilin Chen.  Projection metric learning on grassmann manifold with application to video based face recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 140–149, 2015.
  * [37] Rui Wang, Xiao-Jun Wu, and Josef Kittler.  Graph embedding multi-kernel metric learning for image set classification with grassmannian manifold-valued features.  IEEE Transactions on Multimedia, 23:228–242, 2020.
  * [38] Wen Wang, Ruiping Wang, Zhiwu Huang, Shiguang Shan, and Xilin Chen.  Discriminant analysis on riemannian manifold of gaussian distributions for face recognition with image sets.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2048–2057, 2015.
  * [39] Vincent Arsigny, Pierre Fillard, Xavier Pennec, and Nicholas Ayache.  Geometric means in a novel vector space structure on symmetric positive-definite matrices.  SIAM journal on matrix analysis and applications, 29(1):328–347, 2007.
  * [40] Xavier Pennec, Pierre Fillard, and Nicholas Ayache.  A riemannian framework for tensor computing.  International Journal of computer vision, 66(1):41–66, 2006.
  * [41] Yu-jie Zheng, Jian Yang, Jing-yu Yang, and Xiao-jun Wu.  A reformative kernel fisher discriminant algorithm and its application to face recognition.  Neurocomputing, 69(13-15):1806–1810, 2006.
  * [42] Mehrtash T Harandi, Conrad Sanderson, Richard Hartley, and Brian C Lovell.  Sparse coding and dictionary learning for symmetric positive definite matrices: A kernel approach.  In European Conference on Computer Vision, pages 216–229. Springer, 2012.
  * [43] Sadeep Jayasumana, Richard Hartley, Mathieu Salzmann, Hongdong Li, and Mehrtash Harandi.  Kernel methods on the riemannian manifold of symmetric positive definite matrices.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 73–80, 2013.
  * [44] Zhiwu Huang, Ruiping Wang, Shiguang Shan, and Xilin Chen.  Face recognition on large-scale video in the wild with hybrid euclidean-and-riemannian metric learning.  Pattern Recognition, 48(10):3113–3124, 2015.
  * [45] Guo Kai, Prakash Ishwar, and Janusz Konrad.  Action recognition using sparse representation on covariance manifolds of optical flow.  In IEEE International Conference on Advanced Video & Signal Based Surveillance, 2010.
  * [46] Suvrit Sra and Anoop Cherian.  Generalized dictionary learning for symmetric positive definite matrices with application to nearest neighbor retrieval.  In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 318–332. Springer, 2011.
  * [47] Peihua Li, Qilong Wang, Wangmeng Zuo, and Lei Zhang.  Log-euclidean kernels for sparse representation and dictionary learning.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1601–1608, 2013.
  * [48] Mehrtash T Harandi, Richard Hartley, Brian Lovell, and Conrad Sanderson.  Sparse coding on symmetric positive definite manifolds using bregman divergences.  IEEE transactions on neural networks and learning systems, 27(6):1294–1306, 2016.
  * [49] Vincent Arsigny, Pierre Fillard, Xavier Pennec, and Nicholas Ayache.  Log-euclidean metrics for fast and simple calculus on diffusion tensors.  Magnetic Resonance in Medicine: An Official Journal of the International Society for Magnetic Resonance in Medicine, 56(2):411–421, 2006\.
  * [50] Biao Wang, Weifeng Li, Norman Poh, and Qingmin Liao.  Kernel collaborative representation-based classifier for face recognition.  In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 2877–2881. IEEE, 2013.
  * [51] Shenghua Gao, Ivor Wai-Hung Tsang, and Liang-Tien Chia.  Sparse representation with kernels.  IEEE Transactions on Image Processing, 22(2):423–434, 2012.
  * [52] Yuwei Wu, Yunde Jia, Peihua Li, Jian Zhang, and Junsong Yuan.  Manifold kernel sparse representation of symmetric positive-definite matrices and its applications.  IEEE Transactions on Image Processing, 24(11):3729–3741, 2015.
  * [53] Dong Wei, Xiaobo Shen, Quansen Sun, Xizhan Gao, and Wenzhu Yan.  Prototype learning and collaborative representation using grassmann manifolds for image set classification.  Pattern Recognition, 100:107123, 2020.
  * [54] Pengfei Zhu, Wangmeng Zuo, Lei Zhang, Simon Chi-Keung Shiu, and David Zhang.  Image set-based collaborative representation for face recognition.  IEEE transactions on information forensics and security, 9(7):1120–1132, 2014.
  * [55] Bastian Leibe and Bernt Schiele.  Analyzing appearance and contour based methods for object categorization.  In Computer Vision and Pattern Recognition, 2003. Proceedings. 2003 IEEE Computer Society Conference on, volume 2, pages II–409. IEEE, 2003\.
  * [56] Nitesh Shroff, Pavan Turaga, and Rama Chellappa.  Moving vistas: Exploiting motion for describing scenes.  In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pages 1911–1918. IEEE, 2010.
  * [57] Gustaf Kylberg, Mats Uppström, and Ida-Maria Sintorn.  Virus texture analysis using local binary patterns and radial density profiles.  In Iberoamerican Congress on Pattern Recognition, pages 573–580. Springer, 2011.