[1,3,5]Muhammad Usman

[1]Department of Information & Computer Science, King Fahd University of
Petroleum & Minerals (KFUPM), Dhahran, Saudi Arabia

2]College of Computer and Information Sciences, Jouf University, Sakaka, Saudi
Arabia

3]SDAIA-KFUPM Joint Research Center for Artificial Intelligence, King Fahd
University of Petroleum & Minerals (KFUPM), Dhahran, Saudi Arabia

4]Faculty of Computing and Information Technology, King Abdulaziz University
(KAU), Jeddah, Saudi Arabia

5]Center for Intelligent Secure Systems, King Fahd University of Petroleum &
Minerals (KFUPM), Dhahran, Saudi Arabia

# Deep Models for Multi-View 3D Object Recognition: A Review

Mona Alzahrani<EMAIL_ADDRESS><EMAIL_ADDRESS>Salma
Kammoun<EMAIL_ADDRESS>Saeed Anwar<EMAIL_ADDRESS>Tarek
Helmy<EMAIL_ADDRESS>*  [  [  [  [

###### Abstract

Human decision-making often relies on visual information from multiple
perspectives or views. In contrast, machine learning-based object recognition
utilizes information from a single image of the object. However, the
information conveyed by a single image may not be sufficient for accurate
decision-making, particularly in complex recognition problems. The utilization
of multi-view 3D representations for object recognition has thus far
demonstrated the most promising results for achieving state-of-the-art
performance. This review paper comprehensively covers recent progress in
multi-view 3D object recognition methods for 3D classification and retrieval
tasks. Specifically, we focus on deep learning-based and transformer-based
techniques, as they are widely utilized and have achieved state-of-the-art
performance. We provide detailed information about existing deep learning-
based and transformer-based multi-view 3D object recognition models, including
the most commonly used 3D datasets, camera configurations and number of views,
view selection strategies, pre-trained CNN architectures, fusion strategies,
and recognition performance on 3D classification and 3D retrieval tasks.
Additionally, we examine various computer vision applications that use multi-
view classification. Finally, we highlight key findings and future directions
for developing multi-view 3D object recognition methods to provide readers
with a comprehensive understanding of the field.

###### keywords:

3D object recognition, Multi-view object recognition, Multi-view conventional
neural network, 3D object classification, 3D object retrieval

##  1 Introduction

Deep Learning (DL) has lately been popular to solve many research problems
involving image, sound, text, or graph processing. Significant advances in the
DL networks have been successfully proposed and implemented for one-
dimensional (1D) and two-dimensional (2D) data (Ioannidou ., 2017). This led
to remarkable results in different computer vision tasks such as recognition
(classification and/and retrieval), detection and localization, segmentation,
and scene understanding (Ahmed ., 2018). Due to the huge success of Deep
Neural Networks (DNNs) in many 2D computer vision tasks, there have been
recent efforts to use them for tackling tasks in more challenging scenarios
and complicated settings such as Three-dimensional (3D) data processing
(Ioannidou ., 2017; Ahmed ., 2018).

Three-dimensional data, such as 3D scenes or objects, is considered a
priceless resource in the computer vision field. The reason behind that is
because the 3D data, unlike the widely available 2D data, has rich information
about the full geometry of the observed scenes and objects, thus providing a
chance for the machines to understand the environment better (Ahmed ., 2018).
Because different sensors capture 3D data, it comes in different
representations, as shown in Fig. 1, e.g., polygon meshes, voxels, point
clouds, multi-view renderings images, etc. (Gezawa ., 2020). These 3D
representations vary in both properties and structure. Hence, understanding
their structure is essential for analyzing the 3D objects for various
applications (Ahmed ., 2018).

Figure 1: Various representations of the 3D data.

The availability of 3D data and advancements in DL has led to increased
exploration by researchers in the application of DNNs for solving various
computer vision problems involving 3D scene understanding, such as 3D object
recognition, 3D object classification, 3D shape retrieval, detection, and
segmentation (Ioannidou ., 2017; Ahmed ., 2018; S. Qi ., 2021; Hamdi ., 2021).
However, extending DNNs to work with 3D data is challenging and non-trivial
since DL networks have traditionally been designed to process 1D or 2D data
(Ioannidou ., 2017), and 3D objects possess complex geometric features with
significant variations in their various 3D representations. As a result, DL on
3D data has emerged as a separate research field, referred to as 3D Deep
Learning (3D-DL), within the broader domain of computer vision (Ahmed .,
2018).

Figure 2: 3D object recognition tasks: (a) 3D object classification, and (b)
3D object retrieval. Figure 3: Example of multi-view 3D object representation.

One of the crucial domains of study within the field of 3D computer vision is
the recognition of 3D objects, also called 3D shape recognition. This area of
research encompasses both 3D classification and 3D retrieval techniques, as
highlighted in recent literature (S. Qi ., 2021; H. Su ., 2015). The 3D object
classification task involves classifying a given 3D object by identifying its
category (i.e., class label) as shown in Fig. 2.a. In contrast, the 3D object
retrieval task involves giving a 3D object as a query to find all the 3D
objects in the dataset as the retrieved shapes that have the same predicted
category as shown in Fig. 2.b (Ahmed ., 2018; S. Qi ., 2021; Bustos ., 2005).
The recent attention to building recognition models directly from 3D data has
appeared for several reasons. One of them is the improvements in 3D imaging
sensors (e.g., Microsoft’s Kinect and LiDAR sensors) (S. Qi ., 2021; Bustos .,
2005), which made the technology practical for real-world applications such as
medical image analysis, automated driving, intelligence robots,
virtual/augmented reality, archaeology, and others (S. Qi ., 2021; Wei .,
2020). Another reason is the availability of the academic datasets of 3D
objects such as ModelNet (Wu ., 20151) and ShapeNet (Savva ., 2017), and
online repositories such as TurboSquid 111 TurboSquid. Retrieved on September
$17$, $2023$ from: https://www.turbosquid.com/Search/3D-Models., and 3D
Warehouse 222 3D Warehouse. Retrieved on September $17$, $2023$ from:
https://3dwarehouse.sketchup.com., which allow for training more complex
recognition models than before (S. Qi ., 2021; Wei ., 2020).

The researchers have classified previous methods of 3D object recognition into
three distinct categories based on their representation of input data (S. Qi
., 2021; Wei ., 2020): i) point-based methods (C.R. Qi, Su ., 2017; C.R. Qi,
Yi ., 2017; Y. Li, Bu ., 2018; Y. Liu ., 2019; Y. Wang ., 2019), ii)
volumetric-based methods (Wu ., 20152; Maturana Scherer, 2015; Furuya Ohbuchi,
2016; Zhi ., 2018; Brock ., 2016), and iii) view-based methods (H. Su ., 2015;
Feng ., 2018; Kanezaki ., 2018; Wei ., 2020; Hamdi ., 2021). The researchers
evaluated the techniques utilized for 3D object recognition within the third
classification. The justification for this focus is that view-based methods
have demonstrated superior performance in 3D object recognition, thereby
achieving the current state-of-the-art results (Feng ., 2018), and they are
more similar to the human visual system (Alam ., 2021). In these methods, the
3D objects are rendered through their projected 2D multi-view images (i.e.,
views) from different viewpoints and angles, as shown in Fig. 3.

###  1.1 Previous Surveys and Our Contributions

Many surveys are found in the literature (Ioannidou ., 2017; Vodrahalli
Bhowmik, 2017; Ahmed ., 2018; Guo ., 2020; Gezawa ., 2020; S. Qi ., 2021)
covering the advances of DL methods on 3D object recognition. Table 1
summarized these surveys in terms of their published years, titles, the
computer vision tasks, the targeted 3D data representations, the DL methods
categorization employed for 3D object recognition, and the time period. These
surveys were either focused explicitly on a singular 3D data representation,
such as 3D multi-view data (S. Qi ., 2021) or point cloud data (Guo ., 2020)
or even encompassed a broader scope and included a variety of different 3D
data representations (Ioannidou ., 2017; Vodrahalli Bhowmik, 2017; Ahmed .,
2018; Gezawa ., 2020).

Table 1: Summary of the existed 3D object recognition surveys

No. Reference & Year Title Computer Vision Tasks 3D Data DL Method
Categorization Time Range 1 (Ioannidou ., 2017) Deep learning advances in
computer vision with 3d data: A survey 3D scene segmentation, 3D object
classification and retrieval General 3D data Direct 3D data, descriptors,
RGB-D, 2D projections/views, and hyperspectral DL methods Until 2016 2
(Vodrahalli Bhowmik, 2017) 3D computer vision based on machine learning with
deep neural networks: A review Classification and segmentation, and generation
General 3D data - Until 2017 3 (Ahmed ., 2018) A survey on deep learning
advances on different 3d data representations 3D object classification,
segmentation, and retrieval, 3D correspondence General 3D data Euclidean and
non-Euclidean DL methods Until 2018 4 (Guo ., 2020) Deep learning for 3d point
clouds: A survey 3D object detection and tracking, 3D shape classification,
and 3D point cloud segmentation Point clouds Point-based and view-based DL
methods Until 2020 5 (Gezawa ., 2020) A review on deep learning approaches for
3d data representations in retrieval and classifications 3D object
classification and retrieval General 3D data Raw data, surfaces, solids, high-
level structures, and multi-view DL methods Until 2020 6 (S. Qi ., 2021)
Review of multi-view 3d object recognition methods based on deep learning 3D
object classification and retrieval Multi-view Model-based and view-based CNN
methods Until 2021 7 Our Survey Deep Models for Multi-View 3D Object
Recognition: A Review 3D object classification and retrieval Multi-view Model-
based and view-based CNN methods Until present

Table 2: Competitive analysis of the existed 3D object recognition surveys and
their covered topics

No. Covered Topics (Ioannidou ., 2017) (Vodrahalli Bhowmik, 2017) (Ahmed .,
2018) (Guo ., 2020) (Gezawa ., 2020) (S. Qi ., 2021) Our Survey 1 3D Data
Categorization ✓ ✓ ✓ ✓ ✓ 2 3D Object Recognition Method
Categorization ✓ ✓ ✓ ✓ ✓ ✓ 3 General 3D Object Recognition Pipeline ✓ ✓ ✓ 4
Summary of 3D Datasets ✓ ✓ ✓ ✓ ✓ ✓ 5 Comparison of the Methods Based on
Experimented 3D Datasets ✓ 6 Summary of Camera Configurations ✓ ✓ 7 Comparison
of the Methods Based on Experimented Camera Configurations ✓ 8 Summary of
Views Selection Strategies ✓ ✓ 9 Summary of Pre-trained CNN Architectures ✓ ✓
10 Comparison of the Methods Based on Experimented Pre-trained CNN
Architectures ✓ 11 Summary of Fusion Strategies ✓ 12 Summary of Evaluation
Metrics ✓ ✓ 13 Comparison of the Methods Based on
Recognition Performance ✓ ✓ ✓ ✓ ✓ ✓ 14 Relevant Computer Vision Applications ✓
15 Factors impacting the recognition performance ✓ ✓ 16 Limitations and future
research ✓ ✓ ✓ ✓

Regarding the surveys that generally covered different 3D data representations
(Ioannidou ., 2017; Vodrahalli Bhowmik, 2017; Ahmed ., 2018; Gezawa ., 2020),
(Ioannidou ., 2017) review the existing DL methods applied to 3D data.
Specifically, the authors surveyed the main techniques that tackle the 3D data
manipulation steps, which include segmenting the 3D scene, detecting the 3D
key point, extracting the 3D descriptors, and retrieving and classifying the
3D object. They also detailed the most utilized architectures in 3D computer
vision tasks, which are Convolutional Neural Networks (CNNs), Recurrent Neural
Networks (RNNs), Deep Belief Networks (DBNs) and Autoencoders (AEs). The DL
methods were classified into five categories based on the 3D representation
utilized as input to the deep neural network. These 3D representations are
descriptors extracted from 3D and RGB-D data by separating depth and color
channels, direct access to the 3D data, one or more 2D projections/views of
the 3D object, and HyperSpectral data.

For each 3D data representation, the authors overview the most popular methods
for 3D shape classification and 3D shape retrieval. Similarly, (Vodrahalli
Bhowmik, 2017) provide a review of the general DL methods in the computer
vision field, highlighting Classification, segmentation, and generation tasks
that involve 3D visual data. In addition, the different deep networks that
have been suggested for 3D vision understanding are summarized. Also, the
intersection between this field and neuroscience and psychology has been
explained to understand the human visual system better. The authors expanded
their review with the current major challenges in this field and concluded
their survey with future research directions that were important in that year.
In 2018, (Ahmed ., 2018) categorized the 3D data representation into two main
categories. The first category is the 3D Euclidean representations, which have
a fundamental grid structure that accepts a common coordinates system and a
global parameterization such as 3D descriptors, multi-view data, 3D
projections, RGB-D data, and volumetric data (voxel and octree). The second
category is the non-Euclidean representations, where no global
parameterization lacks the gridded array structure, such as point clouds,
graphs, and polygon meshes. The authors also detailed the DL techniques for
different 3D data representations under these two categories. In 2020, (Gezawa
., 2020) reviewed the performance of DL methods on several 3D data
representations categorized as solids (e.g., octree and voxels), surfaces as a
polygon mesh, multi-views, high-level structures (e.g., 3D descriptors and
graph), and raw data (e.g., RGB-D data, point cloud, and 3D projections). The
performance of these representations was reviewed for 3D object retrieval and
classification tasks. The same year, in 2020, (Guo ., 2020) presented a review
covering three tasks, including 3D object detection and tracking, 3D shape
classification, and 3D point cloud segmentation. It mainly focuses on the
recent progress of DL methods, which take point cloud as the 3D data
representation, and only a few multi-view-based techniques were mentioned in
that work. The point-based methods were categorized based on the network
architecture utilized for the learning, including pointwise MLP, graph-based,
hierarchical data structure-based, convolution-based, and other standard
methods.

Regarding the surveys that focused on the 3D multi-view data, (S. Qi ., 2021)
presented an inclusive review of studies published until 2021 on the latest
progress in the DL methods employed for multi-view 3D object recognition. The
authors divided these methods into three categories depending on their input:
voxel-based methods, pointset-based methods, and view-based methods, where
they focus on the last category. They introduce the most used 3D datasets and
the evaluation metrics for classification and retrieval. Also, they provide an
analysis of the existing methods in terms of the utlized viewpoints setting
and input modes, backbone network, feature extraction, fusion strategies, and
viewpoints selection mechanisms. Towards the end, they summarized the
performance of the existing methods on several famous datasets.

Prior studies, as seen in Table 1, have either encompassed a broad and
outdated range of 3D data representations (Ioannidou ., 2017; Vodrahalli
Bhowmik, 2017; Ahmed ., 2018; Gezawa ., 2020) or solely focused on a single
aspect of 3D data representation, excluding 3D multi-view data (Guo ., 2020).
The survey that focused on the 3D multi-view data (S. Qi ., 2021) also
covering different DL methods and according to what they mention in their
paper was conducted on a very narrow time range from 2018 to 2021 as shown in
Fig. 4. In addition, based on the competitive analysis of the existing 3D
object recognition surveys presented in Table 2, different surveys lack
various importance topics such as 3D data categorization (Guo ., 2020), 3D
datasets summarizing and recognition performance comparison (Vodrahalli
Bhowmik, 2017), multi-view camera configurations, Views selection strategies,
pre-trained CNN architectures, evaluation metrics (Ioannidou ., 2017;
Vodrahalli Bhowmik, 2017; Cao ., 2020; Gezawa ., 2020), fusion strategies
(Ioannidou ., 2017; Vodrahalli Bhowmik, 2017; Cao ., 2020; Gezawa ., 2020; S.
Qi ., 2021), factors impacting the recognition performance (Ioannidou ., 2017;
Vodrahalli Bhowmik, 2017; Ahmed ., 2018; Guo ., 2020; Gezawa ., 2020), future
works (Ioannidou ., 2017; Ahmed ., 2018; S. Qi ., 2021), relevant computer
vision applications (Ioannidou ., 2017; Vodrahalli Bhowmik, 2017; Cao ., 2020;
Gezawa ., 2020; S. Qi ., 2021), and the most important the general pipeline of
the multi-view based methods and its steps (Vodrahalli Bhowmik, 2017; Guo .,
2020; Gezawa ., 2020; S. Qi ., 2021). Therefore, in-depth evaluation of the
most efficient 3D object recognition techniques from their inception is
imperative, specifically focusing on 3D multi-view data representation and
utilizing the popular and widely applied DL networks.

Figure 4: The timeline shows the covered period of the existing 3D object
recognition survey and the most relevant DL-based and transformer-based multi-
view 3D object recognition methods developed in recent years. The timeline
from 2015 (the first developed multi-view 3D object recognition method) until
the present.

In comparison to existing surveys (Ioannidou ., 2017; Cao ., 2020; Ahmed .,
2018; Gezawa ., 2020; S. Qi ., 2021), the primary contributions of this
research can be summarized as follows:

  * •

This review paper comprehensively covers the most recent progress of DL-based
and transformer-based 3D object recognition methods on 3D multi-view data,
highlighting the state-of-the-art techniques in the field and summarizing them
for the readers.

  * •

Unlike existing reviews (Ioannidou ., 2017; Cao ., 2020; Ahmed ., 2018; Gezawa
., 2020; S. Qi ., 2021), this research specifically focuses on DL-based and
transformer-based methods. Among these methods, it focuses on the techniques
that take 3D multi-view data as input representation.

  * •

It presents the analyses of the general pipeline for DL-based multi-view 3D
object recognition, including a summary of the various techniques employed at
each stage of the pipeline.

  * •

It contains the last developments of the transformers and attention mechanisms
in the multi-view 3D object recognition since 2021, which are not covered by
the existing reviews (Ioannidou ., 2017; Cao ., 2020; Ahmed ., 2018; Gezawa .,
2020; S. Qi ., 2021).

  * •

The comprehensive details of existing DL-based and transformer-based multi-
view 3D object recognition models are analyzed and discussed in terms of the
primarily used 3D datasets, camera configurations and the number of views,
view selection strategies, pre-trained CNN architectures, fusion strategies,
and recognition performance (e.g., Tables 4, 5, 6, 7, and 9).

  * •

In addition, this review showcases various computer vision applications that
incorporate multi-view classification through the utilization of CNNs.
Furthermore, contrary to the survey (S. Qi ., 2021) that has a narrow time
range, this review provides an overview of the advancements in multi-view 3D
object recognition methods, starting from the inception in 2015 until the
present day as seen in Fig. 4.

  * •

To provide the readers with future directions and ideas, and several facts
about how the various factors impact the recognition performance and future
trends for the development of multi-view 3D object recognition methods are
highlighted in terms of viewpoints and number of views, backbone and feature
extracting, feature fusion and view selection, light direction and object
color, and number of transformer blocks.

###  1.2 Paper Organization

The remainder of this paper is organized as follows. In Section 2, different
3D data representations are introduced, multi-view learning is defined, more
details on the 3D multi-view data and the use of multi-view in computer vision
fields are presented, and the reasons that led to most of the 3D object
recognition methods to choose the multi-view among all the deep architectures
are specified. Sections 3 and 4, the cores of this paper, start by comparing
different 3D object recognition methods, emphasizing the analysis of up-to-
date DL-based and transformer-based multi-view recognition methods. Section 5
covered various relevant computer vision applications that employed the DL-
based multi-view for image classification. Section 6 provides the readers with
ideas where several recommendations and directions for developing multi-view
3D object recognition methods are highlighted. Finally, this review is
concluded in Section 7.

##  2 Overview

This research reviews the 3D object recognition methods with multi-view 3D
data representation. Hence, in this section, some background is provided to
the readers to help them understand the different 3D data representations in
general and multi-view data representation in specific as it is the concern
representation in this study. In addition, it gives an overview of multi-view
learning from computer vision and machine learning perspectives. In
comparison, the latter focuses on the multi-view conventional neural network.

###  2.1 3D Data Representation

Due to the high significance of 3D-DL in complex tasks such as self-driving
cars, robotics, and virtual and augmented reality, it has attracted increasing
attention in the last few years (Hamdi ., 2021). However, in the 3D-DL field,
two fundamental challenges exist: i) the 3D data representation and ii) the
adopted network architectures. To get a successful deep model, the usability,
simplicity, and efficiency of the 3D data representations and their
fundamental properties need to be considered (Gezawa ., 2020). Hence, various
3D data representations, as shown in Fig. 1, were used by different 3D-DL
methods.

3D data come in different representations depending on the scanning device
used to capture the 3D object. Therefore, the 3D representations vary in
structure and properties (Ahmed ., 2018; Gezawa ., 2020). Several 3D
representations encouraged the researchers to exploit these data on their
tasks and select the suitable one (Gezawa ., 2020; Ioannidou ., 2017). These
representations can be categorized into two main categories based on (Ahmed .,
2018) taxonomy. The first category is the 3D Euclidean representations, which
have a fundamental grid structure that accepts a common coordinates system and
a global parameterization such as 3D descriptors, multi-view data, 3D
projections, RGB-D data, and volumetric data (voxel and octree).

On the other hand, the second category is the non-Euclidean representations,
where there is no global parameterization that lacks the gridded array
structure, such as point clouds, graphs, and polygon meshes. Another 3D data
categorization is the taxonomy introduced by (Gezawa ., 2020). They classified
the 3D data into five categories as follows: i) Raw data includes the 3D
projections, RGB-D data, and point cloud; ii) Solids representations such as
voxels and octree; iii) Surfaces that represent the polygon meshes as the most
popular representation in this category, iv) Multi-views data, and v) High-
level structures such as 3D descriptors and graph. Despite these different
taxonomies, each 3D data representation has a typical structure, properties,
ways for acquisitions, advantages, and limitations, as summarized clearly by
(Gezawa ., 2020). It should be noted that all of these 3D data representations
are vastly grown research areas and no direct winner among them (Gezawa .,
2020). But each has its advantages over the other, and choosing the suitable
one is crucial to the performance of a particular method.

###  2.2 Multi-View Learning

Multi-view Learning (MVL) is a technique that involves the utilization of
classifiers to extract common patterns or feature spaces from various separate
feature subsets or sources of the same data (Yan ., 2021; Alam ., 2021; Y. Li,
Yang Zhang, 2018) and is based on the acquisition of data from multiple
sources, which results in the creation of various views. For example, multi-
media data can be represented as video and audio and website data as text,
images, and hyperlinks. Moreover, due to the advancement of data technology in
real-world applications, vast amounts of multi-view data are generated.
Compared to the single-view learning methods, data from various views mostly
contain complementary information; hence, MVL utilizes this to learn more
comprehensive representations (Y. Li, Yang Zhang, 2018; Alam ., 2021).

MVL has obtained significant importance in the last decades due to its
remarkable performance in machine learning and computer vision fields (Yan .,
2021). In machine learning, it has been successfully applied in several areas,
such as clustering, supervised learning, ensemble learning, semi-supervised
learning, dimensionality reduction, reinforcement learning, etc. In computer
vision, MVL has achieved significant performance in applications such as image
recognition, handwritten digit recognition, human gait recognition, etc. (Alam
., 2021). The rest of this section presents multi-view use in computer vision
fields and more details on the 3D multi-view data. In addition, the reasons
that led most of the 3D object recognition methods to choose the multi-view
CNN among all the deep architectures are specified.

####  2.2.1 Multi-view Computer Vision

In the computer vision field, the view refers to an image displaying the same
object sample (as shown in Fig. 3) or part of it (as shown in Fig. 5) from a
specific perspective. The collection of images is the combination of different
images (or views) that were captured from the same object but different
viewpoints (from different cameras) (Alam ., 2021; Seeland Mäder, 2021).

Figure 5: Example of the multi-view of same object in the computer vision
field.

A significant task of the multi-view computer vision field is the multi-view
classification, which is motivated by human behavior. For example, a typical
use-case for multi-view classification is plant species identification (see
Fig 5), where a human watches different features of an unfamiliar plant to be
able to identify its species. Because it could be hard to distinguish between
different plant species by their flowers alone, it requires additional views,
e.g., their leaves (Seeland Mäder, 2021; Besl Jain, 1985).

(Kertész Vámossy, 2015) categorized the computer vision systems that have
multiple camera views in different ways. One category was based on the number
of views as shown in Fig. 6 .a and summarized as follows: i) Single view
system (Fig. 6 .a.1) when it uses only one camera, ii) Stereo view system
(Fig. 6 .a.2) when it uses two cameras that are necessarily next to each
other, and iii) Multi-view system (Fig. 6 .a.3) when the system uses more than
two cameras which is our concern in this review. Another important category
was based on Fields of View (FOV) overlapping as shown in Fig. 6 .b and listed
as follows: i) Overlapping fields (Fig. 6 .b.1) when the cameras are focused
on the same targeted object, ii) Non-overlapping fields (Fig. 6 .b.2) when the
cameras not capturing the same object on two simultaneous views, and iii)
Small overlapping fields (Fig. 6 .b.3) when the views are complete each other.

Figure 6: Computer vision systems in terms of: a) Number of views, and b)
Field of view overlapping. In terms of the number of views: a.1) Single view
system, a.2) Stereo view system, and a.3) Multi-view system. In terms of the
field of view overlapping: b.1) Overlapping fields, b.2) Non-overlapping
fields, and b.3) Small overlapping fields

####  2.2.2 Multi-view Convolutional Neural Network

Due to their powerful feature extraction ability, DL methods have succeeded in
many application domains with excellent performance, such as artificial
intelligence and computer vision. DL methods can efficiently learn the
targeted data’s non-linear, abstract, and complex representations by utilizing
several hierarchical layers (Voulodimos ., 2018; Yan ., 2021). Also, the deep
MVL methods have been progressively used with encouraging results (H. Su .,
2015; Feng ., 2018; Kanezaki ., 2018; Wei ., 2020; Hamdi ., 2021). And they
enhanced the performance of the traditional machine learning algorithm (Alam
., 2021). (Yan ., 2021) provides a comprehensive review of deep MVL methods;
however, the CNN-based multi-view models are the most extensively utilized DL-
based models in computer vision. The standard CNNs can be employed for scene
labeling, face recognition, image classification, human pose estimation,
action recognition, document analysis, etc. (Alam ., 2021). CNN represents an
advanced method and works equally and sometimes better than humans in various
tasks, especially for classification problems (Seeland Mäder, 2021). In
addition, they significantly outperformed the traditional machine learning
methods in a broad range of tasks, especially for image analysis (Voulodimos
., 2018; Alam ., 2021). CNN is a neural network that contains several layers.
Its layers involve two roles: the first is feature extraction (feature
learning), and the second handles the classification (Alam ., 2021). CNN
consists of a mixture of three main types of layers (see Fig.7):

  * •

Convolutional layers: It is the first step of feature extraction. In these
layers, several utilized filters (a.k.a. kernels) are used to convolve the
whole inputted image or the output of the previous layer to extract and
distinct several features and generate the different feature maps (Voulodimos
., 2018; Ioannidou ., 2017). Filters are also called feature detectors because
they can detect the features within an image, such as vertical and horizontal
lines (Alam ., 2021). So, the feature maps are generated as the output of the
convolutional layer.

  * •

Pooling or subsampling layers: It is responsible for reducing the size or
spatial dimensions (width $\times$ height) of the inputted feature maps, which
is why it is called downsampling or subsampling. This size reduction is
beneficial for reducing the network’s computational costs and avoiding
overfitting (Voulodimos ., 2018; Alam ., 2021). This layer does not impact the
depth dimension of the feature maps, resulting in the same feature map number
passed to the next layer as input (Voulodimos ., 2018; Ioannidou ., 2017). The
pooling operation is precise in the features that exist in the region of the
feature map (Alam ., 2021). There are different types of pooling operations,
but the most frequently utilized are max pooling or average pooling
(Voulodimos ., 2018; Alam ., 2021). So, the pooled feature maps are generated
as the output of the pooling layer.

  * •

Fully Connected (FC) layers: After a stack of multiple convolutional layers
and pooling layers, one or more FC layers (Ioannidou ., 2017; Voulodimos .,
2018). In the FC layer, neurons have full connections to all activation
functions in the previous layer (Voulodimos ., 2018). The purpose of FC layers
is to transfer the 2D pooled feature map into a flattened structure to get a
single 1D feature vector for further processing (Alam ., 2021; Voulodimos .,
2018). Usually, in classification and recognition tasks, the final FC layer is
connected to a classifier such as softmax that produces the response, e.g.,
class label, to the initial inputted image (Ioannidou ., 2017).

Figure 7: The standard Convolutional Neural Network (CNN) architecture
consists of multiple layers, including convolutional layers that extract
features from input images and pooling layers that reduce spatial dimensions.
Fully connected layers at the end process these features for classification,
making CNNs highly effective in various computer vision tasks such as image
recognition and object detection.

As a typical single-view CNN architecture, it aims for feature representation
learning with multiple parameter optimization, which in various domains has
exhibited excellent performance (Yan ., 2021). On the other hand, the multi-
view CNNs have access to multi-view information of the same data, which aims
to integrate helpful information from multiple views to learn more
comprehensive and discriminative representations to yield a more effective
classifier (Alam ., 2021). The current multi-view CNN architectures are
classified into the following two types as illustrated in Fig. 8: i) One-view-
one-network strategy (Fig. 8.a) that uses one CNN for each view to extract
feature representation separately, then the multiple representations from all
the views are fused (merged) through the later part of the network (Yan .,
2021). Several efforts have been made to design multi-view CNN architectures
with this type of strategy (e.g., the famous MVCNN (H. Su ., 2015),
RotationNet (Kanezaki ., 2018), and GVCNN (Feng ., 2018) models) for 3D object
recognition; and ii) Multi-view-one network strategy (Fig. 8. b) that uses a
single CNN to feed it with multi-view data to get the final representation
(Yan ., 2021). An example of this approach is the architecture of (Y. Wang .,
2020), which was proposed for breast cancer classification.

Figure 8: The two types of multi-view CNNs: a) One-view-one-network strategy,
and b) Multi-view-one-network strategy.

The CNN-based methods have been extensively utilized for multi-view 3D object
recognition for several reasons. One reason is that multi-view 3D object
recognition is based on image classification, which is a hard problem that
requires better image classification techniques. The CNN shows robust
performance in automatic feature extraction and classification, which makes it
a broadly employed classifier (L. Sun ., 2019; Alam ., 2021). Also, CNNs are
invariant to transformations, which means they can detect the class label even
if we translate the inputs. This property is useful for specific computer
vision applications, including the classification tasks (Voulodimos ., 2018).
The CNN-based methods proved to have improved feature representative
capability than the standard methods for recognition problems (C. Lin Kumar,
20181). Despite this, CNNs still suffer from some limitations. It heavily
depends on the availability of labeled data. In addition, insufficient or
small training data, even after data augmentation, can result in poor
performance and overfitting problems (Voulodimos ., 2018; C. Lin Kumar,
20181). Furthermore, in the training phase, it is computationally expensive
(Voulodimos ., 2018) and needs to learn a large number of parameters (L. Sun
., 2019). To avoid this limitation, previous multi-view 3D object recognition
approaches have been using transfer learning methods, data augmentation, and
pre-training CNN architecture such as AlexNet (Krizhevsky ., 2012), VGG-M
(Chatfield ., 2014), VGG-VD (Simonyan Zisserman, 2014), GoogLeNet (Szegedy .,
2015), or ResNet (He ., 2016) that already initialized with pre-trained
parameters instead of setting ones (Voulodimos ., 2018; L. Sun ., 2019).

Among the CNN-based 3D object recognition methods, the view-based networks for
3D object recognition have performed best to achieve the current state-of-the-
art performance. Using the DL technique for multi-view 3D object recognition
has become one of the rigorously researched directions due to the advanced and
successful pre-trained classification network that can be used directly as the
backbone network, and multiple viewpoints render multiple views of the object,
which can complete each other’s detailed. However, there still exist some
challenges in this area (S. Qi ., 2021). Hence, many DL-based methods have
been designed to solve the problems relating to this research direction.

The first idea is to collect information from multiple 2D views of a 3D object
into a global shape descriptor for 3D object recognition via a novel CNN
architecture called Multi-View Convolutional Neural Network (MVCNN) (H. Su .,
2015). In that work, (H. Su ., 2015) proved that 3D objects can be represented
effectively using view-based 2D representations, which could be utilized for
classification and retrieval. They contribute to the field by generating
global shape descriptors and leveraging image and 3D shape datasets to improve
performance.

##  3 DL-based Multi-View 3D Object Recognition

This review focuses on the recognition of 3D objects, which encompasses both
3D object classification and retrieval tasks because these tasks were the
first to be effectively addressed using the MVCNN method (H. Su ., 2015).
MVCNN method later served as the milestone for using deep 2D CNNs for multi-
view classification (S. Qi ., 2021). In addition, with the development of 3D
data and their processing techniques in computer vision, 3D object recognition
become an essential task with a critical role in wide applications (Feng .,
2018; Wei ., 2020).

Methods of 3D object recognition using DL have been classified into three
categories based on the input data representation (see Fig 9) (S. Qi ., 2021;
Wei ., 2020):

  1. 1.

Pointset or point-based methods: in this category, strategies are implemented
directly on the 3D objects, which are usually represented as point clouds or
sets of unordered points (Hamdi ., 2021). The point cloud is a collection of
points in the 3D space used to describe the geometric shape of the object’s
surface. Each point is defined by 3D coordinates (x, y, and z), gray value,
RGB color value, and depth data (Cao ., 2020), which was later analyzed by a
3D-DL network to learn the features for recognition (S. Qi ., 2021; Wei .,
2020). To generate point cloud representation, devices such as LiDAR and 3D
scanners can be utilized to generate laser light that hits the object’s
surface. It is worth noticing that the 3D reconstruction is the process of
reconstructing the number of points of the 3D object as shown in Fig. 10.b,
and more point clouds led to more created accurate models (Cao ., 2020). Some
of the methods achieving improved performance are PointNet (C.R. Qi, Su .,
2017) as the first point cloud CNN model, PointNet++ (C.R. Qi, Yi ., 2017),
PointCNN (Y. Li, Bu ., 2018), RS-CNN(Y. Liu ., 2019), and DGCNN (Y. Wang .,
2019).

  2. 2.

Voxel or volumetric-based methods: point cloud data can be used to construct a
3D regular grid representation called a voxel. In 3D space segmentation, the
voxel represents the tiniest unit of digital data. Each unit can be seen as a
fixed coordinates grid. It also has a resolution as the 2D image; the divided
3D space with better quality led to a smaller grid with better resolution (Cao
., 2020) as seen in Fig. 10.c. For the first category, these methods were
implemented directly on the 3D objects, which usually represented them as a
collection of voxels in the 3D Euclidean space (Hamdi ., 2021), and then
analyzed them by a 3D-DL network to learn the features for recognition (S. Qi
., 2021; Wei ., 2020). Most popular voxel-based methods are: 3D ShapeNets (Wu
., 20152) as the first volumetric CNN model, VoxNet (Maturana Scherer, 2015),
DLAN (Furuya Ohbuchi, 2016), LightNet (Zhi ., 2018) and VRN (Brock ., 2016).

  3. 3.

View-based methods: Multi-view images are the oldest representation of the 3D
data (Cao ., 2020). These methods rendered the 3D objects through its
projected 2D multi-view images (also called views) from different viewpoints
and angles as shown in Fig. 10.a. In this method, the 3D geometry is expressed
more simply in (S. Qi ., 2021; Wei ., 2020; Feng ., 2018; Cao ., 2020). Then,
feed these views, which are the input data to the 3D-DL network (mostly CNN)
to gather multi-view features later fused as global descriptors utilized for
recognition (Wei ., 2020). Among these networks, the milestone view-based
methods for 3D shape recognition are MVCNN (H. Su ., 2015) as the first multi-
view CNN model, GVCNN (Feng ., 2018), RotationNet (Kanezaki ., 2018), View-GCN
(Wei ., 2020), and MVTN (Hamdi ., 2021).

Figure 9: Categories of 3D object recognition methods based on input
representation of 3D data. The three types are Voxel-based, Point-based and
View-based Methods. Figure 10: The three most investigated 3D data
representations are (a) Multi-view, (b) Point cloud, and (c) Voxels object
representation.

Both first and second categories fall under one umbrella called model-based
methods (Feng ., 2018) which convolve a 3D shape in the 3D space using a 3D
convolution filter, consequently producing a 3D representation straightly from
the 3D data. Although these model-based methods are effective and use the
spatial information and all of the information about the object when the 3D
representation on a full resolution is used (H. Su ., 2015; Wei ., 2020),
however, they suffer from some disadvantages such as: i) Time complexity:
voxel resolution which needs to be significantly reduced to use for training a
deep network reasonable amount of time with the available samples (Wei .,
2020; H. Su ., 2015), ii) Computational complexity: voxelization of the shape
surface caused computational complexity and data sparsity (Wei ., 2020), iii)
Memory complexity: 3D CNNs are allowed to be used in voxel-based networks,
however, it suffers from cubic memory complexity (Hamdi ., 2021), iv)
Application limitation: the computational cost limits their practical
applications (S. Qi ., 2021), v) Point clouds are sparse, disordered, and
express limited data information (Cao ., 2020), and vi) Voxel with no
rotational invariance (Cao ., 2020).

Since the method’s effectiveness is dependent on the chosen 3D data
representation (Cao ., 2020), this review is focused on the view-based
approaches that are used for the classification and retrieval tasks for the
following advantages: i) They have performed best so far among all the 3D
object recognition methods (Feng ., 2018; H. Su ., 2015; S. Qi ., 2021), ii)
They achieved excellent accuracy for 3D shape classification because it takes
advantage of the pre-trained 2D classification networks (Wei ., 2020; S. Qi .,
2021; Hamdi ., 2021; H. Su ., 2015) with larger image datasets (e.g. ImageNet
(Russakovsky ., 2015)), iii) They do not depend on the complicated 3D features
(S. Qi ., 2021), iv) Capturing the inputs of these methods which are the views
is easy compared to other methods which take polygon mesh or point cloud
structure as input (S. Qi ., 2021; Feng ., 2018), v) The views captured from
several viewpoints can complete each other’s detailed features of the object
in case of occlusion (S. Qi ., 2021), vi) They can be adapted to specific
applications to produce feature representations from multi-view images that
have superior performance compared to other features in a variety of setups
(H. Su ., 2015), vii) They can efficiently learn an object representation that
gathers information from any inputted views number without any particular
order, and continually outputs a global descriptor of the same size (H. Su .,
2015), viii) A large amount of both 2D image and 3D object datasets can be
used to train the view-based methods (H. Su ., 2015; S. Qi ., 2021), ix) They
are more similar to the human visual system, which is fed with a flow of
rendered 2D images rather than more detailed 3D representations (Hamdi .,
2021), x) They solve a 3D task using 2D deep architectures which narrow the
gap between 3D and 2D learning (Hamdi ., 2021), and xi) 3D data with high
resolution can be utilize due to the low memory requirement of the rendered 2D
images (Gezawa ., 2020).

In 2D computer vision, the data structure is very clear-cut, which is images
constructed of dense pixels that are ordered carefully and evenly into a
precise grid. However, the 3D data world has no such consistency. 3D data can
be represented as multi-view image sets, voxels, point clouds, meshes, etc.
So, choosing a suitable data representation is half the battle if it will be
used in the 3D-DL research field. Consequently, in the following subsections,
based on what is mentioned above, this review work focused on the view-based
approaches employed for 3D object recognition. Although DL-based methods have
been varying in their architecture design, all the DL-based multi-view 3D
object recognition models share a general pipeline described in the next
subsection in detail. In addition, different 3D datasets, camera
configurations, view selection approaches, pre-trained CNN architectures, and
fusion strategies used with these approaches are also reviewed.

###  3.1 General Pipeline

Fig. 11 shows the general pipeline of any DL-based multi-view 3D object
recognition model. At the beginning of the model, 3D objects from 3D datasets
are needed for training purposes. Then, several virtual cameras are placed
around the 3D object to render the views. These virtual cameras are placed
according to pre-defined camera configurations and settings. And the number of
the rendered views is the same as the number of these virtual cameras. After
that, each rendered view is fed to a backbone network, which is pre-defined
CNN for feature extraction. Then, feature fusion must be performed using all
the extracted features to create a global shape descriptor. This global
descriptor is utilized later by the same pre-defined CNN for recognition. The
recognition performance can be measured depending on the recognition tasks
(classification or retrieval) using well-known evaluation metrics. It is worth
mentioning that there is another step on the pipeline, which is view selection
strategy that removes irrelevant and redundant views. The view selection step
is not shown in Fig. 11 because its place changes according to the type of
selection, which is discussed later. In this paper, each step of this pipeline
is detailed in subsequent subsections to describe the different choices of
each step and to review the DL-based multi-view models accordingly.

Figure 11: The general deep learning-based multi-view 3D object recognition
pipeline begins with acquiring multiple views of a 3D object and preprocessing
them for consistency. These views are then fed into a deep neural network that
extracts relevant features, fused, and again passed through a pre-trained CNN.
Lastly, the object labels are predicted, enabling accurate recognition from
various angles and perspectives.

####  3.1.1 3D Object Recognition Datasets

Real-world and synthetic datasets are essential for conducting fair
experiments and comparisons between various learning models (Gezawa ., 2020).
However, supervised learning requires a large amount of labeled data. Several
labeled image datasets are available, but they represent the objects by only a
single image instead of a collection of views. The limitation of the
unavailable collection data has prompted some studies to explore implementing
new multi-view or 3D datasets as alternative sources (Seeland Mäder, 2021).
These 3D datasets are ModelNet10 and ModelNet40 (Wu ., 20151), ScanObjectNN
(Uy ., 2019), ShapeNet core55 (Savva ., 2017), RGB-D (Lai ., 2011), and MIRO
(Kanezaki ., 2018). Table 3 summarizes the details of each of these datasets
in terms of their types, number of classes, number of samples, and the
standard splitting of samples for training, validation, and testing. Table 4
shows the view-based related works that experimented with these datasets for
3D object recognition. Due to space constraints within the table, and for easy
access, Table 11 in Appendix A presents a concise overview of these view-based
deep models, along with their corresponding references.

ModelNet is a large-scale dataset of 3D objects provided by (Wu ., 20151). The
dataset currently contains clean 127,915 3D computer-aided design (CAD) models
belonging to 662 object categories. The 3D CAD models were obtained by
querying each object category from online search engines such as 3D Warehouse
333 3D Warehouse. Retrieved on February 17, 2022, from:
https://3dwarehouse.sketchup.com. and manually labeling the data. Examples of
3D objects from the ModelNet dataset are shown in Fig. 12. The ModelNet
dataset has two subsets, which are ModelNet-10 and ModelNet-40 that are mostly
used in semantic segmentation, object classification, and recognition tasks
(Cao ., 2020) with standard splitting for training and testing. In addition to
the original “unaligned” ModelNet40 and ModelNet10, both datasets have the
“aligned” version that is publicly available on the ModelNet website 444
Princeton ModelNet. Retrieved on May 8, 2022, from:
https://modelnet.cs.princeton.edu/.. ModelNet40 (Wu ., 20151) contains 12,311
manually cleaned 3D objects (split as 9,843 for training and 2,468 for
testing) that belong to 40 class categories such as chair, airplane, guitar,
and cup. These 3D objects are without color information, and there is a
diverse number of objects across diverse categories. Modelnet40 contains 40
categories of 3D CAD models and is a common dataset for evaluating the
classification and semantic segmentation of 3D-DL models (Cao ., 2020). On the
other hand, ModelNet10 (Wu ., 20151) contains 4,899 manually cleaned 3D
objects (split as 3,991 for training and 908 for testing) that belong to 10
class categories. The current state-of-the-art performances on 3D object
classification, when the ModelNet40 benchmark dataset was experimented with,
are reported by ViewFormer (H. Sun ., 2023) with 98.90% overall accuracy among
all the classification task methods, where the ViewFormer learning model is
based on the multi-view approach. On the other hand, the current state-of-the-
art performances on 3D object classification when the ModelNet10 benchmark
dataset was experimented with, DRCNN (K. Sun ., 2020) reported 99.34% overall
accuracy, which is also a learning model on the multi-view approach.

ScanObjectNN is a point cloud dataset recently released by (Uy ., 2019). This
dataset has been used for 3D classification. It is more challenging and
realistic than ModelNet40 because it considers occlusions and includes
background. The dataset contains 2,902 point cloud objects collected from
real-world indoor scans that belong to 15 class categories. Examples of 3D
objects from the ScanObjectNN dataset are shown in Fig. 13. Three main
variants of this dataset exist to represent different levels of complexity to
explore the classification methods’ robustness. These variants are OBJ_ONLY
(only segmented objects), OBJ_BG (objects are captured with their background),
and the most challenging variant is PB (with rotating, translating, and
scaling the ground truth bounding boxes). This dataset has been utilized by
MVTN (Hamdi ., 2021) to test the generalization capability of the learning
model in more realistic settings.

Figure 12: Examples of 3D models from the ModelNet dataset (Wu ., 20151).
Figure 13: Examples of 3D objects from the ScanObjectNN dataset (Uy ., 2019).

ShapeNet Core55 (Savva ., 2017) is a subset of a large-scale 3D image ShapeNet
dataset (Chang ., 2015) that was developed by a team of researchers from the
TTIC Institute at Princeton and Stanford universities. It comprises 51,162
clean 3D mesh objects belonging to 55 class categories, further subdivided
into 203 subcategories. Different categories have different numbers of
objects. The 51,162 objects are split into 35764 for training (70%), 5133 for
validation (10%), and 10265 for testing (20%). ShapeNet Core55 is a
challenging 3D dataset designed annually for a shape retrieval challenge
called Shape Retrieval Contest (SHREC). ShapeNet Core55 was provided in the
SHREC’17 track 3. Two versions of the ShapeNet Core55 dataset exist (Kanezaki
., 2018): i) normal dataset (consistently aligned), and ii) perturbed dataset
(randomly rotated). 3D model samples of the ShapeNet Core55 dataset are shown
in Fig. 14. The ShapeNet Core55 is commonly employed for retrieval tasks as
done by PANORAMA-ENN (Sfikas ., 2018), SeqViews2SeqLabels (Han ., 2018), (Ma
., 2018), MVSG-DNN (HY. Zhou ., 2019), G-CNN (Esteves ., 2019), and HMVCM (AA.
Liu ., 2021). However, a few works, such as 3D2SeqViews (Han ., 2019) and
View-GCN (Wei ., 2020), have been using it for evaluating their models for
classification tasks.

RGB-D dataset by (Lai ., 2011) contains real images captured from multiple
viewpoints of objects placed on a one-dimensional rotation table. This dataset
includes RGB and depth multi-view images of 300 object samples belonging to 51
categories. The objects are captured from 30∘, 45∘, and 60∘ elevation angles
above the horizon to end up with 250,000 RGB-D images. However, the RGB-D
dataset has some drawbacks that make it not suitable for training multi-view-
based CNNs, which are i) the number of object samples per category is
insufficient (a minimum of 2 for training), ii) it conflicts with the upright
orientation assumption where the viewpoints of object samples are inconsistent
against the rotation axis in some categories and iii) it does not capture the
bottom faces of the objects on the turning table (Kanezaki ., 2018). Samples
of objects from the RGB-D dataset are shown in Fig. 15.

Figure 14: Examples of 3D models from ShapeNet Core55 dataset (Wu ., 20152).
Figure 15: Samples of objects captured from the RGB-D dataset (Lai ., 2011).

Multi-view images of rotated objects (MIRO) is a self-made dataset introduced
by (Kanezaki ., 2018) to test their RotationNet model. This dataset provides
multi-view images captured of rotated objects using Ortery’s 3D MFP studio. It
contains 120 objects belonging to 12 class categories. Each category has ten
objects. At the same time, each object was captured with ten elevation angles
and 16 azimuth angles to obtain a total of 160 views. However, this dataset
has only been used by RotationNet (Kanezaki ., 2018), and it doesn’t consider
a mainstream dataset yet (S. Qi ., 2021). Samples of multi-view images of 3D
objects from the MIRO dataset are shown in Fig. 16.

Figure 16: Samples of 20 multi-view images of 3D objects from the MIRO dataset (Kanezaki ., 2018) captured using the spherical configuration. Table 3: Details of the 3D datasets that used for multi-view 3D object recognition 3D Dataset Name |  |  Type |  Classes |  Objects |  Training |  Validation |  Testing
---|---|---|---|---|---|---|---
RGB-D (Lai ., 2011) |  |  Real |  51 |  300 |  - |  - |  -
ModelNet40 (Wu ., 20151) |  |  Synthetic |  40 |  12311 |  9843 |  - |  2468
ModelNet10 (Wu ., 20151) |  |  Synthetic |  10 |  4899 |  3991 |  - |  908
ShapeNet Core55 (Savva ., 2017) |  |  Synthetic |  55 |  51162 |  35764 |  5133 |  10265
MIRO (Kanezaki ., 2018) |  |  Synthetic |  12 |  120 |  - |  - |  -
ScanObjectNN (Uy ., 2019) |  |  Real |  15 |  2902 |  - |  - |  -

####  3.1.2 Camera Configurations

The multi-view networks for 3D object recognition captured the views using
some of the five mostly experimented pre-defined viewpoints shown in Fig. 17
(also called camera settings or multi-view configurations). These five
viewpoints are:

  * •

Circular configuration: the first camera setup is the regular circle. Where in
this setup, as shown in Fig. 17.a, virtual cameras are uniformly placed on a
horizontal circular path around the tested object and raised with elevation
$\phi$ equal to $None$ from the ground level and directed at the centroid of
the object’s mass (H. Su ., 2015; S. Qi ., 2021; Wei ., 2020; Feng ., 2018).
The weighted average is calculated as the object’s centroid where the weights
are the face areas (H. Su ., 2015). This setup is commonly beneficial to
capture views of real and aligned objects initially acquired with one-
dimensional turning tables. In other words, it is useful when the objects are
assumed to be with an upright orientation by a consistent axis (e.g., z-axis)
as the rotation axis that identified the upright orientation where the virtual
cameras are distributed over $None$ at intervals of the azimuth angle $\Theta$
around the axis (Kanezaki ., 2018; Hamdi ., 2021). In most of the related
works such as (H. Su ., 2015; Feng ., 2018; Wei ., 2020; Kanezaki ., 2018;
Hamdi ., 2021; Ma ., 2018), they set the azimuth angle $\Theta$ equal to
$None$ as default, which means locating 12 virtual cameras that extracted 12
rendered views from an object that is inputted simultaneously. Or even set the
azimuth angle $\Theta$ equal to $None$ as in (Feng ., 2018) that extracted
eight rendered views from an object.

  * •

Spherical (dodecahedral) configuration: the second camera setup is the
irregular spherical where this setup is without the consistent upright
orientation assumption of shapes when the objects are unaligned and not in the
same vertical direction (Kanezaki ., 2018). As shown in Fig. 17.b, irregularly
placed virtual cameras with equal spaces on a vertex of a dodecahedron/sphere
surrounding the object (Wei ., 2020; Hamdi ., 2021; Kanezaki ., 2018; S. Qi .,
2021). In this configuration, the camera viewpoints can be equally spread in
3D because a dodecahedron has the greatest vertices among regular polyhedral
(Kanezaki ., 2018). View-GCN (Wei ., 2020) experimented with this
configuration by locating 20 virtual cameras on the dodecahedron’s vertices
surrounding the object to render 20 views. Also, RotationNet (Kanezaki .,
2018) was experimented with this configuration but with three diverse patterns
of rotation degrees for each of the 20 viewpoints corresponding to the three
edges connected to each vertex to yield 60 candidates’ viewpoints. On the
other hand, MVCNN (H. Su ., 2015) extracted four rendered views from each one
of the 20 viewpoints using the following degrees rotation $None$, $None$,
$None$, and $None$ to obtain a total of 80 views (H. Su ., 2015; Ma ., 2018).

  * •

Random configuration: this setup is flexibly extended from the previous
spherical configuration where 12 irregular views were randomly selected by
perturbed in coordinates from the previous spherical configuration as shown in
Fig. 17.c (S. Qi ., 2021; Wei ., 2020).

  * •

Circular with elevation configuration: this setup is extended from the
previous circular configuration. Where several elevation levels of viewpoints
exist, as shown in Fig. 17.d. This configuration is also based on the upright
orientation assumption as the first configuration. However, the elevation
angle $\phi$ is not fixed and it is in [$None$, $None$] where it ends up with
a total of $None$/ $\Theta$ viewpoints (Kanezaki ., 2018; Parisotto ., 2023).

  * •

Cylinder (panoramic) configuration: a cylinder setup is a configuration whose
axis is aligned to one of the principal axes of coordinate space (e.g., the
z-axis). A panoramic view is then obtained by projecting the 3D model to the
lateral surface of a cylinder of radius R and height H equal to 2R. The
cylinder is centered at the origin as in Fig. 17.e (Sfikas ., 2018, 2017;
Papadakis ., 2010).

Figure 17: The five mostly experimented camera configurations: (a) Circular,
(b) Spherical (dodecahedral), (c) Random, (d) Circular with elevations, and
(e) Clinder (panoramic).

Tables 5 and 6 summarize the experimented camera configurations and the
experimented numbers of views, respectively, since they are essential factors
that affect the performance of the recognition model (Han ., 2018). From Table
5, it could be noticed that circular and spherical are the most common
configurations that were experimented with by 25 and 14 related works,
respectively, out of 37 works. Even more, from Table 6 also, it is obvious
that out of these works, 30 works have experimented with 12 views, 18 works
have experimented with six views, 13 works have experimented with three views,
and ten works have experimented 20 views. Decreasing the number of views, such
as 1 or 2, will decrease the classification performance because much of the
information will be lost, which is reasonable. However, when the number of
views increases, such as 4, 6, 8, or more, the classification performance
increases fast and becomes much more stable (T. Yu ., 2018; Kanezaki ., 2018).
Even more, using 20 views, the state-of-the-art performance was achieved by
RotationNet (Kanezaki ., 2018), OVPT (W. Wang, Chen ., 2022), MVT (Chen .,
2021), view-GCN (Wei ., 2020), and ViewFormer (H. Sun ., 2023). Otherwise,
worse classification performance is achieved when the model randomly selects
the views. If these selected views are randomly captured from a close or
identical direction, this will significantly reduce the performance because it
will have the same effect as using a few views (Feng ., 2018).

Table 4: 3D datasets that have been experimented with the view-based 3D object
recognition methods

No. 3D Dataset: ModelNet40 ModelNet10 ScanObjectNN ShapeNet Core55 RGB-D MIRO
1 MVCNN ✓ 2 Pairwise ✓ ✓ 3 VolumetricMVCNN ✓ 4 GIFT ✓ ✓ 5 PANORAMA-NN ✓ ✓ 6
PANORAMA-ENN ✓ 7 MHBN ✓ ✓ 8 GVCNN ✓ 9 MVCNN(GoogLeNet) ✓ 10 SeqViews2SeqLabels
✓ ✓ 11 RotationNet ✓ ✓ ✓ ✓ 12 Ma et al. ✓ ✓ 13 VERAM ✓ ✓ 14 MVCNN-new ✓ 15
iMHL ✓ 16 MVSG-DNN ✓ ✓ 17 RN ✓ ✓ 18 MLVCNN ✓ 19 G-CNN ✓ ✓ 20 DeepCCFV ✓ 21
3D2SeqViews ✓ ✓ ✓ 22 DomSetClust ✓ 23 HGNN ✓ 24 DRCNN ✓ ✓ 25 View-GCN ✓ ✓ ✓ 26
HMVCM ✓ ✓ 27 MVTN ✓ ✓ 28 MVT ✓ ✓ 29 OVPT ✓ ✓ 30 MVDAN ✓ 31 MVMSAN ✓ ✓ 32 Yu
and Cao ✓ ✓ 33 VCGR-Net ✓ ✓ 34 ViewFormer ✓ ✓ ✓ ✓ 35 MVContrast ✓ ✓ 36 MORE ✓
✓ 37 MVCVT ✓ ✓

Table 5: Different camera configurations (see Fig. 17) that have been used on
the view-based 3D object recognition methods

No. Multi-view Configurations: (A) Circular (B) Dodecahedral/ Spherical (C)
Random (D) Circular with Elevation (E) Cylinder/ Panoramic 1 MVCNN ✓ 2
Pairwise ✓ 3 VolumetricMVCNN ✓ ✓ 4 GIFT ✓ 5 PANORAMA-NN ✓ 6 PANORAMA-ENN ✓ 7
MHBN ✓ 8 GVCNN ✓ 9 MVCNN(GoogLeNet) ✓ 10 SeqViews2SeqLabels ✓ 11 RotationNet ✓
✓ ✓ 12 Ma et al. ✓ ✓ ✓ 13 VERAM ✓ 14 MVCNN-new ✓ 15 iMHL ✓ 16 MVSG-DNN ✓ ✓ 17
RN ✓ 18 MLVCNN ✓ 19 G-CNN ✓ 20 DeepCCFV ✓ 21 3D2SeqViews ✓ 22 DomSetClust ✓ 23
HGNN ✓ 24 View-GCN ✓ ✓ ✓ 25 HMVCM ✓ 26 MVTN ✓ ✓ 27 MVT ✓ ✓ 28 OVPT ✓ 29 MVDAN
✓ 30 MVMSAN ✓ 31 Yu and Cao ✓ ✓ 32 VCGR-Net ✓ 33 ViewFormer ✓ 34 MVContrast ✓
35 MORE ✓ 36 MVCVT ✓

Table 6: Different numbers of the extracted views that have been experimented
with the view-based 3D object recognition methods

No. Number of Views: 1 2 3 4 6 8 9 10 12 16 20 24 60 64 80 1 MVCNN ✓ ✓ 2
Pairwise ✓ ✓ ✓ 3 VolumetricMVCNN ✓ 4 GIFT ✓ 5 PANORAMA-NN ✓ 6 PANORAMA-ENN ✓ 7
MHBN ✓ ✓ ✓ 8 GVCNN ✓ ✓ 9 MVCNN(GoogLeNet) ✓ ✓ 10 SeqViews2SeqLabels ✓ ✓ ✓ ✓ 11
RotationNet ✓ ✓ 12 Ma et al ✓ ✓ ✓ 13 VERAM ✓ ✓ ✓ 14 MVCNN-new ✓ 15 iMHL ✓ 16
MVSG-DNN ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ 17 RN ✓ ✓ ✓ 18 MLVCNN ✓ ✓ ✓ ✓ ✓ 19 G-CNN ✓ ✓ ✓ 20
DeepCCFV ✓ ✓ ✓ ✓ ✓ 21 3D2SeqViews ✓ ✓ ✓ ✓ 22 DomSetClust ✓ ✓ ✓ ✓ 23 HGNN ✓ 24
DRCNN ✓ ✓ ✓ 25 View-GCN ✓ ✓ 26 HMVCM ✓ 27 MVTN ✓ ✓ ✓ 28 MVT ✓ ✓ ✓ ✓ ✓ 29 OVPT
✓ ✓ ✓ ✓ ✓ 30 MVDAN ✓ ✓ ✓ 31 MVMSAN ✓ ✓ ✓ ✓ 32 Yu and Cao ✓ ✓ 33 VCGR-Net ✓ ✓ ✓
34 ViewFormer ✓ ✓ ✓ ✓ ✓ ✓ 35 MVContrast ✓ ✓ ✓ 36 MORE ✓ 37 MVCVT ✓

####  3.1.3 Views Selection Strategies

A key challenge for the 3D object recognition methods is to use the features
extracted by the deep network and implement a feature fusion to produce a
differentiated global descriptor for the object that will be used later to
gain accurate classification. Feature fusion is the process of combining
multiple views’ features into a discriminated global descriptor. The original
3D object recognition methods utilized all views captured from all viewpoints
for feature fusion (S. Qi ., 2021). However, because only some views are
practical for recognition, some views are more discriminative, and processing
all views needs heavy computation and causes overhead. Therefore, a more
robust view selection mechanism is needed (S. Qi ., 2021; Feng ., 2018; Gezawa
., 2020; W. Wang, Chen ., 2022). Therefore, some 3D object recognition methods
presented view selection techniques to remove irrelevant and redundant views.
If a view or its feature is poorly related to the class information, it is
considered irrelevant. If two views or their features have an almost similar
distribution, one is redundant (Mangai ., 2010). The selected views are then
regarded as qualified samples for training to gain a better feature set,
achieving better performance. According to (S. Qi ., 2021), the different
selection strategies of existing feature fusion methods can be divided into
Passive Views Selection (PVS), Active Views Selection (AVS), and Active Camera
Viewpoints Selection (ACVS) strategies.

The PVS strategy uses feature selection techniques after involving all the
captured views. So, the views are captured after setting all viewpoints
earlier and then feeding all these views into the network. However, the PVS
strategy is practical and straightforward, but it is computationally expensive
and only accomplishes the best performance if it uses all views. Some
techniques of this strategy are the max-pooling, the view-pooling, and the
maxout layers (S. Qi ., 2021). MVCNN (H. Su ., 2015) and GVCNN (Feng ., 2018)
were using this type of selection. (H. Su ., 2015) used the first CNN of the
MVCNN model to generate the features of each captured view and then aggregated
all the extracted features (descriptors) in the view-pooling layer to merge
them into one general 3D shape descriptor that the second CNN of the network
uses for 3D object recognition and sketch recognition tasks. Later, (JC. Su .,
2018) enhanced the object centering and switched to a black rendering
background, which helped improve MVCNN performance. However, the same
selection strategy on feature fusion was used and not improved (S. Qi .,
2021).

Similarly, (Feng ., 2018) employed the same selection strategy in the view-
pooling layer of the GVCNN model but in two steps. First, with GVCNN pooling,
all the views within the same group are pooled to merge into a group-level
descriptor. Then, these group-level descriptions are fused later to produce
the final shape-level 3D object recognition. Yet, the PVS strategies depend on
pooling, which only selects the average or maximum descriptor of all views.
Moreover, according to (S. Qi ., 2021), it has a permutation invariance, which
means it pays no attention to approximately all views’ content and spatial
information between them.

The ACVS strategy does not use all views collected from all viewpoints because
not every view is necessary for the network. So, it uses only a subset of
views captured from specific viewpoints to feed them into the network. This
strategy uses as few views as possible to reduce the computational cost and
tackle the occlusion problem, which is significant for real-world scenarios
and applications (S. Qi ., 2021). RotationNet (Kanezaki ., 2018) and MVTN
(Hamdi ., 2021) are differentiable and use this selection strategy. The idea
of the RotationNet method (Kanezaki ., 2018) is that it will find the best
pose (viewpoint variables) for each category label through unsupervised
learning; hence, viewpoint variables will be dealt with as latent variables
that are optimized in the training phase and then predicted with the category
on the testing phase. On the other hand, MVTN (Hamdi ., 2021) tackles the
drawback of the previously proposed methods that rely on a fixed set of
captured views using fixed camera viewpoints for all 3D objects in the
datasets. Rather than the previous approaches where their scene parameters are
pre-defined and selected for the entire 3D dataset using the most common
camera viewpoints configurations which are circular and spherical around the
object, MVTN has been proposed as an adaptive approach to learning viewpoints
per object.

On the other hand, the AVS strategy also does not consider every view for
recognition, but even so, it needs to feed all views captured from all
viewpoints into the network. Then, it automatically selects the best views
using the network to improve performance and prevent the computational cost.
AVS is not only suffering from the same drawbacks but also for solving
practical problems (e.g., occlusion); it is complex (S. Qi ., 2021). (Wei .,
2020) proposed view-GCN for 3D object recognition based on the view selection
idea. View-GCN implemented the widely used Farthest Point Sampling (FPS) in
GCN (C.R. Qi, Yi ., 2017) and then utilize a selective view-sampling strategy
to sample representative views by view-selectors to build the next level view-
graph. View-selectors are components, each learning to select the
discriminative view from a set of local neighboring views. All the learned
features in different levels are combined by performing max-pooling on the
updated node features to be a pooled descriptor, and then the concatenation of
all the pooled features at all levels is the final global shape descriptor.
Despite the excellent performance of view-GCN, it still needs to feed the
network with all views captured from all viewpoints. Also, OVPT (W. Wang, Chen
., 2022) used this type of technique with the help of information entropy to
evaluate view quality. Again, based on high information entropy values, the
best-view prediction model by MORE (Parisotto ., 2023) trained to provide a
set of best-selected views. Moreover, VCGR-Net (Xu ., 2023) applied this type
of selection through two stages. In the first stage, the model highlighted the
discriminative region (features) using a spatial attention mechanism to
decrease the redundant information between views. In the second stage, it
removed the untrustworthy features using non-local neural networks by
calculating the connections between all possible features across and within
views.

####  3.1.4 Pre-trained CNN Architectures

CNN-based models have been utilize in computer vision and image processing
problems. Hence, there are multiple architectures and variants of the basic
CNN (Bhatt ., 2021). Consequently, the multi-view 3D object recognition
methods take advantage of the CNN architectures pre-trained by the excellent
publicly available large-scale 2D datasets such as ImageNet (Russakovsky .,
2015) and directly use them as a backbone network. So, the backbone network is
a CNN architecture that has performed well in 2D classification tasks. The two
fundamental roles of this pre-trained CNN backbone in the 3D recognition task
are to extract the features from 2D rendered views at the beginning of the
recognition model and to use the fused global shape descriptor for
classification at the end of the recognition model. Using these CNNs as the
backbone networks will save training time, reduce the complicated 3D
recognition task to a simplified 2D recognition task, and improve the
classification accuracy (S. Qi ., 2021; Voulodimos ., 2018; L. Sun ., 2019).
The most used CNN architectures as backbone networks on the multi-view 3D
object recognition methods (H. Su ., 2015; Wei ., 2020; JC. Su ., 2018; Feng
., 2018; Kanezaki ., 2018) are summarized in Table 7. It has been noticed from
these related works that AlexNet (Krizhevsky ., 2012), VGG (Simonyan
Zisserman, 2014), GoogLeNet (Szegedy ., 2015), and ResNet (He ., 2016)
architectures and their variants are the most popular networks that are
employed as the backbone in this field. Therefore, Table 8 summarizes these
CNN variants in terms of their released year, main role or importance, number
of parameters, and their advantages and disadvantages as listed by (Bhatt .,
2021).

Table 7: Different CNN architectures that have been (✗: chosen) or (✓: just
experimented) as backbones on the related works

No. CNN architecture: Noval CNN VGG-M VGG-C,D VGG-A VGG-E GoogLe-Net AlexNet
ResNet-18 ResNet-34 ResNet-50 ResNet-152 1 MVCNN ✓ ✗ 2 Pairwise ✗ 3
VolumetricMVCNN ✗ 4 GIFT ✗ 5 PANORAMA-NN ✗ 6 PANORAMA-ENN ✗ 7 MHBN ✗ 8 GVCNN ✗
9 MVCNN(GoogLeNet) ✗ 10 SeqViews2SeqLabels ✗ 11 RotationNet ✓ ✗ ✓ 12 Ma et al.
✗ 13 VERAM ✗ ✓ 14 MVCNN-new ✓ ✗ ✓ ✓ ✓ 15 MVSG-DNN ✗ 16 RN ✗ 17 MLVCNN ✗ 18
G-CNN ✗ 19 DeepCCFV ✓ ✗ 20 3D2SeqViews ✓ ✗ ✓ 21 DomSetClust ✗ 22 DRCNN ✓ ✓ ✓ ✓
✗ 23 View-GCN ✓ ✗ ✓ 24 HMVCM ✓ ✗ ✓ 25 MVTN ✗ ✓ ✓ 26 MVT ✗ 27 OVPT ✗ 28 MVDAN ✓
✓ ✗ 29 MVMSAN ✓ ✓ 30 VCGR-Net ✗ 31 ViewFormer ✗ ✓ ✓ 32 MVContrast ✓ ✗ ✓ 33
MORE ✓ 34 MVCVT ✓ ✓ ✗ ✓

Table 8: Summary of various CNN architectures with their advantages and
disadvantages (ER is the error rate)

CNN Architecture Dataset Importance Parameters Advantages Disadvantages
AlexNet
(Krizhevsky ., 2012) ImageNet ILSVRC 2012

(ER= 16.4) \- Wider and deeper compared to standard CNN

\- Used dropout, RELU activation function, and overlap pooling 60 M \- Can
extract high, mid, and low-level features using small and large size filters
on the early and final layers

-It introduced CNN regularization -It was first to use parallel GPUs as an accelerator to handle the complex architectures \- It contains dormant neurons in the first two layers

\- Extreme filter size which leads to undersampling of the trained feature
maps VGG
(Simonyan Zisserman, 2014) ImageNet ILSVRC 2012

(ER= 7.3) \- It had a homogeneous topology

\- It used kernels with a small size 138 M \- It introduced the effective
receptive field concept

\- It introduced the homogeneous and simple topology concept

\- It is available in different configurations based on its deep layers: VGG-M
(8 layers), VGG-A (11 layers), VGG-B (13 layers), VGG-C (16 layers), VGG-D (16
layers), VGG-E (19 layers) \- It uses fully linked layers which are
computationally expensive GoogLeNet
(Szegedy ., 2015) ImageNet ILSVRC 2014

(ER= 6.7) \- It was the first to propose the block concept

\- It used the idea of split transform and before merge 4 M \- It introduced
the multi-scale filters utilizing concepts within the layers

\- It introduced the divide, convert, and merge concept

\- It enhances convergence rate using auxiliary classifiers \- It uses
heterogeneous topology which makes parameter tuning difficult

\- Valuable information may be lost as a result of a representational
bottleneck ResNet
(He ., 2016) ImageNet ILSVRC 2015

(ER= 3.6) Identified residual learning with mapping-based skip connections 1.7
M to 25.6 M \- It reduced the error rate for deeper networks

\- It introduced the residual learning concept

\- It alleviated the vanishing gradient problem effect \- It is complicated

\- In feed forwarding, it reduced the feature-map information

\- Because of module stacking, it over-adapts the hyper-parameters for a
particular job

####  3.1.5 Fusion Strategies

The most vital information for any successful recognition task is the
features. However, if the features are confusing, overlapping, noisy, or
distorted, they will constrain any classifier’s performance. Hence, selecting
a feature set is vital for any successful single or multiple classifier(s)
(Mangai ., 2010). In the view-based methods, when the CNNs extract the view-
level features, feature fusion is required to create a global shape descriptor
to provide an accurate object recognition (S. Qi ., 2021). Feature fusion
aggregates multiple view-level features for information synthesized from all
views into a single shape descriptor (H. Su ., 2015; Q. Zhang Chan, 2022). An
early study in 2021 conducted by Seeland and Mäder (Seeland Mäder, 2021)
investigated the three fusion strategies utilize in DL-based multi-view
recognition models. These strategies are early, late, and score fusions as
shown in Fig. 18.

Figure 18: The three fusion strategies that have been used in DL-based multi-
view classification which are a) Early, b) Late, and c) Score fusions.

In the early fusion strategy, the convolutional feature maps are stacked after
they are produced from the multiple CNNs and then subsequently processed
together. In this strategy, the depth of the stacked feature maps needs to be
reduced to a single-view feature map. There are two different approaches for
depth reduction: i) early fusion (max), where a max-pooling operation is
performed on the stacked feature maps, and ii) early fusion (conv), where a
1$\times$1 convolution operation is performed instead of the max-pooling
operation (Seeland Mäder, 2021). On the other hand, the late fusion strategy
depends on gathering the output of the last layer before the layer that does
classification. There are two approaches to applying the late fusion strategy,
which are: i) late fusion (max), where max-pooling operation is performed to
the stacked feature vectors, and ii) late fusion (fc), where concatenation and
fully connected fusion operation are performed (Seeland Mäder, 2021). In
contrast, the score fusion strategy is operated by element-wise aggregation of
the softmax classification scores per image view. Where one of the following
aggregation functions is performed: i) sum-score fusion where the scores of
the views are summation, ii) product-score fusion where the scores of the
views are multiplication, and iii) max-score fusion where the scores are
maximum across views (Seeland Mäder, 2021).

####  3.1.6 Evaluation Metrics

Typically, the evaluation of 3D object recognition methods encompasses two
distinct categories of tasks, namely 3D classification and retrieval tasks.

  * •

3D object classification: test the proposed model on a 3D object
classification task which involves classifying a given object by identifying
its category (a.k.a. class label) as shown in Fig. 2.a. Two metrics have been
used as criteria for object classification accuracy (S. Qi ., 2021; Gezawa .,
2020) to evaluate the classification performance of any 3D object recognition
model:

    * –

Overall/Instance accuracy (OA): also called average instance accuracy or
instance-level accuracy. Which is the objects/samples that classified
correctly to the total number of testing objects/samples (S. Qi ., 2021; Hamdi
., 2021; T. Yu ., 2018; Chen ., 2018). This accuracy is intuitive since it
reflects the number of the wrongly classified objects/samples (S. Qi ., 2021;
Gezawa ., 2020). Instance accuracy can be calculated using Equation 1.

$InstanceAccuracy=\frac{\sum_{i=1}^{C}TP_{i}+TN_{i}}{\sum_{i=1}^{C}P_{i}+N_{i}}$
(1)

    * –

Average/Class Accuracy (AA): a.k.a. average per-class accuracy or class-level
accuracy. Which is the mean or average of all the correctly classified class
accuracy (S. Qi ., 2021; Hamdi ., 2021; T. Yu ., 2018; Chen ., 2018). In other
words, it is the mean of the instance accuracy among all classes. This
accuracy is objective since the number of the tested objects from different
categories is unequal (S. Qi ., 2021). Class accuracy can be calculated using
Equation 2 (S. Qi ., 2021).

$ClassAccuracy=\frac{1}{C}\sum_{i=1}^{C}\frac{TP_{i}+TN_{i}}{P_{i}+N_{i}}$ (2)

Where: C is the total number of experimented classes/categories P and N are
the numbers of positive and negative experimented samples, respectively. TP
and TN are the numbers of true positive and true negative experimented
samples, respectively. i is the corresponding category.

  * •

3D object retrieval: test the proposed model on a 3D object retrieval task,
which involves finding all the closest 3D objects in the dataset that match a
given querying 3D object (having the same category as in Fig. 2.b). To
evaluate the retrieval performance of any 3D object retrieval model, the mean
average precision has been utilize as criteria for object retrieval accuracy
(S. Qi ., 2021; Gezawa ., 2020):

    * –

Mean Average Precision (mAP): which is the average value of Average Precision
(AP) in each category (k) and can be calculated using Equation 3. AP is
defined in Equation 4 as the area under the PR curve (P as in Equation 5 is
the precision value, and $\Delta$r(k) is the recall value change from k-1 to
k) (S. Qi ., 2021; Hamdi ., 2021).

$mAP=\frac{\sum_{k=1}^{C}AP(k)}{C}$ (3)

$AP=\sum_{k=1}^{N}Precision(k)\Delta r(k)$ (4)

$Precision=\frac{TP}{TP+FN}$ (5)

####  3.1.7 Recognition Performance

Several deep networks have been investigated for 3D object classification and
retrieval. Among these networks, the view-based networks have performed best
so far to achieve the current state-of-the-art performance (Feng ., 2018).
Table 9 summarizes the classification and retrieval performance of the view-
based networks on the well known ModelNet40 and ModelNet10 datasets.

From the classification performance of the view-based models, it can be seen
that the best ModelNet40 results obtained by RotationNet (Kanezaki ., 2018),
OVPT (W. Wang, Chen ., 2022), MVT (Chen ., 2021), View-GCN (Wei ., 2020), and
ViewFormer (H. Sun ., 2023) with 97.37%, 97.48%, 97.50%, 97.60%, and 98.8% OA,
respectively, where all of them were using 20 views with spherical
configuration and a selection mechanism except ViewFormer (H. Sun ., 2023)
that employed random configuration. Note that OVPT (W. Wang, Chen ., 2022)
starts with 20 views and then is reduced to 6. On the other hand, from the
ModelNet10 results, it can be noticed that the excellent classification
performance obtained by ViewFormer (H. Sun ., 2023), MVT (Chen ., 2021), DRCNN
(K. Sun ., 2020), and OVPT (W. Wang, Chen ., 2022), with 99.30%, 99.30%,
99.34%, and 99.57% OA, respectively. However, they were using different
numbers of views.

Further, it is evident from the retrieval performance analysis that DRCNN (K.
Sun ., 2020), G-CNN (Esteves ., 2019) and (L. Yu Cao, 2023) achieved the
highest results on both datasets, with 93.90%, 93.56%, and 95.98% mAP on
ModelNet40, and with 96.15%, 96.18%, and 98.23% mAP on ModelNet10,
respectively. The DRCNN (K. Sun ., 2020) and (L. Yu Cao, 2023) utilized a
spherical configuration with 20 views, whereas the G-CNN (Esteves ., 2019)
employed a circular configuration with 12 views.

It is worth mentioning that increasing the number of training samples will
improve the recognition performance, as proven by (Parisotto ., 2023) where
they trained their MORE model with a random fraction (full, third, and
twentieth) of the ModelNet10 and ModelNet40 datasets separately to test
whether subsets of it would achieve similar performance while minimizing the
required training time. Finally, they show that increasing the training data
size will lead to an improvement in performance.

Even more, two different training configurations affect the recognition
performance, which is using i) the pre-trained backbone network on ImageNet
images (Hendrycks Dietterich, 2019) as it is, or ii) fine-tuned the pre-
trained backbone network on ModelNet dataset (Wu ., 20151). The two
configurations have been experimented with by MVCNN (H. Su ., 2015) and GVCNN
(Feng ., 2018), where both of them proven that fine-tuning significantly
improves the recognition performance by a margin of $+6.2\%$. Hence, the later
proposed methods, such as view-GCN (Wei ., 2020) and RotationNet (Kanezaki .,
2018), fine-tuned the weights of the backbone network. However, some work,
such as MVTN (Hamdi ., 2021) and VERAM (Chen ., 2018), refrain from fine-
tuning the weights of the backbone network to avoid the slow training progress
due to the forward and backward propagation needed in each step.

Table 9: Performance summary (OA: Overall Accuracy, and AA: Average Accuracy)
of different view-based 3D object recognition methods achieved on the
ModelNet40 (Wu ., 20151) and ModelNet10 (Wu ., 20151) datasets

No. Method Name Year # of Views ModelNet40 ModelNet10 AA OA mAP AA OA mAP 1
MVCNN 2015 12 89.50% - 80.20% - - - 2 Pairwise 2016 12 - 90.70% - - 94% - 3
VolumetricMVCNN 2016 20 89.70% 92% - - - - 4 GIFT 2016 64 89.50% - 81.94%
91.50% - 91.12% 5 PANORAMA-NN 2017 6 - 90.70% - - 91.12% - 6 PANORAMA-ENN 2018
1 - 95.56% 86.34% - 96.85% 93.28% 7 MHBN 2018 6 93.10% 94.70% - 95% 95% - 8
GVCNN 2018 8 90.70% 93.10% 84.50% - - - 9 MVCNN(GoogLeNet) 2018 12 - 92.20%
83% - - - 10 SeqViews2SeqLabels 2018 12 91.10% 93.30% 89% 94.80% 94.82% 91.43%
11 RotationNet 2018 20 96.29% 97.37% - - 98.90% - 12 Ma et al. 2018 12 -
91.05% - - 95.30% - 13 VERAM 2018 9 92.10% 93.70% - 95.30% 95.50% - 14 MVCNN-
new 2018 6 92.40% 95% - - - - 15 iMHL 2018 12 - 97.20% - - - - 16 MVSG-DNN
2019 12 92.30% - 83.70% 94% - 93.90% 17 RN 2019 12 92.30% 94.30% 86.70% 95.10%
95.30% - 18 MLVCNN 2019 12 94.16% 92.84% 92.84% - - - 19 G-CNN 2019 20 92.60%
94.67% 93.56% - 96.78% 96.18% 20 DeepCCFV 2019 12 - 92.49% - - - - 21
3D2SeqViews 2019 12 91.50% 93.40% - - - 92.12% 22 DomSetClust 2019 12 92.80%
93.80% - - - - 23 HGNN 2019 12 - 96.70% - - - - 24 DRCNN 2020 12 94.86% 96.84%
93.90% 99.30% 99.34% 96.15% 25 View-GCN 2020 20 96.50% 97.60% - - - - 26 HMVCM
2021 12 94.57% 92.80% 92.80% 95.7% - 93.90% 27 MVTN 2021 20 92.20% 93.50%
92.90% - - - 28 MVT 2021 20 - 97.50% - - 99.30% - 29 OVPT 2022 6 96.74% 97.48%
- 99.21% 99.33% - 30 MVDAN 2022 12 95.50% 96.60% - - - - 31 MVMSAN 2022 20
95.68% 96.96% - 98.42% 98.57% - 32 Yu and Cao 2023 20 - 97.13% 95.98% - 99%
98.23% 33 VCGR-Net 2023 12 93.33% 95.62% - 97.24% 97.79% - 34 ViewFormer 2023
20 98.90% 98.80% - 100% 99.30% - 35 MVContrast 2023 12 90.24% 92.54% 90.30% -
- - 36 MORE 2023 1 - 97.16% - - 98.26% - 37 MVCVT 2023 12 - 95.40% - - 98.68%
-

##  4 Transformer-based Multi-View 3D Object Recognition

Transformer is an attention-based block that simulates a human visual system
by not processing an entire scene; instead, it focuses on salient parts
selectively in order to improve the capturing of visual structure and better
represent the interests (Woo ., 2018). Transformers have proved recently their
excellent performance on a broad range of Natural Language Processing (NLP)
applications such as machine translation, text classification, and question
answering. This attracts the interest to adapt these models in the computer
vision field, especially for multi-learning tasks (T. Lin ., 2022). One of
these multi-learning tasks is the multi-view 3D object recognition that
benefits from different transformer-based models such as MVT (Chen ., 2021),
MVDAN (W. Wang, Cai Wang, 2022), OVPT (W. Wang, Chen ., 2022), MVMSAN (W.
Wang, Wang ., 2022), MVCVT (J. Li ., 2023), and (L. Yu Cao, 2023).

Multi-view DL-based models that used view-based pooling such as MVCNN (H. Su
., 2015) and GVCNN (Feng ., 2018), or patch-based pooling such as View-GCN
(Wei ., 2020) did not have visual features interactions between multiple views
in previous layers but only apply visual features fusion from multiple views
in the last pooling layer. This configuration limits its effectiveness and
fails to distinguish patches from different views. Hence, this problem
inspired (Chen ., 2021) to propose a Multi-view Vision Transformer (MVT) for
3D object classification. MVT involves two parts. The first part splits each
view individually into non-overlap patches, and for patches in each, it
generates the low-level features view using a local transformer encoder. The
second part concatenates the low-level patch features from all views as a
global set and then feeds the concatenated set into the global transformer
encoder to communicate the visual features in patches from multiple views.
After that, the output patch features are sum-pooled into a global descriptor
that feeds the MLP head for classification. MVT has experimented with the two
camera settings, 12-view and 20-view, on the two 3D datasets: ModelNet40 and
ModelNet10. MVT outperformed other transformer-based approaches such as OVPT
(W. Wang, Chen ., 2022), MVDAN (W. Wang, Cai Wang, 2022), and MVMSAN (W. Wang,
Wang ., 2022) with 97.5% OA On ModelNet40 and with 99.3% OA On ModelNet10
using 20-view camera setting.

Some of the existing multi-view 3D object recognition methods ignore the
views’ distinguishability and the hierarchical correlation between them.
Hence, (W. Wang, Cai Wang, 2022) proposed a Multi-View Dual Attention Network
(MVDAN) for the task of 3D object classification. MVDAN contains two basic
blocks: View Channel Attention Block (VCAB) and View Space Attention Block
(VSAB). VCAB focuses on the critical views in the same category by finding the
relationship between multiple views and highlighting important perspectives.
In comparison, VSAB focuses on the more discriminatory details between the
views of different categories. MVDAN was evaluated with a circular camera
configuration of 12 views on ModelNet40 objects and Resnet50 as the backbone
network. The MVDAN method achieves 96.6% OA and 95.5% AA while reaching 96.1%
and 96.3% AA when experimented with 3 and 6 views, respectively.

Another work of multi-view 3D object classification that tackles the
correlation between views is the Optimal Viewset Pooling Transformer (OVPT)
proposed by (W. Wang, Chen ., 2022). OVPT aims to improve the recognition
performance by reducing the views’ redundancy. OVPT has three modules; the
first module captured the 20 spherical views from each 3D object and utilized
information entropy as a selection mechanism to reduce the redundant views and
gain the optimal view set. Then, the second module inputs the optimal views to
pre-trained resnet34 as the backbone network for feature extraction that is
later flattened into a local view token sequence so it can be input to the
transformer. Finally, the last module is the pooling transformer that
generates the global descriptors used for classification. OVPT has been
evaluated on both ModelNet10 and ModelNet40. As a result, it achieves 97.48%
OA and 96.74% AA on ModelNet40 with only six best views, and it achieves
99.33% OA and 99.21% AA on ModelNet10 with nine best views. Compared with
other DL-based methods such as iMHL (Z. Zhang ., 2018) and RotationNet
(Kanezaki ., 2018) and other transformer-based methods such as MVDAN (W. Wang,
Cai Wang, 2022) and MVMSAN (W. Wang, Wang ., 2022), OVPT achieves state-of-
the-art performance in the multi-view 3D object classification accuracy with
less redundant views and less computational resources.

In order to improve the classification accuracy by solving the problems of
poor generalization ability and insufficient extracted features of the
existing multi-view 3D network model, (W. Wang, Wang ., 2022) have proposed a
Multi-View SoftPool Attention convolutional Network framework (MVMSAN) for the
3D classification task. The MVMSAN model has three parts; the first part uses
ResNest as a backbone network and adaptive average pooling module to extract
multi-view features from 20 spherical views captured from each object. The
second part used the extracted multi-view feature as the Query for their
proposed SoftPool attention convolution method to extract refined view feature
information during downsampling and calculate attention scores by Query and
Key that feed to the last part of the model. Finally, the last part fused
refined view features and generated a 3D global descriptor for classification.
MVMSAN was evaluated on both ModelNet40 and ModelNet10. MVMSAN outperformed
some deep-based multi-view 3D object recognition methods such as MVCNN (H. Su
., 2015), MHBN (T. Yu ., 2018), VERAM (Chen ., 2018), and RN (Z. Yang Wang,
2019). Chosen ResNest14d as their backbone network, they achieved on
ModelNet40 96.96% OA and 95.68% AA, and on ModelNet10 98.57% OA and 98.42% AA.

More recently, another transformer-based works (L. Yu Cao, 2023; H. Sun .,
2023; J. Li ., 2023) were proposed to handle the problem of ignoring the
correlation and connection between the multi-views that suffered by most
traditional deep-based multi-view 3D object recognition methods such as (H. Su
., 2015; Feng ., 2018). (L. Yu Cao, 2023) proposed a view self-attention
network model for 3D object classification and retrieval. The self-attention
approach solves the problem of input order in the sequence views by
calculating the correlation between them and focusing only on the important
views to extract the global information of the object better. To apply this
approach, Yu and Cao’s model consists of two units: i) the single-view self-
attention network, which discovers the connection between features of a single
view, and ii) the multi-view self-attention network, which discovers the
relationship between the different views to reduce information redundancy by
finding the most critical views. They validate their model with the ModelNet40
and ModelNet10 datasets with 12-view and 20-view camera settings. On
ModelNet40, their method achieved 95.30% OA and 92.26% mAP with 12-views and
97.13% OA and 95.98% mAP with 20-views. While on ModelNet10, their method
achieved 97.23% OA and 95.92% mAP with 12-views and 99% OA and 98.23% mAP with
20-views. In the same year, ViewFormer is proposed by (H. Sun ., 2023). to
handle the limitation of different ways when multi-view are aggregated, such
as when the aggregated views are insufficient (e. g., MVCNN (H. Su ., 2015)),
collected from sequence around the 3D object which is invalid assumption (e.
g., SeqViews2SeqLabels (Han ., 2018)), or when introducing additional
computational overheads using graph construction e.g., view-GCN (Wei .,
2020)). ViewFormer, a view set attention model for multi-view 3D object
classification and retrieval, presented a View Set perspective where several
views of a 3D object are organized, which makes the elements’ permutation
invariant. ViewFormer has four units: i) Initializer is AlexNet to initialize
the views representations, ii) Encoder is modified from the standard
Transformer encoder, iii) Transition summarizes the learned higher-order
correlations to View-Former’s understanding and expresses the 3D object. iv)
The decoder is designed for classification and retrieval tasks. The ViewFormer
is evaluated on the three classification datasets: ModelNet10, ModelNet40, and
RGB-D, and the ShapeNet Core55 as the retrieval dataset. ViewFormer used
random viewpoints of the views with random numbers; however, 20 views reached
the optimal performance. Compared to attention-based methods such as VERAM
(Chen ., 2018), SeqViews2SeqLabels (Han ., 2018), and 3D2SeqViews (Han .,
2019), these methods do not yield acceptable results since the highest
accuracy on ModelNet40 is 93.7% OA, whereas ViewFormer reaches 98.9% AA and
98.8% OA. Additionally, compared to transformer-based methods such as MVT
(Chen ., 2021) on ModelNet10, ViewFormer shows that taking the patch-level
interactions into account is unnecessary to accomplish the 99.30% OA. Later
on, Multi-View Convolutional Vision Transformer (MVCVT) (J. Li ., 2023) was
proposed for 3D object classification to tackle the challenge of forming a
represented global shape descriptor since MVT (Chen ., 2021) uses a
transformer for feature extraction which is computationally expensive and
ignores the specifics of the local features. MVCVT was designed as a hybrid
CNN-transformer model where it used a GoogleNet as a backbone CNN for feature
extraction of different views and then used a transformer for highlighting the
feature relationship between these views. It used a circular camera setting to
capture 12 views around the 3D object. When evaluated on ModelNet10 and
ModelNet40, it achieved 98.68% and 95.4% OA, respectively, which is
competitive to MVT (Chen ., 2021) results.

It is worth mentioning that hybrid deep-transformer models such as MVT (Chen
., 2021), OVPT (W. Wang, Chen ., 2022), MVDAN (W. Wang, Cai Wang, 2022),
MVMSAN (W. Wang, Wang ., 2022), and MVCVT (J. Li ., 2023), that used both
transformer and CNN for 3D object recognition are also applied most of the
steps in the general pipeline in Section 3.1.

##  5 Relevant Computer Vision Applications and Tasks

The multi-view recognition methods can be extended to be generalized to
various computer vision applications that are based on image classification
such as plant species identification (Seeland Mäder, 2021), ant genera
identification (Seeland Mäder, 2021), fingerprint identification (C. Lin
Kumar, 20181), hand-drawn sketch recognition (H. Su ., 2015), car
categorization (Seeland Mäder, 2021), breast cancer classification (L. Sun .,
2019), etc. The general multi-view classification methods used DL can be
classified into two categories based on the computer vision
application/domains it serves. The first category is the methods implemented
on specific computer vision applications (C. Lin Kumar, 20181; L. Sun .,
2019), whereas the second category is those generalized to more than two
different computer vision applications (Seeland Mäder, 2021; H. Su ., 2015).

When human specialists distinguish the 3D fingerprint, they identify the 3D
fingerprint’s global shape and local surface features from different
viewpoints and angles. Inspired by that, multi-view DL approaches have been
investigated for recognizing 3D fingerprints. Such approaches can offer more
accurate 3D fingerprint recognition results. (C. Lin Kumar, 20181) present a
method as the first attempt to investigate the problem of contactless and
partial 3D fingerprint identification since it is more global and accurate
personal identification than contact-based fingerprint recognition. Despite
its benefits, partial 3D fingerprints often result from contactless 3D
capturing, significantly reducing the matching accuracy. Hence, the authors
proposed a fine-tuned multi-view CNN-based network with two channels to learn
discriminative feature representation for solving this challenging problem.
For training, three images containing two side-view and one top-view (as in
Fig. 19 are fed as inputs to the three Siamese networks. This trained multi-
view CNN learns the 3D fingerprint concatenated features containing texture
ridge-valley details, depth, and shape discriminative information. On the
other hand, DL also has been broadly applied to medical screenings such as
mammographic image classification. However, the current methods are mainly
based on single-view mammography, which can’t extract sufficient
discriminative features, resulting in unacceptable classification accuracy.
(L. Sun ., 2019) solved this problem and proposed a novel multi-view
mammography approach using CNN to classify the breast mass as benign or
malignant. Using two mammographic views of each breast in their approach, they
obtained breast mass complementary information that defines its shape,
density, edge morphology, degree of calcification, and location. These two
views, as shown in Fig. 20 are mediolateral oblique (MLO), which is a lateral
view and craniocaudal (CC) which is a top head-to-toe view.

Figure 19: Samples of the multi-view fingerprint database (W. Zhou ., 2014)
with three views (a) Top-view, (b) & (c) Two side-views.
Figure 20: Samples of breast mammographic images from DDSM database (PUB .,
2000). (a) and (c) are the MLO views, (b) and (d) are the CC views of the left
and right breasts, respectively.

The implementation of multi-view classification techniques primarily focuses
on particular applications within computer vision. However, (Seeland Mäder,
2021) generalized their multi-view classification method to other
applications. They present the first systematic investigation of the
effectiveness of three different feature fusion strategies for multi-view
classification in the machine learning area and generalize it to various
computer vision applications. Specifically, these three fusion strategies are
fused early, fused late, and fused with a score. The fusion strategies were
generalized on fine-grained applications such as car categorization, plant
identification, ant genera identification, and plant species identification.
These classification applications were evaluated on four multi-view datasets:
CompCars (L. Yang ., 2015) (see Fig. 21.a) that contain five views (front,
rear, side, front-side, or rear-side), PlantCLEF (Goëau ., 2016) (see Fig.
21.b) that contains two views (flower and leaf), AntWeb (AntWeb, 2021) (see
Fig. 21.c) that contain three views (head, profile view, and dorsal), and data
captured using Flora Incognita (Mäder ., 2021) and Flora Capture (Boho .,
2020) smartphone applications that contain two views (side-view of their
flowers and top-view of their leaves). Moreover, in addition to 3D object
recognition, (H. Su ., 2015) generalize their well-known MVCNN to be used for
hand-drawn sketch classification and sketch-based shape retrieval with data
jittering/augmentation to simulate the effect of views. They used data
jittering to obtain additional samples (multi-view images) from a given 2D
sketch. In their work, each hand-drawn sketch was in-plane rotated with three
angles: $None$, $None$, $None$, and also horizontally reflected to end up with
six samples per 2D sketch image.

Figure 21: Samples of the multi-view datasets: (a) CompCars (L. Yang ., 2015)
with five views, (b) PlantCLEF (Goëau ., 2016) with two views, and (c) AntWeb
(AntWeb, 2021) with three views.

All the above-proposed multi-view DL models (Seeland Mäder, 2021; C. Lin
Kumar, 20181; H. Su ., 2015; L. Sun ., 2019) were capturing the multi-view of
the object from different angles or orientations. On the other hand, (Alam .,
2021) considered the views as the vertical partition of the extracted features
from CNN using the random split technique. This work is regarded as the first
to use feature set partitioning with CNN for different image classification
applications. The Multi-view Convolutional Neural Network (MvCNN) used CNN for
feature extraction from the inputted image. Then, they applied a random split
for the view construction using the extracted features.

Table 10 summarizes multi-view DL models (Seeland Mäder, 2021; C. Lin Kumar,
20181; H. Su ., 2015; L. Sun ., 2019; Alam ., 2021) and the applied datasets
with its computer vision application. The datasets that used for different
multi-view computer vision applications with their references are shown in
Table 12 in Appendix B for lack of space in Table 10 and for easy access. It
has been noticed that the multi-view datasets that were used for evaluation
contained either 3D data, multi-view images, or 2D images with data
augmentation to obtain multi-view images.

Even though the focus of this review is on multi-view 3D object recognition
including 3D classification and 3D retrieval tasks, there have been some other
important and relevant 3D tasks that are out of the scope of this review but
worth exploring such as: multi-view 3D object detection (Y. Li ., 2023; Y. Liu
., 2022) that used mainly by the recent autonomous driving system to detect 3D
objects from different views and perceive the environment, multi-view 3D
semantic scene segmentation (Robert ., 2022) for understanding of visual data
and improving the automated analysis of considerable 3D scenes in indoor
environments, and multi-view 3D reconstruction (D. Wang ., 2021) where 3D
objects are represented from different views is essential task for virtual
reality, 3D modeling, and computer animation.

Table 10: Summary of some computer vision applications that used multi-view DL
models for classification (ACC= Classification Accuracy, and 2D$+$Augment= 2D
with data augmentation)

Refrence Applications Dataset Name Dataset Type Views Classes Samples Metric
ACC -Car categorization CompCars (2015) Multi-view 5 601 8,183 Top-1 96.72%
-Plant
identification PlantCLEF (2016) Multi-view 2 53 1,839 Top-1 94.25% (Seeland
Mäder, 2021) -Ant genera
identification AntWeb (2021) Multi-view 3 82 38,914 Top-1 94.54% -Plant
species identification Flora Incognita (2021) Multi-view 2 775 8,557 Top-1
96.07% (C. Lin Kumar, 20181) -Contactless and partial 3D
fingerprint
identification Contactless 3D fingerprint Database (2008) 3D - 336 2016 Top-1
99.89% Multi-view Fingerprint Database (2014) Multi-view 3 1500 3000 Top-1
85.40% -3D object
recognition ModelNet40 (2015) 3D 80 40 12,311 OA 90.10% (H. Su ., 2015)
-Sketch
classification SketchClean (2014) 2D$+$Augment 6 160 8960 OA 87.20% -Sketch-
based shape retrieval SketchClean (2014) $+$ ModelNet40 (2015) 2D and 3D,
respectively 12 10 193 mAP 36.10% (L. Sun ., 2019) -Breast cancer
classification DDSM (2000) Multi-view 2 2 1,445 AA 82.20% MIAS (1994)
2D$+$Augment 2 2 322 AA 63.06% -Digit recognition MNIST (1998) 10 70,000 MA
96.80% -Gender
classification Men-Women Classification (2019) 2 2893 MA 68.60% -Image
classification CIFAR-10 (2009) 10 60,000 MA 64.90% -Image
classification CIFAR-100 (2009) 100 60,000 MA 45.60% (Alam ., 2021) -Fashion
product classification Fashion-MNIST (2017) 2D with partition of extracted
features 2-6 10 60,000 MA 93.10% -Sign language recognition American Sign
Language (2019) 57 2515 MA 97.60% -Fruit recognition Fruit 360 (2018) 131
67,692 MA 82.00% -Tomato plant
disease
classification Tomato Disease
Classification (2019) 10 18,345 MA 99.90% -Plant disease
classification Plant Disease (2019) 15 20,638 MA 84.90%

##  6 Factors Impacting the Recognition Performance and Future Directions

To provide the readers with more ideas, several facts and future directions
for the development of multi-view 3D object recognition methods are
highlighted as follows:

##### Viewpoints and number of views

For multi-view methods, if the number of views from various perspectives is
reduced, for example, to two, then the recognition performance will decrease
due to the loss of much of the information that makes sense. On the other
hand, if the number of views from various perspectives increases, then the
recognition performance will improve and be much more stable (Cao ., 2020; T.
Yu ., 2018; Kanezaki ., 2018; H. Sun ., 2023; L. Wang ., 2023). Even more,
using 20 views, the state-of-the-art performance was achieved by RotationNet
(Kanezaki ., 2018), view-GCN (Wei ., 2020), OVPT (W. Wang, Chen ., 2022), and
ViewFormer(H. Sun ., 2023). In addition, when the model randomly selects the
views, worse classification performance is achieved. If these selected views
are randomly captured from a close or identical direction, this will
significantly reduce the performance because it will have the same effect as
using a few views (Feng ., 2018). According to (Gezawa ., 2020), methods to
select the relevant number of views are an open research question.

##### Backbone and feature extracting

Including more complex or deeper backbones for feature extraction does not
improve the performance (Hamdi ., 2021). However, performing fine-tuning on
the network, as done by MVCNN (H. Su ., 2015), RotationNet (Kanezaki ., 2018),
and View-GCN (Wei ., 2020), can significantly enhance the recognition
performance. Hence, analyzing and comparing the effect of fine-tuning in
different backbone architectures could be investigated more in the future.

##### Feature fusion and view selection

Most of the previously proposed methods did astonishing work; however, they
fused features from a fixed set (e.g., 12, 20, or 80) of captured views using
fixed camera viewpoints for all 3D objects in the datasets. The scene
parameters are pre-defined and fixed for the entire 3D dataset (Hamdi .,
2021). The most common camera viewpoint configurations are circular and
spherical around the object. Fusing all these views could confuse the
classifier and be misleading for some classes (e.g., looking from the bottom
at a bed). Moreover, in real-world circumference, and due to occlusions,
objects can be observed from limited viewpoints and angles, making it hard to
rely on multi-view images learned with the whole scenarios (Kanezaki ., 2018).
Even more, from the classification performance of the view-based models when
they were evaluated on the well-known ModelNet40 dataset, the best results
were obtained by RotationNet (Kanezaki ., 2018) and View-GCN (Wei ., 2020),
which used a selection mechanism with 97.37% and 97.6% accuracy, respectively.
The selection mechanism is used because not every view is useful for
classification; some views are more discriminative for object classification,
and processing all views needs a heavy computation (S. Qi ., 2021; Feng .,
2018). What types and techniques are better for selection with a focus on view
selection and qualification techniques is still open research to be
investigated.

##### Backbone and feature extracting

Including more complex or deeper backbones for feature extraction does not
improve the performance (Hamdi ., 2021). However, performing fine-tuning on
the network, as done by MVCNN (H. Su ., 2015), RotationNet (Kanezaki ., 2018),
and View-GCN (Wei ., 2020), can significantly enhance the recognition
performance. Hence, analyzing and comparing the effect of fine-tuning in
different backbone architectures could be investigated more in the future.

##### Feature fusion and view selection

Most of the previously proposed methods did astonishing work; however, they
fused features from a fixed set (e.g., 12, 20, or 80) of captured views using
fixed camera viewpoints for all 3D objects in the datasets. The scene
parameters are pre-defined and fixed for the entire 3D dataset (Hamdi .,
2021). The most common camera viewpoint configurations are circular and
spherical around the object. Fusing all these views could confuse the
classifier and be misleading for some classes (e.g., looking from the bottom
at a bed). Moreover, in real-world circumference, and due to occlusions,
objects can be observed from limited viewpoints and angles, making it hard to
rely on multi-view images learned with the whole scenarios (Kanezaki ., 2018).
Even more, from the classification performance of the view-based models when
they were evaluated on the well-known ModelNet40 dataset, the best results
were obtained by RotationNet (Kanezaki ., 2018) and View-GCN (Wei ., 2020),
which used a selection mechanism with 97.37% and 97.6% accuracy, respectively.
The selection mechanism is used because not every view is useful for
classification; some are more discriminative for object classification;
however, processing all of them needs a heavy computation (S. Qi ., 2021; Feng
., 2018). What types and techniques are better for selection with a focus on
views selection and qualification techniques is still open research to be
investigated.

##### Light Direction and Object Color

(Hamdi ., 2021) studied the impact of light’s direction on the recognition
performance and found that selecting a random light in the training phase
benefits the network to generalize to the test set where it can reduce the
overfitting for more significant views number. However, in the testing phase,
relative light should be used to stabilize the performance. In addition, when
they studied the impact of the objects’ color on the recognition performance,
they also found that using random objects’ colors in the training phase can
help to enhance the test accuracy. This fact could be investigated in more
detail in future research.

##### Number of transformer blocks

If the multi-view 3D object recognition model is transformer-based, then the
number of transformer blocks is another factor that could impact the
recognition performance. The transformer-based models such as MVT (Chen .,
2021), OVPT (W. Wang, Chen ., 2022), and MVCVT (J. Li ., 2023) experimented
with the influence of different numbers of transformer blocks. These methods
agree that when the number of transformer blocks increases, it does not always
lead to better recognition performance but will require more training time,
more trainable parameters, and more floating-point operations. However, OVPT
(W. Wang, Chen ., 2022) and MVCVT (J. Li ., 2023) experimented on the
ModelNet10 dataset with 0 to 6 gradually increased transformer blocks and
concluded that three transformer blocks as an optimal choice with better
accuracy and less computational cost. However, MVT (Chen ., 2021) experimented
on the same dataset with the same setting of transformer blocks and concluded
that four were the optimal choice. This makes transformer block factors
required to be investigated from various transformer-based models in the
future.

##  7 Conclusion

The field of 3D data representation has experienced substantial growth in
recent years, focusing on multi-view representations as the input for 3D
object recognition methods due to their proven efficacy. The scope of this
review is limited to DL-based and transformer-based methods, specifically
those that use multi-view data as input representation. The paper provides a
comprehensive overview of the general pipeline of DL-based multi-view 3D
object recognition models, summarizing the various techniques utilized at each
stage. Additionally, the article delves into the details of existing DL-based
multi-view 3D object recognition models, including commonly used 3D datasets,
camera configurations, view selection strategies, pre-trained CNN
architectures, fusion strategies, and recognition performance. In addition, it
contains the latest developments in transformers and attention mechanisms in
multi-view 3D object recognition, which are not covered by the existing
reviews. Finally, the paper highlights several key insights and future
directions for developing multi-view 3D object recognition methods.

## Declarations

  * •

Conflict of interest. The authors have no conflicts of interest to declare
that are relevant to the content of this article.

##  Appendix A Related Deep Models

The view-based deep models that proposed for 3D object recognition methods are
shown in Table 11 along with their references.

Table 11: Related view-based deep models for 3D object recognition methods and their references |  |
---|---|---
No. |  Deep Model |  Reference
1 |  MVCNN |  (H. Su ., 2015)
2 |  Pairwise |  (Johns ., 2016)
3 |  VolumetricMVCNN |  (C.R. Qi ., 2016)
4 |  GIFT |  (Bai ., 2016)
5 |  PANORAMA-NN |  (Sfikas ., 2017)
6 |  PANORAMA-ENN |  (Sfikas ., 2018)
7 |  MHBN |  (T. Yu ., 2018)
8 |  GVCNN |  (Feng ., 2018)
9 |  MVCNN(GoogLeNet) |  (Feng ., 2018)
10 |  SeqViews2SeqLabels |  (Han ., 2018)
11 |  RotationNet |  (Kanezaki ., 2018)
12 |  Ma et al. |  (Ma ., 2018)
13 |  VERAM |  (Chen ., 2018)
14 |  MVCNN-new |  (JC. Su ., 2018)
15 |  iMHL |  (Z. Zhang ., 2018)
16 |  MVSG-DNN |  (HY. Zhou ., 2019)
17 |  RN |  (Z. Yang Wang, 2019)
18 |  MLVCNN |  (Jiang ., 2019)
19 |  G-CNN |  (Esteves ., 2019)
20 |  DeepCCFV |  (Huang ., 2019)
21 |  3D2SeqViews |  (Han ., 2019)
22 |  DomSetClust |  (C. Wang ., 2019)
23 |  HGNN |  (Feng ., 2019)
24 |  DRCNN |  (K. Sun ., 2020)
25 |  View-GCN |  (Wei ., 2020)
26 |  HMVCMx |  (Wei ., 2020)
27 |  MVTN |  (Hamdi ., 2021)
28 |  MVT |  (Chen ., 2021)
29 |  OVPT |  (W. Wang, Chen ., 2022)
30 |  MVDAN |  (W. Wang, Cai Wang, 2022)
31 |  MVMSAN |  (W. Wang, Wang ., 2022)
32 |  Yu and Cao |  (L. Yu Cao, 2023)
33 |  VCGR-Net |  (Xu ., 2023)
34 |  ViewFormer |  (H. Sun ., 2023)
35 |  MVContrast |  (L. Wang ., 2023)
36 |  MORE |  (Parisotto ., 2023)
37 |  MVCVT |  (J. Li ., 2023)

##  Appendix B Multi-View Datasets

The datasets that used for different multi-view computer vision applications
are shown in Table 12 along with their references.

Table 12: Datasets that used for different multi-view computer vision applications and their references |  |
---|---|---
No. |  Multi-view Dataset |  Reference
1 |  CompCars |  (L. Yang ., 2015)
2 |  PlantCLEF |  (Goëau ., 2016)
3 |  AntWeb |  (AntWeb, 2021)
4 |  Flora Incognita |  (Bai ., 2016)
5 |  Contactless 3D fingerprint Database |  (C. Lin Kumar, 20182)
6 |  Multi-view Contactless Fingerprint Database |  (W. Zhou ., 2014)
7 |  ModelNet40 |  (T. Yu ., 2018)
8 |  SketchClean |  (Schneider Tuytelaars, 2014)
9 |  DDSM |  (PUB ., 2000)
10 |  MIAS |  (Suckling J, 1994)
11 |  MNIST |  (LeCun ., 1998)
12 |  Men-Women Classification |  (Orsolini, 2019)
13 |  CIFAR-10 |  (Krizhevsky ., 2009)
14 |  CIFAR-100 |  (Krizhevsky ., 2009)
15 |  Fashion-MNIST |  (Xiao ., 2017)
16 |  American Sign Language |  (Thakur, 2019)
17 |  Fruit 360 |  (Mureşan Oltean, 2018)
18 |  Tomato Disease Classification |  (Lamrahi, 2019)
19 |  Plant Disease |  (Sharma, 2019)

## References

  *   * Ahmed . (2018) survey2018Ahmed, E., Saint, A., Shabayek, A.E.R., Cherenkova, K., Das, R., Gusev, G.Ottersten, B.  2018\.  A survey on deep learning advances on different 3D data representations A survey on deep learning advances on different 3d data representations.  arXiv preprint arXiv:1808.01462,
  * Alam . (2021) 9MVclassificationAlam, M.T., Kumar, V. Kumar, A.  2021\.  A Multi-view Convolutional Neural Network Approach for Image Data Classification A multi-view convolutional neural network approach for image data classification.  2021 International Conference on Communication information and Computing Technology (ICCICT) 2021 international conference on communication information and computing technology (iccict) ( 1–6).
  * AntWeb (2021) AntWebAntWeb  2021\.  AntWeb. Version 8.66 Antweb. version 8.66.  California Academy of Science,
  * Bai . (2016) GIFTBai, S., Bai, X., Zhou, Z., Zhang, Z. Jan Latecki, L.  2016\.  Gift: A real-time and scalable 3d shape search engine Gift: A real-time and scalable 3d shape search engine.  Proceedings of the IEEE conference on computer vision and pattern recognition Proceedings of the ieee conference on computer vision and pattern recognition ( 5023–5032).
  * Besl Jain (1985) ACM1Besl, P.J. Jain, R.C.  1985\.  Three-dimensional object recognition Three-dimensional object recognition.  ACM Computing Surveys (CSUR)17175–145,
  * Bhatt . (2021) CNNVariantsBhatt, D., Patel, C., Talsania, H., Patel, J., Vaghela, R., Pandya, S.Ghayvat, H.  2021\.  CNN Variants for Computer Vision: History, Architecture, Application, Challenges and Future Scope Cnn variants for computer vision: History, architecture, application, challenges and future scope.  Electronics10202470,
  * Boho . (2020) FloraCaptureBoho, D., Rzanny, M., Wäldchen, J., Nitsche, F., Deggelmann, A., Wittich, H.C.Mäder, P.  2020\.  Flora Capture: a citizen science application for collecting structured plant observations Flora capture: a citizen science application for collecting structured plant observations.  BMC bioinformatics2111–11,
  * Brock . (2016) VRNBrock, A., Lim, T., Ritchie, J.M. Weston, N.  2016\.  Generative and discriminative voxel modeling with convolutional neural networks Generative and discriminative voxel modeling with convolutional neural networks.  arXiv preprint arXiv:1608.04236,
  * Bustos . (2005) ACM2Bustos, B., Keim, D.A., Saupe, D., Schreck, T. Vranić, D.V.  2005\.  Feature-based similarity search in 3D object databases Feature-based similarity search in 3d object databases.  ACM Computing Surveys (CSUR)374345–387,
  * Cao . (2020) review2020Cao, W., Liu, Q. He, Z.  2020\.  Review of pavement defect detection methods Review of pavement defect detection methods.  Ieee Access814531–14544,
  * Chang . (2015) ShapeNetChang, A.X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z.others  2015\.  Shapenet: An information-rich 3d model repository Shapenet: An information-rich 3d model repository.  arXiv preprint arXiv:1512.03012,
  * Chatfield . (2014) VGG-MChatfield, K., Simonyan, K., Vedaldi, A. Zisserman, A.  2014\.  Return of the devil in the details: Delving deep into convolutional nets Return of the devil in the details: Delving deep into convolutional nets.  arXiv preprint arXiv:1405.3531,
  * Chen . (2021) MVTChen, S., Yu, T. Li, P.  2021\.  Mvt: Multi-view vision transformer for 3d object recognition Mvt: Multi-view vision transformer for 3d object recognition.  arXiv preprint arXiv:2110.13083,
  * Chen . (2018) VERAMChen, S., Zheng, L., Zhang, Y., Sun, Z. Xu, K.  2018\.  Veram: View-enhanced recurrent attention model for 3d shape classification Veram: View-enhanced recurrent attention model for 3d shape classification.  IEEE transactions on visualization and computer graphics25123244–3257,
  * Esteves . (2019) G-CNNEsteves, C., Xu, Y., Allen-Blanchette, C. Daniilidis, K.  2019\.  Equivariant multi-view networks Equivariant multi-view networks.  Proceedings of the IEEE/CVF International Conference on Computer Vision Proceedings of the ieee/cvf international conference on computer vision ( 1568–1577).
  * Feng . (2019) HGNNFeng, Y., You, H., Zhang, Z., Ji, R. Gao, Y.  2019\.  Hypergraph neural networks Hypergraph neural networks.  Proceedings of the AAAI Conference on Artificial Intelligence Proceedings of the aaai conference on artificial intelligence ( 33, 3558–3565).
  * Feng . (2018) GVCNNFeng, Y., Zhang, Z., Zhao, X., Ji, R. Gao, Y.  2018\.  GVCNN: Group-view convolutional neural networks for 3D shape recognition Gvcnn: Group-view convolutional neural networks for 3d shape recognition.  Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Proceedings of the ieee conference on computer vision and pattern recognition ( 264–272).
  * Furuya Ohbuchi (2016) DLANFuruya, T. Ohbuchi, R.  2016\.  Deep Aggregation of Local 3D Geometric Features for 3D Model Retrieval. Deep aggregation of local 3d geometric features for 3d model retrieval.  BMVC Bmvc ( 7, 8).
  * Gezawa . (2020) review2020-2Gezawa, A.S., Zhang, Y., Wang, Q. Yunqi, L.  2020\.  A review on deep learning approaches for 3d data representations in retrieval and classifications A review on deep learning approaches for 3d data representations in retrieval and classifications.  IEEE access857566–57593,
  * Goëau . (2016) PlantCLEFGoëau, H., Bonnet, P. Joly, A.  2016\.  Plant identification in an open-world (lifeclef 2016) Plant identification in an open-world (lifeclef 2016).  CLEF: Conference and Labs of the Evaluation Forum Clef: Conference and labs of the evaluation forum ( 428–439).
  * Guo . (2020) Survey2020Guo, Y., Wang, H., Hu, Q., Liu, H., Liu, L. Bennamoun, M.  2020\.  Deep learning for 3d point clouds: A survey Deep learning for 3d point clouds: A survey.  IEEE transactions on pattern analysis and machine intelligence,
  * Hamdi . (2021) MVTNHamdi, A., Giancola, S. Ghanem, B.  2021\.  MVTN: Multi-View Transformation Network for 3D Shape Recognition Mvtn: Multi-view transformation network for 3d shape recognition.  Proceedings of the IEEE/CVF International Conference on Computer Vision Proceedings of the ieee/cvf international conference on computer vision ( 1–11).
  * Han . (2019) 3D2SeqViewsHan, Z., Lu, H., Liu, Z., Vong, CM., Liu, YS., Zwicker, M.Chen, C.P.  2019\.  3D2SeqViews: Aggregating sequential views for 3D global feature learning by CNN with hierarchical attention aggregation 3d2seqviews: Aggregating sequential views for 3d global feature learning by cnn with hierarchical attention aggregation.  IEEE Transactions on Image Processing2883986–3999,
  * Han . (2018) SeqViews2SeqLabelsHan, Z., Shang, M., Liu, Z., Vong, CM., Liu, YS., Zwicker, M.Chen, C.P.  2018\.  SeqViews2SeqLabels: Learning 3D global features via aggregating sequential views by RNN with attention Seqviews2seqlabels: Learning 3d global features via aggregating sequential views by rnn with attention.  IEEE Transactions on Image Processing282658–672,
  * He . (2016) ResNetHe, K., Zhang, X., Ren, S. Sun, J.  2016\.  Deep residual learning for image recognition Deep residual learning for image recognition.  Proceedings of the IEEE conference on computer vision and pattern recognition Proceedings of the ieee conference on computer vision and pattern recognition ( 770–778).
  * Hendrycks Dietterich (2019) ImageNet-CHendrycks, D. Dietterich, T.  2019\.  Benchmarking neural network robustness to common corruptions and perturbations Benchmarking neural network robustness to common corruptions and perturbations.  arXiv preprint arXiv:1903.12261,
  * Huang . (2019) DeepCCFVHuang, Z., Zhao, Z., Zhou, H., Zhao, X. Gao, Y.  2019\.  Deepccfv: Camera constraint-free multi-view convolutional neural network for 3d object retrieval Deepccfv: Camera constraint-free multi-view convolutional neural network for 3d object retrieval.  Proceedings of the AAAI Conference on Artificial Intelligence Proceedings of the aaai conference on artificial intelligence ( 33, 8505–8512).
  * Ioannidou . (2017) survay2017Ioannidou, A., Chatzilari, E., Nikolopoulos, S. Kompatsiaris, I.  2017\.  Deep learning advances in computer vision with 3d data: A survey Deep learning advances in computer vision with 3d data: A survey.  ACM Computing Surveys (CSUR)5021–38,
  * Jiang . (2019) MLVCNNJiang, J., Bao, D., Chen, Z., Zhao, X. Gao, Y.  2019\.  MLVCNN: Multi-loop-view convolutional neural network for 3D shape retrieval Mlvcnn: Multi-loop-view convolutional neural network for 3d shape retrieval.  Proceedings of the AAAI Conference on Artificial Intelligence Proceedings of the aaai conference on artificial intelligence ( 33, 8513–8520).
  * Johns . (2016) PairwiseJohns, E., Leutenegger, S. Davison, A.J.  2016\.  Pairwise decomposition of image sequences for active multi-view recognition Pairwise decomposition of image sequences for active multi-view recognition.  Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Proceedings of the ieee conference on computer vision and pattern recognition ( 3813–3822).
  * Kanezaki . (2018) RotationNetKanezaki, A., Matsushita, Y. Nishida, Y.  2018\.  RotationNet: Joint object categorization and pose estimation using multiviews from unsupervised viewpoints Rotationnet: Joint object categorization and pose estimation using multiviews from unsupervised viewpoints.  Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Proceedings of the ieee conference on computer vision and pattern recognition ( 5010–5019).
  * Kertész Vámossy (2015) challenges2015Kertész, G. Vámossy, Z.  2015\.  Current challenges in multi-view computer vision Current challenges in multi-view computer vision.  2015 IEEE 10th Jubilee International Symposium on Applied Computational Intelligence and Informatics 2015 ieee 10th jubilee international symposium on applied computational intelligence and informatics ( 237–241).
  * Krizhevsky . (2009) CIFAR-DSKrizhevsky, A., Hinton, G. .  2009\.  Learning multiple layers of features from tiny images Learning multiple layers of features from tiny images.
  * Krizhevsky . (2012) AlexNetKrizhevsky, A., Sutskever, I. Hinton, G.E.  2012\.  Imagenet classification with deep convolutional neural networks Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems251097–1105,
  * Lai . (2011) RGBDLai, K., Bo, L., Ren, X. Fox, D.  2011\.  A large-scale hierarchical multi-view rgb-d object dataset A large-scale hierarchical multi-view rgb-d object dataset.  2011 IEEE international conference on robotics and automation 2011 ieee international conference on robotics and automation ( 1817–1824).
  * Lamrahi (2019) Tomato-DSLamrahi, N.  2019\.  Tomato Disease Classification Dataset. Tomato disease classification dataset.  data retrieved from kaggle website: https://www.kaggle.com/noulam/tomato
  * LeCun . (1998) MNISTLeCun, Y., Bottou, L., Bengio, Y. Haffner, P.  1998\.  Gradient-based learning applied to document recognition Gradient-based learning applied to document recognition.  Proceedings of the IEEE86112278–2324,
  * J. Li . (2023) MVCVTLi, J., Liu, Z., Li, L., Lin, J., Yao, J. Tu, J.  2023\.  Multi-view convolutional vision transformer for 3D object recognition Multi-view convolutional vision transformer for 3d object recognition.  Journal of Visual Communication and Image Representation103906,
  * Y. Li, Bu . (2018) pointcnnLi, Y., Bu, R., Sun, M., Wu, W., Di, X. Chen, B.  2018\.  Pointcnn: Convolution on x-transformed points Pointcnn: Convolution on x-transformed points.  Advances in neural information processing systems31820–830,