[1]

[] []

# Advancing 3D Point Cloud Understanding through Deep Transfer Learning: A
Comprehensive Survey

Shahab Saquib Sohail  Yassine Himeur<EMAIL_ADDRESS>Hamza Kheddar  Abbes
Amira  Fodil Fadli  Shadi Atalla  Abigail Copiaco  Wathiq Mansoor  School of
Computing Science and Engineering, VIT Bhopal University, Sehore, MP 466114,
India  College of Engineering and Information Technology, University of Dubai,
Dubai, UAE  LSEA Laboratory, Department of Electrical Engineering, University
of Medea, 26000, Algeria  Department of Computer Science, University of
Sharjah, UAE  Institute of Artificial Intelligence, De Montfort University,
Leicester, United Kingdom  Department of Architecture & Urban Planning, Qatar
University, Doha, 2713, Qatar

###### Abstract

The 3D point cloud (3DPC) has significantly evolved and benefited from the
advance of deep learning (DL). However, the latter faces various issues,
including the lack of data or annotated data, the existence of a significant
gap between training data and test data, and the requirement for high
computational resources. To that end, deep transfer learning (DTL), which
decreases dependency and costs by utilizing knowledge gained from a source
data/task in training a target data/task, has been widely investigated.
Numerous DTL frameworks have been suggested for aligning point clouds obtained
from several scans of the same scene. Additionally, DA, which is a subset of
DTL, has been modified to enhance the point cloud data’s quality by dealing
with noise and missing points. Ultimately, fine-tuning and DA approaches have
demonstrated their effectiveness in addressing the distinct difficulties
inherent in point cloud data. This paper presents the first review shedding
light on this aspect. it provides a comprehensive overview of the latest
techniques for understanding 3DPC using DTL and domain adaptation (DA).
Accordingly, DTL’s background is first presented along with the datasets and
evaluation metrics. A well-defined taxonomy is introduced, and detailed
comparisons are presented, considering different aspects such as different
knowledge transfer strategies, and performance. The paper covers various
applications, such as 3DPC object detection, semantic labeling, segmentation,
classification, registration, downsampling/upsampling, and denoising.
Furthermore, the article discusses the advantages and limitations of the
presented frameworks, identifies open challenges, and suggests potential
research directions.

###### keywords:

3D point cloud Deep transfer learning Domain adaption Fine-tuning
Classification Segmentation and registration

3DPC

    3D point cloud
GAN

    generative adversarial network
CV

    computer vision
DL

    deep learning
ML

    machine learning
DTL

    deep transfer learning
DA

    domain adaptation
UDA

    unsupervised domain adaptation
TD

    target domain
SD

    source omain
CNN

    convolutional neural network
MAE

    multi-scale masked autoencoder
IoU

    intersection over union
SSL

    self-supervised learning
SVCN

    sparse voxel completion network
OA

    overall accuracy
CD

    Chamfer distance
LiDAR

    light detection and ranging
BiLSTM

    bidirectional LSTM
ITL

    inductive transfer learning
SFUDA

    source-free unsupervised DA
PCM

    point cloud mixup
MLP

    multi-layer perceptron
CLIP

    contrastive vision-language pretraining
TTL

    transductive transfer learning
CLDA

    cross-modal learning DA
AE

    auto-encoder
RNN

    recurrent neural network
CRF

    conditional random field

##  1 Introduction

###  1.1 Preliminary

CV continues to attract significant interest as a growing branch of machine
learning (ML), which targets different problems in smart cities, medicine,
autonomous driving, medicine, video surveillance, scene understanding, safety,
and security [1, 2]. With the rapid development of sensor technology, 3D
sensors have recently been widely adopted, which increased the interest of the
computer vision (CV) research community in developing 3D sensor data
processing methodologies [3, 4]. Additionally, with the use of augmented
reality and virtual reality (AR/VR), 3D vision problems become more important
since they provide much richer information than 2D [5, 6]. Typically, numerous
3D sensors for acquiring 3D data have been used, including depth-sensing
cameras (such as Apple Depth, RealSense, and Kinect cameras), light detection
and ranging (LiDAR), which is used for mobile mapping terrestrial laser
scanning, aerial LiDAR [7]. Moreover, generative adversarial networks can be
used to augment data when data scarcity problem occurs. In this context, 3D
data can provide rich scale, shape, and geometric information to be
complemented with 2D images for better representing the surrounding
environment [8, 9].

While there are different approaches to representing 3D data, such as
volumetric grids, meshes, and depth images, the 3D point clouds are the most
used. Typically, a 3DPC representation conserves the original geometric
information in 3D space [10]. Besides, a 3DPC is an ensemble of data points
representing an object’s surfaces in 3D coordinates. In this regard, spatial
coordinates are used to represent data points, surface normals, and color
information and format ( e.g., RGB, HSV, and others) [11].

Additionally, although 3DPC can be considered as non-Euclidean geometric data,
in practice, delineated as small Euclidean subgroups with a standard
coordinates system and global parametrization [12]. The success of this
representation is due to its invariability to transformations and attacks,
including rotation, scaling, translation, etc., which makes it robust to
extracting objects’ features. Moreover, using deep learnings techniques to
extract 3DPC data and perform complex tasks such as detection, classification,
recognition, and retrieval has expanded their adoption in many research and
development areas [13]. Typically, 3DPCs are widely adopted to CV tasks, such
as object recognition, semantic segmentation, and scene understanding [14,
15]. Additionally, they are used to create detailed models of environments for
navigation and localization, detailed maps and models of buildings,
landscapes, and other structures, as well as detailed models of buildings and
other structures for design and planning [16]. Moreover, they can be utilized
to inspect and maintain industrial equipment, infrastructure, and other assets
[17, 18]. Besides, the 3DPC technology helps implement realistic and immersive
experiences in AR and VR applications.

The traditional ML approaches and, recently, DL methods witnessed rapid growth
and have attracted researchers because of their applicability in many real-
life applications, including smart healthcare [19], disease diagnosis and
medical image classifications [4], business and marketing [20], recommender
systems [21], energy [22], agriculture [23], robotics [24], and more. The
primary advantage of these learning algorithms is that they train a model that
learns the hidden pattern and can be exploited for any specific purpose with
high accuracy. However, with time, some issues have been raised by the
research communities. First, these learning algorithms require extensive
training datasets, especially DL algorithms [1]. Second, most DL models are
based on supervised learning and then require huge amounts of ground-truth
data. Third, it is a common assumption that the data that is trained and the
future data to be processed must have the same distribution and be in the same
feature space [25]. It becomes difficult and sometimes impossible to maintain
the aforementioned assumption in several real-life scenarios. For example,
when performing specific tasks, such as classification in a particular domain;
however, we may have the requisite trained data in another domain, where both
datasets may not have the same distribution or same feature space.

Figure 1: Difference between conventional ML and DTL techniques for multiple
tasks: (a) conventional ML and (b) DTL.

Furthermore, an essential factor for ensuring accurate performance of ML
algorithms is consistency in the distribution and feature space of the
datasets used for training and testing. If the distribution of data changes,
it becomes necessary to rebuild the model from scratch by collecting new
training data [26]. However, this process is not only costly but also often
impractical due to the challenges associated with re-collecting training data.
Therefore, there is a need for a mechanism that can reduce the expenses
associated with re-collecting training data, minimize the cost of data
labeling, and still achieve high performance without requiring extensive
amounts of training data. To that end, knowledge transfer has been suggested,
which would fit the above constraints and can significantly improve
performance. This knowledge transfer mechanism is termed  deep transfer
learning (DTL) [1, 26, 27]. Fig. 1 explains briefly the difference between
conventional ML and DTL techniques.

On the one hand, employing DL for 3DPC poses a rather complex challenges
because of: (i) the variability in point density and reflective intensity,
influenced by the distance between objects and LiDAR sensors, (ii) existence
of noise originated from sensors ( e.g., perturbations and outliers), (iii)
data incompleteness caused by occlusion between objects and cluttered
background, and (iv) confusion categories caused by shape-similar or
reflectance-similar objects. On the other hand, challenges of DL/DTL models
include (i) permutation and orientation invariance, (ii) 3D translation and
rotation challenges, (iii) difficulty of handling large-scale datasets, (iv)
securing computational resources, and (v) low performance. Fig. 2 summarizes
the tasks and challenges related to data and DL/DTL-based applications on
3DPCs.

Figure 2: Tasks and challenges related to data and DL/DTL-based applications
on 3DPCs.

###  1.2 Our contributions

Arguably, one of the top success stories of DL is DTL. Accordingly, the
finding that pretraining a DL network on a rich source set (e.g., ImageNet)
can help boost performance once fine-tuned on a usually much smaller target
set has been instrumental to many applications in language and CV. Likewise,
several studies have explored the potential of DTL in 3DPC applications to
address the aforementioned challenges. This offers an opportunity to present
the first review article that comprehensively examines the contributions of
DTL-based approaches in advancing our understanding of 3D scenes, such as 3DPC
segmentation, 3D object detection, 3D object classification, 3DPC
registration, and more. For instance, DTL has been shown to be effective in
various ways, including (i) leveraging knowledge learned from synthetic data
to improve semantic segmentation in real LiDAR 3DPC, (ii) enabling accurate
classification of 3DPCs even with limited training data, (iii) mitigating the
issue of overfitting in 3DPC classification, and (iv) reducing the labor-
intensive process of annotating 3DPC datasets, among other benefits. In this
respect, this paper first presents a background of DTL, where a well-defined
taxonomy is conducted. Moving on, datasets and evaluation metrics used to
evaluate existing DTL-based 3DPCs techniques are discussed. Next, existing
studies are overviewed based on different aspects, and their pros and cons are
identified. After that, the challenges encountered when using DTL for 3DPCs
are identified before deriving future research directions. To summarize, the
main contributions of this review can be stated as follows:

  * •

Presenting, to the best of the authors’ knowledge, the first review article on
using DTL and domain adaptation (DA) for 3DPC applications;

  * •

Discussing existing datasets and evaluation metrics used for assessing the
performance of 3DPC frameworks based in DTL;

  * •

Introducing a well-defined taxonomy to overview existing DTL-based 3DPC
studies;

  * •

Identifying the current challenges encountered when using DTL for 3DPC
understanding tasks; and

  * •

Deriving future research directions that can attract significant research
interest in the near future.

###  1.3 Methodology of the survey

The present review study is based on the protocols and procedure suggested in
[28], which has recently been adopted widely in many studies such as [29].
This study is motivated by the recent developments in 3DPC technology. There
are some reviews available on the theme which has exploited ML [30], DL [31],
and reinforcement learning [32]; however, to the best of our search and
efforts, we could not find a review exclusively exploring DTL for different
3DPC tasks. The proliferation of DTL techniques has influenced researchers by
virtue of which the span of related techniques has been expanding and has
covered much recent AI-driven research. Our study gives the readers an insight
into how DTL is being implemented for performing many 3DPC tasks. This review
aims at finding the answer to research questions presented in Table 1.

Table 1: Research questions covered in this review. No. |  Question |  Objective
---|---|---
RQ1 |  What are the key principles and theoretical foundations of DTL and 3DPC? |  Provide a foundational understanding of the key principles and theories behind DTL and 3DPC. This will aid readers in grasifying the basic tenets upon which the field of study is built and support comprehension of more complex concepts discussed later in the review.
RQ2 |  Why the application of DTL in 3DPC is receiving increasing attention? |  Examine and explain why DTL is increasingly being used for 3DPC. This will help highlight the unique benefits and opportunities offered by DTL in handling 3D data, indicating its growing importance in the field.
RQ3 |  How the different 3DPC tasks can be implemented using DTL? |  Present a comprehensive overview of how DTL can be utilized in various 3DPC tasks. This objective aims to showcase the versatility and wide-ranging applicability of DTL in 3DPC tasks, providing practical insights for researchers and practitioners.
RQ4 |  What are the DTL mechanisms and settings for performing the 3DPC tasks and optimizing their performance? |  Delve into the specific mechanisms and settings of DTL that optimize performance in 3DPC tasks. The objective here is to provide a clear understanding of the operational details of DTL, which can serve as a guide for those intending to implement DTL in their own 3DPC tasks.
RQ5 |  Which DTL models are better appropriate for 3DPC tasks? |  Evaluate and suggest DTL models that are most suitable for 3DPC tasks. By doing this, the review will provide practical advice for readers looking to apply DTL in 3DPC, assisting them in model selection.
RQ6 |  What are the issues in implementing DTL for performing the above tasks, and how to tackle these challenges? |  Identify the key challenges associated with implementing DTL for 3DPC tasks and to propose potential solutions to these challenges. This will help improve the application of DTL in 3DPC by addressing problems head-on, promoting a robust and reliable usage of these techniques.
RQ7 |  What are the future research directions for improving DTL in 3DPC? |  Forecast future research directions for DTL in 3DPC. This objective aims to stimulate new research initiatives by identifying promising avenues for further exploration, driving innovation and development in the field.

itemize

Why the application of DTL in 3DPC is receiving increasing attention?

How the different 3DPC tasks can be implemented using DTL?

What are the DTL mechanisms and settings for performing the 3DPC tasks and
optimizing their performance?

Which DTL models are better appropriate for 3DPC tasks?

What are the issues in implementing DTL for performing the above tasks, and
how to tackle these challenges?

What are the future research directions for improving DTL in 3DPC?

(a)

(b)

Figure 3: Summary of the approach used to search and select articles included
in the review: (a) The adopted search procedure and (b) The selection
criteria.

The bibliometric research is performed in the context of a narrative review.
The recent works related to the 3DPC involving DTL techniques for performing
their respective tasks have been searched. We searched the relevant keywords,
like, "3D point cloud", "3D point cloud segmentation", "3D point cloud
classification", "3D object detection", "deep transfer learning", and "domain
adaptation", with different combinations on Scopus database in titles,
abstracts, and keywords. The adopted search procedure is explained in Figs.
3(a) and 3(b). Typically, in response to the relevant queries, 176 articles
were retrieved, with 149 of them being non-duplicates. These articles
underwent further scrutiny to determine their relevance to the theme of the
present study. Ultimately, 108 papers were selected for inclusion in this
study.

The paper is organized as follows: Section 2 introduces background related to
DTL definitions and terms, pre-trained models, useful datasets, and metrics
used in 3DPC. Section 3 reviews relevant literature and work related to DTL
and DA. Section 4 outlines state-of-the-art proposed applications based on
DTL. Section 5 discusses open challenges. Section 6 suggests recent future
research directions. Finally, Section 7 concludes the survey. Fig. 4
illustrates the survey’s structure, enhancing readability, and offering
guidance to readers.

Figure 4: Survey structure with sections and sub-sections disctribution.

##  2 Background

###  2.1 3D Point Cloud

A 3DPC is a collection of data points in a three-dimensional coordinate
system. Each point in the point cloud is represented as:

$P=\\{p_{1},p_{2},...,p_{n}\\}$

where each point $p_{i}$ is a vector in $\mathbb{R}^{3}$:

$p_{i}=(x_{i},y_{i},z_{i})$

####  2.1.1 Operations and Transformations

Common operations and transformations applied to 3DPCs include:

  * •

Translation: Moving the point cloud by adding a constant vector
$\mathbf{t}=(t_{x},t_{y},t_{z})$ to each point:

$p_{i}^{\prime}=p_{i}+\mathbf{t}$

  * •

Rotation: Rotating the point cloud using a rotation matrix $\mathbf{R}$. The
rotation matrix is a 3x3 matrix:

$p_{i}^{\prime}=\mathbf{R}p_{i}$

Rotation matrices can be derived from Euler angles, axis-angle, or quaternion
representations.

  * •

Scaling: Changing the size by scaling each point by a scalar $s$ or different
scalars for each axis:

$p_{i}^{\prime}=s\cdot p_{i}\text{ or
}p_{i}^{\prime}=(s_{x}x_{i},s_{y}y_{i},s_{z}z_{i})$

  * •

Transformation: A combination of translation, rotation, and scaling,
represented as matrix multiplication in homogeneous coordinates:

$p_{i}^{\prime}=\mathbf{T}\begin{bmatrix}x_{i}\\\ y_{i}\\\ z_{i}\\\
1\end{bmatrix}$

where $\mathbf{T}$ is a 4x4 transformation matrix:

$\mathbf{T}=\begin{bmatrix}\mathbf{R}&\mathbf{t}\\\ 0&1\end{bmatrix}$

###  2.2 Definitions pertaining to DTL

This section presents the main definitions used in DTL-based 3DPC
applications.

Def. 1 - Domain: Consider a dataset $X$ consisting of $n$ observations,
$x_{1},\cdots,x_{n}$, in a feature space $\chi$. The marginal probability
distribution of $X$ is represented by $P(X)$. A domain, denoted as
$\mathbb{D}$, is defined as the set containing $X$ and $P(X)$. In the field of
DTL, the domain containing the initial knowledge is referred to as the source
omain (SD), represented by $\mathbb{D}_{S}$, while the domain containing the
unknown knowledge to be learned is called the target domain (TD) and
represented by $\mathbb{D}_{T}$ .

Def. 2 - Task: The dataset $X$ contains $n$ observations,
$x_{1},\cdots,x_{n}$, in a feature space $\chi$, and it is associated with a
set of labels $Y$, $y_{1},\cdots,y_{n}$, in a label space $\gamma$. A task can
be defined as a set containing the labels $Y$ and a learning objective
predictive function $\mathbb{F}(X)$, represented by
$\mathbb{T}=\\{Y,\mathbb{F}(X)\\}$. This function is also denoted as the
conditional distribution $P(Y|X)$. In accordance with this definition of task,
the label spaces of the SD and TD are represented as $\gamma_{S}$ and
$\gamma_{T}$ respectively [33].

One way to classify DTL methods is based on their approach to transferring
knowledge, which can be broken down into what, when, and how knowledge is
transferred.

(a) What knowledge is transferred: The classification of DTL methods based on
"what knowledge is transferred" examines the specific characteristics of
knowledge that can be transferred across domains or tasks. Some information is
unique to a particular domain or task, while other knowledge is general and
can improve the performance of the TD or task. Based on this criterion, DTL
methods can be categorized as model-based, relation-based, instance-based, and
feature-based [34].

(b) How knowledge is transferred: The classification of DTL methods based on
this question focuses on the specific algorithms or techniques used to
transfer knowledge across domains or tasks.

(c) When knowledge is transferred: inquires as to when and under what
circumstances knowledge should or should not be transferred.

###  2.3 Theoretical taxonomy of DTL

In this section, a clear categorization of DTL methodologies used for 3DPCs is
presented. The proposed classification, shown in Fig. 5, is arranged based on
the following criteria: (i) learning style, (ii) methodology, (iii) DTL type,
(iv) data annotation, and (v) widely used DTL models. DTL techniques can
generally be divided into different groups depending on whether the source and
TDs and tasks are alike or not. The mathematical descriptions of the different
DTL groups and their distinctions are presented in Fig. 6. While this
information is discussed in the literature review, aiming to simplify it and
present it in a more reader-friendly manner in this study.

Figure 5: Proposed taxonomy of existing DTL algorithms for 3DPCs.  Figure 6:
The DTL classification is determined by how similar the source and TDs and
tasks are. The symbol ($\varsubsetneq$) is used to indicate that the
domains/tasks are distinct yet related. The symbol ($\exists!$) indicates the
presence of a single unique domain/task. While ($\cong$) denotes that the
domains, tasks, or spaces do not match.

####  2.3.1 Inductive DTL

The goal of inductive DTL is to improve the target prediction function
$\mathbb{F}T$ in the TDs when compared to classical ML. This is achieved even
when the target tasks $\mathbb{T}{T}$ are different from the source tasks
$\mathbb{T}{S}$. However, the SD $\mathbb{D}{S}$ and TD $\mathbb{D}_{T}$ may
not always be identical (as shown in Fig. 6). Inductive DTL can take two
forms, depending on the availability of labeled or unlabeled data:

(a) Multi-task DTL: This approach is used when the SDs has a large labeled
dataset ($X_{S}$ labeled with $Y_{S}$). This is a specific form of multi-task
learning where multiple tasks $(T_{1},T_{2},\dots,T_{n})$ are learned
simultaneously (in parallel), including both the source and target tasks[35].

(b) Sequential learning: also known as self-taught learning, is a method used
when the dataset in the SD is unlabeled. It relies on (i) transferring the
feature representation learned from a large collection of unlabeled datasets,
and (ii) applying the learned representation to labeled data for
classification tasks. This DTL method involves sequentially learning multiple
tasks where the gaps between the SDs and TDs may differ. For instance, assume
we have a pre-trained model (PTM) $M$ and we apply DTL to several tasks
$(T_{1},T_{2},\dots,T_{n})$. In this approach, a specific task
$\mathbb{T}_{T}$ is learned at each time step $t$ and it is slower than multi-
task learning, however, when not all the tasks are present at the time of
training, it may be advantageous. Sequential learning can be further
classified into different types [36].

  * 1-

Fine-tuning: involves training a new function, $\mathbb{F}_{T}$, that adapts
the parameters of a PTM $M$ from a source task, $\mathbb{T}_{S}$, to a target
task, $\mathbb{T}_{T}$, by translating the weights of the source task,
$W_{S}$, to the weights of the target task, $W_{T}$. This can be done across
all layers or just a subset of them, and the learning rate for each layer can
be adjusted independently (known as discriminative fine-tuning). Additionally,
new parameters, $K$, can be added to the model to improve its performance on
the target task [37].

$\mathbb{F}_{T}(W_{T},K)=W_{S}\times K$ (1)

  * 2-

Adapter modules: they are designed to take a PTM, $M_{S}$, and adapt its
weights, $W_{S}$, to a target task, $\mathbb{T}_{T}$. The adapter module
achieves this by introducing a new set of parameters, $K$, that are smaller in
size compared to $W_{S}$, i.e. $K\ll W_{S}$. Both $K$ and $W_{S}$ are
decomposed into smaller, more compact modules, such that $W_{S}={w}_{n}$ and
$K={k}_{n}$. This allows the adapter module to learn a new function,
$\mathbb{F}_{T}$, which adapts the model to the target task.

$\mathbb{F}_{T}(K,W_{S})=k_{1}^{\prime}\times w_{1}\times\cdots
k_{n}^{\prime}\times w_{n}$ (2)

The equation (2) illustrates the procedure of adapting a model to a new task
by dynamic weight adjustment, original weights $W_{S}={w}_{n}$ remain
unchanged, whereas the set of weights $K$ are updated to
$K^{\prime}={k^{\prime}}_{n}$. This principle of Dynamic DA is illustrated in
Fig. 7.

  * 3-

Feature based: it focuses on learning concepts and representations at various
levels of an image, such as corners or interest points, blobs or regions of
interest points, ridges, or edges $E$. In this approach, the collection of $E$
obtained from a PTM $M$ is kept unchanged and only $W^{\prime}$ is fine-tuned,
such that the function $\mathbb{F}_{T}$ can be represented as $E\times
W^{\prime}$. The idea is that $E$ are the feature learned from PTM and fine-
tuning only the last layer $W^{\prime}$ to adapt to the new task.

  * 4-

Zero-shot: is the simplest approach among all the others. It does not involve
modifying or adding new parameters to a PTM by assuming that the existing
parameters, denoted as $W_{S}$, cannot be changed. Essentially, zero-shot does
not require any training to optimize or learn new parameters."

Figure 7: Example of DTL models used in 3DPCs: (a) fine-tuning, and (b) deep
DA.

####  2.3.2 Transductive DTL

In comparison to traditional ML, which can be used as a benchmark for DA and
DTL, DTL addresses the scenario where the TD data, denoted as $\mathbb{D}{T}$,
differs from the SDs data, denoted as $\mathbb{D}{S}$. While the SDs has
annotated data ($X_{S}$ paired with $Y_{S}$), the TD has no labeled data. The
source and target tasks are similar as outlined in Fig. 6. Transductive DTL
aims at constructing a target prediction function $\mathbb{F}_{T}$ using the
knowledge from both the SD and TD. Additionally, transductive DTL can be
further categorized into two groups based on the relationship between the SD
and TD, as described in [38].

(a) Deep domain adaptation (DDA): refers to the situation where the feature
spaces of the SD, denoted as $\chi_{S}$, and the TDs, denoted as $\chi_{T}$,
are the same. However, the probability distributions of the input data are
different, with $P(Y_{S}/X_{S})\neq P(Y_{T}/X_{T})$ as described in [39]. DDA
is particularly useful when the target task has a unique distribution or
limited labeled data, as highlighted in [40].

(b) Cross-modality DTL: most DTL techniques require some form of relationship
between feature spaces (or label spaces) of the source and TDs, i.e.,
$\mathbb{D}{S}$ and $\mathbb{D}{T}$. This means that DTL can only be applied
when the source and target have the same modality, such as text, speech, or
video. In contrast, cross-modality DTL is one of the most challenging areas of
DTL, as it assumes that the feature spaces of the source and TDs are
completely different ($\chi_{S}\neq\chi_{T}$), such as speech-to-image, image-
to-text, and text-to-speech. Additionally, the label spaces of the source
$Y_{S}$ and target $Y_{S}$ domains may also differ ($Y_{S}\neq Y_{T}$) as
described in [41].

(c) Unsupervised DTL: is a method of improving the learning of the target
prediction function $\mathbb{F}T$ in the TD $\mathbb{D}{T}$ by using knowledge
from the SDs $\mathbb{D}{S}$ and the source task $\mathbb{T}{S}$, even when
the labels $Y_{S}$ and $Y_{T}$ are not present. It is important to note that
the source and target tasks, $\mathbb{T}{S}$ and $\mathbb{T}{T}$, are related
but distinct. Such kind of approach is useful when labels are not available
for TD data, as stated in [42].

####  2.3.3 Adversarial DTL

Adversarial learning, as introduced in [43], is a method that helps to learn
more transferable and discriminative representations. The first method using
this approach, the domain-adversarial neural network (DANN) was presented in
[44]. Unlike traditional methods that use predefined distance functions, DANN
utilizes a domain-adversarial loss within the network. This approach has shown
to improve the network’s ability to learn discriminative data and has been
used in many visual surveillance studies [45, 46, 47, 48]. However, prior
works in DANN have not considered the different effects of marginal and
conditional distributions. An alternative approach, dynamic distribution
alignment, was proposed in [49] that dynamically evaluates the importance of
each distribution, thus providing a more nuanced approach." Another approach
called the adversarial scoring network (ASNet) was introduced in [50] to
bridge the gap between domains at different levels of granularity. This
approach uses adversarial learning to align the SDs with the TD in the global
and local feature space during the coarse-grained stage. The transferability
of source attributes is then evaluated in fine-grained stage by comparing the
similarity between the source and target samples at multiple levels, utilizing
the generative probability obtained in the coarse stage. The transferable
elements are then selectively used to assist the DTL adaptation process. This
coarse-to-fine architecture effectively reduces the problem of domain
disparity. Specifically, photographs are encoded into density maps by a
generator, and then classified as SDs or TD using a dual-discriminator.
Adversarial training is employed between the dual-discriminator and generator
to bring the domains’ distributions closer together. The dual-discriminator
also generates four different scores which are used as a signal to optimize
the density of the SDs during adaptation for fine-grained transfer as stated
in [50].

In [51], the authors studied the adversarial robustness in 3DPC recognition
using three different architectures: multi-layer perceptron (MLP) network
(PointNet), convolutional network (DGCNN), and transformer-based network. They
employed two methods: adversarial pretraining for fine-tuning, where self-
supervised learning (SSL) tasks are utilized for pretraining, and adversarial
joint training, where the self-supervised task is trained alongside the
recognition task, as illustrated in Fig. 8.

Figure 8: Flowchart of the adversarial TL-based 3DPC classification approach
proposed in [51].

###  2.4 Popular DL-based PC models

The development of DL-based 3DPC approaches is characterized by a rich
diversity of deep learning models each aimed at overcoming specific challenges
related to point cloud data processing. While these models offer significant
improvements in processing speed, accuracy, and applicability, they also
underscore the ongoing need for models that can generalize across different
environments and handle the inherent complexities of 3D point data more
effectively.

####  2.4.1 Feature Extraction and Geometric Detail Enhancement Models

Models such as PointNet [52] and PointNet++ [53] spearheaded the direct
processing of point clouds by respecting permutation invariance and enhancing
the capture of hierarchical structures, respectively. They are pivotal in
tasks like object classification and segmentation, though they struggle with
local detail due to uniform point processing. PointVGG [54] and PointPAVGG
[55] extend these capabilities by adapting image-based convolutional and
attention mechanisms to point clouds, striving to bridge gaps in capturing
intricate geometric details but facing challenges in computational efficiency.
SpiderCNN [56] and PointCNN [57] further innovate by adapting traditional CNN
transformations to unordered point data, enhancing the models’ ability to
learn from complex datasets.

####  2.4.2 Segmentation and Object Detection Models

Models like SplatNet [58], SGPN [59], and FoldingNet [60] represent advances
in segmentation and unsupervised learning. SplatNet leverages sparse bilateral
convolutional layers for processing high-dimensional lattices in point clouds,
suitable for segmentation but hampered by scaling issues. SGPN focuses on
instance segmentation by predicting semantic classes and point groupings,
which becomes complex in cluttered environments. FoldingNet explores
unsupervised learning with a folding-based decoder that may not generalize
across all point cloud types.

####  2.4.3 Data Integration and Upsampling Models

PU-Net [61] and PointGrid [62] focus on upsampling and integrating grid-based
approaches with point processing. PU-Net enhances point cloud density using
multi-level feature integration, while PointGrid blends point and grid-based
methods for recognizing 3D models, though it struggles with sparse point
clouds.

####  2.4.4 Specialized Application Models

Models such as Hand PointNet [63] and PointNetVLAD [64] are tailored for
specific applications like 3D hand pose estimation and global descriptor
extraction, critical for place recognition. However, these models often face
limitations when applied outside their intended scope, such as dynamic
environments or varying object types.

####  2.4.5 Innovative Approaches in Object Detection and Registration

Emerging models like PREDATOR [65] and 3DIoUMatch [66] showcase the potential
of deep learning in object detection and registration. PREDATOR addresses low-
overlap point cloud registration with a deep attention mechanism, suited for
benchmark scenarios but limited in broader applications. 3DIoUMatch leverages
a semi-supervised learning approach to enhance 3D object detection, showing
dependency on initial label quality for performance efficacy. Table LABEL:tab2
summarizes the most popular 3DPC models proposed in the literature.

Table 2: Summary of popular DL-based 3DPC understanding models.  Ref. |  Model Name |  Contribution Description |  Dataset |  Application |  Limitation
---|---|---|---|---|---
[52] |  PointNet |  Introduced a neural network that processes point clouds directly, respecting permutation invariance. |  Various 3D benchmarks |  Object classification, part segmentation, semantic parsing |  Does not capture local structural details.
[67] |  ScanNet |  Developed an RGB-D video dataset with extensive annotations for deep learning applications. |  ScanNet dataset |  3D object classification, semantic voxel labeling, CAD model retrieval |  Limited diversity in scene views and semantic annotations.
[68] |  OctNet |  Proposed a sparse 3D data representation that allows for deep, high-resolution 3D convolutional networks. |  Not specified |  3D object classification, orientation estimation, point cloud labeling |  Focus on sparse data might not generalize to denser datasets.
[53] |  PointNet++ |  Extended PointNet to capture local structures using a hierarchical neural network. |  Challenging benchmarks of 3DPCs |  Enhanced 3D recognition tasks |  Performance drops with non-uniform point densities.
[54] |  PointVGG |  Introduced point convolution and pooling methods to adapt image-based techniques for point clouds. |  Challenging benchmarks of 3DPCs |  Object classification, part segmentation |  May not fully capture intricate geometric details.
[55] |  PointPAVGG |  A VGG-based network that incorporates a point attention mechanism for feature extraction from point clouds. |  ShapeNet, ModelNet |  Point cloud classification, segmentation |  Increased computational demand due to complex feature integration.
[58] |  SplatNet |  Utilized sparse bilateral convolutional layers for processing point clouds in a high-dimensional lattice. |  Not specified |  3D segmentation |  Scaling issues with memory and computational cost in larger lattices.
[60] |  FoldingNet |  Developed a deep auto-encoder with a folding-based decoder for unsupervised learning on point clouds. |  Not specified |  Unsupervised learning, 3D object reconstruction |  Generic decoder structure may not work for all point cloud types.
[61] |  PU-Net |  Presented a method for upsampling 3DPCs using multi-level features. |  Synthesis and scan data |  Point cloud upsampling |  Focus on upsampling may not improve other manipulations.
[69] |  SO-Net |  Built a permutation invariant architecture using Self-Organizing Maps for point cloud processing. |  Not specified |  Point cloud reconstruction, classification, segmentation, shape retrieval |  Requires tuning of the network’s receptive field.
[70] |  PIXOR |  Developed a real-time 3D object detection system from point clouds using BEV, optimized for autonomous driving. |  KITTI, large-scale 3D vehicle detection benchmark |  Real-time 3D object detection in autonomous driving |  Primarily optimized for vehicle detection, may not generalize to other object types.
[59] |  SGPN |  Introduced a network for 3D instance segmentation by predicting point groupings and semantic classes. |  Various 3D scenes |  3D instance segmentation, object detection, semantic segmentation |  May struggle with highly cluttered or complex environments.
[71] |  VoxelNet |  Unified feature extraction and bounding box prediction for 3DPCs into a single deep network. |  KITTI |  3D detection of cars, pedestrians, and cyclists |  High computational cost due to dense voxelization of point clouds.
[63] |  Hand PointNet |  Developed a hand pose regression network using 3DPCs to capture complex hand structures. |  Three challenging hand pose datasets |  3D hand pose estimation |  Focuses on hand pose, limiting its applicability to other forms of point cloud processing.
[64] |  PointNetVLAD |  Proposed a network for global descriptor extraction from point clouds for place recognition. |  Created benchmark datasets for point cloud based retrieval |  Place recognition from point clouds |  May not be as effective in highly dynamic environments.
[72] |  PPFNet |  Introduced a network for learning globally informed 3D local feature descriptors from point clouds. |  Not specified |  Finding correspondences in unorganized point clouds |  Dependency on the quality of local features and global context understanding.
[62] |  PointGrid |  Combined point and grid-based approaches to recognize 3D models from point clouds. |  Popular shape recognition benchmarks |  3D model recognition, classification, and segmentation |  Might not handle extremely sparse or irregular point clouds effectively.
[73] |  PointFusion |  Leverages image and point cloud data for 3D object detection without dataset-specific tuning. |  KITTI, SUN-RGBD |  Generic 3D object detection across diverse environments |  The fusion process can be complex and computationally intensive.
[74] |  Frustum PointNets |  Operates on raw point clouds for 3D object detection, combining 2D and 3D detection methods. |  KITTI, SUN RGB-D |  3D object detection in both indoor and outdoor scenes |  Might encounter difficulties with very sparse point clouds or heavy occlusions.
[75] |  3DFeat-Net |  Learns 3D feature detectors and descriptors for point cloud matching using weak supervision. |  Outdoor Lidar datasets |  Point cloud matching for localization and mapping |  Performance highly dependent on the effectiveness of weak supervision learning mechanisms.
[56] |  SpiderCNN |  Developed SpiderConv units to handle 3DPCs |  ModelNet40 |  3DPC classification and segmentation |  May struggle with very large or noisy datasets
[57] |  PointCNN |  Introduced a X-transformation to handle unordered point clouds for CNNs |  Multiple challenging benchmark datasets |  Feature learning from point clouds |  Dependent on the effectiveness of the X-transformation
[76] |  SqueezeSeg |  Created a CNN pipeline for semantic segmentation of LiDAR data |  KITTI, GTA-V (simulated) |  Semantic segmentation in autonomous driving |  Reliance on synthetic data for improved accuracy
[77] |  2DPASS |  Boosted point cloud learning via 2D image fusion during training |  SemanticKITTI, NuScenes |  Semantic segmentation of point clouds |  Requires multi-modal data during training, complex implementation
[65] |  PREDATOR |  Focused on low-overlap point cloud registration with deep attention |  3DMatch benchmark |  Point cloud registration |  Limited to pairwise registration, may not generalize beyond benchmark scenarios
[66] |  3DIoUMatch |  Implemented a semi-supervised 3D object detection with teacher-student learning |  ScanNet, SUN-RGBD, KITTI |  3D object detection |  High task complexity and dependency on initial label quality
[78] |  PointPoseNet |  Developed a pipeline for 6D object pose estimation from point clouds |  LINEMOD, Occlusion LINEMOD |  6D pose estimation |  Limited to known objects in complex scenes, sensitivity to occlusions
[79] |  ASAP-Net |  Enhanced spatio-temporal modeling of point clouds |  Synthia, SemanticKITTI |  Point cloud sequence segmentation |  Requires specific attention and structure-aware algorithms for optimal performance
[80] |  MUSCLE |  Proposed a compression algorithm for LiDAR data using spatio-temporal relationships |  UrbanCity, SemanticKITTI |  LiDAR data compression |  Efficiency depends on the variability of the input data
[81] |  PC-RGNN |  Addressed sparse and partial point clouds for 3D detection using GNNs |  KITTI |  3D object detection |  Highly dependent on the quality of initial point cloud data

###  2.5 Pre-trained 2D models

Xu et al. [82] demonstrate how pretrained 2D image models can be adapted for
3DPC understanding with minimal modification. By extending 2D ConvNets and
vision transformers to handle 3D data, the method involves inflating 2D
filters to 3D and only finetuning specific layers like the input, output, and
normalization layers. This approach leverages the deep feature representations
learned from large-scale 2D image datasets, enabling the models to perform
competitively on 3DPC tasks such as classification and segmentation, while
significantly reducing the training time and data requirements compared to
training from scratch. The transferability is facilitated by the similarity in
feature representation between the 2D and 3D tasks, despite their differences
in data modality.

Besides, shape classification and part segmentation pose significant
challenges in CV. While trained convolutional neural networks have shown
impressive performance on regular grid data like images, accurately capturing
shape information and geometric representation from irregular and disordered
point clouds is problematic. To address this, [54] proposes point convolution
(Pconv) and point pooling (Ppool) techniques inspired by convolution and
pooling in image processing, specifically designed for point clouds to learn
high-level features. Pconv gradually magnifies receptive fields to capture
local geometric information, while Ppool tackles the disorder of point clouds
using a symmetric function that aggregates points progressively for a more
detailed local geometric representation. Our novel network, named PointVGG,
incorporates Pconv, Ppool, and a graph structure for feature learning in point
clouds, and is applied to object classification and part segmentation tasks.
Experimental results demonstrate that PointVGG achieves state-of-the-art
performance on challenging benchmarks of 3DPC. Moreover, extracting high-level
features from disordered point cloud data using pre-trained 2D CNN remains
challenging. To address this, [55] proposes a VGG-based network called point
positional attention VGG (PointPAVGG), inspired by the classical VGG network.
Our approach combines global and local features by extracting local geometric
information from every sphere domain and analyzing the global position score
using our point attention (PA) module. PointPAVGG, with its graph structure
point cloud feature extraction and PA, is applied to point cloud
classification and segmentation tasks. Through comprehensive experiments on
ShapeNet and ModelNet, our method demonstrates superior performance, achieving
state-of-the-art results in classification and segmentation tasks.

The work in [83] presents a method for automatic detection of manhole covers
from mobile mapping point cloud data, which are large-scale spatial databases
used for various purposes. The method uses a fully CNN with DTL and a
simplified class activation mapping (CAM) location algorithm to accurately
determine the position of manhole covers. Different source model
architectures, such as AlexNet, VGG-16, Inception-v3, and ResNet-101, are
assessed. Results showed that VGG-16 achieved the best detection performance
among others, with recall, precision, and F2-score of 0.973 each. The approach
also achieves a horizontal 95% confidence interval of 16.5 cm for location
performance using VGG-16 architecture. The study highlights the importance of
incorporating geometric information channels in the ground image for improved
detection and location accuracy. Aerial imaging using drones, an efficient
timely data collection after natural hazards for post-event management,
provides detailed site characterization with minimal ground support, but
results in large amounts of 2D orthomosaic images and 3DPC. Effective data
processing workflows are needed to identify structural damage states. Liao et
al. [84] introduce two DL models, based on 2D and 3D CNNs, for post-windstorm
classification. DTL from AlexNet and VGGNet is used for the 2D CNNs, while a
3D fully convolutional network (3DFCN) with skip connections is developed and
trained for point cloud data. The models are compared using quantitative
performance measures, and the 3DFCN shows greater robustness in detecting
different damage classes. This highlights the importance of 3D datasets,
particularly depth information, in distinguishing between different damage
states in structures.

In [85], a DL-based approach is proposed, which involves projecting 3DPC into
2D rendering views and then feeding them into a CNN for quality score
prediction. DTL is employed to leverage the capabilities of VGG-16 trained on
the ImageNet database. The performance of the proposed model is evaluated on
two benchmark databases, ICIP2020 and SJTU, and the results show a strong
correlation between the predicted and subjective quality scores, outperforming
state-of-the-art point cloud quality assessment models. [86] addresses the
problem of learning outdoor 3DPC from monocular data using a sparse ground-
truth dataset. A DL-based approach called Pix2Point is proposed, which uses a
2D-3D hybrid neural network architecture and a supervised end-to-end
minimization of an optimal transport divergence between point clouds. The
proposed approach outperformed efficient monocular depth methods when trained
on sparse point clouds. The paper highlights the potential of DL for monocular
3DPC prediction and its ability to handle complete and challenging outdoor
scenes. The encoding block, in the target model, consists of convolution,
pooling, and normalization layers to extract feature descriptions from the RGB
image. These features are then processed by a fully connected layer to obtain
a preliminary set of 3D point coordinates. Several source models, based on
VGG, DenseNet, and ResNet architectures, are explored and compared, which are
referred to as backbones.

Balado et al. [87] propose in 2020, a method that minimizes the use of point
cloud samples for training CNNs by converting point clouds to images (pc-
images). This enables the generation of multiple samples per object through
multi-view, and the combination of pc-images with images from online datasets
such as ImageNet and Google Images. Results suggest keeping some point cloud
images in training; even 10% can lead to high classification accuracy. The
work presented in [88] showcases a prototypical implementation of a service-
oriented architecture for classifying indoor point cloud scenes in office
environments. This approach utilizes multi-view techniques for semantic
enrichment of captured scans and subsequent classification. The approach is
tested using a pretrained CNN model, Inception V3, to classify common office
furniture objects such as chairs, sofas, and desks in 3DPC scans. The results
show that the approach can achieve acceptable accuracy in classifying common
office furniture, based on RGB cubemap images of the octree partitioned areas
of the 3DPC scan. Additional methods for web-based 3D visualization, editing,
and annotation of point clouds are also discussed.

The authors in [89] propose a visual recognition and location method for
object detection in soft robotic manipulation using RGB-D information fusion.
The method involves scanning and reconstructing the environment using ORB-
SLAM2, constructing an object feature database, matching point clouds using
the iterative closest point (ICP) algorithm, identifying regions of interest,
and using the inception-v3 model and DTL for object recognition. The position
of the object relative to the camera is obtained through correspondence
between color information and point cloud data. The method showed that objects
belonging to the same object have much lower matching error compared to those
not belonging to the same object, and successful object identification and
location were achieved through color recognition. Moving on, the authors in
[90] adopt the ResNet50 [91] pretrained on the ImageNet data set to extract
deep features from each feature image and obtain five multi-scale and multi-
view (MSMV) deep features per point.

Table LABEL:tab3 presents a summary of DTL-based 3DPC models.

Table 3: Summary of DTL-based 3DPC models.  Category |  Model |  Highlights |  Limitations
---|---|---|---
Point-based methods |  AltasNet [12] |  In AtlasNet, a surface representation is inferred by regarding a 3D shape as a collection of parametric surface elements. |  The reiterated many times largely determines the reconstruction.
MSN [92] |  A sampling algorithm combines a set of parametric surface elements, which MSN predicts, with the partial input. |  Fine-grained details of object shape aren’t generated successfully.
ASHF-Net [93] |  A hierarchical folding decoder with the gated skip-attention and multi-resolution completion target to exploit the local structure details of the incomplete inputs is proposed by ASHF-Net. |  The decoder [113] obtains unstructured predictions, and the surface of the results doesn’t remain smooth.
Point-based methods |  PCN [94] |  The coarse-to-fine completion is performed by PCN, which combines the fully connected network and FoldingNet. |  The synthesis of shape details is not achievable.
SA-Net [95] |  Hierarchical folding in the multi-stage points generation decoder is proposed by SA-Net. |  The implicit representation of the target shape from the intermediate layer, which helps refine the shape in the local region, is difficult to interpret and constrain.
FoldingNet [60] |  It is commonly assumed in a two-stage generation process that a 2D-manifold can recover 3D objects. |  Explicitly constraining the implicit intermediate is challenging.
SK-PCN [96] |  The global structure is acquired by predicting the 3D skeleton with SK-PCN, and the surface completion is achieved by learning the displacements of skeletal points. |  The overall shapes are the sole focus of the meso-skeleton.
Point-based methods |  GRNet [97] |  Unordered point clouds are regularized by GRNet, which introduces 3D grids as intermediate representations. |  The resolution still governs it. A significant computational cost is incurred when a higher resolution is used.
VE-PCN [98] |  The structure information is incorporated into the shape completion by VE-PCN through the utilization of edge generation. |  High frequency components are the sole focus of the edges.
Point-PEFT [99] |  Introduces a Parameter-Efficient Fine-Tuning method for 3D models, minimizing adaptation costs for downstream tasks with minimal trainable parameters. |  While effective, the specialized method’s broader applicability and long-term adaptability across diverse domains remain unproven.
DAPT [100] |  Implement a Dynamic Adapter for point cloud analysis, offering efficient parameter use and reducing training resources significantly. |  Although reducing trainable parameters, the adaptability and effectiveness in extremely diverse environments is yet to be fully assessed.
AgileGAN3D [101] |  A novel framework for 3D artistic portrait stylization using unpaired 2D exemplars and advanced 3D GAN models. |  The dependency on the quality and diversity of the 2D exemplars might limit the model’s versatility in less controlled scenarios.
|  3D-TRAM [102] |  Combines transfer learning with a memory component to enhance 3D reconstruction from 2D images, leveraging CAD models for better accuracy. |  The method’s performance can vary significantly with the complexity of the scene and the quality of the available CAD models.

###  2.6 Pre-trained 3D models

Implementing a standardized approach to 3DPC neural network design has the
potential to yield comparable advancements seen in the extensive research
conducted on pre-training visual models, especially in the image domain.
However, compared to the 2D domain, the design of neural networks for point
cloud data is less mature, as evidenced by the numerous new architectures
proposed recently. This is due to several factors, including the challenge of
processing unordered sets [103], the choice of neighborhood aggregation
mechanism, which could be hierarchical [53, 104, 105], spatial CNN-like [106,
56, 57, 107], spectral [108, 109], or graph-based [110, 111], and the fact
that points are discrete samples of an underlying surface, which has led to
the consideration of continuous convolutions [112, 113].

While pre-training image models have been successful in achieving high levels
of prosperity, pre-training 3D models is still in the developmental stage. To
address this issue, numerous researchers have explored various SSL mechanisms
that utilize different pretext tasks, such as solving jigsaw puzzles [114],
estimating orientation [115], and reconstructing deformations [116]. Drawing
inspiration from pre-training strategies in the image domain, several
approaches have been proposed in the 3D domain, including point contrast
[117], which uses a contrastive learning principle, and OcCo [118], Point-BERT
[119], and Point-M2AE [120], which introduce reconstruction pretext tasks to
facilitate better representation learning. However, the lack of available data
in the 3D domain remains a significant obstacle in developing more effective
pre-training strategies.

black For instance, [117] focuses on developing effective representation
learning techniques for point cloud data. The authors draw inspiration from
the success of standardized neural architectures in the 2D domain, such as
VGGNet [simonyan2014very] and ResNet/ResNeXt [91], and conjecture that a
similar standardization of point cloud neural network design could enable
similar progress. However, compared to the 2D domain, the design of neural
networks for point cloud data is less mature, as evidenced by the numerous new
architectures proposed recently. This is due to several factors, including the
challenge of processing unordered sets [ravanbakhsh2016deep, 52, 103], the
choice of neighborhood aggregation mechanism (e.g., hierarchical [53, 104,
105], spatial CNN-like [106, 56, 57, 107], spectral [108, 109], or graph-based
[110, 111]), and the fact that points are discrete samples of an underlying
surface, which has led to the consideration of continuous convolutions [112,
113].

In this respect, Choy et al. proposed the Minkowski Engine [121], which is an
extension of sub-manifold sparse convolutional networks [122] to higher
dimensions. By facilitating the adoption of common deep architectures from 2D
vision, sparse convolutional networks could help standardize DL for point
cloud. In [117], the authors use a unified UNet [123] architecture built with
Minkowski Engine as the backbone network in all our experiments and show that
it can seamlessly transfer between tasks and datasets. [120] The paper
proposes Point-M2AE scheme, a multi-scale masked autoencoder (MAE) pre-
training framework for SSL of 3DPC. Unlike standard auto-encoder (AE)-based
transformers, Point-M2AE modifies the encoder and decoder into pyramid
architectures to model spatial geometries and capture fine-grained and high-
level semantics of 3D shapes. The encoder uses a multi-scale masking strategy
for consistent visible regions across scales and a local spatial self-
attention [124] mechanism during fine-tuning. The lightweight decoder
gradually upsamples point tokens with skip connections from the encoder,
promoting reconstruction from a global-to-local perspective. Point-M2AE
achieves 92.9% accuracy on ModelNet40 using a linear SVM. Fine-tuning enhances
performance to 86.43% on ScanObjectNN and provides benefits in various tasks,
such as few-shot classification, part segmentation, and 3D object detection.

The reference [125] introduces the PointCLIP method, which encodes point
clouds by projecting them into multi-view depth maps and aggregates view-wise
zero-shot predictions for knowledge transfer from 2D to 3D. An inter-view
adapter is designed to extract global features and adaptively fuse few-shot
knowledge from 3D into CLIP pre-trained in 2D. Fine-tuning the lightweight
adapter in few-shot settings significantly improves PointCLIP’s performance.
PointCLIP also exhibits complementary properties with classical 3D-supervised
networks, and ensembling with baseline models further boosts performance,
surpassing state-of-the-art models. PointCLIP is a promising alternative for
effective 3DPC understanding with low resource cost and data regime, as
demonstrated through experiments on ModelNet10, ModelNet40, and ScanObjectNN
datasets. Huang et al. [126] proposes a method called MF-PointNN for surrogate
modeling of melt pool in metallic additive manufacturing. Melt pool modeling
is important for uncertainty quantification and quality control in metal
additive manufacturing, but finite element simulation for thermal modeling can
be time-consuming. MF-PointNN is a multi-fidelity approach that combines low-
fidelity analytical models and high-fidelity finite element simulation data
using DTL. A basic PointNN is first trained with low-fidelity data to
establish correlation between inputs and thermal field of analytical models.
Then, the basic PointNN is updated and fine-tuned using a small amount of
high-fidelity data to build the MF-PointNN. This latter efficiently maps input
variables and spatial positions to thermal histories, allowing for efficient
prediction of the three-dimensional melt pool. Results of melt pool modeling
for Ti-6Al-4V in electron beam additive manufacturing under uncertainty show
the effectiveness of the proposed approach.

###  2.7 Evaluation metrics

Several metrics have been reported to evaluate the performance of different
DTL approaches for various 3DPC tasks. F1 measures and overall accuracy (OA),
are the most frequently used measures to assess the performance of algorithms
over benchmark datasets. These metrics are used for many purposes, including
segmentation, classification, and registration. F1 measures give an idea about
the behavior of the precision and recall curve, whereas OA conveys the mean
accuracy for instances of the test. Besides, intersection over union (IoU) and
mean IoU have been extensively used, especially for object detection and
segmentation purpose. For instance, in [127], authors have used mean IoU
between point-wise ground truth and prediction. In [128], authors have
computed IoU for different shapes. It is calculated as the average of IoUs of
all parts in a shape. Furthermore, for some particular categories, mean IoUs
(mIoUs) are obtained by finding out the average of all IoUs shapes. Other
metrics, instance mIoU (Ins. mIoU) and category mIoU (Cat. mIoU) are also
introduced and computed by calculating the average of all shapes and the
average of mIoUs for all categories, respectively.

Metrics such as OA and mIoU provide a holistic view of the model’s performance
across different categories. Additionally, the frame per second (FPS) metric
evaluates the efficiency and speed of real-time applications, while root mean
squared error (RMSE) and mean absolute percentage error (MAPE) are used for
precision in measurement tasks. Moving on, consistency rate (CR), consistency
proportion (CP), and weight coverage (WCov) are advanced evaluation metrics
used to assess the quality of 3D point cloud reconstructions. CR measures the
rate at which a reconstructed scene maintains geometric consistency across
different views or instances. CP evaluates the proportion of consistent data
points within the entire dataset, providing a sense of how uniformly the model
performs. WCov assesses the coverage of the weighted areas in the point cloud,
indicating how well the model captures the essential features and structures
of the scene.

To evaluate the performance or semantic segmentation, authors in [129] have
used errors of commission and errors of omission metrics over the CMP façade
dataset. Frame per second (FPS) metric is used by [130] on the ouster LiDAR-64
dataset. In addition to this, Zong et al. [131] have used average precision
(mPrec) and average recall (mRec) to test their segmentation method. Moreover,
the best performance for OA on the Shapenet dataset is 94.5% [132]. However,
the highest precision (85.99%) is achieved on CMP facade dataset [129]. The F1
score has been calculated on different datasets, viz. CMP facade, ISPRS,
Semantic 3D dataset, ModelNet4, and other. The best performance has been
reported by [133] on ModelNet40. The aforementioned metrics are not only
applicable to 3DPC tasks but also to other artificial intelligence-related
tasks such as classification, segmentation, and other. However, subsequent
subsections thoroughly elaborate on specific metrics relevant to 3DPC tasks.

####  2.7.1 Distance metrics

Distance metrics play a pivotal role in the processing of point clouds for
tasks such as nearest-neighbor search, clustering, and segmentation. They
provide a measure of similarity or dissimilarity between points in space,
influencing the outcomes of many algorithms.

####  2.7.2 Euclidean Distance

Defined as the square root of the sum of the squared differences between the
coordinates of two points:

$d(\mathbf{p},\mathbf{q})=\sqrt{(p_{1}-q_{1})^{2}+(p_{2}-q_{2})^{2}+\cdots+(p_{n}-q_{n})^{2}}$

where $\mathbf{p}=(p_{1},p_{2},\ldots,p_{n})$ and
$\mathbf{q}=(q_{1},q_{2},\ldots,q_{n})$ represent the coordinates of two
points in an $n$-dimensional space. The Euclidean distance
$d(\mathbf{p},\mathbf{q})$ calculates the "as-the-crow-flies" distance between
the two points, effectively measuring the length of the straight line segment
that connects them.

####  2.7.3 Manhattan Distance

Also known as taxicab or city block distance, it is the sum of the absolute
differences of their coordinates:

$d(\mathbf{p},\mathbf{q})=|p_{1}-q_{1}|+|p_{2}-q_{2}|+\cdots+|p_{n}-q_{n}|$

This metric is ideal for grid-based and urban environments where travel paths
are constrained to grid layouts.

####  2.7.4 Mahalanobis Distance

Takes into account the correlations of the dataset and is scale-invariant,
defined as:

$d(\mathbf{x},\mathbf{\mu})=\sqrt{(\mathbf{x}-\mathbf{\mu})^{T}\mathbf{S}^{-1}(\mathbf{x}-\mathbf{\mu})}$

where:

  * •

$\mathbf{x}$ is the vector representing the point whose distance from the
distribution is being measured.

  * •

$\mathbf{\mu}$ is the mean vector of the distribution, representing the
central tendency.

  * •

$\mathbf{S}^{-1}$ is the inverse of the covariance matrix $\mathbf{S}$ of the
distribution, which adjusts the distance measure to account for the spread and
orientation of the data points.

This formulation accounts for the shape of the data distribution, correcting
distances based on how data spreads and correlates across dimensions, thus
providing a more nuanced measure of distance.

####  2.7.5 Hausdorff distance

Let us assume that there are two point clouds $S_{1}$ and $S_{2}$ in $R^{3}$
with $N_{1}$ and $N_{2}$ points in each cloud, respectively. In order to
compare these point sets, one of the earliest methods that was proposed is the
Hausdorff distance ($\mathcal{D}_{H}$), which is based on finding the minimum
distance from a point to a set [134]:

$d(x,y)=\left\|x-y\right\|_{2}$ (3) $D(x,S)=\underset{y\in S}{\text{min
}}d(x,y)$ (4)

and calculates a symmetric max min distance [135]:

$\mathcal{D}_{H}(S_{1},S_{2})=\max\left\\{\underset{a\in
S_{1}}{\max}D(a,S_{2}),\underset{b\in S_{2}}{\max}D(b,S_{1})\right\\}$ (5)

####  2.7.6 Chamfer distance

The Chamfer distance (CD) is a metric used to evaluate the similarity between
two sets of points in space. It considers the distance between each point in
both sets and finds the nearest point in the other set, then sums the square
of these distances.CD is commonly used in the ShapeNet’s shape reconstruction
challenge. The CD between two point clouds, S1 and S2, is defined as [136]:

$\mathcal{D}_{CD}\left(S_{1},S_{2}\right)=\frac{1}{\left|S_{1}\right|}\sum_{x\in
S_{1}}\underset{y\in
S_{2}}{\min}\left\|x-y\right\|_{2}^{2}+\frac{1}{\left|S_{2}\right|}\sum_{y\in
S_{2}}\underset{x\in S_{1}}{\min}\left\|x-y\right\|_{2}^{2}$ (6)

####  2.7.7 Earth mover’s distance

In contrast to the Hausdorff distance and its derivative methods that involve
identifying the closest neighboring point for each point, the Earth mover’s
distance (EMD) or the Wasserstein distance operates by establishing a one-to-
one correspondence (i.e., bijection represented by $\zeta$) between the two
sets of points. The objective is to minimize the total distance between the
corresponding points in the two sets [134].

$\mathcal{D}_{EMD}(S_{1},S_{2})=\underset{\zeta:S_{1}\longrightarrow
S_{2}}{\min}\sum_{a\in S_{1}}\left\|a-\zeta(a)\right\|_{2}$ (7)

###  2.8 Datasets

There is a considerable number of datasets available for performing various
3DPC tasks using state-of-the-art techniques. A detail about datasets on which
DL has been implemented is reported in [31]. However, in our study, we have
considered the datasets for which DTL has been incorporated to carry out 3DPC
tasks. These include widely used benchmarks like, PointNet [137], ShapeNet
[117], ModelNet [128], ScanNet [117], ISPRS [138], KITTI 3D object detection
[130], Campus3D [139] and semantic 3D dataset [140]. Apart from these well-
known benchmarks, in this study, newly introduced datasets are also included.
These comprises of Scaffolds dataset [140], CMP façade dataset [129], DFC
[138], Ouster LiDAR-64 [130], Santa Monica point cloud data [141], UTD-MHAD
[142], PointDA-10 [143], NYU [144], SHREC2018 [145], and EndoSLAM dataset
[146].

These datasets are used for 3DPC segmentation (semantic, panoptic, and
instance), classification, and object detection and tracking and contain
features beneficial for carrying out related tasks. For instance, datasets
like ModelNet 40, ShapeNet and ScanNet, among others, are used for shape
classification and contain columns like sample size, classes, training,
testing data percentage, and other useful features. In addition to this,
SHREC2018 and EndoSLAM are medical-related datasets. The latter is created
through the recording of multiple endoscope cameras for six porcine organs, as
well as synthetically generated records. However, the former, the SHREC2018
protein dataset, contains 2267 protein details. To record the details, the
protein data bank (PDB) format has been used. The datasets, references, and
available links are given in Table 4.

Table 4: Dataset and evaluation metrics used by DTL methods for various 3DPC tasks.  Ref. |  Dataset |  Dataset availability |  Evaluation metric |  Best performance (%)
---|---|---|---|---
[117] |  ShapeNet, ScanNet |  http://www.scan-net.org/#code-and-data |  - |  -
[137] |  PointNet [52], ShapeNet [147], and ModelNet-40 |  http://stanford.edu/rqi/pointnet/
https://shapenet.org/
https://modelnet.cs.princeton.edu/ |  - |  -
[148] |  UTD-MHAD |  http://www.utdallas.edu/kehtar/Kinect2Dataset.zip |  Recognition rate |  92.50
[133] |  ModelNet40 |  https://modelnet.cs.princeton.edu/ |  F1, precision, recall |  F1 (91)
[149] |  ModelNet |  https://modelnet.cs.princeton.edu/ |  MAE-T, MAE-F |  CA (97.6)
[128] |  ShapeNet, ModelNet |  https://shapenet.org/
https://modelnet.cs.princeton.edu/ |  OA |  90.40
[138] |  ISPRS, DFC |  https://www.isprs.org/data/ |  F1, OA |  F1 (83.62), OA (89.84)
[130] |  KITTI 3D object detection, Ouster LiDAR-64 |  https://ouster.com/resources/lidar-sample-data/ |  FPS (frame per second) |  FPS (30.6)
[150] |  ImageNet, KITTI 2D, VLP-16 |  http://www.image-net.org/about-stats |  IoU, F1, mAP |  mAP(81.27), IoU(65.11), F1 (0.82)
[140] |  Semantic3D dataset [22], Scaffolds dataset |  http://www.semantic3d.net/view_dbase.php?chl=1 |  F1, precision, recall |  F1 (90.84)
[139] |  Campus3D |  https://3d.nus.app/ |  OA, IoU, mIoU, CR, CP and WCov |  OA(90.9), mIoU(61.5)
[129] |  CMP façade dataset |  https://cmp.felk.cvut.cz/tylecr1/facade/ |  Precision, recall, F1, Errors of commission, Errors of omission |  Precision (85.97), recall (89.80), F1 (87.85)
[141] |  Santa Monica point cloud data, KITTI 2D, VLP-16 |  http://www.cvlibs.net/datasets/kitti/ |  F1, precision, recall |  -
[143] |  PointDA-10 |  - |  Accuracy |  49.70
[145] |  SHREC2018 |  - |  precision, recall, NN, T1, T2, EM, DCG |  -
[146] |  EndoSLAM dataset |  https://github.com/CapsuleEndoscope/EndoSLAM |  RMSE |  -
[131] |  3DPC tunnel dataset |  - |  Average precision (mPrec), average recall (mRec) |  mPrec (71.2)
[151] |  2014 scans |  - |  MAPE, RMSE |  MAPE (0.416), RMSE (0.112)
[152] |  NYU |  - |  RMSE, Accuracy |  (81.8),
[132] |  ShapeNetPart |  https://shapenet.org/ |  OA, mIoU, Cat. mIoU, Ins. mIoU |  OA (94.5)

##  3 Overview of DTL

Arguably one of the top success stories of DL is DTL. Many applications in
language and vision have benefited from the discovery that pretraining a
network on a rich source set (e.g., ImageNet) can help boost performance once
fine-tuned on a typically much smaller target set. Yet, very little is known
about its usefulness in 3DPC understanding. We see this as an opportunity
considering the effort required for annotating data in 3D.

###  3.1 Domain adaptation

DA has shown significant improvements in various ML and CV tasks, such as
classification, detection, and segmentation. However, there are limited
methods that have achieved DA directly on 3DPC data, to the best of our
knowledge. The challenge of point cloud data lies in its rich spatial
geometric information, where the semantics of the entire object are
contributed by regional geometric structures. Most general-purpose DA methods
that focus on global feature alignment and disregard local geometric
information may not be suitable for 3D domain alignment. Wu and their
colleagues [153] discusse the challenges of generating and annotating large
amounts of real-world data for DL-based approaches in robotics CV tasks. To
overcome this, the authors propose using simulation-to-reality (sim2real) DTL
for point cloud data in an industrial application case. They provide insights
on generating and processing synthetic point cloud data to improve model
performance when transferred to real-world data. The issue of imbalanced
learning is also investigated, and the authors propose a novel patch-based
attention network as a strategy to address this problem. Another work in
[154], presents a new method called DTL-based sampling-attention network
(TSANet) for semantic segmentation of 3D urban point clouds to facilitate the
development of smart cities. The method includes a segmentation model with
point downsampling–upsampling structure, embedding method, attention
mechanism, and focal loss for feature processing and learning. The DTL
technique aims to reduce data requirements and labeling efforts by leveraging
prior knowledge. The method is evaluated on a realistic point cloud dataset of
Cambridge and Birmingham cities, demonstrating promising performance,
surpassing other state-of-the-art models in terms of accuracy and mean IoU.
[116] introduces SSL for DA in 3D perception problems, specifically on point
clouds. It describes a new family of pretext tasks called deformation
reconstruction, inspired by sim-to-real transformations, and proposes a novel
training procedure called point cloud mixup (PCM) motivated by the MixUp
method for labeled point cloud data. Evaluations on DA datasets for
classification and segmentation show significant improvement over existing and
baseline methods, demonstrating the effectiveness of SSL-PCM-DA on point
clouds. Similarly, [155] introduces a new model called SqueezeSegV2 for point
cloud segmentation that is more robust to dropout noise in LiDAR point clouds.
The improved model structure, training loss, batch normalization, and
additional input channel result in significant accuracy improvement when
trained on real data. To overcome the challenge of limited labeled point-cloud
data, the proposed scheme employs DA training pipeline, consisting of learned
intensity rendering, geodesic correlation alignment, and progressive domain
calibration. When trained on real data, the new model exhibits significant
segmentation accuracy improvements over the original SqueezeSeg. Moreover,
when trained on synthetic data using the proposed DA pipeline, the test
accuracy on real-world data nearly doubles. A similar scheme for segmentation
is proposed in [156], introducing LiDARNet, a boundary-aware DA model tailored
for semantic segmentation of LiDAR point cloud data. The model uses a two-
branch structure to extract domain private and shared features, and
incorporates Gated-SCNN to learn boundary information during segmentation. The
domain gap is further reduced by learning a mapping between domains using
shared and private features. They also introduce a new dataset, SemanticUSL,
for DA in LiDAR semantic segmentation, which has the same format and ontology
as Semantic KITTI. Experiments on real-world datasets show that LiDARNet
achieves comparable performance on the SD and significant performance
improvement (8-22% mIoU) on the TDs after adaptation.

Other researchers have advanced unsupervised DA approaches, demonstrated by
the method outlined in [157], aimed at enhancing semantic labeling accuracy
for 3DPC in autonomous driving scenarios. The proposed approach uses a
complete and label approach, leveraging a sparse voxel completion network
(SVCN) to recover underlying surfaces of sparse point clouds and transfer
semantic labels across different LiDAR sensors. The approach does not require
manual labeling for training pairs and introduces local adversarial learning
to the model surface prior. Experimental results on a new benchmark dataset
show significant performance improvements ranging from 8.2% to 36.6% compared
to previous DA methods. Likewise, [158] introduces PointDAN, a 3D DA network
tailored for point cloud data, aligning global and local features at multiple
levels to achieve domain alignment. It introduces a Self-adaptive node module
for local alignment, which models discriminative local structures, and a node-
attention module for hierarchical feature representation. For global
alignment, an adversarial-training strategy is employed. PointDAN outperforms
state-of-the-art DA methods in terms of adapting 3DPC data across domains, as
demonstrated on the benchmark dataset PointDA-10, created by the authors. Zhao
et al. [159] proposes an end-to-end framework called ePointDA for simulation-
to-real DA (SRDA) in LiDAR point cloud segmentation. ePointDA consists of
three modules: self-supervised dropout noise rendering, statistics-invariant
and spatially-adaptive feature alignment, and transferable segmentation
learning. The framework bridges the domain shift at pixel-level and feature-
level, without requiring real-world statistics. Experimental results on
adapting from synthetic to real datasets demonstrate the superiority of
ePointDA in LiDAR point cloud segmentation. Xu and their colleagues [160]
propose semantic point generation to enhance the reliability of LiDAR-based
object detectors against domain shifts in autonomous driving. The scheme
generates semantic points to recover missing parts of foreground objects
caused by occlusions, low reflectance, or weather interference. By merging the
semantic points with the original points, an augmented point cloud is
obtained, which significantly improves the performance of modern LiDAR-based
detectors in unsupervised DA tasks. The method also benefits object detection
in the original domain, surpassing KITTI when combined with PV-RCNN.

Lang et al. [161] utilize PointNets to represent point clouds organized in
vertical columns. The scheme, called PointPillars, achieves superior speed and
accuracy, surpassing KITTI benchmarks while running at a significantly higher
frame rate of 62 Hz. A faster version achieves state-of-the-art performance at
105 Hz, making it a suitable encoding approach for object detection in point
clouds in robotics applications like autonomous driving. For object detection
from 3DPC, the authors [162] propose a Semi-Supervised DA method for 3D object
detection that leverages a small amount of labeled target data to improve
adaptation performance. The method consists of an inter-DA stage, which uses a
Point-CutMix module to align point cloud distribution across domains, and an
intra-domain generalization stage, which employs intra-domain Point-MixUp in
semi-supervised learning to enhance model generalization on the unlabeled
target set. Experimental results show that the proposed scheme, with only 10%
labeled target data, outperforms a fully-supervised oracle model with 100%
target labels on the Waymo to nuScenes domain shift. However, Du et al. [163]
tackle the challenge of enhancing feature representation robustness in 3DPC by
employing the DA approach. The severe spatial occlusion and point density
variance in point cloud data make designing robust features crucial. The
proposed approach bridges the gap between the perceptual domain (real scene)
and the conceptual domain (augmented scene with non-occluded point clouds),
mimicking the functionality of human perception. Experimental results show
that this simple yet effective approach significantly improves the performance
of 3DPC object detection, achieving state-of-the-art results.

###  3.2 Fine-tuning

Fine-tuning stands as a ubiquitous DTL approach, aiding pretrained models to
adapt to new tasks through iterative training, often supplemented with
additional layers known as the target model. [164]. The recent research
studies, such as [165, 166, 167], have argued that even if the target task and
source task are different, transferring features via DTL approaches
outperforms random features selections. This characteristic has led to the
great success of DTL in different spheres of real-life applications such as
medical imaging [168, 4], recognition tasks like speech recognition [26],
hand-gesture [169], face recognition [170], and rcently, in the processing of
3DPC data [130, 171]. One of the major concerns raised in fine-tuning a
pretrained model is to identify which layer to fine-tune. The authors in [172]
have explicitly added regularization terms to the loss function for obtaining
the parameters of the fine-tuned model close to the original pretrained model.
AdaFilter [164], is an adaptive fine-tuning approach, which considers
criterion for optimization by selecting only a part of the convolutional
filters in the pretrained model. Furthermore, they have exploited a recurrent
gated network and considered activation of the previous layer for carefully
fine-tuning the desired convolutional filters only. The adaptive fine-tuning
scheme considers the similarity between the source and target tasks and
datasets to reuse more pretrained filters.

Many works that involve processing 3DPC data have exploited DTL for many
tasks, include 3D reconstructions of scaffolds [140], 3DPC instance
segmentation [131], classification of soft-story buildings using 3DPC data
[141], 3DPC understanding [117], 3D orientation recognition [149], SSL on
3DPCs [148], and remaining useful life prediction [151], among others.
Furthermore, some fine-tuning strategies involve all the pretrained
parameters, whereas others fine-tuning the last few layers [164], as illusted
in Fig. 7 (a). G Diraco et al. [151] have performed an extensive experiment
for a particular scenario in which the amount of data in the TD is small and
explored how it behaves if only part of the network is fine-tuned on the given
target dataset. They have updated the decoder of their model, which has been
trained on SDs and evaluated on the target dataset.

One factor contributing to the success of DTL is its ability to perform well
with smaller datasets. In support of this, the authors [141] have employed DTL
to address the challenge of training large parameters in deep convolutional
networks. Their experimental results demonstrate the transferability of DTL
from SDs to TD, as the dataset under consideration lacked sufficient data to
train large parameters. Additionally, compared to traditional training
methods, DTL consumes less time, making it a more efficient approach. To
support this claim, the authors noted that models such as VGGNet, Inception
and ResNet, which were trained using fine-tuning techniques, exhibited shorter
training times by triggering the early stopping mechanism. This improvement in
training time has enabled the workflow to quickly identify soft-story
buildings on a city scale. Furthermore, the authors have shown that DTL can
effectively reduce the risk of overfitting that can occur with DL techniques.
By leveraging a pretrained network, DTL can transfer knowledge and prevent the
model from memorizing the training data, leading to more robust and
generalizable performance. Moving on, Xiu et al. [173] propose using airborne
LiDAR to detect collapsed buildings during earthquake emergency response, as
DTL-based damage detection with aerial images has limitations in detecting
collapsed buildings with undamaged roofs. The authors develop a 3DPC-based
dataset for building damage detection and propose a general extension
framework and a visual explanation method to validate model decisions. The
results conducted using PointNet [52], PointNet++ [53] and DGCNN [174] show
that 3DPC-based methods can achieve high accuracy and are robust even with
reduced training data. The model also achieves moderate accuracy on another
dataset with different architectural styles without additional training.

###  3.3 Unsupervised DTL

Unsupervised DTL is mainly based on unsupervised domain adaptation (UDA) to
remove the need or addressing the issue of lack for labeled data and allows
any image to be used as a datapoint for any CV tasks [175, 176]. Numerous
methods have been proposed for performing UDA on 2D images, which can be
divided into two main categories: methods based on domain-invariant feature
learning and methods for learning domain mapping. The former [177, 178, 179,
180, 181] aim to minimize the discrepancy between two distributions in the
feature space, while the latter [182, 183, 184] use neural networks, such as
CycleGAN [53], to directly learn the translation from the SD to the TDs. In
[185], 2D translation is extended to depth images using a differential
contrastive learning strategy for preserving underlying geometries. Despite
their differences, these methods widely exploit domain adversarial training.
Additionally, several useful techniques, such as pseudo-labeling [186] and
batch normalization tailored for DA [187], have also been proposed.

Although there have been significant efforts made in UDA for 2D images and
depth, UDA on 3DPC is still in its early stages. UDA on point clouds involves
extending domain adversarial training from 2D images to 3DPC to align features
on both local and global levels [188]. Nevertheless, adversarial methods on
3DPC struggle to balance local geometry alignment and global semantic
alignment. CycleGAN [189] is utilized by both [159] and [190] to generate more
realistic LiDAR point clouds from synthetic data. This Sim2Real approach is
used to minimize feature distances between the SD and TD. The work in [157],
on the other hand, leverages segmentation on completed surface reconstructed
from sparse point cloud for better adaptation.

For object-level tasks, [191, 158] align global and local features, while
[155] and [190] project point clouds to 2D and birds-eye view, respectively,
to reduce sparsity. [163] creates a car model set and adapts their features
for detection object features, but only targets general car 3D detection on a
single point cloud domain. Recently, [192] published the first study targeting
UDA for 3D LiDAR detection. They identify the vehicle size as the domain gap
between KITTI [193] and other datasets and resize the vehicles in the data. In
contrast, [160] identifies point cloud quality as the major domain gap between
Waymo’s two datasets [194] and proposes a learning-based approach to close the
gap.

Recent works on UDA on point clouds mainly focus on designing suitable self-
supervised tasks on point clouds to facilitate learning domain invariant
features, which are discussed in detail in the following subsection. In
addition to UDA on object point clouds, several methods have been proposed to
address specific domain gaps on LiDAR point clouds. These methods commonly
address depth missing and sampling difference between sensors. ST3D [195]
presents a task-specific self-training pipeline with curriculum data
augmentation to further improve the adaptation process

Alternatively, most existing UDA approaches focus on uni-modal data, despite
the availability of multi-modal datasets. In [196], the authors propose a
cross-modal UDA (xMUDA) approach for 3D semantic segmentation, where both 2D
images and 3DPC are utilized. This is challenging as the two input modalities
are heterogeneous and can be affected differently by domain shift. In xMUDA,
the modalities learn from each other through mutual mimicking, separate from
the segmentation objective, to prevent the stronger modality from adopting
false predictions from the weaker one. The proposed xMUDA approach is
evaluated on various UDA scenarios, such as day-to-night, country-to-country,
and dataset-to-dataset shifts, using recent autonomous driving datasets.
Results show that xMUDA significantly improves over uni-modal UDA in all
tested scenarios and complements state-of-the-art UDA techniques. Saltori et
al. [197] introduces a novel source-free UDA (SF-UDA) framework for 3D object
detection using LiDAR point clouds, which does not require any annotations
from the SD or images/annotations from the TD. It addresses domain shift in
LiDAR data, which is not only due to changes in environment and object
appearances, but also to geometry (e.g., point density variations). SF-UDA3D
utilizes pseudo-annotations, reversible scale-transformations, and motion
coherency for DA. Experimental results on large-scale datasets, KITTI and
nuScenes, show that SF-UDA3D outperforms previous methods based on feature
alignment and state-of-the-art 3D object detection methods that use few-shot
target annotations or target annotation statistics. Also, [198] introduces
RefRec, an approach that investigate pseudo-labels and self-training for UDA
in point cloud classification. Instead of relying on multi-task learning,
RefRec proposes two innovations for effective self-training on 3D data. First,
it refines noisy pseudo-labels by matching shape descriptors learned from the
unsupervised task of shape reconstruction on both domains. Second, it proposes
a novel self-training protocol that learns domain-specific decision boundaries
and mitigates the negative impact of mislabelled target samples and in-domain
intra-class variability. RefRec achieves state-of-the-art performance on
standard benchmarks for UDA in point cloud classification, demonstrating the
effectiveness of self-training for this emerging research problem.
Additionally, [199] focuses on UDA in 3D CV tasks, specifically point cloud
visual tasks. The proposed approach introduces a dual-branch feature alignment
network (DFAN) architecture that leverages the characteristics of local and
global features in point clouds. The approach utilizes different strategies
for feature extraction and alignment in each branch, complementing each other.
Hierarchical alignment for local features and distribution alignment for
global features are also introduced. Experimental results on benchmark
datasets demonstrate that the proposed approach achieves state-of-the-art
performance in point cloud classification and segmentation tasks.

Previous studies on unsupervised 3D learning have principally concentrated on
ShapeNet [147], which is a repository of single-object computer aided design
(CAD) models. Typically, the idea is to use ShapeNet as the ImageNet
counterpart in 3D so that the characteristics learned on single synthetic
objects can be transferred to other real-world applications. In this regard,
numerous works have then been proposed. For instance, [200] introduces a
point-level DA by searched transformations. Typically, transformations of
3DPCs are learned via the search of the best combination of operations on
3DPCs, which transfers data from the SD to the TD while keeping the TD
unlabeled. Besides, most 3DPC-based UDA techniques focus on extracting domain-
invariant features in different domains for feature alignment. In this regard,
UDA is advocated for the detection of 3D objects in the context of semantic
point generation, as proposed in [160].

Moving on, in [157], a UDA problem approach for the semantic labeling of 3DPCs
is proposed, focusing on domain gap reduction of LiDAR sensors. Based on the
observation that sparse 3DPCs are sampled from 3D surfaces, a SVCN was
designed to complete the 3D surfaces of a sparse 3DPC. Unlike semantic labels,
obtaining training pairs for SVCN requires no manual labeling. A local
adversarial learning to model the surface prior was also introduced. The
recovered 3D surfaces serve as a canonical domain from which semantic labels
can transfer across different LiDAR sensors. Experiments and ablation studies
with benchmark for cross-domain semantic labeling of LiDAR data show that this
approach provides 6.3-37.6% better performance than previous DA methods. Fig.
9 portrays the block diagram of the the UDA scheme introduced in [157] for
semantic segmentation of LiDAR 3DPCs.

Figure 9: Flowchart of the UDA approach proposed in [157] for semantic
segmentation of LiDAR 3DPCs

###  3.4 Semi-supervised DTL

The integration of semi-supervised DTL into 3DPC understanding for tasks like
object detection and segmentation is rapidly transforming the field,
particularly in contexts where labeled data are scarce. This approach
leverages both labeled and unlabeled data, enhancing learning efficiency and
model performance across various applications, from autonomous driving to
tunnel monitoring. Typically, Tang et al. [201] develope a semi-supervised 3D
object detection model that utilizes a novel network to transfer knowledge
from well-labeled object classes to weakly labeled classes, improving
detection in classes with only 2D labels. Moving on, Ji et al. [5] propose the
Semi-supervised Learning-based Point Cloud Network (SPCNet) which integrates
various learning modules to enhance tunnel scene segmentation from 3DPCs,
significantly reducing the reliance on extensive labeled datasets. Imad et al.
[130]: Focused on utilizing 3D LiDAR data for autonomous driving perception,
enhancing object detection through a semi-supervised learning framework that
minimizes the need for large-scale annotated datasets.

In this same direction, Huang et al. [202] present a point cloud registration
framework that minimizes a feature-metric projection error without needing
correspondences, using a semi-supervised approach. This method is particularly
robust to noise and density differences in point clouds. Horache et al. [203]
propose a method for generalizing deep learning for 3DPC registration on
entirely different datasets using a combination of Multi-Scale Sparse Voxel
Convolution and an unsupervised transfer algorithm, UDGE, enhancing
adaptability across varied real-world datasets. Li et al. [204] introduce a
semi-supervised point cloud segmentation method that utilizes both labeled and
unlabeled data. By employing adversarial architecture for confidence
discrimination of label predictions on unlabeled point clouds, their approach
enhances segmentation performance.

Besides, Chen et al. [205] propose a multimodal semi-supervised learning
framework that utilizes instance-level consistency and a novel multimodal
contrastive prototype loss to enforce consistent representations across
different 3D data modalities of the same object. Similarly, Xiao et al. [206]
tackle the synthetic-to-real data gap in 3DPCs segmentation by developing a
large-scale synthetic dataset and a novel translation method that decomposes
and addresses the differences in appearance and sparsity between synthetic and
real datasets. Moving on, Mei et al. [207] develop a system for the semantic
segmentation of 3D LiDAR data in dynamic scenes, using a semi-supervised
learning strategy that combines limited manual annotations with large amounts
of constraint data to enhance scene adaptability. Additionally, Huang et al.
[208] introduce a spatio-temporal representation learning framework for
learning from unlabeled 3DPCs in a self-supervised manner, facilitating the
generalization of pre-trained models to a variety of downstream tasks.
Moreover, Qin et al. [209] propose a weakly supervised framework for 3D object
detection that leverages unsupervised 3D proposal generation and cross-modal
knowledge distillation, reducing the dependency on annotated 3D bounding
boxes.

On the other hand, Yu et al. [210] tackle the data scarcity challenge in 3D
tasks by transferring knowledge from robust 2D models to augment RGB-D images
with pseudo-labels, significantly improving the pre-training of 3D models with
limited labeled data. Xu et al. [211] develop a hierarchical point-based
active learning strategy for 3DPC segmentation, which measures uncertainty at
multiple levels and selects important points for manual labeling, effectively
utilizing limited annotations. Zhang et al. [212] study weakly semi-supervised
3D object detection with point annotations to generate high-quality pseudo-
bounding boxes, enabling 3D detectors to perform comparably to fully-
supervised models with substantially fewer labeled data. Lastly, Wang et al.
[213] introduce a Semi-Supervised Domain Adaptation method for 3D object
detection (SSDA3D) that employs an Inter-domain Point-CutMix module to align
point cloud distributions across domains and an Intra-domain Point-MixUp for
enhancing model generalization on unlabeled target data. Table 5 summarizes
and compares the above-discussed studies based on several aspects. This
comparison underscores the dynamic evolution of semi-supervised DTL techniques
in handling 3DPCs, which is pivotal for applications ranging from autonomous
vehicles to environmental scanning and medical imaging. These advancements
promise to drive significant improvements in how 3D data is processed and
utilized across various fields.

Table 5: Comparison of Studies on Semi-Supervised DTL in 3DPCs Ref. |  ML Model |  Dataset |  Application / Task |  Advantage |  Limitation
---|---|---|---|---|---
[201] |  Semi-supervised 3D Object Detection |  SUN-RGBD, KITTI |  3D object detection from 2D and 3D labels |  Efficiently transfers 3D info from strong to weak classes |  Requires sufficient data in strong classes for effective transfer
[205] |  Multimodal Semi-supervised Learning |  ModelNet10, ModelNet40 |  3D classification and retrieval |  Improves data efficiency using multimodal data consistency |  Performance depends on the quality of multimodal data integration
[5] |  SPCNet |  Real tunnel point clouds |  Multi-class object segmentation in tunnel scenes |  Reduces reliance on labeled data, enhances segmentation performance |  Specific to tunnel environments, may not generalize
[130] |  DTL based Semantic Segmentation |  KITTI, Ouster LiDAR-64 |  3D object detection in autonomous driving |  Reduces need for large-scale datasets, fast processing |  Limited by the initial data quality and transfer efficiency
[213] |  Semi-Supervised Domain Adaptation (SSDA3D) |  Waymo, nuScenes |  3D object detection in diverse conditions |  Adapts to new domains with minimal labeled data |  Performance may degrade with severe domain shifts
[206] |  DTL with PCT |  SynLiDAR |  3DPC segmentation |  Bridges the gap between synthetic and real data, extensive dataset |  Focuses on segmentation; may not directly apply to other 3D tasks
[202] |  Semi-supervised or unsupervised feature-metric registration |  N/A |  Point cloud registration |  Robust to noise and does not require correspondences; fast optimization |  Limited details on specific dataset adaptability and real-world implementation
[203] |  MS-SVConv and UDGE |  3DMatch, ETH, TUM |  3DPC registration |  Generalizes across different datasets using unsupervised DTL |  May require substantial computational resources; specific adaptation challenges not addressed
[207] |  CNN-based classifier |  Custom LiDAR dataset |  Semantic segmentation of dynamic scenes |  Combines few annotations with large constraint data for improved adaptability |  Primarily tailored to dynamic scenes, may not generalize to static or varied environments
[208] |  Spatio-Temporal Representation Learning (STRL) |  Synthetic, indoor, outdoor datasets |  3D scene understanding |  Learns from unlabeled data; generalizes to multiple 3D tasks |  Dependence on temporal correlation which may not be present in all datasets
[210] |  DTL from 2D to 3D models |  ScanNet |  Semantic segmentation |  Uses pseudo-labels for pre-training 3D models; enhances data efficiency |  Relies on the quality and relevance of 2D model training to 3D tasks
[209] |  Weakly supervised 3D object detection |  KITTI |  3D object detection |  Reduces need for detailed annotations; uses unsupervised proposal generation |  Performance may lag behind fully-supervised methods; adaptation to other datasets not detailed
[204] |  Semi-supervised point cloud segmentation |  N/A |  Point cloud segmentation |  Utilizes both labeled and unlabeled data; improves with self-training |  Specifics on dataset and environmental adaptability are not detailed
[211] |  Hierarchical point-based active learning |  S3DIS, ScanNetV2 |  Point cloud semantic segmentation |  Efficient use of very few labeled data; incorporates active learning |  May require intricate setup for uncertainty measurement and point selection
[212] |  Vision Transformer-based WSS3D |  SUN RGBD, KITTI |  3D object detection |  Low reliance on fully labeled data; uses point annotations effectively |  Challenges with varying detector compatibility and scene diversity

###  3.5 Inductive transfer learning

In inductive transfer learning (ITL), usually SD and TD remain the same,
however target task differs from the source task. In ITL setting, knowledge is
transferred from source task to attain high performance in the target task. As
suggested in [214, 215], the target is labeled, whereas source may be labeled,
unlabeled, or both. However, in the field of 3DPC, it is observed that only
labeled source has been explored.

For instance, Chen et al. [141] focus on classifying soft-story buildings
using CNNs and density features extracted from 3DPCs. More specifically, Once
the 3DPC data of Santa Monica city is collected, density features are
extracted and converted into 2D imagery data. The task of identifying a soft-
story building is then approached as a binary classification problem, which
has effectively been addressed using CNN models, including VGG, Inception,
ResNet and Naive CNN. They have trained the SD with labeled settings
exploiting more than 1.2 million images and 1,000 labeled categories, and
surprisingly VGGNet has proved to be the best in 2014, with its different
versions namely, VGG19 and VGG16. In addition to this, 138 million parameters
are trained in the network. VGGNet uses 3 x 3 filters, contrary to this,
Inception uses 1 × 1 filters, limiting the number of input channels. Hence,
the trainable parameters for Inception is low, approximately 6.4 million
trainable parameters. In an another work, Murtiyoso et al. [129], have taken
advantage of DeepLabv3+ network which they trained on labeled dataset of
building façade images. The trained network is deployed on TD, 2D orthoimages
received from photogrammetry. Similarly, a DL-based encoder-decoder
lightweight neural architecture, RandLA-Net, for the semantic segmentation of
3DPC data, has been utilized in [140] for performing per-point segmentation of
large-scale 3DPCs, as detailed in [216], and demonstrated a good semantic
segmentation performance in the Semantic 3D benchmark [140, 217].

In [218], authors have exploited ITL for 3DPC classification using an airborne
laser scanning method. They have extracted deep features using deep residual
network, and CNN was dedicated to classification task. To exploit CNN further,
the unevenly distributed 3DPC is transformed into voxels [219]. However, this
transformation can lead to information redundancy as well as some feature
loss.PointNet and later PointNet++ were proposed in [52] and [53],
respectively. They successfully directly classify the original 3DPC, leading
to the proliferation of PointNet-like approaches for 3DPC classification.
However, classification accuracy is affected when small range of multi-view
projection influence the detailed description on each 3D point. In addition,
there is a need of huge computing time for generating feature maps.

Lie and his team [138] have used DensNet201 for obtaining deep features, and
make use of an improved fully CNN by incorporating it into the ITL. The result
was very promising and superior to state-of-the-art on ISPRS dataset for OA.
In many cases weight transfer has found to be instrumental in classification
performance. For instance, Kamil and Marian [142] has suggested weights
transfer while they use ITL to improve the classification accuracy of
bidirectional LSTM (BiLSTM)-based approach for identifying human activities.
Similarly, Sun et al. [220] has incorporated deep ITL to explore the options
for solving the issues that are encountered with SAE networks in the
prediction of remaining useful life by utilizing weight and feature transfer.

For semantic labeling of heritage, Arnold et al. [133] has utilized ITL by
considering geometric shape characteristic for the purpose of segmentation of
memorial objects from the scene. Further, they pretrained CNN on a model from
ModelNet’s labeled dataset. Similarly, for body measurement of cattle, Huang
et al. [221] have pretrained the Kd-network by obtaining 3D deep model’s
initial parameters from ShapeNet. The authors extracted the 3DPC spatial
feature information which are transferred for identifying cattle body
silhouette. In addition, their ITL method is applicable even if the data
distribution is different for source and target data.

###  3.6 Transductive transfer learning

The transductive transfer learning (TTL) was introduced by Arnold et al. [222]
to solve a task in which domains are different however the source and target
tasks remain same . The authors advocate that at the training time, all
unlabeled data must be available in the TD, contrary to this, Pan and Yang
[215] urges that for finding out marginal probability, it would be enough to
seeing part of the unlabeled target data at the training time.

Applying DTL for different domain encounters many issues. One of these issues
for SDs and TDs is distribution mismatching. There are few works based on UDA
methods that have been reported in the literature for the adaptation of a
model to an unlabeled TD from a labeled SD for various different applications
[223, 224, 177, 225, 226]. In addition to UDA, authors in [143] has suggested
source-free unsupervised DA (SFUDA). They have used virtual domain modeling to
address of one the most talked issue in SFUDA without requiring original
source, reducing source and task data mismatching. To fill the gap between SD
and TD distributions, they have introduced an intermediate virtual domain,
reducing the distribution mismatch into two steps, minimizing the domain gap
between SD and virtual domains and then between the virtual and TDs. To
achieve this goal, the authors have generated the virtual domain samples in
the feature space using an approximated Gaussian mixture model (GMM) and the
pretrained source model, so that the virtual domain maintains a similar
distribution to the SD without access to the original source data. On the
other hand, they have proposed an effective distribution alignment method that
gradually improves the compactness of the TD distribution through model
learning to reduce the aforementioned distribution gap. However, the first
SFUDA framework was proposed in [197], where the authors exploited the source
model to fine-tune the TD data without access to the source data for 3D object
detection. The paper [116] discussed the use of SSL to learn useful
representations from unlabeled data in DA for 3D perception problems. The
authors proposed a new family of pretext tasks employing SSL for DA on point
clouds, called deformation reconstruction, along with a novel training
procedure called point cloud mixup (PCM) for labeled point cloud data. The
results demonstrated that employing SSL with this technique significantly
improves classification and segmentation in DA datasets.

As the research on DA for 3DPC is evolving, researchers are working to improve
the method to reduce the SD’ differences from TD, and to maximize the domain
adaptability so that it can be generalized. However, most of the research
addresses DA through experimentation over different domain meant for different
but related task. Table LABEL:UDA summarizes some of the relevant studies,
compares their characteristics and identifies their pros and cons.

The works [227, 228] require handling preprocessing tasks and additional data,
which in turn increases complexity. Rist et al. [227] exploit cross-sensor DA
via 3D voxels, while Wang et al. [228] utilize adversarial networks for global
adaptation by exploiting cross-range adaptation. In this context, the authors
in [229] present a potentially significant contribution. Their approach
circumvents complex procedures and the need for additional data. Instead, they
leverage the KITTI benchmark dataset to align strong-weak features and enhance
feature representation, supplementing it with available data when necessary.
Specifically, they focus on the ’car’ example from the benchmark dataset,
considering ’near-range’ and ’far-range’ objects as the SD and TD,
respectively. In addition to the underlying issues in handling different
domain for DA problems, distribution mismatch within a domain has also been
raised by researchers. In this regard, to adapt the different scenario within
a single domain, Zhang et al. [230] have taken the advantage of cross-dataset
and considered several adaptation scenario like day-night adaptation and
adaptation of different scenes (say, from different places) as in the case
with nuScenes dataset consisting of driving scenes from different geographical
locations, Boston and Singapore. Their extensive experiments enable
understanding how researchers can leverage cross-datasets for UDA. Moreover,
they have introduced range-aware and scale-aware detection mechanism for 3DPC
tasks. In the similar direction, Jaritz et al. [196] has considered the same
scenario, i.e., day-night adaptation and Boston-Singapore scenario, however
hey have primarily focused on cross-modality adaptation using their xMUDA
model. By the virtue of their proposed method, they have shown a better
performance can be achieved for cross-modality. Moreover, A2D2 and Semantic
KITTI datasets were used as source and target respectively. Moving forward,
Nunes et al. [231] present a new contrastive learning approach for
representation learning of 3DPs point cloud data in the context of autonomous
driving. The approach extracts class-agnostic segments and applies contrastive
loss to discriminate between similar and dissimilar structures. The method is
applied on data recorded with a 3D LiDAR and achieves competitive performance
in comparison to other self-supervised contrastive point cloud methods. Fig.
10 presents an example of fine-tuning in 3DPC segmentation [196].

Figure 10: Example of using fine-tuning in 3DPC segmentation: (A) Data
augmentation is used to generate the augmented views $\mathcal{P}^{q}$ and
$\mathcal{P}^{k}$ from a point cloud $\mathcal{P}$, and class-agnostic
segments $S$ are extracted from $\mathcal{P}$. The augmented segments
$\mathcal{S}^{q}$ and $\mathcal{S}^{k}$ are determined with their point-wise
features using the point indexes of $\mathcal{S}$ extracted from
$\mathcal{P}$. Point-wise features $\mathcal{F}^{q}$ and $\mathcal{F}^{k}$ are
computed, followed by dropout and global max pooling over each segment. The
segment feature vectors are projected using the projection head to obtain the
final features $\textbf{s}^{q}_{m}$ and $\textbf{s}^{k}_{m}$ from the $M$
segments, and the contrastive loss is computed, and (B) The pre-trained
backbone is fine-tuned for the downstream task, i.e., semantic segmentation
[231].

A UDA-based semantic labeling scheme of 3DPCs is proposed in [157], where
domain discrepancies induced by different LiDAR sensors have been addressed.
Typically, a complete and label technique that recovers 3D surfaces is
developed before passing them to a segmentation network. More precisely, a
SVCN to complete the 3D surfaces of a sparse 3DPC is designed. By contrast to
semantic labeling, obtaining training pairs for SVCN does not require any
manual labeling. Moreover, local adversarial learning has been introduced for
modeling the surface priors. The recovered 3D surfaces serve as a canonical
domain, from which semantic labels can transfer across different LiDAR
sensors.

Table 6: A summary of unsupervised DA frameworks and their characteristics used in 3DPC understanding. Work |  SD |  TD |  ML model |  SoDTs |  SoDDs |  Pros and cons
---|---|---|---|---|---|---
[116] |  PointSegDA |  PointSegDA |  DefRec |  S |  D |  The first study of SSL for DA on 3DPCs.
[142] |  UTD-MHAD |  UTD-MHAD |  N/A |  S |  S |  Computationally efficient and enable human activity detection using depth maps
[143] |  ModelNet40 |  ShapeNet |  VDM-DA |  S |  D |  Enable virtual domain modeling for source data-free DA.
[155] |  GTA-LiDAR |  KITTI |  SqueezesegV2 |  S |  D |  Enhance model structure and UDA for road-object segmentation.
[156] |  Semantic (KITTI, POSS, USL) |  Semantic (KITTI, POSS, USL) |  LiDARNet |  S |  D |  Preserve almost the same performance on the SD after adaptation and achieve 8%-22% mIoU performance increase in the TD.
[157] |  Waymo |  KITTI;
nuScenes |  SVCN |  S |  D |  Provide 8.2-36.6% better performance than previous DA methods.
[158] |  pointDA-10 |  pointDA-10 |  pointDAN |  S |  S |  3DPC representation using a multi-scale 3D DA network.
[195] |  Waymo |  KITTI, nuSenses, Lyft |  ST3D |  S |  D |  Exceed fully supervised results on KITTI 3D object detection benchmark.
[197] |  nuScenes |  KITTI |  SF-UDA |  S |  D |  Enable SFUDA but requires to be tested beyond cars, to detect other types of objects.
[198] |  PointDA-10; ScanObjectNN |  ShapeNet, ModelNet40, ScanNet, ScanObjectNN |  RefRec |  S |  D |  Refine pseudolabels, offline and online, by leveraging shape descriptors learned to solve shape reconstruction on both domains
[228] |  KITTI |  nuScenes |  CrAF |  S |  D |  Conduct more challenging cross-device adaptation.
[230] |  PreSIL; nuScenes |  KITTI; nuScenes |  SRDAN |  S |  D |  Demosntrate the significance of geometric characteristics for cross-dataset 3D object detection.
[232] |  labeled data, VirtualKITTi |  Semantic KITTI |  CLDA |  D |  D |  Improve segmentation performance for rare classes but still needs more enhancement.
[233] |  PointDA-10 |  PointDA-10 |  BADM;
PointDA-10 |  S |  S |  Achieve state-of-the-art performance and outperform other 3D DA techniques but not appropriate for different tasks or different domains.
[234] |  bSSFP-MRI; MM-WHS |  LGE-MRI; MRI and CT |  UDA-GAN |  D |  D |  Provide promising performance, compared to the state-of-the-art.
[235] |  Video data |  LiDAR |  PALMAR |  S |  D |  63% improvement of multi-person tTracking than state-of-the-art frameworks while maintaining efficient computation on the edge devices.
[236] |  KITTI object; near-range |  KITTI object; far-range |  SCNET |  S |  S |  Achieve a remarkable improvement in the adaptation capabilities but without enable knowledge transfer to different tasks.
[237] |  GTA-V |  Oxford RobotCar |  vLPD-Net R;
vLPD-Net V |  S |  D |  Achieve state-of-the-art performance on the real-world Oxford RobotCar dataset but the investigation of the loop closure and re-localization in real-world is needed.
[238] |  CadData |  CamData |  DANN;
SSLPC;
PoinDAN |  S |  D |  Less time-consuming for implementation in production.
[239] |  PointDA-10 |  PointDA-10 |  DSDAN |  S |  S |  Exceed the state-of-the-art performance of cross-dataset 3DPC recognition tasks.
[240] |  High resolution LiDar data |  Low resolution LiDar data |  LAMAR |  S |  D |  94% Human activity recognition performance in multiple-inhabitant scenario.
[241] |  A2D2 |  Semantic KITTI |  xMUDA |  S |  D |  Enable cross-modal learning DA (CLDA) in 3D semantic segmentation; however, knowledge cannot be transferred across different ttasks.
[242] |  KITTI |  MDLS |  CycleGAN;
YOLOv3 |  S |  D |  Outperformed other compared baseline approaches with more than 39% improvement in F 1 -Measure score.

Abbreviations: Same or different tasks (SoDTs); Same or different domains
(SoDDs); Different (D); Same(S).

##  4 Applications of TL-based 3DPCs

TL-based 3DPC can be applied in various applications, including robotics and
autonomous systems, augmented reality and virtual reality, medical imaging,
geo-spatial data analysis, industrial inspection, building information
modeling, etc. In this section, we focus on some of the main applications that
attract increasing research effort. Table LABEL:3dpc_tasks summarizes some of
the pertinent DTL-based 3DPC frameworks proposed to perform different tasks.

###  4.1 Semantic labeling and segmentation

Semantic labeling of 3DPC is crucial for performing segmentation tasks to
represent scans accurately, using manual as well as automatic labeling. For
instance, recently, Xie et al. [117] have used the Stanford large-scale 3D
indoor spaces (S3DIS) dataset for transferring knowledge to the target
dataset. However, it contains a much smaller repository as compared to source
dataset (ScanNet), and manual semantic labeling has been done with 13
categories. Arnold et al. [133] used simple MLP to learn the diversity of the
feature space and acquire knowledge for semantic labeling.

One of the prime concerns is to identify similar labels and different data
from the same labels separately. To address this, authors [243] have suggested
a domain-independent approach for semantic labeling. Their proposed model
considers similarity metrics as features to infer the correct semantic labels
and learns a similarity function to identify if attributes have the same
labels. Because the matching function is unrelated to specific labels, their
model does not depend upon the label and thus are independent of domain
ontologies. A similar approach for bridge 3DPC has been suggested in [244].
The authors have received an improved IoU with 94.29%.

The ability of DTL to reduce the need for large-scale dataset can be
economical and hence can be exploited in 3DPC tasks for performing accurately
without requiring more extensive data. For example, with the help of DTL,
authors in [245] have explored the analogy between real and synthetic data for
semantic segmentation and tried reducing their gap. Based on deep CNN (DCNN),
authors in [246] have suggested a weakly-supervised semantic segmentation
technique. Unlike others, they have considered auxiliary segmentation
annotations for distinct categories to guide segmentation on pictures with
only image-level class labels. They have used a decoupled encoder-decoder
architecture with an attention model allowing segmentation knowledge to be
transferable across categories.

Semantic segmentation is conducted in [206] by transferring knowledge from
synthetic to real LiDAR 3DPCs. Typically, a large-scale synthetic LiDAR
dataset, is first collected from multiple virtual environments with rich
layouts and scenes. It includes point-wise labeled 3DPCs with accurate
geometric shapes and comprehensive semantic classes. Moving on, a 3DPC
translator (PCT) is designed to mitigate the discrepancy between real and
synthetic 3DPCs. Typically, the synthetic-to-real discrepancy is decomposed
into a sparsity and an appearance component before separately handling them.
Fig. 11 explains the PCT approach adopted in [206], disentangling 3DPC
translation into appearance and sparsity translation tasks. Accordingly, dense
3DPCs having similar appearances are first learned within the appearance
translation. The sparsity translation then learns real sparsity distribution
in 2D space and fuses it with the reconstructed 3DPC in 3D space. Then, real
sparsity distributions in 2D space are learned within the sparsity translation
before being fused with the reconstructed 3DPC in 3D space.

Figure 11: The PCT method separates the task of translating 3DPC into two
parts: appearance translation and sparsity translation. Using synthetic 3DPC
as input, the appearance translation learns to create dense 3DPC that look
similar to real ones. The sparsity translation then learns the typical
sparsity patterns found in real 2D point clouds, and combines these patterns
with the dense 3DPC in 3D space. The end result is a 3DPC that looks and is
sparse similarly to real 3DPC.

Moving forward, A. Murtiyoso et al. [129] has used DTL on a photogrammetric
orthoimage by exploiting labeled and rectified images of building facades for
training neural networks on them. Next, they allow the transition from 2D
orthoimage to 3DPC by another program. With their promising results for
photogrammetric data, their proposed work is a potential option to assist in
automatic 3DPC semantic segmentation. Similarly, authors in [171] have claimed
that semantic segmentation can perform equally competently to the latest ML
approaches in modeling cultural heritage.

On the other hand, 3DPC instance segmentation is equally employed for many
applications. For example, in [131], authors have taken a problem of height
detection of a catenary conductor. For 3DPC instance segmentation, they have
incorporated DTL and 3D-BoNet model with a multiscale grouping (MSG) structure
on a smaller dataset, with loss curve converging better for their model. In
addition to its applications in building, tunneling, and construction, DTL for
3DPC segmentation has found utility in robotics. Specifically, it is utilized
for automatic toolpath generation to enable tasks such as masking, deburring,
and polishing in various industrial applications. Z. Xie and his team [137]
have utilized 3DPC segmentation and classification to extract the object
features. Furthermore, they have exploited DTL to enhance performance and
avoid investing much time in training data. Their toolpath recommendation
automatically suggests patterns to choose from and does not require manual
efforts. To identify human activities, authors [142] have carried out
segmentation to separate the human figure from the background so that the
descriptor can only be determined for a human figure.

###  4.2 Classification

3DPC classification is the process of classifying the features of 3D objects
and categorizing the classes to which they belong. The 3DPC segmentation types
and classification are illustrated in Fig. 12. In the diagram, it is shown how
a building, with the help of CNN, can be classified based on different
classifications e.g., concerning time (period), region, and structure.
Recently, researchers have explored different ways of directly classifying
3DPC, mainly to avoid information loss during the conversion of 3DPC into
non-3DPC. Automated classification of point cloud data has great potential for
various applications, but a limited number of labelled points can lead to
overfitting and poor generalization in ML models.

One of the pioneered works in the field is done by Qi et al. [52], reported
being the first DL model to classify the original 3DPC directly. Their
PointNet model is based on CNN, and its success has attracted many works to
follow their proposed architecture. CNN has been widely used for
classification tasks. For example, authors [129] have used CNN with DTL to
classify improved fisher vectors used to represent 3DPC. The work [247]
introduces a method called Atrous XCRF that induces controlled noise by
invoking conditional random field similarity penalties using nearby features
to improve generalization. The method achieves high OA and F1 score in a
benchmark study using the ISPRS 3D labeling dataset and is on par with the
current best model. Using DTL with Bergen 2018 dataset shows improvement in
accuracy, but more work is needed to address generalization issues in DA and
DTL. To overcome the requirement of large dataset, Lei et al. [138] have
proposed a 3DPC classification method that integrates an improved CNN into
DTL. They have used the DensNet201 model as a pretrained model for finding
deep features able to accurately characterize the object. In addition, The
authors [218] have proposed an ALS 3DPC classification method based on DTL
using CNN. In [138], an ALS 3DPC classification approach for integrating an
enhanced fully-CNN into DTL with multi-view and multiscale and deep features
is proposed. Typically, the shallow features of the ALS 3DPC, such as height,
intensity, and curvature change, are extracted to generate feature maps by
multiscale voxel and multi-view projection. Second, these feature maps are fed
into the pretrained DenseNet201 model to derive deep features. Experimental
results show that OA and the average F1 scores obtained by the proposed method
are 89.84% and 83.62%, respectively.

Figure 12: The flow chart of the proposed classification method.

###  4.3 Point-set registration

With the increasing number of 3D sensors and 3D data production, consolidating
overlapping 3DPCs, which is called registration, has become a significant
issue in many applications [248]. Registration can be used online (e.g., in a
LiDAR SLAM pipeline for loop closure detection) or offline (e.g., in 3D
reconstructions of RGB-D indoor scenes [249] or for outdoor LiDAR map building
for autonomous vehicles). However, 3DPC registration can be challenging with
real-world scans because of noise, incomplete 3D scans, outliers, etc. There
are numerous existing approaches to this issue; however, recently, DL
approaches have become very popular, especially for learning descriptors and
computing transformations. These data-driven approaches, especially those that
involve learning on large datasets with the ground truth pose as supervision,
have effectively solved registration problems [250, 237]. Specifically, in
[237], a registration-aided DA model for 3DPC-based place recognition is
developed, called vLPD-Net. The method utilizes virtual large-scale point
cloud descriptors and a structure-aware registration network, along with
adversarial training, to minimize the gap between synthetic and real-world
domains.

###  4.4 Scene understanding

The method proposed by Xu et al. [82] uses DTL technique by inflating weights
to transform an image-pretrained model into a 3DPC model. Authors in [251]
have proposed a model ’Shape Self-Correction’ for 3DPC analysis where they
have explored how unlabeled data can be utilized for a better 3DPC
understanding. In addition, their approach can identify distorted points and
automatically correct them by producing state-of-the-art results. Moreover,
this model can be used as a pretrained model to be fine-tuned for related
tasks. Going forward, Zhang et al. [252] also tried to exploit a pretrained
large-scale 2D dataset to generalize it to 3DPC understanding using a
contrastive vision-language pretraining (CLIP). Their model, PointCLIP,
projects 3DPC into multi-view depth maps by encoding it, and successfully,
view-wise zeroshot prediction is aggregated to transfer knowledge from 2D to
3D. Furthermore, many works exploit unsupervised approaches for 3DPC
understanding. For example, Xie et al. [117] have suggested an unsupervised
pretraining framework. To avoid the domain gap and utilize their approach
across the different domains in the real world, they have directly made use of
a complex network to pretrain the model. The major contribution they have is
the examination of the transferability of learned representations
generalization across the domain for 3DPC understanding. In addition, their
architecture can also represent point-level information by capturing local
features. To further exploit the geometric structure of the 3DPC and
relationships of local features, Liu et al. [253] have suggested a
convolution-based geometric-aware approach. Moreover, they proposed a
filtering mechanism that identifies noise and outliers for a high-level
understanding.

###  4.5 Object detection and denoising

The exploitation of DTL for various applications of 3DPC in 3D object
detection and recognition has recently attracted researchers, ranging from
medical science to biological research and robotics. For instance, authors in
[145] have used DTL for protein structure indexing using 3DPC. The suggested
approach saves training time by fine-tuning the pretrained network, using
Euclidean distance to extract shape-matching, with generic 3D objects.
Furthermore, authors in [131] have proposed a detection method incorporating
instance segmentation and DTL for overhead catenary height detection of a