# CP3: Unifying Point Cloud Completion by Pretrain-Prompt-Predict Paradigm

Mingye Xu, Yali Wang, Yihao Liu, Yu Qiao, M. Xu, Y. Wang and Y. Liu are with
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences,
Shenzhen, 518055. Email<EMAIL_ADDRESS>M. Xu and Y. Liu are also with the
University of Chinese Academy of Sciences, Beijing, 100049\. Y. Wang and
Y.Qiao are with with Shanghai AI Laboratory. Y. Wang and Y.Qiao are co-
corresponding authors.

###### Abstract

Point cloud completion aims to predict complete shape from its partial
observation. Current approaches mainly consist of generation and refinement
stages in a coarse-to-fine style. However, the generation stage often lacks
robustness to tackle different incomplete variations, while the refinement
stage blindly recovers point clouds without the semantic awareness. To tackle
these challenges, we unify point cloud Completion by a generic Pretrain-
Prompt-Predict paradigm, namely CP3. Inspired by prompting approaches from
NLP, we creatively reinterpret point cloud generation and refinement as the
prompting and predicting stages, respectively. Then, we introduce a concise
self-supervised pretraining stage before prompting. It can effectively
increase robustness of point cloud generation, by an Incompletion-Of-
Incompletion (IOI) pretext task. Moreover, we develop a novel Semantic
Conditional Refinement (SCR) network at the predicting stage. It can
discriminatively modulate multi-scale refinement with the guidance of
semantics. Finally, extensive experiments demonstrate that our CP3 outperforms
the state-of-the-art methods with a large margin.

###### Index Terms:

Point Cloud Completion, Prompting, Self-supervised Pretraining, Semantic
Refinement.

##  1 Introduction

Point cloud analysis has attracted a lot of research interest in computer
vision and robotics. Unfortunately, the scanned point cloud is usually
incomplete in practice, due to complex occlusion, light reflection, limited
sensor resolution, etc. Therefore, the completion task has gradually become
important. The existing approaches [1, 2, 3, 4, 5] are mainly based on point
cloud generation and refinement, which first generates the coarse completion
from partial point cloud, and then recovers the fine completion from the
coarse one. However, these approaches focus on designing complicated models,
while ignoring the fundamental problems about how to effectively use point
cloud at different stages.

Figure 1:  Our CP3 paradigm. We unify point cloud completion by the recent
prompting scheme in NLP, since we find the incomplete point cloud plays a
similar role as the incomplete sentence, e.g., “I love this movie. Overall, it
was a $\\_\\_$ movie”. Following this line, we reinterpret point cloud
generation and refinement as the prompting and predicting stages, which first
generates the imperfect point cloud (like answer: “good / fantastic” ) from
the incomplete one, and then refines target point cloud (like output:
“++(positive)” ) from the imperfect one. But different from NLP, point cloud
generation is not pretrained, and refinement lacks semantic awareness. Hence,
we propose a concise IOI pretraining to boost generation robustness in a self-
supervised way, and design a novel semantic guidance to modulate refinement
for shape confusion reduction.

At the generation stage, the core problem is to learn a robust incomplete-to-
complete mapping for tackling different partial observations of point cloud.
Most methods mainly work on developing deterministic or probabilistic
generation networks [2, 4, 5], while lacking insightful consideration about
data itself. In fact, the training pairs of incomplete and complete point
cloud are predetermined. With such limited data diversity, it is often
difficult to generate robust completion from new incomplete variations, by
training those complex generation networks from scratch. At the refinement
stage, the key is to recover the plausible shape of the complete point cloud,
based on the coarse (or imperfect) completion from the generation stage.
However, the existing refinement approaches [6, 7, 4] ignore the semantic
information of point cloud, which is an important clue of shape recovery. As a
result, these refinement networks are unsatisfactory on point clouds which
have similar shapes but belong to different semantic categories, e.g., chair
and table in Fig. 1.

To tackle these problems, we propose to unify point cloud Completion
frameworks by a generic Pretrain-Prompt-Predict paradigm, namely CP3, which
can achieve robust generation and discriminative refinement by self-supervised
pretraining and semantic-guided predicting. As shown in Fig. 1, our
inspiration mainly comes from the recent prompting scheme in NLP [8, 9, 10].
Specifically, as Fig. 1 shows, we flexibly reinterpret the generation stage as
the prompting stage, since both stages aim at learning on new scenarios with
limited data diversity. But differently, the pretrained model is ready in NLP.
Therefore, its goal is to design a prompt function that modifies $input$ as
$prompt$ for answer search. Alternatively, point cloud generation is not
pretrained. Hence, our goal is to design an inverse prompt function which aims
at constructing the diversified $input$ from the given $prompt$ (i.e.,
incomplete point cloud) for pretraining. In particular, we introduce a concise
Incompletion-Of-Incompletion (IOI) sampling mechanism as inverse prompt
function, where we randomly crop incomplete point clouds into new incomplete-
of-incomplete ones. Subsequently, we pretrain the generation network with a
self-supervised pretext task, which applies the IOI point cloud as input to
recover the original incomplete point cloud. This design contains two main
advantages. First, this pretraining task is similar to the downstream
generation task, which can reduce the task gap for model transfer [8]. Second,
IOI pretraining can effectively increase robustness of generation network, by
pretraining on rich variations of incomplete point clouds.

Moreover, we reinterpret the refinement stage as the predicting stage, since
this stage is used to map the imperfect point cloud (i.e., $answer$ of
$prompt$) into the refined one (i.e., the target $output$). But different from
answer mapping in NLP, point cloud refinement requires extra semantic
knowledge, since the shape of point cloud is highly relevant to its high-level
semantics, such as category information. Based on this observation, we propose
a novel Semantic Conditional Refinement (SCR) network, which adaptively
leverage semantic information as discriminative guidance for refining point
cloud. Specifically, it mainly consists of two distinct blocks, i.e.,
semantic-guided modulation and multi-scale point deconvolution blocks. The
semantic-guided modulation block smartly converts the semantic information of
point cloud into a dynamical filter to modulate the point-wise features.
Conditioned on these semantic-modulated features, the point deconvolution
block progressively refines point cloud by multi-scale relation learning.
Hence, unlike the previous refinement networks [5, 7, 11], our SCR network can
effectively alleviate shape confusion among various categories.

Finally, we evaluate our CP3 on the widely-used point cloud completion
benchmarks such as MVP [5] and PCN [1]. The extensive experiments show that
our IOI pretraining and SCR predicting can effectively boost point cloud
generation and refinement. Consequently, CP3 outperforms a number of state-of-
the-art approaches with a large margin, e.g., it achieves the new SOTA result
on both MVP and PCN datasets, with 2.27 CD loss on MVP (the current SOTA [5]
is 3.06) and 7.02 CD loss on PCN (the current SOTA [7] is 7.21). We will
release our code afterwards.

##  2 Related Work

###  2.1 3D shape Completion

Most traditional completion methods rely on shape continuity inferred from
input scan. Some methods [12, 13, 14, 15, 16] are mainly based on hand-crafted
rules or smooth interpolation/extrapolation to characterize the missing
regions. Generative based methods [17, 18, 19] is to exploit the large-scale
shape database to search for the similar shapes/patches to fill the missing
regions of 3D shapes. However, the generalization capability of these methods
to diverse partial structures is usually limited. In contrast, deep learning
methods can learn more flexible representations for more complicated shapes.
The existing deep learning methods are mainly based on the crose-to-fine
style, which generates a coarse one and refines the geometry details. PCN [1]
first operates on raw point clouds and generates a coarse completion based on
learned global representation and folding based decoding [20]. Based on PCN,
many methods [1, 2, 3, 4] achieve more detailed point cloud completion with
better completion results, but usually lack the long-range correlations in
local regions. [5, 11, 7] adopt transformers [21] to learn the structural
information of pairwise interactions and context correlations. However, they
pay much attention to the complicated model structures and usually integrate
the generation designs and refinement designs in a single framework, while
ignoring deep investigation of distinct data problems in each stage. In
addition, a self-supervised approach has been introduced for scene completion
with RGB-D scans [22], while different data formulation and task definition
make it hard for traditional point cloud completion. Different from the
previous approaches, we creatively reinterpret the existing completion process
as prompt (generation) and predict (refinement), inspired by the “pretrain-
prompt-predict” paradigm in NLP [8]. Moreover, we propose an IOI pretraining
for data-diversified generation, and develop a discriminative refinement
network with semantic guidance.

###  2.2 Conditional Feature Modulation

Conditional feature modulation has been used extensively in prior and
concurrent works including discrete labels [23, 24], text [25], and images
[26]. For example, Condition GAN [23] can generate the controllable images
conditioned on class labels. AdaIN [27] can effectively achieve image transfer
by matching feature statistics of content and style. StyleGANs [28, 29] design
modulations in the generator to adjust the ‘style’ of an image. [30] propose a
global feature modulation method for photo retouching. Inspired by these
modulation methods, we introduce a semantic-guided modulation for point cloud
completion, which can effectively reduce shape confusion for discriminative
refinement.

Figure 2:  IOI pretraining and prompting.

###  2.3 Prompting Paradigm

Recently, the prompting paradigm has attracted a lot of attention in NLP [8,
31, 32]. The conventional training paradigm refers to pretrain-finetune, while
the prompting paradigm is characterized as pretrain-prompt-predict.
Specifically, it is used to rebuild the input text through some prompt
information, and transfer the predetermined task to the prompting task, e.g.,
“I love this movie.” is changed into “I love this movie. Overall, it was a __
movie.”. Unlike supervised learning, prompting dose not need to fine tune the
whole model, but to design a text prompt to rebuild the model. Specifically,
it transfers the knowledge in a large number of pretrained models and
rearranges prompting tasks to make them closer to pre-determined tasks. By
designing appropriate prompts, a number of works [33, 10, 34] generalizes the
model behavior well, so that the pretrained language model can be effectively
adapted to predict the desired output in NLP. Moreover, such prompting
paradigm has been also introduced for vision-language modeling in computer
vision [35]. To our best knowledge, we are the first to leverage such
“Pretrain-Prompt-Predict” paradigm for point cloud completion, where we unify
point cloud generation and refinement as the prompting and predicting stage,
and introduce a self-supervised pretraining stage with the help of our inverse
prompt function.

##  3 Method

Overall Paradigm of Our CP3. In this section, we introduce the proposed CP3
paradigm in details. As shown in Fig. 1, we unify point cloud completion by
the recent prompting paradigm in NLP. Specifically, point cloud generation
refers to the prompting stage. Given an incomplete point cloud (i.e., prompt),
the generation network is to produce an imperfect completion (i.e., answer).
Since the training pairs in this stage are predefined with limited data
diversity, learning the generation network from scratch leads to
unsatisfactory overfitting. To tackle the above problem, we introduce a
pretraining stage for point cloud generation. In particular, we design an
inverse prompt function, i.e., Incompletion-Of-Incompletion (IOI) sampling,
which randomly crops incomplete point clouds (i.e., prompt) into a large
number of diversified IOI point clouds (i.e., input). Subsequently, we
pretrain the generation network with a self-supervised pretext task, where we
use IOI point cloud as input to reconstruct the corresponding incomplete one.
Finally, we reinterpret the refinement stage as the predicting stage, where we
predict the target point cloud (i.e., output) from the imperfect or coarse one
(i.e. answer in the prompting stage). To enhance such an answer mapping, we
introduce a novel Semantic Conditional Refinement (SCR) network. It can
effectively reduce shape confusion by progressively refining multi-scale point
cloud with its semantic category.

Figure 3: The crop procedure of IOI sampling.  Figure 4: The structure of SCR
network with Semantic-Guided Modulation and Multi-scale Point Deconvolution.

###  3.1 Pretrain & Prompt: Point Cloud Generation with IOI-Pretrain

As discussed before, we rethink point cloud generation by the prompting stage
in NLP, where an incomplete point cloud is analogous to an incomplete
sentence, e.g., “I love this movie. Overall, it was a $\\_\\_$ moive”. But
differently, a well-pretrained language model is ready for the prompting stage
in NLP, while such a model is not available for point cloud generation. In
fact, the existing generation models [5, 4, 1] are trained from scratch, with
limited diversity of training pairs. As a result, its performance would be
restricted, when tackling new incomplete variations. To enhance its
robustness, we propose a self-supervised pretraining stage for the generation
model, based on a concise Incompletion-Of-Incompletion (IOI) sampling
mechanism.

IOI Sampling as Inverse Prompt Function. To construct the diversified inputs
for pretraining, we propose a concise IOI sampling mechanism, which generates
incomplete-of-incomplete point clouds (i.e., input) from incomplete ones
(i.e., prompt). We call IOI as inverse prompt function, with inspiration of
prompt function in NLP. The difference is that, the prompt function in NLP is
to convert input as prompt for answer search, while our inverse prompt
function is to construct diversified input from the given prompt.
Specifically, the IOI sampling is based on the following procedure in right
part of Fig. 3. First, we randomly determine an crop plane through the given
incomplete point cloud. In particular, a random vector $v$ is ascertained from
the object center through two random angle $\phi$ and $\theta$, where
${v}=\\{{\sin(\theta)\cos(\phi)},\sin(\theta)\sin(\phi),\cos(\theta)\\}$. Then
we find the crop plane which is perpendicular to this vector $v$ and passes
through the center of point cloud (i.e., yellow plane in Fig. 3). Second, we
calculate the projection distance ${d}_{i}$ from each point ${p}_{i}$ to this
crop plane,

${d}_{i}=({p}_{i}-\frac{1}{N}\sum\nolimits_{j=1}^{N}p_{j})\circ{v},$ (1)

where $\circ$ is the hadamard product. Finally, we use the projection distance
${d}_{i}$ as sampling guidance, and drop the farthest $rN$ points from the
crop panel, i.e., red points in Fig. 3. The crop rate is set as
$r\subset[0,\Gamma]$, where $\Gamma$ is the maximun threshold. It is worth
mentioning that, our IOI sampling brings the following benefits. On one hand,
the random crop of incomplete point cloud can generate rich data diversity for
pretraining. On the other hand, it naturally builds up a pretext task for
self-supervised pretraining as follows.

Self-Supervised Pretraining. By IOI sampling, we have incomplete point clouds
and their incomplete-of-incomplete variations on hand. Hence, we can pretrain
the generation network by self-supervised learning of incomplete point cloud,
i.e., we recover the incomplete point cloud from its IOI point cloud, as shown
in Fig. 2. Moreover, we would like to emphasize that, our pretraining task is
actually similar to the original generation task (i.e., the prompting stage),
by analogy with the“incomplete-to-complete” mapping. Hence, such design can
effectively reduce the task gap for model transfer, which perfectly matches
the insight of prompting in NLP [8, 31]. Finally, we choose the recent VRCNet
[5] as an exemplar of the generation network for pretraining. Note that, other
generation networks also work with our pretraining design. More details can be
found in Subsection 4.3 and in Table VIII.

Prompting. At the generation stage, we are actually given with limited
training pairs of incomplete and complete point clouds. Hence, we simply fine-
tune the generation network with these pairs, after pretraining. Consequently,
we use the well-tuned network for prompting, i.e., generating a coarse
complete one (i.e., anwser) from an incomplete point cloud (i.e., prompt).

###  3.2 Predict: Point Cloud Refinement with Semantic Guidance

After obtaining the coarse point cloud (i.e., answer) from the incomplete one
(i.e., prompt), we next map answer into output, i.e., the target complete
point cloud. As discussed in the introduction, this predicting stage refers to
point cloud refinement. However, the existing refinement networks [4, 6]
ignore semantic understanding when recovering the shape of point cloud. This
leads to unsatisfactory completion on those confused categories. To tackle
this problem, we propose a novel Semantic Conditional Refinement Network
(SCRNet), which progressively modulates multi-scale point cloud
representation, with guidance of high-level semantics. As shown in Fig. 4, it
mainly consists of two core blocks, i.e., Semantic-Guided Modulation (SGM)
block and Multi-scale Point Deconvolution (MPD) block. In each refinement
unit, the SGM block first adjusts point-wise representation discriminatively,
by a dynamic convolution of semantic information. Then, the MPD block further
refines the modulated representation of point cloud, by multi-scale
aggregation of geometrical context.

Semantic-Guided Modulation (SGM) Block. To reduce shape confusion, we
introduce a new Semantic-Guided Modulation (SGM) block, which leverages
semantics to module point cloud discriminatively. We mainly use the category
as the semantic information for illustration. The semantics can be also a
learnable global representation of point cloud, as shown in Table X. For
convenience, we omit the subscript $i$ for the $i$-th refinement unit in the
following. As shown in Fig. 4, we introduce a distinct convolution with the
semantic-guided filter $\mathbf{W}\in\mathrm{R}^{{C_{in}\times C_{out}}}$ in
the SGM block. We use this convolution to learn the semantic-relevant feature
$\mathcal{\hat{K}}\in\mathrm{R}^{{N}\times C_{out}}$ from the displacement
feature $\mathcal{{K}}\in\mathrm{R}^{{N}\times C_{in}}$,

$\mathcal{\hat{K}}(j,q)=\sum\nolimits_{p=1}^{C_{in}}\mathbf{W}(p,q)\cdot\mathcal{{K}}(j,p),$
(2)

where $j$ enumerates the number of points $N$, $p$ enumerates the number of
input channels, and $q$ enumerates the number of output channels. Moreover,
the displacement feature $\mathcal{{K}}$ is initialized with the point-wise
feature from generation.

More specifically, the semantic-guided filter $\mathbf{W}$ is constructed with
the following steps. First, we use the one-hot label vector $S$ of point cloud
to construct a category-relevant scale vector
$\vartheta(S)\in\mathrm{R}^{C_{in}}$, where $\vartheta(.)$ refers to the MLP
layer. Second, we use this scale vector as semantic guidance to generate the
filter matrix $\mathbf{B}\in\mathrm{R}^{{C_{in}\times C_{out}}}$,

$\mathbf{B}(p,q)=\vartheta(S)_{p}\cdot\mathbf{A}(p,q),$ (3)

where $\mathbf{A}\in\mathrm{R}^{{C_{in}\times C_{out}}}$ is a trainable
parameter matrix. By this operation, $\vartheta(S)_{p}$ can modulate the
$p$-th feature channel of input point cloud. Finally, we normalize the filter
matrix for optimization stability,

$\mathbf{W}(p,q)=\mathbf{B}(p,q)/\sqrt{\sum\nolimits_{p=1}^{C_{in}}{\mathbf{B}(p,q)^{2}+\epsilon}},$
(4)

where $\epsilon$ is a small constant to avoid numerical issues. Based on this
normalized filter $\mathbf{W}$, we can modulate different feature channels
according to semantic category. As a result, our SGM block makes the
refinement stage more discriminative, which can effectively distinguish point
clouds that have similar local shapes but belong to different categories.

Multi-scale Point Deconvolution (MPD) Block. After obtaining the modulated
feature, we use it for point cloud refinement. As shown in Fig. 4, we choose
snowflake point deconvolution [7] as our base operation, which learns the
point displacement progressively to refine the shape. For the $i$-th block, it
firstly constructs the input feature $\mathcal{Q}_{i}$, by concatenating the
predicted point cloud $\mathcal{P}_{i}$ with its global feature from PointNet
[36]. Second, it sends $\mathcal{Q}_{i}$ and $\mathcal{\hat{K}}_{i}$ into the
skip transformers [7] for point relation learning, where
$\mathcal{\hat{K}}_{i}$ is the modulated displacement feature from the
previous block. The output of skip transformers is denoted as
$\mathcal{H}_{i}$. Third, it upsamples $\mathcal{H}_{i}$ as displacement
feature $\mathcal{K}_{i+1}$ for next layer and generates the coordinate
variation $\Delta\mathcal{P}_{i+1}$ in the current block, by point-wise
splitting operation in [7]. Finally, it adds the coordinate variation
$\Delta\mathcal{P}_{i+1}$ with the duplicated previously-predicted point cloud
$\mathcal{P}_{i}$, which generates the refined point cloud $\mathcal{P}_{i+1}$
in the current block.

Note that, our MPD block follows the refinement style of snowflake point
deconvolution. But differently, it is equipped with our semantic-guided
modulation block and multi-scale relation learning. First, we use the
modulated feature $\mathcal{\hat{K}}_{i}$ of SGM block as input to skip
transformer, instead of using the original displacement feature
$\mathcal{K}_{i}$. In this way, we flexibly integrate semantic category
information into relation learning, which enhances the shape-context feature
$\mathcal{H}_{i}$ for discirminative refinement. Second, we introduce multi-
scale learning strategy for skip transformers, instead of building point
relations in a single scale. For example, we use skip transformers with two
different local regions to learn shape context features $\mathcal{H}_{i}^{1}$
and $\mathcal{H}_{i}^{2}$. Next, we fuse these context features to obtain the
displacement feature $\mathcal{K}_{i+1}$ in the current block,

$\mathcal{K}_{i+1}=\operatorname{MLP}({\varphi}(\mathcal{H}_{i}^{1}\oplus\mathcal{H}_{i}^{2}))$
(5)

where $\oplus$ is the concatenation operation, and $\varphi$ is the point-wise
splitting operation [7] for upsampling. Via such multi-scale design, our MPD
block can effectively capture complicated shape variations in the point
clouds.

Method | airp. | cabinet | car | chair | lamp | sofa | table | waterc. | bed | bench | books. | bus | guitar | motor. | pistol | skateb. | Avg.
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
PCN [1] | 2.95 | 4.13 | 3.04 | 7.07 | 14.93 | 5.56 | 7.06 | 6.08 | 12.72 | 5.73 | 6.91 | 2.46 | 1.02 | 3.53 | 3.28 | 2.99 | 6.02
TopNet[37] | 2.72 | 4.25 | 3.40 | 7.95 | 17.01 | 6.04 | 7.42 | 6.04 | 11.60 | 5.62 | 8.22 | 2.37 | 1.33 | 3.90 | 3.97 | 2.09 | 6.36
MSN [3] | 2.07 | 3.82 | 2.76 | 6.21 | 12.72 | 4.74 | 5.32 | 4.80 | 9.93 | 3.89 | 5.85 | 2.12 | 0.69 | 2.48 | 2.91 | 1.58 | 4.90
Wang et. al. [2] | 1.59 | 3.64 | 2.60 | 5.24 | 9.02 | 4.42 | 5.45 | 4.26 | 9.56 | 3.67 | 5.34 | 2.23 | 0.79 | 2.23 | 2.86 | 2.13 | 4.30
ECG [4] | 1.41 | 3.44 | 2.36 | 4.58 | 6.95 | 3.81 | 4.27 | 3.38 | 7.46 | 3.10 | 4.82 | 1.99 | 0.59 | 2.05 | 2.31 | 1.66 | 3.58
GRNet [6] | 1.61 | 4.66 | 3.10 | 4.72 | 5.66 | 4.61 | 4.85 | 3.53 | 7.82 | 2.96 | 4.58 | 2.97 | 1.28 | 2.24 | 2.11 | 1.61 | 3.87
NSFA [38] | 1.51 | 4.24 | 2.75 | 4.68 | 6.04 | 4.29 | 4.84 | 3.02 | 7.93 | 3.87 | 5.99 | 2.21 | 0.78 | 1.73 | 2.04 | 2.14 | 3.77
SnowFlakeNet [7] | 1.16 | 3.32 | 2.41 | 2.97 | 5.46 | 3.58 | 3.82 | 2.72 | 6.72 | 2.80 | 4.10 | 1.80 | 0.52 | 1.90 | 1.82 | 2.70 | 3.18
VRCNet [5] | 1.15 | 3.20 | 2.14 | 3.58 | 5.57 | 3.58 | 4.17 | 2.47 | 6.90 | 2.76 | 3.45 | 1.78 | 0.59 | 1.52 | 1.83 | 1.57 | 3.06
Ours | 0.74 | 2.94 | 2.25 | 2.78 | 2.54 | 2.87 | 2.84 | 2.00 | 5.24 | 1.98 | 2.87 | 1.67 | 0.45 | 1.45 | 1.23 | 0.92 | 2.27
TABLE I: Completion results (L2 Chamfer distance $\times 10^{4}$) on MVP dataset (16,384 points). Method | airp. | cabinet | car | chair | lamp | sofa | table | waterc. | bed | bench | books. | bus | guitar | motor. | pistol | skateb. | Avg.
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
PCN [1] | 0.82 | 0.61 | 0.69 | 0.52 | 0.46 | 0.55 | 0.65 | 0.63 | 0.45 | 0.69 | 0.55 | 0.78 | 0.91 | 0.67 | 0.77 | 0.86 | 0.638
TopNet[37] | 0.79 | 0.62 | 0.61 | 0.44 | 0.39 | 0.51 | 0.64 | 0.61 | 0.41 | 0.68 | 0.52 | 0.77 | 0.87 | 0.62 | 0.73 | 0.84 | 0.601
MSN [3] | 0.88 | 0.69 | 0.69 | 0.60 | 0.60 | 0.63 | 0.73 | 0.80 | 0.57 | 0.80 | 0.64 | 0.81 | 0.94 | 0.73 | 0.81 | 0.89 | 0.710
Wang et. al. [2] | 0.90 | 0.69 | 0.73 | 0.67 | 0.68 | 0.64 | 0.75 | 0.74 | 0.60 | 0.80 | 0.66 | 0.80 | 0.93 | 0.77 | 0.84 | 0.90 | 0.740
ECG [4] | 0.91 | 0.68 | 0.72 | 0.68 | 0.73 | 0.65 | 0.77 | 0.75 | 0.64 | 0.82 | 0.71 | 0.80 | 0.95 | 0.78 | 0.84 | 0.90 | 0.753
GRNet [6] | 0.85 | 0.58 | 0.65 | 0.64 | 0.71 | 0.58 | 0.69 | 0.72 | 0.59 | 0.77 | 0.64 | 0.68 | 0.87 | 0.74 | 0.79 | 0.85 | 0.692
NSFA [38] | 0.90 | 0.69 | 0.72 | 0.74 | 0.78 | 0.71 | 0.82 | 0.80 | 0.69 | 0.85 | 0.75 | 0.82 | 0.93 | 0.82 | 0.86 | 0.89 | 0.783
SnowFlakeNet [7] | 0.92 | 0.70 | 0.73 | 0.72 | 0.78 | 0.70 | 0.80 | 0.79 | 0.68 | 0.85 | 0.73 | 0.82 | 0.95 | 0.80 | 0.87 | 0.92 | 0.782
VRCNet [5] | 0.93 | 0.72 | 0.76 | 0.74 | 0.79 | 0.70 | 0.81 | 0.80 | 0.65 | 0.86 | 0.76 | 0.83 | 0.96 | 0.83 | 0.89 | 0.93 | 0.796
Ours | 0.94 | 0.74 | 0.75 | 0.77 | 0.84 | 0.74 | 0.82 | 0.82 | 0.72 | 0.87 | 0.77 | 0.84 | 0.97 | 0.85 | 0.90 | 0.93 | 0.814
TABLE II: Point cloud completion results (F-Score@ 1%, higher is better) on MVP dataset (16,384 points). # Points | 2,048 | 4,096 | 8,192 | 16,384
---|---|---|---|---
CD | F1 | CD | F1 | CD | F1 | CD | F1
PCN [1] | 9.77 | 0.320 | 7.96 | 0.458 | 6.99 | 0.563 | 6.02 | 0.638
TopNet [37] | 10.11 | 0.308 | 8.20 | 0.440 | 7.00 | 0.533 | 6.36 | 0.601
MSN [3] | 7.90 | 0.432 | 6.17 | 0.585 | 5.42 | 0.659 | 4.90 | 0.710
Wang et. al. [2] | 7.25 | 0.434 | 5.83 | 0.569 | 4.90 | 0.680 | 4.30 | 0.740
ECG[4] | 6.64 | 0.476 | 5.41 | 0.585 | 4.18 | 0.690 | 3.58 | 0.753
VRCNet [5] | 5.96 | 0.499 | 4.70 | 0.636 | 3.64 | 0.727 | 3.06 | 0.796
SnowFlakeNet [7] | 6.05 | 0.500 | 4.77 | 0.651 | 3.80 | 0.747 | 3.18 | 0.782
Ours | 5.10 | 0.526 | 3.49 | 0.682 | 3.14 | 0.756 | 2.27 | 0.814
TABLE III: Comparisons on MVP dataset with various resolutions.

##  4 Experiments

Figure 5: Qualitative completion results on the MVP dataset by different
methods.

###  4.1 Implementation Details

In this subsection, we will introduce the specific implementation details of
pretrain, prompt and predict in our CP3 paradigm, as well as the training
details of the whole framework. Firstly, we use inverse prompt function to
construct the pretraining pairs, where the threshold $\Gamma$ of IOI
pretraining is set to 0.9. Then we pretrain the generation network in the
self-supervised way. After training the generation stage, we fix it to extract
the global feature and point-wise feature of the last layer, and feed them to
train the refinement stage. In semantic-guided predict stage, we use three
blocks to recover the local details iteratively, based on the imperfect point
cloud from prompt stage, as shown in Fig. 4. For various resolutions of
datasets, we can adjust the upsampling multiplier in the MPD block to upsample
at different scales in training, e.g., for 2,048 points of imperfect point
cloud and 16,384 points of target point cloud, we can set the multipliers of
three MPD blocks to $[1,1,8]$ and choose local regions of multi-scale
transformer with 12 and 24. The loss functions of generation stage is the KL
divergence loss [5] and CD loss, the refinement loss function is CD loss.

For training details, the batch size is set to 64. The optimizer is Adam
optimizer with initial learning rate $1\times 10^{-4}$ (decayed by 0.7 every
40 epochs). For the environment, our framework is implemented by PyTorch 1.5
and RTX 8000 GPUs (each with 48 GB memory consumption). As for resource usage,
our modulation module contains only less than 43k parameters. The model
parameters / inference time of our SCRNet is 20.7M / 0.52s, PoinTr [11] is
30.9M / 0.65s, SnowFlakeNet [7] is 19.2M / 0.61s, but we have the best
completion result (as Table III and IV shows).

###  4.2 Comparison with State-Of-The-Art

Evaluation on MVP Dataset. The MVP dataset is generated by CAD models selected
from ShapeNet [39]. Compared with other datasets [1, 37], the MVP dataset is
more challenging in incomplete variations (partial point clouds rendered from
26 views) and rich categories (16 categories). In order to accurately evaluate
the quality of the completions at different resolutions, it also provides
complete point clouds with different resolutions, including 2048 (1x), 4096
(2x), 8192 (4x) and 16384 (8x). For evaluation, we follow the same metrics
with VRCNet [5] to fairly compare our method with other methods, where L2
version of Chamfer Distance (CD) [37] is used to evaluate average closest
point distance and align with previous studies, and F-score [40] evaluates the
distance between object surfaces, which is defined as the harmonic mean
between precision and recall. The evaluated CD and F-score for all evaluated
methods (16,384 points) are reported in Table I and II, respectively.
Additionally, we re-implemented the official code of SnowFlakeNet [7] for fair
comparison. We can find that our method can achieve the best performance in CD
loss with 2.27 and F-score @ 1% with 0.814. We compare our methods with
existing methods that support multi-resolution completion in Table III. Our
method outperforms all the other methods with a large margin.

Evaluation on PCN Dataset. The widely-used PCN [1] dataset is a subset of the
ShapeNet dataset [39] with 8 categories. The incomplete shapes are generated
by backprojecting complete shapes into 8 different partial views. For each
complete shape, 16,384 points are evenly sampled from the shape surface. For
fair comparison, we adopt the L1 version of chamfer distance as the evaluation
metric, which follows the same practice as previous methods [1, 11, 7]. The
results are summarized in Table IV, our method can achieve 7.02 performance
and outperform other state-of-the-art methods in most categories.

Methods | Plane | Cabinet | Car | Chair | Lamp | Couch | Table | Boat | Avg.
---|---|---|---|---|---|---|---|---|---
FoldingNet [20] | 9.49 | 15.80 | 12.61 | 15.55 | 16.41 | 15.97 | 13.65 | 14.99 | 14.31
TopNet [37] | 7.61 | 13.31 | 10.90 | 13.82 | 14.44 | 14.78 | 11.22 | 11.12 | 12.15
AtlasNet [41] | 6.37 | 11.94 | 10.10 | 12.06 | 12.37 | 12.99 | 10.33 | 10.61 | 10.85
PCN[1] | 5.50 | 22.70 | 10.63 | 8.70 | 11.00 | 11.34 | 11.68 | 8.59 | 9.64
GRNet [6] | 6.45 | 10.37 | 9.45 | 9.41 | 7.96 | 10.51 | 8.44 | 8.04 | 8.83
CDN [2] | 4.79 | 9.97 | 8.31 | 9.49 | 8.94 | 10.69 | 7.81 | 8.05 | 8.51
PMP-Net[42] | 5.65 | 11.24 | 9.64 | 9.51 | 6.95 | 10.83 | 8.72 | 7.25 | 8.73
NSFA [38] | 4.76 | 10.18 | 8.63 | 8.53 | 7.03 | 10.53 | 7.35 | 7.48 | 8.06
PoinTr [11] | 4.75 | 10.47 | 8.68 | 9.39 | 7.75 | 10.93 | 7.78 | 7.29 | 8.38
SCRN [43] | 4.80 | 9.94 | 9.31 | 8.78 | 8.66 | 9.38 | 7.20 | 7.91 | 8.29
SnowFlakeNet [7] | 4.29 | 9.16 | 8.08 | 7.89 | 6.07 | 9.23 | 6.55 | 6.40 | 7.21
Ours | 4.34 | 9.02 | 7.90 | 7.41 | 6.35 | 8.52 | 6.32 | 6.26 | 7.02
TABLE IV: Comparisons on PCN dataset (L1 Chamfer distance $\times 10^{3}$).

Qualitative Analysis of different methods.  To make comparisons under the
challenging situations, we visualize the completion results of different
methods on MVP dataset with 16,384 points. The qualitative results are shown
in Fig. 5. Compared to other methods, our proposed CP3 can recover better
complete shapes with fine details. Our method largely improves completion for
confused classes (e.g., table & chair). Especially for objects with
challenging partial areas and complex geometry, our method can achieve the
precise and reasonable completion, e.g., for the lamp with fine geometric
details, we recover the details to the greatest extent. For the pistol and
motorbike with large partial areas, our compltions are not only detailed, but
also has less noise and a more uniform local density, we can see that there is
less noise at the trigger and motor tires.

###  4.3 Analysis

Figure 6: Qualitative completion results (2,048 points) by different ablation
settings.

####  4.3.1 Ablation Studies of Overall Framework

In order to examine the effectiveness of our designs, we conducted detailed
ablation studies based on our framework for MVP dataset (2048 points). The
results are summarized in Table V. ($i$) only has the generation stage
(prompt), where the result is 5.96. Based on the prompt, ($ii$) adds the
predict stage to point cloud refinement. The result improves to 5.80 due to
our semantic modulation strategy for discriminative shape recovery. ($iii$)
and ($iv$) add the IOI pretraining paradigm. When the completion model only
includes the generation stage, the result is improved to 5.69. While we use
the entire framework of CP3, the result can be improved to 5.10. It can be
concluded that our proposed IOI pretraining and semantic-guided refinement can
greatly improve the performance of point cloud completion.

To qualitatively validate our framework, we conduct a visual comparison of
some of the models in Table V. The results are shown in Figure 6. We can
clearly observe that complete results of $i$ (baseline) have some defects,
e.g., the density of incomplete area and complete area is inconsistent, noise
points will be generated in the hollowed out area and some missing regions can
not be complete enough. With the IOI pretrain $iii$ , the large missing area
can be more complete. e.g., compared with setting $i$ , the incomplete area of
the bench is more complete in setting $iii$ . When adding the semantic
conditional network for refinement (setting $iv$), the density problem can be
solved effectively and the missing areas can be more complete.

CP3 | Pretrain (IOI) | Prompt | Predict | CD
---|---|---|---|---
$i$ | $\times$ | ✓ | $\times$ | 5.96
$ii$ | $\times$ | ✓ | ✓ | 5.80
$iii$ | ✓ | ✓ | $\times$ | 5.69
$iv$ | ✓ | ✓ | ✓ | 5.10
TABLE V: Ablation studies of the CP3 framework on MVP dataset with 2048 points (CD multiplied by $10^{4}$), “$\checkmark$” means that we use the corresponding strategies, while “$\times$” means that the relevant strategies is not applicable. | Pretraining input | Pretraining Target | CD
---|---|---|---
Pre-1 | Mirrored Incompletion | Completion | 7.59
Pre-2 | Jittered Incompletion | Completion | 7.16
Pre-3 | Incompletion | Completion | 5.96
Pre-4 | Incompletion | Incompletion | 5.78
Pre-5 | Incompletion-of-Incompletion | Completion | 6.03
Pre-6 |  hybrid training as SCRN[43] |  | 5.75
Pre-7 | Incompletion-of-Incompletion | Incompletion | 5.69
TABLE VI: Ablation tudies of IOI pretraining: different pretraining settings. threshold $\Gamma$ | 0.1 | 0.3 | 0.5 | 0.7 | 0.9
---|---|---|---|---|---
CD | 5.80 | 5.78 | 5.73 | 5.71 | 5.69
TABLE VII: Ablation studies of IOI pretraining: different maximun threshold of the crop rate. Generation Network | w/o IOI Pre. | w IOI Pre.
---|---|---
PCN [1] | 9.77 | 8.51
ECG [4] | 6.64 | 6.34
VRCNet [5] | 5.96 | 5.69
TABLE VIII: Ablation studies of IOI pretraining: different generation
networks. Figure 7:  Evaluating with wrong category guidance. Left:
Qualitative evaluation results of different setting to the category guidance.
Right: Comparisons of correct/wrong category guidance for evaluation on MVP
dataset (Lower is better).

####  4.3.2 Ablation Studies of IOI Pretraining

Why does the IOI pretraining work? To verify the effectiveness of our IOI
pretraining, we compare a variety of pretraining pairs in Table VI. When we
use the incompletion to reconstruct itself (“Pre-4”), the result can improve
to 5.78, because the model learns the distribution of each incomplete point
cloud independently, and then plays a better generalization in the generation
stage. Next we try to use completion as a pretext task (“Pre-5 ” and “Pre-7”).
Compared with incomplete point cloud, the incomplete of incomplete point cloud
has a greater degree and more diverse partial areas, because it has undergone
the twice incomplete treatment. It is difficult for the network to complete
the IOI input to the complete target. Therefore, it is more appropriate to
process the augment input to the original incomplete point cloud. As we can
see, “Pre-5 ” is 6.03, while “Pre-7 ” improves to 5.69. We also consider some
commonly used data diversified operations in point cloud, e.g., mirroring and
jittering the point cloud, demonstrating by the “Pre-1” and “Pre-2”. As for
comparison with other pretraining method for generation stage, we used the
same pretraining strategy as SCRN [43]. SCRN is a hybrid training of shape
completion (“IOI” to “I”) and partial reconstruction (“IOI” to “IOI”) in a
parallel manner. Such pretraining may mislead the model, by using the similar
input but with different supervision. In contrast, our CP3 follows a serial
training manner from NLP, with “IOI” to “I” for pretraining and then “I” to
“C” for prompting. To show our effectiveness, we conduct such parallel style
of SCRN as “Pre-6 ”. The CD loss ($\times 10^{4}$) of parallel manner is worse
(parallel vs our serial: 5.75 vs 5.69). In addition, based on “Pre-7”, we also
compare the maximum threshold $\Gamma$ of the crop rate based on the IOI
pretraining, which is shown in Table VII. Through the series of experiments,
we finally determine the threshold $\Gamma$ is 0.9.

Figure 8: Left: Comparisons of different semantic guidance operation. Right:
Details of feature affination.

Can IOI pretrain paradigm work for other networks?  Here we compare the
effectiveness of IOI pretraining for different networks (e.g., PCN [1], ECG
[4] and VRCNet [5]) in Table VIII. With IOI pretraining, the performance of
such mainstream completion networks is greatly improved in varying degrees. It
can be concluded that the IOI pretraining is a general and effective new
training paradigm for point cloud completion task.

####  4.3.3 Ablation Studies of Semantic Refinement

Why does the semantic refinement work? To validate the designs of our predict
(semantic-guided refinement) stage, we first conduct three different
variations as Table IX shows. When we use the basic refinement network without
semantic-guided modulation and multi-scale transformer strategy, the result is
5.60. With the semantic modulation, the charmer distance reduces to 5.17.
Subsequently, when adding the multi-scale strategy in the MPD blocks, the
final result can achieve to 5.10. Moreover, we investigate different
conditional information in Table X. It can be concluded from the experiments
that under the modulation of category semantics and global learnable semantic,
the performance can be greatly improved. Specially, the category semantics is
more effective as the conditional information. In order to further validate
Semantic-Guided Modulation (SGM) block, we conduct a series of comparative
experiments of whether to use the SGM block in each layer of our SCRNet, which
is shown in Table XI. It is relatively more efficient to use semantic-guided
modulation blocks at the front layers. Moreover, for the multi-scale strategy,
we conduct different settings for the transformer scale in Table XII, which
indicates that it is more effective to choose a larger local pattern of
fusion.

Why semantic label contributes to completion task? Analysis from the
perspective of point-wise representation, our SGM modulate seach point-wise
feature with semantic label. This can effectively distinguish point clouds
which have similar local details but belong to different categories. In Fig.
9, we show point-wise feature distribution of two point clouds (chair and
table), Clearly, the point-wise features of two point clouds (chair and table)
are distinguished after using our SGM.

Figure 9: TSNE visualization of features before and after semantic guided
modulation.

Evaluating with wrong category guidance. To verify the effectiveness of
semantic-guided modulation on different categories, we use model $iv$ in Table
V to evaluate our CP3 with different strategies of category guidance. The
results are shown in Fig. 7. When we randomly replace the categories of all
test point clouds, and the CD loss becomes larger (i.e., 5.68). For further
analysis, we show the test results for all categories. We can see that, for a
small number of categories with high discrimination (e.g., airplane,
watercraft, guitar), their completions are not greatly affected by the wrong
guidance. While for most confusing categories with similar structures, such as
chair sofa and table, the evaluation results are heavily deteriorated by the
wrong guidance. The visualization of chair and table in Fig. 7 can also verify
this viewpoint. The local area will have more noise points under the wrong
guidance than the correct category guidance. It shows that our semantic
conditional modulation network is effective to reduce shape confusion.

Semantic modulation | Multiscale operation | CD
---|---|---
$\times$ | $\times$ | 5.60
$\checkmark$ | $\times$ | 5.17
$\checkmark$ | $\checkmark$ | 5.10
TABLE IX: Ablation studies of our SCRNet for refnement: SGM and MPD blocks in the refinement.  Conditional information | CD
---|---
Without semantics | 5.60
Category semantics | 5.17
Global learnable semantics | 5.25
Category + Global learnable semantics | 5.22
TABLE X: Ablation studies of our SCRNet for refnement: conditional information in our semantic-guided modulation.  1-st layer | 2-nd layer | 3-rd layer | CD
---|---|---|---
$\checkmark$ | $\times$ | $\times$ | 5.17
$\times$ | $\checkmark$ | $\times$ | 5.18
$\times$ | $\times$ | $\checkmark$ | 5.20
TABLE XI: Ablation studies of our SCRNet for refnement: whether to use the semantic-guided modulation block in each layer. Strategy |  local regions ($k$) | CD
---|---|---
single-scale | [12] | 5.17
multi-scale | [12, 24] | 5.10
multi-scale | [6,12] | 5.23
TABLE XII: Ablation studies of our SCRNet for refnement: $k$-nearest neighbours of the multi-scale strategy.  Method | CD
---|---
PCN [1] | 16.56
ECG [4] | 15.23
PoinTr [11] | 10.31
VRCNet [5] | 9.08
SnowFlakeNet [7] | 8.25
Ours | 7.64
TABLE XIII: Generalization to novel unseen categories on MVP dataset.  |  PCN [1] |  FoldingNet [20] |  NSFA[38] |  TopNet[37] |  GRNet[6] | Ours
---|---|---|---|---|---|---
Consistency $\downarrow$ | 1.56 | 1.05 | - | 0.57 | 0.31 | 0.29
Fidelity error $\downarrow$ | 0.034 | 0.036 | 0.026 | 0.031 | - | 0.013
TABLE XIV: Quantitative comparisons on KITTI dataset.  Figure 10:  Qualitative
results on the KITTI dataset. Our method generates more complete and locally
detailed shapes for the real scanned cars.

Comparison with other semantic guidance operations. Here we investigate
different operations for feature modulation. In addition to our semantic
guided modualtion, we also try concatenation and feature affination which
inspired by [30, 44]. As Figure 8 (right) shows, feature affination aims at
scaling and shifting the intermediate features $\mathcal{{K}}$ by the semantic
information. This operation can be formulated as:
$\mathcal{\hat{K}}=\sigma(\mathcal{{K}})\odot{\alpha}+\beta$ , where $\odot$
denotes the element-wise multiplication operation and $\sigma$ is MLP layers,
$\mathcal{{K}}\in\mathrm{R}^{N\times C}$ is the intermediate displacement
features from MPD block, ${\alpha},\beta\in\mathrm{R}^{1\times C}$ are affine
parameters that are estimated from the point cloud category labels
$S\in\mathrm{R}^{1\times C_{cate}}$ via MLP layers. We use conditional vector
$\alpha$ to affect the cluster centers of the local representation and use
conditional vector $\beta$ to fine-tune the variance in the feature space.
Thus, we can achieve point feature global adjustment with only few parameters.
The comparisons of different modulate operations are reported in Fig. 8
(left), From the looks of it, PointNet [36] simply concatenates the category
and point-wise features, which ignores the discriminative interaction between
them. Alternatively, our SGM converts category into a dynamical filter to
modulate all the point-wise features with semantic information. Our semantic
guided modulation and feature affination provide a performance boost.
Therefore, the model is not easy to be confused with the similar local
structures of different semantic information.

####  4.3.4 Generalization Analysis to novel unseen categories

To verify the generalization ability to novel unseen categories, following the
the setting of [11], we take the first 8 categories of MVP [5] for training
and the rest 8 categories of the test data for evaluation. As shown in Table
XIII, we compare our method with some sota methods, e.g., PoinTr [11], VRCNet
[5], SnowFlakeNet [7]. Under such challenging situation, our method can
achieve the best performance of 7.64. While VRCNet only achieves 9.08 and
SnowFlakeNet achieves 8.25.

####  4.3.5 Completion on real-world Kitti dataset

In order to validate the performance of our CP3 in real-world scenarios, we
fine tune our model on ShapeNet-car dataset and test on KITTI-cars [45],
following [5, 1]. To show the performance of our CP3 in real-world KITTI-cars
dataset, we report the Fidelity error and Consistency in Table XIV. Fidelity
is calculated from the average distance from each point in the partial input
to its nearest neighbour in the output. This measures how well the input is
preserved. Consistency is calculated from the average CD distance between the
outputs of the same object in successive frames. More details on the
evaluation metrics can be found in [1, 6]. Compared to other methods, our CP3
has better consistency between the two consecutive frames and better fidelity
error. Moreover, we also show the qualitative visualization results are shown
in Fig. 10, our results are more complete overall and less noisy locally. The
bodywork of the car has also been refined in detail.

##  5 Conclusion

In this paper, we creatively propose a new paradigm called CP3 for point cloud
completion, including IOI pretraining, prompting (generation) and predicting
(refinement). IOI pretraining boosts the robustness of generation network by
learning a pretext task internally in a self-supervised way. Moreover, we
design a novel semantic conditional modulation for point cloud refinement,
which uses semantic information as guidance to adaptively modulate point cloud
representation for discriminative recovery. Extensive experimental results and
visualizations validate the effectiveness of our approach.

Broader Impact. The core of our CP3 is a generic training paradigm, inspired
by the well-established natural language processing. Hence, it would be
interesting to extend this paradigm for other 3D tasks like point cloud scene
understanding. Furthermore, it is worth investigating whether it is an
effective method for traditional vision tasks like image inpainting.
Hopefully, such paradigm can leverage benefits from both CV and NLP for better
representation learning in a unified manner.

## References

  * [1] W. Yuan, T. Khot, D. Held, C. Mertz, and M. Hebert, “Pcn: Point completion network,” in _3DV_ , 2018.
  * [2] X. Wang, M. H. Ang Jr, and G. H. Lee, “Cascaded refinement network for point cloud completion,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020.
  * [3] M. Liu, L. Sheng, S. Yang, J. Shao, and S.-M. Hu, “Morphing and sampling network for dense point cloud completion,” in _Proceedings of the AAAI conference on artificial intelligence_ , 2020.
  * [4] L. Pan, “Ecg: Edge-aware point cloud completion with graph convolution,” _IEEE Robotics and Automation Letters_ , 2020.
  * [5] L. Pan, X. Chen, Z. Cai, J. Zhang, H. Zhao, S. Yi, and Z. Liu, “Variational relational point completion network,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021.
  * [6] H. Xie, H. Yao, S. Zhou, J. Mao, S. Zhang, and W. Sun, “Grnet: Gridding residual network for dense point cloud completion,” in _European Conference on Computer Vision_ , 2020.
  * [7] P. Xiang, X. Wen, Y.-S. Liu, Y.-P. Cao, P. Wan, W. Zheng, and Z. Han, “Snowflakenet: Point cloud completion by snowflake point deconvolution with skip-transformer,” _arXiv preprint arXiv:2108.04444_ , 2021.
  * [8] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” _arXiv preprint arXiv:2107.13586_ , 2021.
  * [9] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, “Finetuned language models are zero-shot learners,” _arXiv preprint arXiv:2109.01652_ , 2021.
  * [10] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell _et al._ , “Language models are few-shot learners,” _Advances in neural information processing systems_ , 2020.
  * [11] X. Yu, Y. Rao, Z. Wang, Z. Liu, J. Lu, and J. Zhou, “Pointr: Diverse point cloud completion with geometry-aware transformers,” in _Proceedings of the IEEE international conference on computer vision_ , 2021.
  * [12] S. Shalom, A. Shamir, H. Zhang, and D. Cohen-Or, “Cone carving for surface reconstruction,” in _ACM SIGGRAPH Asia_ , 2010.
  * [13] M. Attene, “A lightweight approach to repairing digitized polygon meshes,” _The visual computer_ , 2010.
  * [14] J. Lin, X. Jin, and C. C. Wang, “Fusion of disconnected mesh components with branching shapes,” _The Visual Computer_ , 2010.
  * [15] W. Hu, Z. Fu, and Z. Guo, “Local frequency interpretation and non-local self-similarity on graph for point cloud inpainting,” _IEEE Transactions on Image Processing_ , 2019.
  * [16] D. T. Nguyen, B.-S. Hua, K. Tran, Q.-H. Pham, and S.-K. Yeung, “A field model for repairing 3d shapes,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2016.
  * [17] C.-H. Shen, H. Fu, K. Chen, and S.-M. Hu, “Structure recovery by part assembly,” _ACM Transactions on Graphics (TOG)_ , 2012.
  * [18] T. Shao, W. Xu, K. Zhou, J. Wang, D. Li, and B. Guo, “An interactive approach to semantic modeling of indoor scenes with an rgbd camera,” _ACM Transactions on Graphics (TOG)_ , 2012.
  * [19] A. Martinovic and L. Van Gool, “Bayesian grammar learning for inverse procedural modeling,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2013.
  * [20] Y. Yang, C. Feng, Y. Shen, and D. Tian, “Foldingnet: Point cloud auto-encoder via deep grid deformation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2018.
  * [21] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017.
  * [22] A. Dai, C. Diller, and M. Nießner, “Sg-nn: Sparse generative neural networks for self-supervised scene completion of rgb-d scans,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020.
  * [23] M. Mirza and S. Osindero, “Conditional generative adversarial nets,” _arXiv preprint arXiv:1411.1784_ , 2014.
  * [24] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” _Advances in neural information processing systems_ , 2014.
  * [25] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee, “Generative adversarial text to image synthesis,” in _ICML_ , 2016.
  * [26] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2017.
  * [27] X. Huang and S. Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” in _Proceedings of the IEEE international conference on computer vision_ , 2017.
  * [28] T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019.
  * [29] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, “Analyzing and improving the image quality of stylegan,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020.
  * [30] J. He, Y. Liu, Y. Qiao, and C. Dong, “Conditional sequential modulation for efficient global image retouching,” in _European Conference on Computer Vision_ , 2020.
  * [31] B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” _arXiv preprint arXiv:2104.08691_ , 2021\.
  * [32] Y. Lu, M. Bartolo, A. Moore, S. Riedel, and P. Stenetorp, “Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity,” _arXiv preprint arXiv:2104.08786_ , 2021.
  * [33] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever _et al._ , “Language models are unsupervised multitask learners,” _OpenAI blog_ , 2019\.
  * [34] F. Petroni, T. Rocktäschel, P. Lewis, A. Bakhtin, Y. Wu, A. H. Miller, and S. Riedel, “Language models as knowledge bases?” _arXiv preprint arXiv:1909.01066_ , 2019.
  * [35] K. Zhou, J. Yang, C. C. Loy, and Z. Liu, “Learning to prompt for vision-language models,” _arXiv preprint arXiv:2109.01134_ , 2021.
  * [36] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2017.
  * [37] L. P. Tchapmi, V. Kosaraju, H. Rezatofighi, I. Reid, and S. Savarese, “Topnet: Structural point cloud decoder,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019.
  * [38] W. Zhang, Q. Yan, and C. Xiao, “Detail preserved point cloud completion via separated feature aggregation,” in _European Conference on Computer Vision_ , 2020.
  * [39] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao, “3d shapenets: A deep representation for volumetric shapes,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2015.
  * [40] A. Knapitsch, J. Park, Q.-Y. Zhou, and V. Koltun, “Tanks and temples: Benchmarking large-scale scene reconstruction,” _ACM Transactions on Graphics (ToG)_ , 2017.
  * [41] T. Groueix, M. Fisher, V. G. Kim, B. C. Russell, and M. Aubry, “A papier-mâché approach to learning 3d surface generation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2018.
  * [42] X. Wen, P. Xiang, Z. Han, Y.-P. Cao, P. Wan, W. Zheng, and Y.-S. Liu, “Pmp-net: Point cloud completion by learning multi-step point moving paths,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_.
  * [43] X. Wang, M. H. Ang, and G. Lee, “Cascaded refinement network for point cloud completion with self-supervision,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * [44] Y. Liu, J. He, X. Chen, Z. Zhang, H. Zhao, C. Dong, and Y. Qiao, “Very lightweight photo retouching network with conditional sequential modulation,” _arXiv preprint arXiv:2104.06279_ , 2021.
  * [45] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. IEEE, 2012.


# Coordinate Interleaved Orthogonal Design with Media-Based Modulation

Ibrahim Yildirim, , Ertugrul Basar, and
Ibrahim Altunbas This work was supported by the Scientific and Technological
Research Council of Turkey (TUBITAK) under Grant 117E869.I. Yildirim and I.
Altunbas are with Istanbul Technical University, Istanbul, Turkey. (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>E. Basar is with Koç
University, Istanbul, Turkey. (e-mail<EMAIL_ADDRESS>
###### Abstract

In this work, we propose a new multiple-input multiple-output (MIMO) concept,
which is called coordinate interleaved orthogonal design with media-based
modulation (CIOD-MBM). The proposed two novel CIOD-MBM schemes provide
improved data rates as well as diversity gain while enabling hardware
simplicity using a single radio frequency (RF) chain. Moreover, using the
equivalent channel model, a reduced complexity can be obtained for maximum
likelihood (ML) detection of the proposed system. Using computer simulations,
it is shown that CIOD-MBM schemes provide remarkably better performance
against the conventional MBM and CIOD systems.

###### Index Terms:

Space-time coding, coordinate interleaved orthogonal design, index modulation,
media-based modulation.

##  I Introduction

With the proliferation of applications using wireless communication systems in
recent years, a huge increase in the number of users and data traffic has
emerged. It is envisaged that next-generation wireless communication
applications such as Internet-of-Things, vehicle-to-everything communications
and molecular communications will be introduced into daily life in the near
future [1]. Despite the increasing demand from day to day, the available
resources remain limited when considering the frequency spectrum and physical
hardware implementations. In next-generation communication systems, under such
resource constraints, it is aimed to develop methods with high band
efficiency, channel capacity and energy efficiency. All around the world,
researchers have eagerly worked to meet these growing demands. As a result of
these efforts, many versatile and highly successful methods have been put
forward.

Index modulation (IM) schemes, which achieve high data rates while reducing
hardware costs in the transmitter, have recently received remarkable
attention. Spatial modulation (SM), the well-known member of the IM family,
conveys information by transmit antenna indices as well as conventional phase
shift keying (PSK) or quadrature amplitude modulation (QAM) methods [2]. When
high data rates are desired in SM systems, it is required to make giant
increases in the number of transmitting antennas. From this point of view, the
media-based modulation (MBM) scheme, which is proposed as an avant-garde
digital transmission method for rich scattering environments, is considered as
an alternative to SM [3]. In MBM systems, the current distributions of the
reconfigurable antennas are changed by means of various elements, such as PIN
diodes, varactors, micro-electro-mechanical switches, and so on, hence,
different channel fading realizations, which is called as channel states, are
obtained. Using the on/off status of the parasitic elements called radio
frequency (RF) mirrors, different radiation patterns are acquired by varying
current characteristics. If these antenna radiation patterns are sufficiently
uncorrelated, distinguishable channel fading realizations can be obtained and
used as an additional information source in a rich scattering environment.

Since a single transmit antenna will be active in SM and MBM systems during
the transmission, inter-channel interference (ICI) problem is eliminated. On
the other hand, the transmission via a single active antenna means that a
single RF chain is used in the transmitter part, which greatly reduces the
hardware cost. Furthermore, unlike SM, since the spectral efficiency in the
MBM varies linearly proportional to the number of RF mirrors, it is sufficient
to increase the number of RF mirrors in a bid to achieve high data rates.

Existing multiple-input multiple-output (MIMO) techniques such as SM,
quadrature SM (QSM) and generalized SM (GSM) can be integrated with MBM to
achieve promising results. In [4], GSM is adopted to MBM in order to enhance
spectral efficiency and error performance. Combining QSM and MBM innovatively,
promising results are obtained by quadrature channel modulation (QCM) [5].
Space-time coding concept also can be merged with MBM to increase the
diversity gain. To this end, space-time channel modulation (STCM) method is
proposed in [6], which combines Alamouti’s famous space-time block code (STBC)
with MBM. Additionally, considering the flexibility of IM techniques, time-
indexed based SM, MBM, and SM-MBM schemes are proposed in [7]. In recent past,
since MBM systems are prone to increase physical layer security, various
secret transmission scenarios are proposed to enable high secrecy capacity
using appropriate designs [8, 9]. Differential MBM (DMBM) [10] concept has
alleviated the requirement of channel sounding when it is hard to get accurate
channel state information (CSI) at the destination. In [11], a comprehensive
frame for space-time coded MBM systems is presented and STBC-based MBM scheme,
which uses a single RF chain, is proposed and miscellaneous diversity gains
are obtained. In order to overcome fractional bit problems in IM systems,
authors propose fractional MBM with golden angle modulation (GAM-MBM) in [12]
by making use of the symmetrical structure of GAM.

In this paper, we introduce coordinate interleaved orthogonal design MBM
(CIOD-MBM) concept, which provides diversity gain as well as increased
spectral efficiency while assuring hardware simplicity using a single RF
chain. The focus of this work is to bring the desirable benefits of CIOD, such
as enabling low complexity maximum likelihood (ML) detection opportunity and
simple transmitter structure, into the area of interest of MBM. In this
context, two promising CIOD-MBM schemes, which use a single active antenna
during the transmission, are presented. Furthermore, the theoretical average
bit error probability (ABEP) of the proposed CIOD-MBM schemes are derived.
Afterwards, the supremacy of the proposed CIOD-MBM concept is demonstrated via
computer simulations in terms of error performance and obtained numerical
results are verified with theoretical results.

##  II Coordinate Interleave Orthogonal Designs for Media-Based Modulation

Figure 1: Block diagram of $N_{r}\times N_{t}$ MIMO system which is employed
by CIOD-MBM schemes.

In this section, we introduce the system model and the working principle of
CIOD-MBM schemes under quasi-static flat Rayleigh fading channel. $N_{r}\times
N_{t}$ MIMO transmission scenario is considered by assuming each transmit
antenna has $N_{rf}$ RF mirrors. In the proposed systems, CIOD scheme
contributes to MBM transmissions as an additional source of information.

The essential principle of CIOD is to achieve diversity gain by transmitting
symbols through different channels in two different time slots. Due to the
transmission using a single active antenna, reduced hardware cost is obtained
keeping only one RF chain. Unlike traditional STBCs, signal constellation must
be rotated at a specific angle to ensure the full diversity of CIODs [13]. The
CIOD signal matrix for the system with two transmit antennas is as follows

$\begin{bmatrix}[cc]\tilde{s_{0}}&0\\\ 0&\tilde{s_{1}}\\\ \end{bmatrix}$ (1)

where $\tilde{s_{0}}=x_{0}^{\Re}+jx_{1}^{\Im}$ and
$\tilde{s_{1}}=x_{1}^{\Re}+jx_{0}^{\Im}$ are transmitted symbols, which are
obtained by partitioning $x_{i}$ ($i\in\left\\{0,1\right\\}$) symbol into its
real ($x_{i}^{\Re}$) and imaginary ($x_{i}^{\Im}$) part. Here, $x_{0}$ and
$x_{1}$ are selected from a signal constellation formed by rotating the
ordinary $M$-ary signal constellation at a certain angle. For instance, the
optimum rotation angle in the quadrature PSK (QPSK) constellation is 13.2885°
for the conventional CIOD in (1) [14].

Considering peculiarities of CIOD, two novel CIOD-MBM schemes are designed for
a given MIMO system. The generalized baseband received signal model of the
CIOD-MBM schemes can be expressed as

$\mathbf{Y}=\mathbf{HX}+\mathbf{N}$ (2)

where $\mathbf{Y}\in\mathbb{C}^{N_{r}\times 2}$ is matrix of the received
signals in two different time slots, $\mathbf{H}\in\mathbb{C}^{N_{r}\times
N_{t}2^{N_{rf}}}$ is the channel matrix, and
$\mathbf{N}\in\mathbb{C}^{N_{r}\times 2}$ is the noise sample matrix,
respectively. At the same time,
$\mathbf{X}\in\mathbb{C}^{N_{t}2^{N_{rf}}\times 2}$ represents the CIOD
matrix, which is designed for two different MBM schemes. The elements of
$\mathbf{H}$ and $\mathbf{N}$ follow zero mean independent and identically
complex Gaussian distribution with variances $1$ and $N_{0}$, respectively.

###  II-A CIOD-MBM I Scheme

The block diagram of CIOD-MBM I scheme is given in Fig. 1, where
$\eta_{c}=\log_{2}(N_{t}/2)+N_{rf}$ and $\eta_{s}=2\log_{2}(M)$ respectively
denote the number of index and symbol bits sent in two slots. The spectral
efficieny of CIOD-MBM I scheme is calculated as

$\eta=\frac{\log_{2}(N_{t}/2)+N_{rf}+2\log_{2}(M)}{2}\text{\quad bit/s/Hz}.$
(3)

Before determining the active channel state, available antennas are divided
into two groups. Then, using the first $\log_{2}(N_{t}/2)$ bits, active
antenna index $k_{1}$ is determined for the first time slot. The group, to
which the first active antenna belongs, is omitted in the second selection.
For the determined antenna, active channel state $l$ is chosen depending on
$N_{rf}$ bits. In the second time slot, second active antenna index $k_{2}$ is
determined as $k_{2}=N_{t}/2+k_{1}$. Then, $l$th channel state is also
selected in the second time slot as similar as first time slot. After all,
transmitted symbols $x_{0}$ and $x_{1}$ are chosen by the aid of last
$2\log_{2}(M)$ bits. Following these procedures, the transmission matrix
$\mathbf{X}$ is constructed as

${\footnotesize\mathbf{X}=\begin{bmatrix}[c|c|c|c|c]\dots&\dots\overbrace{\text{$\tilde{s_{0}}$}}^{\text{$l$\text{th
state}}}\dots&\dots&0\quad\dots\quad 0&\dots\\\
\dots&\makebox[0.0pt][l]{$\smash{\underbrace{\phantom{\begin{matrix}\hphantom{0}{\dots\quad
0\quad\dots}\end{matrix}}}_{\text{$k_{1}\text{th
antenna}$}}}$}\hphantom{0}{\dots\quad
0\quad\dots}&\dots&\makebox[0.0pt][l]{$\smash{\underbrace{\phantom{\begin{matrix}\hphantom{0}{\dots\quad{\overbrace{\text{$\tilde{s_{1}}$}}^{\text{$l$\text{th
state}}}}\quad\dots}\end{matrix}}}_{\text{$k_{2}\text{th
antenna}$}}}$}\hphantom{0}{\dots\quad{\overbrace{\text{$\tilde{s_{1}}$}}^{\text{$l$\text{th
state}}}}\quad\dots}&\dots\\\ \\\ \end{bmatrix}^{T}}$ (4)

where $(.)^{T}$ denotes matrix transpose, and possible channel realizations
and time slots are represented by rows and columns, respectively.

The CIOD-MBM I scheme provides full diversity when the minimum determinant of
the distance matrix is maximized [15]. Under the full rank condition, the
minimum coding gain distance is expressed as

$\delta_{\min}=\underset{\mathbf{X},\hat{\mathbf{X}},\mathbf{X}\neq\hat{\mathbf{X}}}{\mathop{\min}}\,\det[(\mathbf{X}-\hat{\mathbf{X}})(\mathbf{X}-\hat{\mathbf{X}})^{H}].$
(5)

In CIOD-MBM I, the rotation angle, which guarantees full diversity, is
calculated as $13.2885$° for QPSK. If the system with $N_{t}=4$, $N_{rf}=1$
and $M=4$ (QPSK) is considered to exemplify the working mechanism of CIOD-MBM
I, spectral efficiency ($\eta$) is calculated as $3$ bit/s/Hz. In two time
slots, $2\eta=6$ incoming bits are processed as follows:

${\footnotesize\mathbf{b}=[\,\underbrace{1\
0}_{\mathop{\log_{2}(N_{t}/2)+N_{rf}}}\,\underbrace{1\
1}_{\mathop{\log_{2}(M)}}\,\underbrace{1\ 0}_{\mathop{\log_{2}({M})}}]}.$ (6)

Using the mapping rule of the CIOD-MBM as shown in Table I, first
$\log_{2}(N_{t}/2)+N_{rf}=2$ bits assign the active antenna index set
$\left\\{k_{1},k_{2}\right\\}$ and channel state $l$, respectively. After the
two active channel states are separately obtained, the next $[1\ 1]$ bits
modulate to determine rotated QPSK symbol $x_{0}=0.2299-j0.9732$. Thereafter,
last two bits $[1\ 0]$ specify the $x_{1}=-0.9732-j0.2299$ from rotated QPSK
symbol set. Finally, CIOD-MBM I matrix is obtained as

$\mathbf{X}=\begin{bmatrix}[cc|cc|cc|cc]0&0&x_{0}^{\Re}+jx_{1}^{\Im}&0&0&0&0&0\\\
0&0&0&0&0&0&x_{1}^{\Re}+jx_{0}^{\Im}&0\\\ \end{bmatrix}^{T}.$ (7)

Table I: CIOD-MBM index mapping rule for $N_{t}=4$ and $N_{rf}=1$ Bits | $\left\\{k_{1},k_{2}\right\\}$ | $l$ | $\mathbf{X}^{T}$
---|---|---|---
00 | $\left\\{1,3\right\\}$ | $1$ | $\begin{bmatrix}[cc|cc|cc|cc]\tilde{s_{0}}&0&0&0&0&0&0&0\\\ 0&0&0&0&\tilde{s_{1}}&0&0&0\\\ \end{bmatrix}$
01 | $\left\\{1,3\right\\}$ | $2$ | $\begin{bmatrix}[cc|cc|cc|cc]0&\tilde{s_{0}}&0&0&0&0&0&0\\\ 0&0&0&0&0&\tilde{s_{1}}&0&0\\\ \end{bmatrix}$
10 | $\left\\{2,4\right\\}$ | $1$ | $\begin{bmatrix}[cc|cc|cc|cc]0&0&\tilde{s_{0}}&0&0&0&0&0\\\ 0&0&0&0&0&0&\tilde{s_{1}}&0\\\ \end{bmatrix}$
11 | $\left\\{2,4\right\\}$ | $2$ | $\begin{bmatrix}[cc|cc|cc|cc]0&0&0&\tilde{s_{0}}&0&0&0&0\\\ 0&0&0&0&0&0&0&\tilde{s_{1}}\\\ \end{bmatrix}$

###  II-B CIOD-MBM II

The main motivation of this system is to increase the spectral efficiency of
the CIOD-MBM I scheme while maintaining diversity gain and other inherent
advantages. The block diagram of CIOD-MBM II can be seen from Fig. 1, which
the terms in parentheses represent active antenna and channel state indices of
CIOD-MBM II, for $\eta_{c}=N_{rf}-1+2\log_{2}(N_{t})$ and
$\eta_{s}=2\log_{2}(M)$. In the CIOD-MBM II scheme, which is inspired by QSM
[16] and QCM [5], active transmit antennas are individually selected for real
and imaginary parts of $\tilde{s_{0}}$ and $\tilde{s_{1}}$ symbols. The
spectral efficiency of the CIOD-MBM II is calculated as

$\eta=\frac{N_{rf}-1+2\log_{2}(N_{t})+2\log_{2}(M)}{2}\text{\quad bit/s/Hz}.$
(8)

In order to guarantee the utilization of a single RF chain at a single time
slot, a total of $2^{N_{rf}}$ channel states are divided into two groups
similar to the CIOD-MBM I. The first $2N_{rf}-1$ bits decide the active
channel state $l_{1}$ for the initial time slot. After determining the group,
to which $l_{1}$ belongs, active second channel state $l_{2}$ is determined as
$l_{2}=2^{N_{rf}}/2+l_{1}$ . Using $2\log_{2}(N_{t})$ bits, active antennas
$k_{1}$ and $k_{2}$ are selected for real and imaginary parts of transmitted
symbols, respectively. Then, the last $2\log_{2}(M)$ bits are used to specify
the modulated symbols $x_{0}$ and $x_{1}$ from rotated $M$-ary constellations.
Generalized transmission matrix is given as

$\displaystyle{\footnotesize\mathbf{X}=\begin{bmatrix}[c|c|c|c|c]\dots&\dots\overbrace{\text{$x_{0}^{\Re}$}}^{\text{$k_{1}$}}\dots\overbrace{\text{$jx_{1}^{\Im}$}}^{\text{$k_{2}$}}\dots&\dots&\dots\quad
0\quad\dots&\dots\\\
\dots&\makebox[0.0pt][l]{$\smash{\underbrace{\phantom{\begin{matrix}\hphantom{0}{\dots\quad
0\quad\dots}\end{matrix}}}_{\text{$l_{1}\text{th
state}$}}}$}\hphantom{0}{\dots\quad
0\quad\dots}&\dots&\makebox[0.0pt][l]{$\smash{\underbrace{\phantom{\begin{matrix}\hphantom{0}{\dots\overbrace{\text{$x_{1}^{\Re}$}}^{\text{$k_{1}$}}\dots\overbrace{\text{$jx_{0}^{\Im}$}}^{\text{$k_{2}$}}\dots}\end{matrix}}}_{\text{$l_{2}\text{th
state}$}}}$}\hphantom{0}{\dots\overbrace{\text{$x_{1}^{\Re}$}}^{\text{$k_{1}$}}\dots\overbrace{\text{$jx_{0}^{\Im}$}}^{\text{$k_{2}$}}\dots}&\dots\\\
\end{bmatrix}^{T}}$ (9)

where possible channel realizations and time slots are represented by rows and
columns, respectively.

Table II: CIOD-MBM II index mapping rule for $N_{t}=2$ and $N_{rf}=2$ Bits |  $\left\\{l_{1},l_{2}\right\\}$-$\left\\{k_{1},k_{2}\right\\}$ | $\mathbf{X}^{T}$
---|---|---
000 |  $\left\\{1,3\right\\}$ \- $\left\\{1,1\right\\}$ | $\begin{bmatrix}[cc|cc|cc|cc]x_{0}^{\Re}+jx_{1}^{\Im}&0&0&0&0&0&0&0\\\ 0&0&0&0&x_{1}^{\Re}+jx_{0}^{\Im}&0&0&0\\\ \end{bmatrix}$
001 |  $\left\\{1,3\right\\}$ \- $\left\\{1,2\right\\}$ | $\begin{bmatrix}[cc|cc|cc|cc]x_{0}^{\Re}+jx_{1}^{\Im}&0&0&0&0&0&0&0\\\ 0&0&0&0&x_{1}^{\Re}&jx_{0}^{\Im}&0&0\\\ \end{bmatrix}$
010 |  $\left\\{1,3\right\\}$ \- $\left\\{2,1\right\\}$ | $\begin{bmatrix}[cc|cc|cc|cc]x_{0}^{\Re}&jx_{1}^{\Im}&0&0&0&0&0&0\\\ 0&0&0&0&x_{1}^{\Re}+jx_{0}^{\Im}&0&0&0\\\ \end{bmatrix}$
011 |  $\left\\{1,3\right\\}$ \- $\left\\{2,2\right\\}$ | $\begin{bmatrix}[cc|cc|cc|cc]x_{0}^{\Re}&jx_{1}^{\Im}&0&0&0&0&0&0\\\ 0&0&0&0&x_{1}^{\Re}&jx_{0}^{\Im}&0&0\\\ \end{bmatrix}$
100 |  $\left\\{2,4\right\\}$ \- $\left\\{1,1\right\\}$ | $\begin{bmatrix}[cc|cc|cc|cc]0&0&x_{0}^{\Re}+jx_{1}^{\Im}&0&0&0&0&0\\\ 0&0&0&0&0&0&x_{1}^{\Re}+jx_{0}^{\Im}&0\\\ \end{bmatrix}$
101 |  $\left\\{2,4\right\\}$ \- $\left\\{1,2\right\\}$ | $\begin{bmatrix}[cc|cc|cc|cc]0&0&x_{0}^{\Re}+jx_{1}^{\Im}&0&0&0&0&0\\\ 0&0&0&0&0&0&x_{1}^{\Re}&jx_{0}^{\Im}\\\ \end{bmatrix}$
110 |  $\left\\{2,4\right\\}$ \- $\left\\{2,1\right\\}$ | $\begin{bmatrix}[cc|cc|cc|cc]0&0&x_{0}^{\Re}&jx_{1}^{\Im}&0&0&0&0\\\ 0&0&0&0&0&0&x_{1}^{\Re}+jx_{0}^{\Im}&0\\\ \end{bmatrix}$
111 |  $\left\\{2,4\right\\}$ \- $\left\\{2,2\right\\}$ | $\begin{bmatrix}[cc|cc|cc|cc]0&0&x_{0}^{\Re}&jx_{1}^{\Im}&0&0&0&0\\\ 0&0&0&0&0&0&x_{1}^{\Re}&jx_{0}^{\Im}\\\ \end{bmatrix}$

The system with $N_{t}=2$, $N_{rf}=2$ and $M=4$ is explained by the following
example. In this scenario, the spectral efficiency is calculated as $\eta=3.5$
bit/s/Hz. $30$° is the optimum angle that ensures full diversity and maximizes
the $\delta_{min}$ for QPSK in CIOD-MBM II while $8.6$° is the optimum angle
for $16$-QAM. For two time slots, it is assumed that incoming bits are
arranged as follows:

${\footnotesize\mathbf{b}=[\,\underbrace{1\ 0\
1}_{\mathop{N_{rf}-1+2\log_{2}(N_{t})}}\,\underbrace{0\
1}_{\mathop{\log_{2}(M)}}\,\underbrace{1\ 1}_{\mathop{\log_{2}({M})}}]}.$ (10)

With the aid of Table II, active antenna set $\left\\{k_{1},k_{2}\right\\}$
and channel index set $\left\\{l_{1},l_{2}\right\\}$ are determined by the
first $\eta_{c}=3$ index bits. Then, using $\eta_{s}$ symbol bits, $x_{0}$ and
$x_{1}$ are chosen from the rotated constellation as $-0.5+j0.866$ and
$0.5-j0.866$, respectively. In the final stage, transmission matrix is
constructed as

$\mathbf{X}=\begin{bmatrix}[cc|cc|cc|cc]0&0&-0.5-j0.866&0&0&0&0&0\\\
0&0&0&0&0&0&0.5&j0.866\\\ \end{bmatrix}^{T}.$ (11)

###  II-C ML Detection for Proposed CIOD-MBM Schemes

As mentioned previous sections, one of the major advantages of CIOD is its
symbol-by-symbol decoding possibility while keeping ML complexity remarkably
reduced. In this sense, in the proposed CIOD-MBM I scheme, the equivalent
system model of (2) is obtained by

$\mathbf{y}_{eq}=\mathbf{H}_{eq}\mathbf{x}_{eq}+\mathbf{n}_{eq}$ (12)

where $\mathbf{y}_{eq}\in\mathbb{C}^{4N_{r}\times 1}$,
$\mathbf{x}_{eq}=[x_{0}^{\Re}\ x_{0}^{\Im}\ x_{1}^{\Re}\ x_{1}^{\Im}]^{T}$ and
$\mathbf{n}_{eq}\in\mathbb{C}^{4N_{r}\times 1}$ represent the equivalent
received signal vector, the equivalent transmitted symbols and the equivalent
noise vector, respectively. The construction of generalized $\mathbf{y}_{eq}$
is shown by

$\mathbf{y}_{eq}=[y_{1,1}^{\Re}\ y_{2,1}^{\Re}\dots y_{N_{r},1}^{\Re}\
y_{1,1}^{\Im}\dots y_{N_{r},1}^{\Im}\dots y_{1,2}^{\Re}\dots
y_{N_{r},2}^{\Im}]^{T}$ (13)

where $y_{r,t}$ stands for received signal at $r$th receive antenna in $t$th
time slot.

$\mathbf{H}_{eq}=\setcounter{MaxMatrixCols}{13}\begin{bmatrix}[ccccccccccccc]h_{1,m_{1}}^{\Re}&h_{2,m_{1}}^{\Re}&\dots&h_{N_{r},m_{1}}^{\Re}&h_{1,m_{1}}^{\Im}&\dots&h_{N_{r},m_{1}}^{\Im}&0&\dots&0&0&\dots&0\\\
0&0&\dots&0&0&\dots&0&h_{1,m_{2}}^{\Re}&\dots&h_{N_{r},m_{2}}^{\Re}&h_{1,m_{2}}^{\Im}&\dots&h_{N_{r},m_{2}}^{\Im}\\\
0&0&\dots&0&0&\dots&0&-h_{1,m_{2}}^{\Im}&\dots&-h_{N_{r},m_{2}}^{\Im}&h_{1,m_{2}}^{\Re}&\dots&h_{N_{r},m_{2}}^{\Re}\\\
-h_{1,m_{1}}^{\Im}&-h_{2,m_{1}}^{\Im}&\dots&-h_{N_{r},m_{1}}^{\Im}&h_{1,m_{1}}^{\Re}&\dots&h_{N_{r},m_{1}}^{\Re}&0&\dots&0&0&\dots&0\\\
\end{bmatrix}^{T}$ (14)

Furthermore, $\mathbf{H}_{eq}\in\mathbb{C}^{4N_{r}\times 4}$ denotes the
equivalent channel matrix, which is specified in (14) for CIOD-MBM I. In this
notation, $h_{r,m_{i}}$ ($i\in\left\\{1,2\right\\}$) represents channel fading
coefficients between the selected active channel state ($m_{i}$) and $r$th
receive antennas, where $m_{i}$ is chosen from the possible $N_{t}2^{N_{r}f}$
channel realizations depending on $k$ and $l$ sets. Because of the
orthogonality of columns, the equivalent channel matrix is decomposed as
$\mathbf{H}_{eq}=[\mathcal{H}_{1}\ \mathcal{H}_{2}]$ where
$\mathcal{H}_{1}\in\mathbb{C}^{4N_{r}\times 2}$ and
$\mathcal{H}_{2}\in\mathbb{C}^{4N_{r}\times 2}$ represent first two and last
two columns of $\mathbf{H}_{eq}$. It can be easily observed that
$\mathcal{H}_{1}$ and $\mathcal{H}_{2}$ are orthogonal each other, therefore,
$x_{0}$ and $x_{1}$ are individually decoded for CIOD-MBM I.

As is common, considering the perfect CSI in CIOD-MBM I, ML detection is
employed at the receiver by

$[\hat{x_{0}},\hat{x_{1}},\hat{k}_{1},\hat{k}_{2},\hat{l}]=\arg\underset{x_{0},x_{1},k_{1},k_{2},l}{\mathop{\min}}\,{{\left\|\mathbf{y}-\mathbf{HX}\right\|}^{2}}$
(15)

where $\Arrowvert.\Arrowvert$ represents the Euclidean norm. Similarly, the
detection of the index bits is performed by minimization over
$k_{1},k_{2},l_{1},l_{2}$ indices in (15) instead of $k_{1},k_{2},l$ in CIOD-
MBM II scheme.

In CIOD-MBM I, considering all possible channel realizations, minimum ML
decision metrics of $x_{0}$ and $x_{1}$ are separately obtained by

$\displaystyle
d_{0}^{(k_{1},l)}=\underset{x_{0}}{\mathop{\min}}\,{{\left\|\mathbf{y}-\mathcal{H}_{1}[x_{0}^{\Re}\
x_{0}^{\Im}]^{T}\right\|}^{2}},$ $\displaystyle
d_{1}^{(k_{2},l)}=\underset{x_{1}}{\mathop{\min}}\,{{\left\|\mathbf{y}-\mathcal{H}_{2}[x_{1}^{\Re}\
x_{1}^{\Im}]^{T}\right\|}^{2}}.$ (16)

Thereafter, the minimum ML decision metric for a corresponding antenna and
antenna state indices $(k_{1},k_{2},l)$ is calculated by
$d^{(k_{1},k_{2},l)}=d_{0}^{(k_{1},l)}+d_{1}^{(k_{2},l)}$. After obtaining all
$d^{(k_{1},k_{2},l)}$, the most likely antenna and antenna state indices
combination is determined by
$(\hat{k}_{1},\hat{k}_{2},\hat{l})=\arg\min_{(k_{1},k_{2},l)}d^{(k_{1},k_{2},l)}$.
Using the most likely channel realizations, the transmitted data symbols are
estimated by
$(\hat{x}_{0},\hat{x}_{1})=(x_{0}^{(k_{1},l)},x_{1}^{(k_{2},l)})$.

After the demapping process, decoded bits are obtained at the receiver. Thanks
to the equivalent model, the number of calculated metrics is reduced
$2^{\eta_{c}}M^{2}$ to $2^{\eta_{c}}2M$ for CIOD-MBM I scheme. Due to the non-
ortogonality of equivalent channel matrix for CIOD-MBM II, brute-force ML
detection is performed as in (15) with a complexity order of
$2^{\eta_{c}}M^{2}$.

##  III Performance Analysis

By the aid of union bounding technique, bit error performance of CIOD-MBM
schemes are obtained [17]. Assuming that $\mathbf{X}$ is transmitted and
incorrectly detected as $\hat{\mathbf{X}}$, the ABEP is approximately
calculated by

${{P}_{b}}\approx\frac{1}{2\eta{{2}^{2\eta}}}\sum\nolimits_{\mathbf{X}}{\sum\nolimits_{\mathbf{\hat{X}}}}{P(\mathbf{X}\to\mathbf{\hat{X}})}\,e(\mathbf{X},\mathbf{\hat{X}})$
(17)

where pairwise bit error probability (PEP) and the number of bit errors are
represented by $P(\mathbf{X}\to\mathbf{\hat{X}})$ and
$e(\mathbf{X},\mathbf{\hat{X}})$ respectively.

The conditional PEP (CPEP) for the CIOD-MBM schemes can be obtained as

$\displaystyle{\small
P(\mathbf{X}\\!\to\\!\mathbf{\hat{X}}|\mathbf{H})\\!=\\!Q\\!\left(\\!\sqrt{\\!{{\Delta}}/{(2{N_{0}})}}\right)}$
(18)

where
$\Delta={\small{\left\|\mathbf{H}(\mathbf{X}-\mathbf{\hat{X}})\right\|}^{2}_{\text{F}}}$,
$\left\|.\right\|^{2}_{\text{F}}$ is the Frobenius norm and $Q(x)$ is the
well-known $Q$-function, which is also represented as
$Q(x)=1/\pi\int_{0}^{\pi/2}e^{-x^{2}/\sin^{2}\theta}d\theta$. By means of this
alternative version of the $Q$-function and the moment generating function of
${{\left\|\mathbf{H}(\mathbf{X}-\mathbf{\hat{X}})\right\|}^{2}_{\text{F}}}$
[18], the unconditional PEP is given by

$\displaystyle P(\mathbf{X}\to$ $\displaystyle\mathbf{\hat{X}})=$
$\displaystyle=$
$\displaystyle\frac{1}{\pi}{{\int\nolimits_{0}^{\frac{\pi}{2}}{\,\left(\frac{{{\sin}^{2}}\theta}{{{\sin}^{2}}\theta+\frac{\lambda_{1}}{4{{N}_{0}}}}\right)}}^{{{N}_{r}}}{\,\left(\frac{{{\sin}^{2}}\theta}{{{\sin}^{2}}\theta+\frac{\lambda_{2}}{4{{N}_{0}}}}\right)}^{N_{r}}}\,d\theta$
(19)

where $\lambda_{d}$, $d\in\left\\{1,2\right\\}$ stand for the eigenvalues of
the $(\mathbf{X}-\mathbf{\hat{X}})(\mathbf{X}-\mathbf{\hat{X}})^{H}$ matrix,
which is not equal to zero.

##  IV Simulation Results

In order to evaluate the error performance of the CIOD-MBM schemes, BER
results are produced using computer simulations under varying system
parameters. Considering conventional MBM [3], CIOD [13] and CIOD-SM [19] as
reference systems, BER results are obtained with respect to $E_{b}/N_{0}$,
where $E_{b}$ is the average transmitted energy per bit.

In Fig. 2, theoretical ABEP results of both CIOD-MBM schemes are obtained to
compare the BER results obtained by computer simulations.As can be clearly
seen, the theoretical curves are perfectly matched with the computer
simulation results at moderate and high $E_{b}/N_{0}$ values. At the same
time, the proposed CIOD-MBM schemes offer a flexible design opportunity for
varying $N_{t}$, $N_{rf}$ and $N_{r}$ values. As expected, the error
performance deteriorates as the number of possible channel realizations
increases when the same $M$-ary constellation is used. For increasing $\eta$
values, CIOD-MBM II scheme provides better error performance than CIOD-MBM I
scheme as clearly seen in Fig. 2.

Figure 2: Theoretical ABEP and computer simulated BER of the CIOD-MBM schemes
under various $N_{t}$, $N_{rf}$, $N_{r}$ and $M$ values.

Fig. 3 depicts the BER of the proposed schemes against the BER of conventional
CIOD, CIOD-SM and MBM for $\eta=4$ bit/s/Hz. Here, degree of SM (DoSM) scheme,
which is defined in [19], is considered as CIOD-SM. It should be noted that
all reference schemes use a single RF chain during the transmission for a fair
comparison. As can be clearly seen, CIOD-MBM I scheme provides around 5 dB
supremacy over CIOD and approximately 10 dB over CIOD-SM and MBM schemes. At
the same time, CIOD-MBM II scheme provides similar error performance with
CIOD-MBM scheme for $\eta=4$ bit/s/Hz while providing better error performance
for higher spectral efficiency values as in Fig. 2.

##  V Conclusions

In this paper, CIOD-MBM concept has been introduced to increase the spectral
efficiency of conventional CIOD and MBM schemes by using CIOD as an additional
information source as well as transmit antennas, channel states, and
amplitude/phase modulation. One of the most weighty benefits of the CIOD-MBM
schemes, which promise high data rates by using a low number of transmit
antennas, is enabling reduced receiver complexity by symbol-by-symbol
detection. As a result of theoretical ABEP and computer simulations, it has
been shown that proposed CIOD-MBM schemes provide superiority to reference
systems in terms of error performance.

Figure 3: The BER of the proposed CIOD-MBM schemes compared to the reference
systems with $N_{r}=2$ receive antennas for about $\eta=4$ bit/s/Hz.

## References

  * [1] J. G. Andrews, S. Buzzi, W. Choi, S. V. Hanly, A. Lozano, A. C. K. Soong, and J. C. Zhang, “What will 5G be?” _IEEE Journal on Selected Areas in Communications_ , vol. 32, no. 6, pp. 1065–1082, June 2014.
  * [2] J. Jeganathan, A. Ghrayeb, and L. Szczecinski, “Spatial modulation: Optimal detection and performance analysis,” _IEEE Trans. Wireless Commun._ , vol. 12, no. 8, pp. 545–547, Aug. 2008.
  * [3] E. Seifi, M. Atamanesh, and A. K. Khandani, “Media-based MIMO: Outperforming known limits in wireless,” in _2016 IEEE Int. Conf. on Commun. (ICC)_ , Kuala Lumpur, Malaysia, May 2016, pp. 1–7.
  * [4] Y. Naresh and A. Chockalingam, “On media-based modulation using RF mirrors,” _IEEE Trans. Veh. Technol._ , vol. PP, no. 99, pp. 1–16, Oct. 2016\.
  * [5] I. Yildirim, E. Basar, and I. Altunbas, “Quadrature channel modulation,” _IEEE Wireless Commun. Lett._ , vol. 6, no. 6, pp. 790–793, Aug. 2017.
  * [6] E. Basar and I. Altunbas, “Space-time channel modulation,” _IEEE Trans. Veh. Technol._ , vol. 66, no. 8, p. 7609–7614, Aug. 2017.
  * [7] B. Shamasundar, S. Bhat, S. Jacob, and A. Chockalingam, “Multidimensional index modulation in wireless communications,” _IEEE Access_ , vol. 6, pp. 589 – 604, Nov. 2017.
  * [8] I. Yildirim, E. Basar, and G. K. Kurt, “Media-based modulation for secrecy communications,” _Electronics Letters_ , vol. 54, no. 12, pp. 789–791, June 2018.
  * [9] L. Zhang and M. Zhao, “Secrecy enhancement for media-based modulation via probabilistic optimization,” _IEEE Commun. Lett._ , vol. 23, no. 7, p. 1149–1152, July 2019.
  * [10] Y. Naresh and A. Chockalingam, “A low-complexity maximum-likelihood detector for differential media-based modulation,” _IEEE Commun. Lett._ , vol. 21, no. 10, p. 2158–2161, Oct. 2017.
  * [11] Z. Yigit and E. Basar, “Space-time media-based modulation,” _IEEE Trans. Signal Process._ , vol. 67, no. 9, pp. 2389–2398, May 2019.
  * [12] I. Yildirim, E. Basar, and I. Altunbas, “Fractional media-based modulation with golden angle modulation,” in _2019 IEEE Wireless Communications and Networking Conference Workshop (WCNCW)_ , Apr. 2019, pp. 1–5.
  * [13] Z. A. Khan and B. S. Rajan, “Space-time block codes from co-ordinate interleaved orthogonal designs,” in _IEEE Int. Symp. Information Theory_ , Lausanne, Switzerland, 2002, p. 275.
  * [14] M. Z. A. Khan and B. S. Rajan, “Single-symbol maximum likelihood decodable linear STBCs,” _IEEE Trans. Inform. Theory_ , vol. 52, no. 5, pp. 2062–2091, Apr. 2006.
  * [15] H. Jafarkhani, _Space-Time Coding_. Cambridge, UK: Cambridge University Press, 2005.
  * [16] R. Mesleh, S. Ikki, and H. Aggoune, “Quadrature spatial modulation,” _IEEE Trans. Veh. Technol._ , vol. 64, no. 6, pp. 2738–2742, June 2015.
  * [17] M. Simon and M. S. Alaouni, _Digital Communications over Fading Channels_. New York: John Wiley & Sons, 2005.
  * [18] G. L. Turin, “The characteristic function of Hermitian quadratic forms in complex normal variables,” _Biometrika_ , vol. 47, no. 1/2, pp. 199–201, June 1960.
  * [19] R. Rajasker and K. V. S. Hari, “Modulation diversity for spatial modulation using complex interleaved orthogonal design,” in _IEEE TENCON_ , Cebu, Philippines, Nov. 2012, pp. 1–6.