# R2-MLP: Round-Roll MLP Architecture for
Multi-View 3D Object Recognition

Shuo Chen, Tan Yu, Ping Li

Cognitive Computing Lab
Baidu Research
10900 NE 8th St. Bellevue, WA 98004, USA

{shanshuo1992, tan.yu1503<EMAIL_ADDRESS>
###### Abstract

Recently, vision architectures based exclusively on multi-layer perceptrons
(MLPs) have gained much attention in the computer vision community. MLP-like
models achieve competitive performance on a single 2D image classification
with less inductive bias without hand-crafted convolution layers. In this
work, we explore the effectiveness of MLP-based architecture for the view-
based 3D object recognition task. We present an MLP-based architecture termed
as Round-Roll MLP (R2-MLP). It extends the spatial-shift MLP backbone by
considering the communications between patches from different views. R2-MLP
rolls part of the channels along the view dimension and promotes information
exchange between neighboring views. We benchmark MLP results on ModelNet10 and
ModelNet40 datasets with ablations in various aspects. The experimental
results show that, with a conceptually simple structure, our R2-MLP achieves
competitive performance compared with existing state-of-the-art methods.

##  1 Introduction

With the increase and maturity of 3D acquisition technology, 3D object
recognition has become one of the most popular research directions in object
recognition. In the past decade, we have witnessed the great success of
convolutional neural networks (CNNs) in image understanding. CNNs have been
widely used in 3D object classification and achieved excellent performance. 3D
object recognition methods based on CNNs can be divided into three research
directions (Yang and Wang, 2019) by the input mode, namely, voxel-based
methods (Wu et al., 2015; Qi et al., 2016), point-based methods (Qi et al.,
2017a, b), and view-based methods (Wang et al., 2017; Feng et al., 2018;
Kanezaki et al., 2018; Han et al., 2019b, a; Wei et al., 2020). Among them,
view-based methods render each 3D objects to 2D images from different
viewpoints and model each view through the CNN originally devised for
modelling 2D images. Since view-based methods can fine-tune from the model
pre-trained on large-scale 2D image recognition datasets such as ImageNet,
they normally perform better than their voxel-based and point-based
counterparts training from scratch.

Many efforts have been devoted to improving the effectiveness of view-based
methods for 3D object recognition in the past few years. MVCNN (Su et al.,
2015) is one of the earliest attempts to adapt CNN to model the projected
views of a 3D object. It directly max-pools the view features from a 2D CNN
into a global 3D object representation for recognition. Following MVCNN (Su et
al., 2015), these works (Feng et al., 2018; Han et al., 2019a, b; Wang et al.,
2017; Kanezaki et al., 2018; Wei et al., 2020) seek to find a more effective
way to aggregate the view features. Specifically, Recurrent Cluster Pooling
CNN (RCPCNN) (Wang et al., 2017) and Group-View CNN (GVCNN) (Feng et al.,
2018) group views into multiple sets and conduct pooling within each set.
SeqViews2SeqLabels (Han et al., 2019b) and 3D2SeqViews (Han et al., 2019a)
model the view order through recurrent neural network. Multi-Loop-View
Convolutional Neural Network (MLVCNN) (Jiang et al., 2019) designs a
hierarchical view-loop-shape architecture and applies max pooling to the
outputs from all hidden layers in the Long Short-Term Memory (LSTM) to obtain
loop-level descriptors. View-based Graph Convolutional Network (View-GCN) (Wei
et al., 2020) models the view-based relations through a recurrent neural
network. Multi-view Harmonized Bilinear Network (MHBN) (Yu et al., 2018) and
Multi-View VLAD Network (MVLADN) (Yu et al., 2021c) observe the limitation of
view-based pooling, formulates the view-based 3D object recognition into a
set-to-set matching problem, and investigates in patch-level pooling. Even
though existing view-based methods have achieved excellent performance in 3D
recognition, they ignore the commutations between patches from different
projected views when generating the patch/view features. As we know, a patch
in a project view might be closely relevant to a patch in another view. A
communication between relevant patches from different projected views might be
beneficial to 3D object recognition.

Observing the limitations of existing methods, in this work, we propose round-
roll MLP (R2-MLP) for achieving communications between patches from different
views. The key novelty of R2-MLP architecture is the round-roll module. It
shifts a part of features from each view to its adjacent views in a round-roll
manner. Through the round-roll shifting, the patches in a specific projected
view can exchange the information with the patches in its neighboring
projected views, making the communications across the views feasible.
Meanwhile, we adopt the spatial-shift operation in S2-MLP architecture for
communications between patches within each view. In the implementation, we
simultaneously achieve the round-roll operation for communications across
views and the spatial-shift operation for communications within views in a
single component, termed the round-roll (R2) module.

Figure 1:  The round-roll module. A 3D object is projected into $v$ views,
generating a feature tensor $\bm{\mathcal{T}}\in\mathbb{R}^{v\times w\times
h\times c}$, where $v,w,h,c$ denote the view number, width, height, and
channel axis, separately. $c$ channels are equally divided into $6$ groups.
The first four groups of channels are shifted along the width and height
dimensions for communications between patches within each view. The last two
groups are shifted along the view dimension for communications between patches
from different views. The shifting operation along the view dimension is in a
round-roll manner. Specifically, in the case visualized in the figure, the
round-roll shift is in this manner: view 1 $\to$ view 2 $\to$ view 3 $\to$
view 1.

Figure 1 illustrates the proposed round-roll module. Based on the R2 module
and MLPs, we build our round-roll MLP architecture. We propose a computation-
free round-roll operation to interact with patch features from different
views. Compared to convolution and self-attention, shifting and rolling have
benefits on computation. The complexity of parameters and FLOPs are shown in
Table 1.

Table 1:  $V$, $H$, $W$, and $C$ are the feature map’s number of views, height, width, and channel numbers. $K$ is the convolutional kernel size. R2-MLP is less computationally intensive than the self-attention layer due to the lack of calculations such as the attention matrix. As for the parameter cardinality, we propose computation-free shifting and rolling operations to interact with the patch features between views. R2-MLP has more advantages in parameters and FLOPs than convolution and self-attention.  Operation | Params | FLOPs
---|---|---
Convolution | $\mathcal{O}(VK^{2}C^{2})$ | $\mathcal{O}(VK^{2}HWC^{2})$
Self-Attention | $\mathcal{O}(3VC^{2})$ | $\mathcal{O}(VH^{2}W^{2}C^{2})$
Round-Roll MLP | $\mathcal{O}(VC^{2})$ | $\mathcal{O}(VHWC^{2})$

In summary, our contributions are three-fold:

  * •

We introduce the round-roll module. It achieves communications between
adjacent views by exchanging a part of features of patches in adjacent views.

  * •

We introduce a conceptually simple MLP-based architecture for 3D object
recognition. As far as we know, this is the first work to utilize MLP-based
models in this field.

  * •

We have conducted experiments on the popular ModelNet40 and ModelNet10
datasets. Our R2-MLP achieves the compelling results on ModelNet40 and
outperforms the state-of-the-art methods on ModelNet10.

##  2 Related Works

###  2.1 3D object recognition

Existing mainstream 3D object recognition methods can be categorized into
three groups according to the input mode: volume-based methods (Wu et al.,
2015; Qi et al., 2016), point-based methods (Qi et al., 2017a, b) and view-
based methods (Wang et al., 2017; Feng et al., 2018; Kanezaki et al., 2018;
Han et al., 2019b, a; Wei et al., 2020). Among them, volume-based methods
quantize the 3D object into regular voxels and conduct 3D convolutions on
voxels. Nevertheless, it is computationally expensive for 3D convolution when
the resolution is high. To improve efficiency, volume-based methods normally
conduct low-resolution quantization. Therefore, there is inevitable
information loss. At the same time, point-based methods directly model the
cloud of points, efficiently achieving competitive performance. Although the
volume-based and point-based methods utilize the spatial information of 3D
objects, their practical applications are limited by the computational cost.
View-based methods do not rely on complex 3D features. They project a 3D
object into multiple 2D views. They model each view through the backbone for
2D image understanding, to obtain view or patch features. View-based methods
have another advantage of utilizing the backbone model trained on the mass
amount of image data. The input of our method is multiple views. Thus, we
mainly review view-based methods in the following.

MVCNN (Su et al., 2015) pioneers exploiting CNN for modelling multiple views.
It aggregates the view features from CNN through max-pooling. As an
improvement to MVCNN (Su et al., 2015), MVCNN-MultiRes (Qi et al., 2016)
exploits views projected from multi-resolution settings, boosting the
recognition accuracy. Different from the methods above taking fixed
viewpoints, Pairwise (Johns et al., 2016) decomposes the sequence of projected
views into several pairs and models the pairs through CNN. GIFT (Bai et al.,
2016) proposes a more comprehensive virtual camera setting. It represents each
3D object by a set of view features. It determines the similarity between two
3D objects by matching two sets of view features through a devised matching
kernel. RotationNet (Kanezaki et al., 2018) allows sequential input of views
and treats viewpoints as latent variables to boost the recognition
performance. RCPCNN (Wang et al., 2017) groups views into multiple sets and
concatenates the set features as the 3D object representation. GVCNN (Feng et
al., 2018) also groups multiple projected views into multiple sets. But it
adaptively assigns a higher weight to the group containing crucial visual
content to suppress the noise. Instead of CNN, Seqviews2seqlabels (Han et al.,
2019b) and 3D2SeqViews (Han et al., 2019a) exploit the view order besides
visual content through recurrent neural network. Later on, View-GCN (Wei et
al., 2020) models the relations between views by a graph convolution network.
MHBN (Yu et al., 2018) and MVLADN (Yu et al., 2021c) investigate pooling
patch-level features to generate the 3D object recognition. Relation Network
(Yang and Wang, 2019) enhances each patch feature by patches from all views
through a reinforcement block plugged in the rear of the network. Besides CNN,
a more recent work MVT (Chen et al., 2021) introduces the Transformer block
and achieves state-of-the-art. Unlike the abovementioned methods, we introduce
a conceptually simple MLP-based architecture for 3D object recognition.

###  2.2 MLP-based architectures

Recently, models with only MLPs and skip connections has been a new trend for
visual recognition tasks (Tolstikhin et al., 2021; Touvron et al., 2022; Yu et
al., 2022, 2021a, 2021b). MLP-based models resort to neither convolution
layers. As a replacement, they use MLP layers to aggregate the spatial
context. MLP-mixer (Tolstikhin et al., 2021) is the first work for exploiting
pure MLP-based vision backbone. It achieves the communications between patches
through a token-mixing MLP built upon two fully-connected layers.
Concurrently, Res-MLP (Touvron et al., 2022) simplifies the token-mixing MLP
to a single fully-connected layer and utilizes more layers to build a deeper
architecture, achieving a higher recognition accuracy. To further improve the
accuracy, Spatial-shift MLP backbone (S2-MLP) (Yu et al., 2022) adopts the
spatial-shift operation for cross-patch communications. AS-MLP (Lian et al.,
2022) also conducts the spatial-shift operation but shifts the features along
two axes separately. S2-MLP v2 (Yu et al., 2021b), an improved version of
S2-MLP, uses a pyramid structure to boost the image recognition performance.
CCS-MLP (Yu et al., 2021a) rethinks the design of token-mixing MLP and
proposes a circulant channel-specific MLP. Specifically, it devises the weight
matrix of token-mixing MLP as a circulant matrix, taking fewer parameters.
Meanwhile, CCS-MLP can efficiently compute the multiplication between vector
and circulant matrix through Fast Fourier Transform (FFT). GFNet (Rao et al.,
2021) adopts 2D FFT to map the feature map into the frequency domain and mix
frequency-domain features through MLP.

Though the existing methods perform well on image recognition, they are not
proposed for multi-view 3D object recognition. Therefore, we explore how to
efficiently aggregate patch features from different views on the MLP-based
architecture. We propose the round-roll module to shift and roll the tensor
consisting of patch features. Then patches can communicate with patches from
the same view and the patches from other views.

##  3 Method

###  3.1 Spatial-shift operation

As is proposed in Yu et al. (2022), the spatial-shift operation is spatial-
agnostic and maintains a local receptive field. Given an input tensor
$\bm{X}\in\mathbb{R}^{w\times h\times c}$, where $w$ is width, $h$ is height
and $c$ represents the channel. First $\bm{X}$ is split into four parts
$\\{\bm{X}_{i}\\}_{i=1}^{4}$ along channel dimension. Then each part is
shifted along one direction:

$\displaystyle\begin{split}\bm{X^{\prime}}[1\\!:,\ :,\
:\\!c/4]&=\bm{X}[:\\!w\\!-\\!1,\ :,\ :\\!c/4],\\\
\bm{X^{\prime}}[:\\!w\\!-\\!1,\ :,\ c/4\\!:\\!c/2]&=\bm{X}[1\\!:,\ :,\
c/4\\!:\\!c/2],\\\ \bm{X^{\prime}}[:,\ 1\\!:,\ c/2\\!:\\!3c/4]&=\bm{X}[:,\
:\\!h\\!-\\!1,\ c/2\\!:\\!3c/4],\\\ \bm{X^{\prime}}[:,\ :\\!h\\!-\\!1,\
3c/4\\!:]&=\bm{X}[:,\ 1\\!:,\ 3c/4\\!:].\end{split}$ (1)

After spatially shifting, each patch absorbs the visual content from its
connecting patches. It is parameter-free and efficient for computation by
simply shifting channels from a patch to its adjoining patches.

Figure 2:  The Round-Roll MLP (R2-MLP) architecture. A 3D object is projected
into multiple views. Each view is split into non-overlap patches. Then we
project each patch into an embedding vector. The patch feature vectors are
processed by a stack of $N$ R2-MLP blocks for communication. Then these
features are pooled into a global feature vector. We finally use the linear
layer head for classification.

###  3.2 Round-Roll MLP Architecture

Figure 2 provides an overview of our method. It contains a patch embedding
layer, a stack of $N$ R2-MLP blocks, a pooling layer, and a fully-connected
layer for classification.

Patch embedding layer. For each 3D object, we project it into $v$ views. Each
view image $V^{j}(j=1,\dots,v)$ is of $W\times H\times 3$ size. Identically to
S2-MLP, we split each view into $w\times h$ patches and project each patch of
size $p\times p\times 3$ to a linear layer and obtain a $c$ dimensional
vector:

$\bm{x}_{i}^{j}=\bm{W}_{0}\bm{p}_{i}^{j}+\bm{b}_{0},~{}i=1\dots wh,~{}j=1\dots
v,$ (2)

where $\bm{W}_{0}\in\mathbb{R}^{c\times 3p^{2}}$ is the weight matrix and
$\bm{b}_{0}$ is the bias vector. In total, each 3D object generates $vwh$
patches.

R2-MLP block. Our architecture stacks $N$ R2-MLP blocks with the same
structure. Each block contains four fully-connected layers, two layer
normalization (LN) layers, two GELU activation layers, and the proposed round-
roll module. Meanwhile, after each LN layer is paralleled with a skip-
connection (He et al., 2016). As we have introduced the details of LN and GELU
in the previous section, we only introduce the round-roll module here.

Similar to the spatial-shift module (Yu et al., 2022), our round-roll module
achieves cross-patch communications through shifting channels. Unlike the S2
module, which only shifts the channels along the height and width dimension,
our round-roll module also shifts channels along the view dimension for
communications between patches across different views. Suppose we have an
input tensor $\bm{\mathcal{T}}\in\mathbb{R}^{v\times w\times h\times c}$,
where $v$ is the number of views for each 3D object model, $w$ denotes the
weight, $h$ is the height, and $c$ is the number of channels. Our proposed
round-roll module groups $c$ into several groups, shifting and rolling
different channel groups in different directions. The feature map, after the
rolling, aligns different views’ token features to the same channel, and then
the interaction of the view information can be realized after channel
projection. The calculation is visualized in Figure 3. Specifically, the
round-roll operation includes three steps: 1) uniformly split the channels
into six groups, 2) shift the first four groups of channels on weight and
height axes, and 3) roll the last two groups of channels on the view axis. To
be specific, we equally split $\bm{\mathcal{T}}$ along channel dimension into
six thinner tensors $\\{\bm{\mathcal{T}}_{t}\\}_{t=1}^{6}$, where
$\bm{\mathcal{T}}_{t}\in\mathbb{R}^{v\times w\times h\times c/6}$. For
$\bm{\mathcal{T}}_{1}$, we shift it along the width axis by +1. Meanwhile, we
shift the $\bm{\mathcal{T}}_{2}$ along the width axis by -1.
$\bm{\mathcal{T}}_{3}$ and $\bm{\mathcal{T}}_{4}$ are shifted by +1 and -1
along the height axis separately. That is, the shift operations on
$\\{\bm{\mathcal{T}}_{t}\\}_{t=1}^{4}$ are similar to that in the spatial-
shift module (Yu et al., 2022). While for $\bm{\mathcal{T}}_{5}$ and
$\bm{\mathcal{T}}_{6}$, we adapt the round-roll operation on these two
components along the view dimension. Specifically, we roll
$\bm{\mathcal{T}}_{5}$ on the view axis by +1. In parallel,
$\bm{\mathcal{T}}_{6}$ is rolled by -1 along the view axis.

Figure 3:  Illustrative round-roll operation of one fully-connected layer. The
light blue, purple, and blue denote the current, previous, and next view
feature channels separately. The yellow is the output features. The dark blue
dot represents the position of interest, the dark red denotes other features
used in the calculation process, and the green represents the corresponding
output feature.“Group” means that the channel is split into different groups.

$\displaystyle\begin{split}\bm{\mathcal{T^{\prime}}}[:,1\\!:,:,:\\!c/6]&=\bm{\mathcal{T}}[:,:\\!w\\!-\\!1,:,\
:\\!c/6],\\\
\bm{\mathcal{T^{\prime}}}[:,:\\!w\\!-\\!1,:,c/6\\!:\\!c/3]&=\bm{\mathcal{T}}[:,1\\!:,:,c/6\\!:\\!c/3],\\\
\bm{\mathcal{T^{\prime}}}[:,:,\\!1\\!:,c/3\\!:\\!c/2]&=\bm{\mathcal{T}}[:,:,:\\!h\\!-\\!1,\\!c/3\\!:\\!c/2],\\\
\bm{\mathcal{T^{\prime}}}[:,:,\>\\!h\\!-\\!1,c/2\\!:\\!2c/3]&=\bm{\mathcal{T}}[:,:,1\\!:,c/2\\!:\\!2c/3],\\\
\bm{\mathcal{T^{\prime}}}[1\\!:,:,:,2c/3\\!:\\!5c/6]&=\bm{\mathcal{T}}[:\\!v\\!-\\!1,:,:,\\!2c/3\\!:\\!5c/6],\\\
\bm{\mathcal{T^{\prime}}}[0,:,:,2c/3\\!:\\!5c/6]&=\bm{\mathcal{T}}[v\\!-\\!1,:,:,\\!2c/3\\!:\\!5c/6],\\\
\bm{\mathcal{T^{\prime}}}[:\\!v\\!-\\!1,:,\>\\!h\\!-\\!1,5c/6\\!:]&=\bm{\mathcal{T}}[1\\!:,,:,1\\!:,5c/6\\!:],\\\
\bm{\mathcal{T^{\prime}}}[v\\!-\\!1,:,:,5c/6\\!:]&=\bm{\mathcal{T}}[0,:,:,5c/6\\!:].\\\
\end{split}$ (3)

##  4 Experiments

###  4.1 Settings

Datasets. We conduct the experiments on the popular benchmark ModelNet40
(Chang et al., 2015) and ModelNet10 (Chang et al., 2015) datasets. ModelNet40
has 40 categories totaling 12311 3D models, divided into 9843 training data
and 2468 test data. ModelNet10 is a subset of ModelNet40, including 10
categories. It has 3991 training data and 908 test data. We have two
viewpoints settings: 12 views and 20 views. For 12 view setting, we render the
3D mesh models by placing 12 centroid pointing virtual cameras around the mesh
every 30 degrees with an elevation of 30 degrees from the ground plane, as
shown in Figure 4(a). We utilize the rendered view images provided by Wang et
al. (2017). This viewpoint setting is adopted when we feed $\leq 12$ for each
3D model. 20 view setting places 20 virtual cameras on vertices of a
dodecahedron encompassing the object, as shown in Figure 4(b). In experiments,
we use the rendered images offered by Kanezaki et al. (2018).

(a) 12 view setting.

(b) 20 view setting.

Figure 4: Graph structure of 12 view and 20 view settings.

Evaluation metric. We use instance accuracy, the ratio of the number of
correctly classified objects to the total number of objects, as the metric of
the classification task.

Implementation details. We utilize the training strategy offered by DeiT
(Touvron et al., 2021). Specifically, we set the batch size as 64. We train
our network with the AdamW optimizer with a learning rate of 0.003 and weight
decay of 0.05. The learning rate is scheduled with a linear warmup and cosine
decay for 1000 epochs. We also use data augmentation methods, viz., label
smoothing, Auto-Augment, Rand-Augment, random erase, mixup and cutmix. We also
adopt Stochastic depth. The customization is set identically as DeiT (Touvron
et al., 2021).

###  4.2 Ablation studies

Unless specified, we use 6 views for each 3D model. We stack 36 layers of
round-roll modules, namely R2-MLP-36, and train it from scratch. We perform
ablation studies on the ModelNet10 dataset.

Ablation on the round-roll module. We first ablate our proposed round-roll
module. We illustrate different settings in Figure 5. In setting (a), we adopt
spatial-shift operation only, which is identical to Yu et al. (2022) and no
interrelation occurs between views. We treat setting (a) as a baseline. We
shift the tensor 1 stride both forward and backward in (b). The accuracy
increases from 92.62% (a) to 92.84% (b). From (c) to (e) we roll the tensor
instead of shift. Rolling operation makes it available for the first view and
last view to interrelate with each other. And we can observe a significant
improvement when adopting rolling operation. All (c), (d), and (e) settings
perform better than (b). Rolling both forward and backward, (d) and (e), reach
a noticeable improvement. But rolling 1 stride on two directions has higher
accuracy than 2 strides. We think this might be because, with less stride, the
view could interrelate with closer adjacent views that are more similar to
themselves. So, stride 1 in (d) could perform better than 2 in (e). And we use
(d), rolling 1 stride on two directions, as the default.

The round-roll module splits the feature tensor into 6 groups along channel
axis $c$. The first four groups are shifted along the width and height
dimensions while the last two groups are shifted along the view dimension.

setting | (a) | (b) | (c) | (d) | (e)
---|---|---|---|---|---
accuracy | 92.62 | 92.84 | 93.94 | 94.71 | 94.49

Figure 5: The different settings to ablate the round-roll module. (a) spatial-shift only and no interrelation between views; (b) shift 1 stride both forward and backward; (c) roll 1 stride forward; (d) roll 1 stride both forward and backward; (e) roll 2 strides both forward and backward. We observe that the best performance is (d) when rolling feature tensor along view dimension 1 stride forward and backward. Table 2:  Ablation on R2-MLP block’s group splitting. We assign four groups for spatial-shift and two groups for round-roll. The channel $c$ is 384 in total. The first row is for each spatial-shift group, and the second is for each round-roll group. Equally divided is the best.  spatial-shift | 0 | 16 | 32 | 48 | 64 | 80 | 96
---|---|---|---|---|---|---|---
round-roll | 192 | 160 | 128 | 96 | 64 | 32 | 0
accuracy | 89.21 | 93.29 | 93.78 | 94.43 | 94.71 | 94.49 | 92.62

We ablate the number of channels assigned to each group in Table 2. The
results show that equally dividing the channels guarantees the best
performance, and we set it as the default.

Table 3: The influence of the number of views on the ModelNet10 dataset with R2-MLP-36 model trained from scratch. views | 1 | 3 | 6
---|---|---|---
acc. | 90.42 | 94.05 | 94.71

The influence of the number of views. Table 3 ablates the effect of the number
of views on the average instance accuracy of our method on the ModelNet10
dataset. More projected views lead to better classification accuracy.
Specifically, using a single view, R2-MLP-36 model without pre-training only
achieves a $90.42\%$ recognition accuracy, whereas using 6 views achieves a
$94.71\%$ recognition accuracy. Better performance is expected since more
views provide more visual information for a 3D object and benefit recognition.

Ablating the depth. We ablate the influence of the network depth on the
computation cost and recognition accuracy. Following (Yu et al., 2022), we
testify three models of variant depth: R2-MLP-12, R2-MLP-24, and R2-MLP-36.
The models are trained from scratch with 6 views on the ModleNet10 dataset. We
also count the number of parameters, FLOPs, and inference throughput of each
model. We set the batch size (the number of 3D objects per batch) as 4. We use
an NVIDIA TITAN X Pascal GPU during the inference. The results are shown in
Table 4. We could see that the deeper model takes more parameters and FLOPs
and achieves higher accuracy. By default, we adopt the $36$-layer.

Table 4: The influence of different architecture. The inference throughput is measured as the number of 3D objects processed per second on an NVIDIA TITAN X Pascal GPU. Model | R2-MLP-12 | R2-MLP-24 | R2-MLP-36
---|---|---|---
layers | 12 | 24 | 36
params | 18.0M | 35.8M | 53.5M
FLOPs | 21B | 42B | 63B
throughput | 69 | 41 | 27
accuracy | 94.38 | 94.49 | 94.71

The influence of pre-training. A straightforward approach to boost the
performance of our R2-MLP is to initialize its weights by the model pre-
trained on a large-scale image recognition dataset. To this end, we use the
weights of S2-MLP-deep (Yu et al., 2022) to initialize our R2-MLP.

Table 5: Influence of ImageNet pre-training. R2-MLP-36 | w/o pre-train | w/ pre-train
---|---|---
accuracy | 94.71 | 95.93

Table 5 shows the results on the ModelNet10 dataset. When fine-tuning the pre-
trained model, the accuracy is 95.93% compared to 94.71% when training from
scratch. Higher accuracy shows that the knowledge from ImageNet benefits our
R2-MLP on the 3D object recognition task.

Hybrid of S2-MLP block and R2-MLP block. To bridge the architecture gap
between the pre-trained S2-MLP model and our R2-MLP model for a better
transfer learning performance, a plausibly more reasonable manner is to use
S2-MLP blocks in the shallow layers and R2-MLP blocks in the deep layers.
Specifically, we stack several S2-MLP blocks and follow R2-MLP blocks. The
results are shown in Table 6.

Table 6: Influence of R2-MLP depth. S2-MLP blocks | 24 | 12 | 0
---|---|---|---
R2-MLP blocks | 12 | 24 | 36
accuracy | 94.99 | 95.48 | 95.93

We keep the total number of blocks as $36$ and gradually increase the R2-MLP
blocks. While we have more R2-MLP blocks, the accuracy increases accordingly.
The R2-MLP only architecture achieves higher performance than the hybrid
architecture. The efforts to bridge the architecture gap from the hybrid
architecture cannot achieve satisfactory results. So we do not adopt this
hybrid architecture by default.

Influence of fusion ways on patch features. After a stack of $N$ R2-MLP
blocks, we obtain a tensor $\bm{\mathcal{T}}\in\mathbb{R}^{v\times w\times
h\times c}$, which contains $vwh$ $c$-dimension patch features. We need to
pool these patch features in the tensor into a $c$-dimension feature for the
final recognition. For the easiness of the illustration, we term the pooling
on the patches along the height dimension and the width dimension as spatial
pooling and term the pooling on patches along the view dimension as view
pooling.

Considering the order and pooling way, we investigate six different
combinations on the spatial level and the view level: (i) mean view pooling
followed by max spatial pooling; (ii) max view pooling followed by mean
spatial pooling; (iii) max view pooling followed by max spatial pooling; (iv)
mean view pooling followed by max spatial pooling; (v) max spatial pooling
followed by mean spatial pooling; (vi) mean spatial pooling followed by max
view pooling.

Table 7: Influence of fusion ways on patch features. fusion way | accuracy
---|---
(i) | view mean $\rightarrow$ spatial mean | 94.11
(ii) | view mean $\rightarrow$ spatial max | 94.49
(iii) | view max $\rightarrow$ spatial max | 94.60
(iv) | view max $\rightarrow$ spatial mean | 94.71
(v) | spatial max $\rightarrow$ view mean | 94.71
(vi) | spatial mean $\rightarrow$ view max | 94.16

Table 7 shows that using only mean pooling or maxing pooling in both spatial
pooling and view pooling can not achieve competitive results. In contrast, the
hybrid version (iv) and (v) achieve the highest accuracy. We set (iv) as the
default.

Table 8: Comparison with the present state-of-the-art methods on the ModelNet40 dataset. The best accuracy is bold. Method | Views | ModelNet40 | ModelNet10
---|---|---|---
MVCNN (Su et al., 2015) | 80 | 90.1 | -
RotationNet (Kanezaki et al., 2018) | 12 | 91.0 | 94.0
GVCNN (Feng et al., 2018) | 12 | 93.1 | -
Relation Network (Yang and Wang, 2019) | 12 | 94.3 | 95.3
3D2SeqViews (Han et al., 2019a) | 12 | 93.4 | 94.7
SeqViews2SeqLabels (Han et al., 2019b) | 12 | 93.4 | 94.8
MLVCNN (Jiang et al., 2019) | 12 | 94.2 | -
CAR-Net (Xu et al., 2021) | 12 | 95.2 | 95.8
MVT-small (Chen et al., 2021) | 12 | 94.4 | 95.3
R2-MLP-36 (Ours) | 6 | 94.7 | 95.9
R2-MLP-36 (Ours) | 12 | 95.0 | 97.4
RotationNet (Kanezaki et al., 2018) | 20 | 97.4 | 98.5
View-GCN (Wei et al., 2020) | 20 | 97.6 | -
CAR-Net (Xu et al., 2021) | 20 | 97.7 | 99.0
MVT-small (Chen et al., 2021) | 20 | 97.5 | 99.3
R2-MLP-36 (Ours) | 20 | 97.7 | 99.6

###  4.3 Comparison with state-of-the-art methods

In Table 8, we compare our method with view-based methods including MVCNN (Su
et al., 2015), RotationNet (Kanezaki et al., 2018), GVCNN (Feng et al., 2018),
Relation Network (Yang and Wang, 2019), 3D2SeqViews (Han et al., 2019a),
SeqViews2SeqLabels (Han et al., 2019b), MLVCNN (Jiang et al., 2019) and CAR-
Net (Xu et al., 2021). With 20 views, our R2-MLP model achieves the best
performance compared with these methods. Specifically, on the ModelNet10
dataset, we achieve the highest recognition accuracy of 99.6%. Furthermore, on
the ModelNet40 dataset, we are the best (97.7%) like CAR-Net (Xu et al.,
2021). With 12 views, we are the best on the ModelNet10 dataset and the
second-best on the ModelNet40 dataset. With 6 views we also achieve comparable
results. It is worth noting that the architecture of our R2-MLP has advantages
in parameters and computation FLOPS than the compared methods using CNNs, e.g.
CARNet (Xu et al., 2021). Meanwhile, our MLP-based architecture takes fewer
parameters and less inference time. We compare the R2-MLP-36 inference time
with its CNN counterparts MVCNN (Su et al., 2015) and CAR-Net (Xu et al.,
2021).

Table 9: Comparison of inference time on ModelNet40. Our model’s inference speed is faster by a large margin. Model | CAR-Net (Xu et al., 2021) | R2-MLP-36 (Ours)
---|---|---
inference | 0.20 s | 0.07 s

Additionally, in Table 9 we measure the time for processing a 3D object of
ModelNet40 on one NVIDIA V100 GPU, following the same setting adopted in (Xu
et al., 2021). The CAR-Net (Xu et al., 2021) is 0.20 s while our R2-MLP-36 is
0.07 s.

###  4.4 3D shape retrieval

Besides the classification task, we also evaluate our R2-MLP on the well-known
3D shape retrieval benchmark: ModelNet40 (Chang et al., 2015). The retrieval
task in ModelNet40 is a general 3D shape retrieval task (within-domain task),
in which the given query objects and target objects are all 3D models. We
adopt the evaluation metric mean Average Precision (mAP) to measure the
retrieval performance. For this task, we follow the retrieval setup of MVCNN
(Su et al., 2015). In particular, we consider the deep feature representation
of the last layer before the classification head. We first use
$L^{2}$-normalization on feature vectors. We fine-tune R2-MLP for
classification; thus, retrieval performance is not optimized directly. Instead
of training it with a different loss suitable for retrieval, we use Local
Fisher Discriminant Analysis for Supervised Dimensionality Reduction (LFDA)
(Sugiyama, 2007) to project the feature into a more expressive space. The
reduced features describe a shape. During the inference, shape representations
are used to retrieve the most similar shapes in the test set according to the
Euclidean distance between pairwise shape-level representative features.

Table 10: 3D shape retrieval comparison with the state-of-the-art methods on the ModelNet40 dataset. R2-MLP achieves the best retrieval performance among recent state-of-the-art methods. Method | Views | mAP
---|---|---
LFD (Chen et al., 2003) | Volume | 40.0
3DShapeNets (Chang et al., 2015) | Volume | 49.2
PVNet (You et al., 2018) | Point | 89.5
MVCNN (Su et al., 2015) | 12 | 80.2
MLVCNN (Jiang et al., 2019) | 24 | 92.2
MVTN (Hamdi et al., 2021) | 12 | 92.9
R2-MLP (Ours) | 12 | 93.0
R2-MLP (Ours) | 20 | 93.1

Table 10 displays the comparison to the state-of-the-art methods. Volume-based
LFD (Chen et al., 2003) and 3DShapeNets (Chang et al., 2015) have poor
performance. In general, view-based methods perform much better than volume-
based methods. Also, the point-based method PVNet (You et al., 2018) achieves
a competitive result. Our method outperforms the MLVCNN (Jiang et al., 2019),
increasing from 92.2% to 93.1%, with fewer views. The outperformance shows our
method’s generality on the 3D shape retrieval task. The qualitative examples
of the retrieved results are shown in Figure 6.

Figure 6:  Qualitative examples for 3D object retrieval on ModelNet40: (left):
we show the query object (sofa) from the test. (right): top five retrieved
objects by different methods. The negatively retrieved results are framed in
red.

##  5 Conclusion

This paper proposes a Round-Roll MLP (R2-MLP) for effective 3D object
recognition. Considering the efficiency, we propose the R2-MLP module, whose
key component is shifting the feature tensor along the spatial dimension and
rolling along the view dimension. The R2-MLP module empowers each view to
communicate with connected adjacent views, overcoming the limitations of
existing S2-MLP architectures with only spatial-shift operations on patches
from the same view. Besides, the R2-MLP module does not need extra computation
cost and is efficient to deploy. Our proposed R2-MLP architecture achieves
SOTA on ModelNet40 and ModelNet10 datasets.

## References

  * Bai et al. [2016] Song Bai, Xiang Bai, Zhichao Zhou, Zhaoxiang Zhang, and Longin Jan Latecki.  GIFT: A real-time and scalable 3d shape search engine.  In _Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5023–5032, Las Vegas, NV, 2016.
  * Chang et al. [2015] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al.  ShapeNet: An information-rich 3D model repository.  _arXiv preprint arXiv:1512.03012_ , 2015.
  * Chen et al. [2003] Ding-Yun Chen, Xiao-Pei Tian, Yu-Te Shen, and Ming Ouhyoung.  On visual similarity based 3d model retrieval.  _Comput. Graph. Forum_ , 22(3):223–232, 2003\.
  * Chen et al. [2021] Shuo Chen, Tan Yu, and Ping Li.  MVT: multi-view vision transformer for 3D object recognition.  In _Proceedings of the 32nd British Machine Vision Conference (BMVC)_ , page 349, Online, 2021.
  * Feng et al. [2018] Yifan Feng, Zizhao Zhang, Xibin Zhao, Rongrong Ji, and Yue Gao.  GVCNN: group-view convolutional neural networks for 3d shape recognition.  In _Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 264–272, Salt Lake City, UT, 2018.
  * Hamdi et al. [2021] Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem.  MVTN: multi-view transformation network for 3d shape recognition.  In _Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 1–11, Montreal, Canada, 2021.
  * Han et al. [2019a] Zhizhong Han, Honglei Lu, Zhenbao Liu, Chi-Man Vong, Yu-Shen Liu, Matthias Zwicker, Junwei Han, and C. L. Philip Chen.  3D2SeqViews: Aggregating sequential views for 3d global feature learning by CNN with hierarchical attention aggregation.  _IEEE Trans. Image Process._ , 28(8):3986–3999, 2019a.
  * Han et al. [2019b] Zhizhong Han, Mingyang Shang, Zhenbao Liu, Chi-Man Vong, Yu-Shen Liu, Matthias Zwicker, Junwei Han, and C. L. Philip Chen.  SeqViews2SeqLabels: Learning 3D global features via aggregating sequential views by RNN with attention.  _IEEE Trans. Image Process._ , 28(2):658–672, 2019b.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778, Las Vegas, NV, 2016.
  * Jiang et al. [2019] Jianwen Jiang, Di Bao, Ziqiang Chen, Xibin Zhao, and Yue Gao.  MLVCNN: multi-loop-view convolutional neural network for 3d shape retrieval.  In _Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI)_ , pages 8513–8520, Honolulu, HI, 2019.
  * Johns et al. [2016] Edward Johns, Stefan Leutenegger, and Andrew J. Davison.  Pairwise decomposition of image sequences for active multi-view recognition.  In _Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 3813–3822, Las Vegas, NV, 2016.
  * Kanezaki et al. [2018] Asako Kanezaki, Yasuyuki Matsushita, and Yoshifumi Nishida.  RotationNet: Joint object categorization and pose estimation using multiviews from unsupervised viewpoints.  In _Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5010–5019, Salt Lake City, UT, 2018.
  * Lian et al. [2022] Dongze Lian, Zehao Yu, Xing Sun, and Shenghua Gao.  AS-MLP: an axial shifted MLP architecture for vision.  In _Proceedings of the Tenth International Conference on Learning Representations (ICLR)_ , Virtual Event, 2022.
  * Qi et al. [2016] Charles Ruizhongtai Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas J. Guibas.  Volumetric and multi-view cnns for object classification on 3d data.  In _Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5648–5656, Las Vegas, 2016.
  * Qi et al. [2017a] Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas.  PointNet: Deep learning on point sets for 3d classification and segmentation.  In _Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 77–85, Honolulu, HI, 2017a.
  * Qi et al. [2017b] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas.  PointNet++: Deep hierarchical feature learning on point sets in a metric space.  In _Advances in Neural Information Processing Systems (NIPS)_ , pages 5099–5108, Long Beach, CA, 2017b.
  * Rao et al. [2021] Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou.  Global filter networks for image classification.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , pages 980–993, virtual, 2021.
  * Su et al. [2015] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik G. Learned-Miller.  Multi-view convolutional neural networks for 3d shape recognition.  In _Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV)_ , pages 945–953, Santiago, Chile, 2015.
  * Sugiyama [2007] Masashi Sugiyama.  Dimensionality reduction of multimodal labeled data by local fisher discriminant analysis.  _J. Mach. Learn. Res._ , 8:1027–1061, 2007.
  * Tolstikhin et al. [2021] Ilya O. Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy.  MLP-Mixer: An all-MLP architecture for vision.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , pages 24261–24272, virtual, 2021.
  * Touvron et al. [2021] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In _Proceedings of the 38th International Conference on Machine Learning (ICML)_ , pages 10347–10357, Virtual Event, 2021.
  * Touvron et al. [2022] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou.  ResMLP: Feedforward networks for image classification with data-efficient training.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * Wang et al. [2017] Chu Wang, Marcello Pelillo, and Kaleem Siddiqi.  Dominant set clustering and pooling for multi-view 3d object recognition.  In _Proceedings of the British Machine Vision Conference (BMVC)_ , London, UK, 2017.
  * Wei et al. [2020] Xin Wei, Ruixuan Yu, and Jian Sun.  View-GCN: View-based graph convolutional network for 3d shape analysis.  In _Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1847–1856, Seattle, WA, 2020.
  * Wu et al. [2015] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao.  3D ShapeNets: A deep representation for volumetric shapes.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1912–1920, Boston, MA, 2015.
  * Xu et al. [2021] Yong Xu, Chaoda Zheng, Ruotao Xu, Yuhui Quan, and Haibin Ling.  Multi-view 3d shape recognition via correspondence-aware deep learning.  _IEEE Trans. Image Process._ , 30:5299–5312, 2021.
  * Yang and Wang [2019] Ze Yang and Liwei Wang.  Learning relationships for multi-view 3d object recognition.  In _Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 7504–7513, Seoul, Korea, 2019.
  * You et al. [2018] Haoxuan You, Yifan Feng, Rongrong Ji, and Yue Gao.  PVNet: A joint convolutional network of point cloud and multi-view for 3D shape recognition.  In _proceedings of the 2018 ACM Multimedia Conference on Multimedia Conference (MM)_ , pages 1310–1318, Seoul, Korea, 2018.
  * Yu et al. [2018] Tan Yu, Jingjing Meng, and Junsong Yuan.  Multi-view harmonized bilinear network for 3d object recognition.  In _Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 186–194, Salt Lake City, UT8, 2018.
  * Yu et al. [2021a] Tan Yu, Xu Li, Yunfeng Cai, Mingming Sun, and Ping Li.  Rethinking token-mixing MLP for mlp-based vision backbone.  In _Proceedings of the 32nd British Machine Vision Conference (BMVC)_ , page 139, Online, 2021a.
  * Yu et al. [2021b] Tan Yu, Xu Li, Yunfeng Cai, Mingming Sun, and Ping Li.  S2-MLPv2: Improved spatial-shift mlp architecture for vision.  _arXiv preprint arXiv:2108.01072_ , 2021b.
  * Yu et al. [2021c] Tan Yu, Jingjing Meng, Ming Yang, and Junsong Yuan.  3D object representation learning: A set-to-set matching perspective.  _IEEE Trans. Image Process._ , 30:2168–2179, 2021c.
  * Yu et al. [2022] Tan Yu, Xu Li, Yunfeng Cai, Mingming Sun, and Ping Li.  S${}^{\mbox{2}}$-MLP: Spatial-shift MLP architecture for vision.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , pages 3615–3624, Waikoloa, HI, 2022\.