# Non-Rigid Shape Registration via
Deep Functional Maps Prior

Puhua Jiang1,2# Mingze Sun1# Ruqi Huang1
1\. Tsinghua Shenzhen International Graduate School, China 2. Peng Cheng Lab,
China $\\#$ indicates equal contribution. Corresponding author:
<EMAIL_ADDRESS>
###### Abstract

In this paper, we propose a learning-based framework for non-rigid shape
registration _without correspondence supervision_. Traditional shape
registration techniques typically rely on correspondences induced by extrinsic
proximity, therefore can fail in the presence of large intrinsic deformations.
Spectral mapping methods overcome this challenge by embedding shapes into,
geometric or learned, high-dimensional spaces, where shapes are easier to
align. However, due to the dependency on abstract, non-linear embedding
schemes, the latter can be vulnerable with respect to perturbed or alien
input. In light of this, our framework takes the best of both worlds. Namely,
we deform source mesh towards the target point cloud, guided by
correspondences induced by high-dimensional embeddings learned from deep
functional maps (DFM). In particular, the correspondences are dynamically
updated according to the intermediate registrations and filtered by
consistency prior, which prominently robustify the overall pipeline. Moreover,
in order to alleviate the requirement of extrinsically aligned input, we train
an orientation regressor on a set of aligned synthetic shapes independent of
the training shapes for DFM. Empirical results show that, with as few as
dozens of training shapes of limited variability, our pipeline achieves state-
of-the-art results on several benchmarks of non-rigid point cloud matching,
but also delivers high-quality correspondences between unseen challenging
shape pairs that undergo both significant extrinsic and intrinsic
deformations, in which case neither traditional registration methods nor
intrinsic methods work. The code is available at
https://github.com/rqhuang88/DFR.

##  1 Introduction

Estimating correspondences between deformable shapes is a fundamental task in
computer vision and graphics, with a significant impact on an array of
applications including robotic vision [50], animation [40], 3D reconstruction
[58], to name a few. In this paper, we tackle the challenging task of
estimating correspondences between _unstructured_ point clouds sampled from
surfaces undergoing _significant_ non-rigid deformations.

In fact, this task has attracted increasing interest [59; 26; 35; 24; 8] in
shape matching community. These methods all follow a data-driven approach to
learn embedding schemes, which project point clouds into certain high-
dimensional spaces. By lifting to high-dimensional spaces, the non-rigid
deformations are better characterized than in the ambient space,
$\mathbb{R}^{3}$. The optimal transformations and therefore the
correspondences are then estimated in the embedded spaces. Despite the great
progresses, such approaches suffer from the following limitations: 1) Their
performance on unseen shapes is largely unwarranted; 2) The learned high-
dimensional embeddings lack intuitive geometric meaning, and the induced
correspondences are difficult to evaluate and analyze without ground-truth
labels.

On the other hand, the above limitations are less of a concern from the
perspective of shape registration, either axiomatic [1; 55] or learning-based
[31]. Unlike the former, the latter deform a source shape to non-rigidly align
with a target explicitly in the ambient space $\mathbb{R}^{3}$. In
consequence, the mapping quality can be directly related to the alignment
quality. Given a deformed source shape and a target, one can evaluate both
qualitatively (by visual comparison) and quantitatively (by computing Chamfer
distance), and, more importantly, further optimize the registration outcome
via the correspondence-label-free quantitative metric. However, shape
registration itself does not serve as a direct rescue to our problem of
interest, as the previous approaches typically rely on the premise that the
undergoing non-rigid deformation can be approximated by a set of local, small-
to-moderate, rigid deformations, which severely hinders their performance in
the presence of large deformations (see Fig. 1) and heterogeneity.

Figure 1: Shape registration methods like NDP [31] and AMM [55] estimate
intermediate correspondences via extrinsic proximity, therefore suffering from
large intrinsic deformations. In contrast, by incorporating a DFM-based point
feature extractor, our method successfully deforms a FAUST template (the left-
most mesh) to another individual of a different pose (the right-most point
cloud).

Motivated by the above observations, we propose a framework to take the best
of classic shape registration and learning-based embedding techniques. In a
nutshell, we leverage the estimated correspondence from the latter to guide
shape deformation via the former iteratively. Our key insight is to enforce
similarity between the deformed source and target in _both ambient space and
learned high-dimensional space_. Intuitively, in the presence of large
deformation, the learned correspondence is more reliable than that obtained by
proximity in the ambient space. On the other hand, properly deforming source
mesh to target point cloud increases spatial similarity in the ambient space,
leading to higher similarity in the embedded space and therefore more accurate
correspondences. In the end, we can compute correspondences between raw point
clouds with a shared source mesh as a hub. As will be shown in Sec. 4, our
method allows for choosing source shape during inference, which can be
independent of the training data.

While being conceptually straightforward, we strive to improve the
performance, robustness and practical utility of our pipeline by introducing
several tailored designs. First of all, the key component of our pipeline is
an embedding scheme for accurately and efficiently estimating correspondences
between the deformed source shape and the target _point cloud_. In particular,
to take advantage of Deep Functional Maps (DFM) [29; 7; 49], the current
state-of-the-art approach on matching _triangular meshes_ , we pre-train an
unsupervised DFM [49] on meshes as a teacher net and then learn a point-based
feature extractor (_i.e._ , an embedding scheme for points) as a student net
on the corresponding vertex sets with a natural self-supervision. Unlike the
approach taken in [8], which heavily relies on DiffusionNet [47] to extract
intricate structural details (e.g., Laplace-Beltrami operators) from both mesh
and point cloud inputs, the teacher-student paradigm allows us to utilize a
more streamlined backbone – DGCNN [53]. Secondly, in contrast to prior works
that either implicitly [31; 55] or explicitly [46; 8; 24] require rigidly
aligned shapes for input/initialization, or demand dense correspondence labels
[35]. We train an orientation regressor on a large set of synthetic, rigidly
aligned shapes to _automatically_ pre-process input point clouds in arbitrary
orientations. Last but not least, we propose a dynamic correspondence updating
scheme, bijectivity-based correspondence filtering, two-stage registration in
our pipeline. We defer the technical details to Sec. 3.

Overall, our framework enjoys the following properties: 1) Due to the hybrid
nature of our framework, it can handle point clouds undergoing significant
deformation and/or heterogeneity; 2) As our feature extractor is self-
supervised by some deep functional maps network, our framework is free of
correspondence annotation throughout; 3) Our core operation is performed in
the ambient space, enabling more efficient, faithful, and straightforward
analysis of registration/mapping results than the purely learning-based
approaches; 4) Based on a data-driven orientation regressor, we achieve an
automatic pipeline for estimating correspondences between deformable point
clouds.

As shown in Fig. 2, trained on the SCAPE [2] dataset, our framework
generalizes well and outperforms the competing methods by a large margin in
the challenging SHREC07-H dataset, which presents prominent variability in
both extrinsic orientation and intrinsic geometry. We conduct a set of
experiments to verify the effectiveness of our pipeline. We highlight that it
achieves state-of-the-art results in matching non-rigid point clouds in both
near-isometric and heterogeneous shape collections. More remarkably, it
generalizes well despite the distinctiveness between the training set and test
set. Moreover, our method, being point-based, is more robust with respect to
topological perturbations within input shapes than the mesh-based baselines.

Figure 2: We estimate correspondences between heterogeneous shapes from
SHREC’07 with four learning-based methods, all trained on the SCAPE_r dataset.
Our method outperforms the competing methods by a large margin. Remarkably,
our method manages to deform a SCAPE template shape to heterogeneous shapes,
as indicated by the blue shapes.

##  2 Related Works

Non-rigid Shape Registration: Unlike the rigid counterpart, non-rigidly
aligning shapes is more challenging because of the perplexity of deformation
models. In general, axiomatic approaches [1] assume the deformation of
interest can be approximated by local, small-to-moderate, rigid deformations,
therefore suffer from large intrinsic deformations. In fact, recent advances
such as [60; 28; 55] mainly focus on improving the efficiency and robustness
regarding noises and outliers. On the other hand, there exists as well a trend
of incorporating deep learning techniques [6; 5; 21]. Among them, NDP [31] is
similar to our method in the sense that the non-learned version follows an
optimization-based approach based on some neural encoding. However, its shape
registration is purely guided by proximity in the ambient space, hindering its
performance in our problem of interest. Perhaps the most relevant approach
with our method along this line is TransMatch [51], which follows a supervised
learning scheme and learns a transformer to predict directly 3D flows between
input point clouds. As demonstrated in Fig. 2 and further in Sec. 4, as a
supervised method, TransMatch generalizes poorly to unseen shapes.

Non-rigid Point Cloud Matching: Unlike the above, several recent approaches
[26; 30; 59] directly establish correspondence between a pair of point clouds.
In general, these methods embed point clouds into a canonical feature space,
and then estimate correspondences via the respective proximity. Leapard [30]
learns such feature extractor under the supervision of ground-truth
correspondences between partial point clouds with potentially low overlaps. On
the other hand, DPC [26] and CorrnetNet3D [59] leverage point cloud
reconstruction as proxy task to learn embeddings without correspondence label.
Since intrinsic information is not explicitly formulated in these methods,
they can suffer from significant intrinsic deformations and often generalize
poorly to unseen shapes.

Deep Functional Maps: Stemming from the seminal work of functional maps [39]
and a series of follow-ups [38; 22; 44; 37; 23], spectral methods have
achieved significant progress in axiomatic non-rigid shape matching problem,
and consequentially laid solid foundation for the more recent advances on Deep
Functional Maps (DFM), which is pioneered by Litany et al. [32]. Unlike
axiomatic functional maps frameworks that aim at optimizing for
correspondences represented in spectral basis (so-called functional maps)
_with hand-crafted features as prior_. DFM takes an inverse viewpoint and aims
to search for the optimal features, such that the induced functional maps
satisfy certain structural priors as well as possible. Recent findings along
this direction empirically suggest that the structural priors suffice to train
an effective feature extractor, without any correspondence supervision [12;
29; 7; 3]. However, because of the heavy dependence of eigenbasis of Laplace-
Beltrami operators, DFM is primarily designed for shapes represented by
triangle meshes and can suffer notable performance drop when applied to point
clouds without adaptation [8].

In fact, inspired by the success of DFM, several approaches [24; 8] have been
proposed to leverage intrinsic geometry information carried by meshes in the
training of feature extractors tailored for non-structural point clouds. Among
these, NIE [24] and SSMSM [8] are closely related to our approach. As all
three works attempt to train an effective, intrinsic-geometry-aware, per-point
feature extractor by leveraging meshes during training. However, unlike the
former two, our method incorporates optimization-based shape registration
techniques, which effectively alleviates the limitation of purely relying on
an abstract high-dimensional embedding scheme (_i.e._ , the learned feature
extractor).

Combination of Extrinsic and Intrinsic Methods: Unlike the above, our method
operates in both extrinsic and intrinsic spaces. From this point of view, the
most relevant work is Smooth Shells [14], which proposes to perform shape
registration under a mixed representation of shapes, including eigenbasis,
coordinates, and normal. Though achieving considerable high-quality
correspondence on standard benchmarks, it heavily relies on the availability
of mesh representation, but also is sensitive to the original extrinsic
alignment of input shapes. These limitations are inherited by Deep Shells [16]
– the learning version of the former. Similarly, NeuralMorph [15] jointly
performs interpolation and correspondence estimation between a pair of shapes,
which inject the intrinsic information by imposing geodesic preservation loss.
As a result, it is non-trivial to lift the requirement of mesh input.

##  3 Methodology

Figure 3: The schematic illustration of our pipeline. ${\mathbf{A}}$ is a pre-
trained orientation regressor for aligning input shapes. Then a pre-trained
feature extractor ${\mathbf{F}}$ embeds them into a high-dimensional canonical
space. During the iterative optimization procedure of registration,
correspondences are dynamically updated according to learned features
(Stage-I) and coordinates (Stage-II) of the intermediate shapes. See more
details in the text.

Given a pair of shapes $\mathcal{S},\mathcal{T}$, our target is to deform
$\mathcal{S}$ to non-rigidly align with $\mathcal{T}$. We assume that
$\mathcal{S}$ is represented as a triangle mesh so that we can effectively
regularize the deformed shape by preserving local intrinsic geometry. On the
other hand, we require _no_ structural information on $\mathcal{T}$ and
generally assume it to be a _point cloud_. Our pipeline can also be extended
to compute correspondences between two raw point clouds
$\mathcal{T}_{1},\mathcal{T}_{2}$. To this end, we fix a template mesh
$\mathcal{S}$, perform respective shape registration between $\mathcal{S}$ and
the target point clouds, and finally compute the map by composition
$T_{12}=T_{s2}\circ T_{1s}$.

Our pipeline is shown in Fig. 3, which consists of three main components: 1)
An orientation regressor, ${\mathbf{A}}$, for _extrinsically_ aligning input
shapes, either mesh or point cloud; 2) A point feature extractor,
${\mathbf{F}}$, trained under deep functional maps scheme; 3) A registration
module that iteratively optimizes for deformations non-rigidly aligning
$\mathcal{S}$ with $\mathcal{T}$. In particular, it takes the rigidly aligned
shapes from 1) as input, and leverages 2) to update correspondences during
optimization.

Though our pipeline leverages a pre-trained feature extractor as registration
prior, we highlight that neither the source nor the target is necessarily
within or close to the respective training set. We provide throughout
experimental evidence in Sec. 4 showing the generalizability of our pipeline.

###  3.1 Orientation Regressor

Our first objective is to align input point clouds with arbitrary orientations
into a canonical frame. We take a data-driven approach by training an
orientation regressor [9] on $5000$ synthetic SURREAL shapes from [52], which
are implicitly aligned to a canonical frame by the corresponding generative
codes. We refer readers to Supplementary Materials for more details.

###  3.2 Point Feature Extractor

In order to efficiently and accurately estimate correspondences between
deformed source shape and the target point cloud during registration, our next
goal is to train a feature extractor for point clouds, which is intrinsic-
geometry aware. The authors of [8] propose a multi-modal feature extractor
based on DiffusionNet [47], which can process point clouds with an extra step
of _graph_ Laplacian construction [48]. Though it has demonstrated satisfying
performance accommodating point cloud representation in [8], the explicit
graph Laplacian construction is computationally heavy. Hence, we adopt the
modified DGCNN proposed in [24] as our backbone, which is lightweight and
robust regarding the sampling density and sizes of point clouds.

In particular, our training follows a teacher-student paradigm. Namely, we
first train a deep functional maps (DFM) network, ${\mathbf{G}}$, on a
collection of meshes. Then we train a DFM network ${\mathbf{F}}$ on the
corresponding vertex sets, with an extra self-supervision according to the
inclusion map between meshes and their vertex sets. In other words, the
feature produced by ${\mathbf{F}}$ is _point-wisely_ aligned with that
produced by ${\mathbf{G}}$.

Training DFM on Meshes:  Our training scheme in principle follows that of
[49], which delivers both accurate map estimation and excellent generalization
performance.

1) We take a pair of shapes $\mathcal{S}_{1},\mathcal{S}_{2}$, and compute the
leading $k$ eigenfunctions of the Laplace-Beltrami operator on each shape. The
eigenfunctions are stored as matrices $\Phi_{i}\in\mathbb{R}^{n_{i}\times
k},i=1,2$.

2) We compute
$G_{1}={\mathbf{G}}(\mathcal{S}_{1}),G_{2}={\mathbf{G}}(\mathcal{S}_{2})$,
projected into the spaces spanned by $\Phi_{1},\Phi_{2}$ respectively, and
leverage the differential FMReg module proposed in [13] to estimate the
functional maps $C_{12},C_{21}$ between the two shapes, then by regularizing
the structure of $C_{12},C_{21}$, say:

$E_{\mbox{bij}}({\mathbf{G}})=\|C_{12}C_{21}-I\|_{F}^{2},E_{\mbox{ortho}}({\mathbf{G}})=\|C_{12}C_{12}^{T}-I\|+\|C_{21}C_{21}^{T}-I\|.$
(1)

3) To enhance ${\mathbf{G}}$, we estimate the correspondences between
$\mathcal{S}_{1},\mathcal{S}_{2}$ via the proximity among the rows of
$G_{1},G_{2}$ respectively. Namely, we compute

$\Pi_{12}(i,j)=\frac{\exp(-{\alpha\delta_{ij}})}{\sum_{j^{\prime}}\exp(-\alpha\delta_{ij^{\prime}})},\forall
i\in[|\mathcal{S}_{1}|],j\in[|\mathcal{S}_{2}|],$ (2)

where $\delta_{ij}=\|G_{1}(i,:)-G_{2}(j,:)\|$ and $\alpha$ is the temperature
parameter, which is increased during training [49]. Then $\Pi_{12}$ is the
soft map between the two shapes. Ideally, it should be consistent with the
functional maps estimated in Eqn.(1), giving rise to the following loss:

$E_{\mbox{align}}({\mathbf{G}})=\|C_{12}-\Phi_{2}^{\dagger}\Pi_{21}\Phi_{1}\|+\|C_{21}-\Phi_{1}^{\dagger}\Pi_{12}\Phi_{2}\|,$
(3)

where $\dagger$ denotes to the pseudo-inverse. To sum up,
${\mathbf{G}}^{*}=\arg\min E_{\mbox{DFM}}({\mathbf{G}})$, which is defined as:

$E_{\mbox{DFM}}({\mathbf{G}})=\lambda_{\mbox{bij}}E_{\mbox{\mbox{bij}}}({\mathbf{G}})+\lambda_{\mbox{orth}}E_{\mbox{orth}}({\mathbf{G}})+\lambda_{\mbox{align}}E_{\mbox{align}}({\mathbf{G}}).$
(4)

Training DFM on Point Clouds: Now we are ready for training ${\mathbf{F}}$
with a modified DGCNN [24] as backbone. We let
$F_{1}=\textbf{F}(\mathcal{S}_{1}),F_{2}=\textbf{F}(\mathcal{S}_{2})$. On top
of $E_{\mbox{DFM}}$ in Eqn. (4), we further enforce the extracted feature from
${\mathbf{F}}$ to be aligned with that from $\textbf{G}^{*}$ via PointInfoNCE
loss [54]:

$E_{\mbox{NCE}}({\mathbf{F}},{\mathbf{G}}^{*})=-\sum_{i=1}\limits^{n_{1}}\log\frac{\exp(\langle
F_{1}(i,:),G^{*}_{1}(i,:)\rangle/\gamma)}{\sum_{j=1}\limits^{n_{1}}\exp(\langle
F_{1}(i,:),G^{*}_{1}(j,:)\rangle/\gamma)}-\sum_{i=1}\limits^{n_{2}}\log\frac{\exp(\langle
F_{2}(i,:),G^{*}_{2}(i,:)\rangle/\gamma)}{\sum_{j=1}\limits^{n_{2}}\exp(\langle
F_{2}(i,:),G^{*}_{2}(j,:)\rangle/\gamma)},$ (5)

where $\langle.\rangle$ refers to the dot products, $\gamma$ is the
temperature parameter, and $n_{i}$ is the number of points of
$\mathcal{S}_{i}$. So the final training loss is defined as:

$E({\mathbf{F}})=E_{\mbox{DFM}}({\mathbf{F}})+\lambda_{\mbox{NCE}}E_{\mbox{NCE}}({\mathbf{F}},{\mathbf{G}}^{*}),$
(6)

which is optimized on all pairs of shapes within the training set.

###  3.3 Shape Registration

Now we are ready to formulate our registration module. We first describe the
deformation graph construction in our pipeline. Then we formulate the involved
energy functions and finally describe the iterative optimization algorithm.
During registration, we denote the deformed source model by
$\mathcal{S}^{k}=\\{\mathcal{V}^{k},\mathcal{E}\\}$,$\mathcal{V}^{k}=\left\\{v_{i}^{k}\mid
i=1,\ldots,N\right\\}$, where $k$ indicates the iteration index and
$v_{i}^{k}$ is the position of the $i-$th vertex at iteration $k$. The target
point cloud is denoted by $\mathcal{T}=\left\\{u_{j}\mid
j=1,\ldots,M\right\\}$.

Deformation Graph:  Following[20], we reduce the vertex number on
$\mathcal{S}$ to $H=[N/2]$ with QSlim algorithm [17]. Then an embedded
deformation graph $\mathcal{DG}$ is parameterized with axis angles
$\Theta\in\mathbb{R}^{H\times 3}$ and translations
$\Delta\in\mathbb{R}^{H\times 3}$: $\textbf{X}=\\{\Theta,\Delta\\}$. The
vertex displacements are then computed from the corresponding deformation
nodes. For a given $\textbf{X}^{k}$, we can compute displaced vertices via:

$\mathcal{V}^{k+1}=\mathcal{DG}(\textbf{X}^{k},\mathcal{V}^{k}).$ (7)

Dynamic correspondence update:  Thanks to our point-based feature extractor
${\mathbf{F}}$, we can freely update correspondences between deforming source
mesh and target point clouds. In practice, for the sake of efficiency, we
update every 100 iterations in optimization (see Alg. 1).

Correspondence Filtering via Bijectivity:  One key step in shape registration
is to update correspondences between the deforming source and the target. It
is then critical to reject erroneous corresponding points to prevent error
accumulation over iterations. In particular, we propose a filter based on the
well-known bijectivity prior. Given $\mathcal{S}^{k}$ and $\mathcal{T}$, we
first compute maps represented as permutation matrices
$\Pi_{\mathcal{S}\mathcal{T}},\Pi_{\mathcal{T}\mathcal{S}}$, either based on
the learned feature or the coordinates. For $v_{i}^{k}\in\mathcal{S}^{k}$, we
compute the geodesic on $\mathcal{S}^{k}$ between $v_{i}^{k}$ and its image
under permutation $\Pi_{\mathcal{S}\mathcal{T}}\Pi_{\mathcal{T}\mathcal{S}}$,
then reject all $v_{i}^{k}$ if the distance is above some threshold. For the
sake of efficiency, we pre-compute the geodesic matrix of $\mathcal{S}$ and
approximate that of $\mathcal{S}^{k}$ with it. Finally, we denote the filtered
set of correspondences by
$\mathcal{C}^{k}=\\{(v_{i_{1}}^{k},u_{j_{1}}),(v_{i_{2}}^{k},u_{j_{2}})\cdots\\}$.

In the following, we introduce the energy terms regarding our registration
pipeline.

Correspondence Term measures the distance of filtered correspondences between
$\mathcal{S}^{k}$ and $\mathcal{T}$, given as:

$E_{\mbox{corr}}=\frac{1}{|\mathcal{C}^{k}|}\sum_{(v_{i}^{k},u_{j})\in\mathcal{C}^{k}}\left\|v^{k}_{i}-u_{j}\right\|_{2}^{2}.$
(8)

Chamfer Distance Term has been widely used [31; 30] to measure the extrinsic
distance between $\mathcal{S}^{k}$ and $\mathcal{T}$:

$E_{\mbox{cd}}=\frac{1}{N}\sum_{i\in[N]}\min_{j\in[M]}\left\|v^{k}_{i}-u_{j}\right\|_{2}^{2}+\frac{1}{M}\sum_{j\in[M]}\min_{i\in[N]}\left\|v^{k}_{i}-u_{j}\right\|_{2}^{2}$
(9)

As-rigid-as-possible Term reflects the deviation of estimated local surface
deformations from rigid transformations. We follow [20; 27] and define it as:

$E_{\mbox{arap}}=\sum_{h\in[H]}\sum_{l\in\psi(h)}(\left\|d_{h,l}(\textbf{X})\right\|_{2}^{2}+\alpha\left\|(R\left(\Theta_{h}\right)-R\left(\Theta_{l}\right)\right\|_{2}^{2})$
(10)
$d_{h,l}(\textbf{X})=d_{h,l}(\Theta,\Delta)=R\left(\Theta_{h}\right)\left(g_{l}-g_{h}\right)+\Delta_{k}+g_{k}-\left(g_{l}+\Delta_{l}\right).$
(11)

Here, $g\in R^{H\times 3}$ are the original positions of the nodes in the
deformation graph $\mathcal{DG}$, and $\psi(h)$ is the 1-ring neighborhood of
the $h-$th deformation node. $R(\cdot)$ is Rodrigues’ rotation formula that
computes a rotation matrix from an axis-angle representation and $\alpha$ is
the weight of the smooth rotation regularization term.

Total cost function: The total cost function $E_{\text{total }}$ combines the
above terms with the weighting factors $\lambda_{cd},\lambda_{corr}$, and
$\lambda_{arap}$ to balance them:

$E_{\mbox{total}}=\lambda_{\mbox{cd}}E_{\mbox{cd}}+\lambda_{\mbox{corr}}E_{{\mbox{corr}}}+\lambda_{\mbox{arap}}E_{\mbox{arap}}$
(12)

Now we are ready to describe our algorithm for minimizing $E_{\mbox{total}}$,
which is shown in Alg. 1.

Two-stage Registration:  Finally, we observe that solely depending on learned
features to infer correspondence is sub-optimal. At the converging point, the
deformed source shape is often at the right pose but has deficiencies in shape
style. To this end, we perform a second-stage registration, based on the
coordinates of deforming source and target. As shown in Sec. 4.2, such a
design is clearly beneficial.

Input: Source mesh $\mathcal{S}=\\{\mathcal{V},\mathcal{E}\\}$ and target
point cloud $\mathcal{T}$;Trained point feature extractor F

Output: $\textbf{X}^{*}$ converging to a local minimum of $E_{\mbox{total}}$;
Deformed source model $\\{\mathcal{V}^{*},\mathcal{E}\\}$; Correspondence
$\Pi_{\mathcal{S}\mathcal{T}}^{*},\Pi_{\mathcal{T}\mathcal{S}}^{*}$ between
$\mathcal{S}$ and $\mathcal{T}$.

1 Initialize deformation graph $\mathcal{DG}$ and $\textbf{X}^{(0)}$ with
identity transformations; $F_{\mathcal{T}}=\textbf{F}(\mathcal{T})$;k = 0;

2while  _True_ do

3 Update source vertex $\mathcal{V}^{(k)}$ by Eqn.(7);

4 if  _$k\%100==0$ and Flag == Stage-I _ then

5 $F^{(k)}_{\mathcal{S}}=\textbf{F}(\mathcal{V}^{(k)})$;
$\Pi_{\mathcal{S}\mathcal{T}}^{(k)}=\textbf{NN}(F^{(k)}_{\mathcal{S}},F_{\mathcal{T}})$
;
$\Pi_{\mathcal{T}\mathcal{S}}^{(k)}=\textbf{NN}(F_{\mathcal{T}},F^{(k)}_{\mathcal{S}})$;

6

7 if  _$k\%100==0$ and Flag == Stage-II _ then

8
$\Pi_{\mathcal{S}\mathcal{T}}^{(k)}=\textbf{NN}(\mathcal{V}^{(k)},\mathcal{T})$
;
$\Pi_{\mathcal{T}\mathcal{S}}^{(k)}=\textbf{NN}(\mathcal{T},\mathcal{V}^{(k)})$;

9

10 Compute the set of filtered correspondences $\mathcal{C}^{k}$ ;

11 $\textbf{X}^{(k+1)}=\arg\min{E_{\mbox{total}}}$ by Eqn.(12)

12 if converged and Flag == Stage-I then Flag = Stage-II;

13 if converged and Flag == Stage-II then return
$\Pi_{\mathcal{S}\mathcal{T}}^{*}$;$\Pi_{\mathcal{T}\mathcal{S}}^{*}$;
$\mathcal{V}^{*}$;

14 k = k + 1;

Algorithm 1 Shape registration pipeline.

##  4 Experiments

Datasets: We evaluate our method and several state-of-the-art techniques for
estimating correspondences between deformable shapes on an array of benchmarks
as follows: FAUST_r: The remeshed version of FAUST dataset [4], which consists
of 100 human shapes (10 individuals performing the same 10 actions). We split
the first 80 as training shapes and the rest as testing shapes; SCAPE_r: The
remeshed version of SCAPE dataset [2], which consists 71 human shapes (same
individual in varying poses). We split the first 51 as training shapes and the
rest as testing shapes; SHREC19_r:  The remehsed version of SHREC19 dataset
[36], which consists of 44 shapes of different identities and poses. We use it
solely in test, and follow the test pair list provided by [36]; SHREC07-H:  A
subset of SHREC07 dataset [18], which consists of 20 heterogeneous human
shapes of the varying number of vertices. We use it solely in test, and use
the accompanied sparse landmark annotations to quantify all the pairwise maps
among them; DT4D-H:  A dataset proposed in [34], which consists of 10
categories of heterogeneous humanoid shapes. We use it solely in testing, and
evaluating the inter-class maps split in [29]; TOPKIDS:  This is a challenging
dataset [25] consisting of 26 shapes of a kid in different poses, which
manifest significant topological perturbations in meshes.

Baselines: We compare our method with an array of competitive baselines,
including axiomatic shape registration methods: Smooth Shells [14], NDP [31],
AMM [55]; learning-based registration methods: 3D-CODED [19], Deep Shells
[16], TransMatch [51], SyNoRiM [21]; deep functional maps frameworks: SURFMNet
[45], WSupFMNet [46], GeomFMaps [13], DiffFMaps [35], NIE [24], ConsistFMaps
[7], SSMSM [8]. According to input requirements, we put those relying on pure
mesh input on the top, and the rest, suitable for point clouds, in the bottom
of both tables.

Train/Test Cases: Throughout this section, we consider a highly challenging
scenario: for each learning-based method, we train two models respectively
with FAUST_r and SCAPE_r dataset, and then we run tests in a range of test
cases including FAUST_r, SCAPE_r, SHREC19_r, SHREC07-H, DT4D-H and TOPKIDS.
The test pairs of each case have been described above. In the following, $A/B$
means we train on dataset $A$ and test on dataset $B$.

Datasets Alignment: There exist extrinsically aligned versions for all the
involved datasets but SHREC07-H. We equally feed in aligned datasets to all
the baselines when available. On the other hand, the original version of
SHREC07-H manifests significant variability of orientations. For the sake of
fairness and simplicity, we apply our orientation regressor to it and provide
baselines for our automatically aligned data. Note that, all the aligned
datasets, as well as the synthetic dataset on which we train our orientation
regressor, roughly share the same canonical orientation, which is defined by
the SMPL [33] model. Finally, we always apply automatic alignment before
inputting shapes into our pipeline, whatever the original orientation they
are.

Choice of Source Shape: As mentioned at the beginning of Sec. 3, we compute
correspondences between two raw input point clouds by associating them via a
common source mesh $\mathcal{S}$. In Tab. 1 and Tab. 2, we indicate our choice
of source mesh by Ours-_name_ , where _name_ indicates the origin of the
source mesh. For simplicity, we fix the source mesh from each dataset and
visualize them in the appendix. On the other hand, when implementing axiomatic
shape registration methods, we only consider deforming the same template we
pick from FAUST_r (resp. SCAPE_r) to every test point cloud in the test set of
FAUST_r (resp. SCAPE_r), and establish correspondence by map composition, in
the same manner as ours.

Metric: Though we primarily focus on matching point clouds, we adopt the
commonly used geodesic error normalized by the square root of the total area
of the mesh, to evaluate all methods, either for mesh or point cloud.

Hyper-Parameters:  We remark that all the hyper-parameters are fixed for _all_
experiments in this section. In particular, we settle them by performing a
grid search with respect to the weights used in the final optimization to seek
the combination that leads to the best registration results (quantitatively in
terms of Chamfer distance and qualitatively by visual inspection) on a few
training shapes. We provide a more detailed discussion and ablation of the
choice of hyper-parameters in the appendix.

###  4.1 Experimental Results

Near-isometric Benchmarks:  As shown in Fig. 1, in the presence of large
deformation between the template and the target, NDP[31] and AMM[55] fail
completely while our method delivers high-quality deformation.

Table 1: Mean geodesic errors (×100) on FAUST_r, SCAPE_r and SHREC19_r. The
best is highlighted.

| Train | FAUST_r |  | SCAPE_r
---|---|---|---|---
Method | Test | FAUST_r | SCAPE_r | SHREC19_r |  | SCAPE_r | FAUST_r | SHREC19_r
Smooth Shells [14] |  | 2.5 | \ | \ |  | 4.7 | \ | \
SURFMNet(U) [45] |  | 15.0 | 32.0 | \ |  | 12.0 | 32.0 | \
NeuroMorph(U) [15] |  | 8.5 | 29.0 | \ |  | 30.0 | 18.0 | \
WSupFMNet(W) [46] |  | 3.3 | 12.0 | \ |  | 7.3 | 6.2 | \
GeomFMaps(S) [13] | mesh | 3.1 | 11.0 | 9.6 |  | 4.4 | 6.0 | 11.4
Deep Shells(W) [16] |  | 1.7 | 5.4 | 26.6 |  | 2.5 | 2.7 | 21.4
ConsistFMaps(U) [7] |  | 1.5 | 7.3 | 20.9 |  | 2.0 | 8.6 | 28.7
AttentiveFMaps(U) [29] |  | 1.9 | 2.6 | 6.2 |  | 2.2 | 2.2 | 9.3
NDP [31] |  | 20.4 | \ | \ |  | 16.2 | \ | \
AMM [55] |  | 14.2 | \ | \ |  | 13.1 | \ | \
CorrNet3D(U) [59] |  | 63.0 | 58.0 | \ |  | 58.0 | 63.0 | \
3D-CODED(S) [19] |  | 2.5 | 31.0 | \ |  | 31.0 | 33.0 | \
SyNoRiM(S) [21] |  | 7.9 | 21.9 | \ |  | 9.5 | 24.6 | \
TransMatch(S) [51] | pcd | 2.7 | 33.6 | 21.0 |  | 18.6 | 18.3 | 38.8
DPC(S) [26] |  | 11.1 | 17.5 | 31.0 |  | 17.3 | 11.2 | 28.7
DiffFMaps(S) [35] |  | 3.6 | 19.0 | 16.4 |  | 12.0 | 12.0 | 17.6
NIE(W) [24] |  | 5.5 | 15.0 | 15.1 |  | 11.0 | 8.7 | 15.6
SSMSM(W) [8] |  | 2.4 | 11.0 | 9.0 |  | 4.1 | 8.5 | 7.3
Ours-SCAPE |  | 3.4 | 5.1 | 5.4 |  | 2.6 | 4.0 | 5.1
Ours-FAUST |  | 3.0 | 6.3 | 5.9 |  | 2.9 | 4.0 | 4.8

Moreover, as illustrated in Table 1, our method achieves the best performance
in 5 out of 6 test cases. Remarkably, in the two most challenging tasks,
FAUST_r/SHREC19_r and FAUST_r/SHREC19_r, our method indeed outperforms _all_
of the baselines, including the state-of-the-art methods that take meshes as
input. Regarding point-based methods, SSMSM [8] performs well in the standard
case and outperforms ours in FASUT_r/FAUST_r, but generalizes poorly to unseen
shapes. Another important observation is that our method manifests robustness
with respect to the choice of template shapes. In fact, the above observations
remain true no matter which template we select.

Non-isometric Benchmarks:  We stress test our method on challenging non-
isometric datasets including SHREC07-H and DT4D-H. We emphasize that these
test shapes are unseen during training.

1) SHREC07-H contains 20 heterogeneous human shapes, whose number of vertices
ranges from $3000$ to $15000$. Moreover, there exists some topological noise
(_e.g._ , the hands of the rightmost shape in Fig. 2 is attached to the thigh
in mesh representation). As shown in Fig. 2, SSMSM [8] barely delivers
reasonable results, which might be due to the sensitivity of graph Laplacian
construction on point clouds. Topological noise, on the other hand, degrades
mesh-based methods like Deep Shells [16]. Meanwhile, as shown in Tab. 2, our
method achieves a performance improvement of over 40% compared to the previous
SOTA approaches ($\mathbf{8.5}$ vs. 15.3; $\mathbf{5.9}$ vs. 13.4), which
again confirms the robustness of our approach. 2) Regarding DT4D-H, We follow
the test setting of AttentiveFMaps [29], and only consider the more
challenging inter-class mapping. Interestingly, our method outperforms the
state-of-the-art point cloud-based approach by approximately 30% and exceeds
the performance of mesh-based state-of-the-art methods by over 70%.
Furthermore, even in the worst case, training on FAUST_r and using SCAPE
template, our error is still the lowest compared to external baselines.

Table 2: Mean geodesic errors (×100) on SHREC’07 and DT4D-H. The best is
highlighted.

| Train | FAUST_r |  | SCAPE_r
---|---|---|---|---
Method | Test | SHREC07-H(*) | DT4D-H |  | SHREC07-H(*) | DT4D-H
GeomFMaps [13] |  | 30.5 | 38.5 |  | 28.9 | 28.6
Deep Shells [16] |  | 30.6 | 35.9 |  | 31.3 | 25.8
ConsistFMaps [7] | mesh | 36.2 | 33.5 |  | 37.3 | 38.6
AttentiveFMaps [29] |  | 16.4 | 11.0 |  | 21.1 | 21.4
TransMatch [51] |  | 25.3 | 26.7 |  | 31.2 | 25.3
DiffFMaps [35] |  | 16.8 | 18.5 |  | 15.4 | 15.9
NIE [24] |  | 15.3 | 13.3 |  | 13.4 | 12.1
SSMSM [8] | pcd | 42.2 | 11.8 |  | 37.7 | 8.0
Ours-SCAPE |  | 9.3 | 9.8 |  | 5.9 | 5.7
Ours-FAUST |  | 8.5 | \ |  | 6.1 | \
Ours-CRYPTO |  | \ | 6.9 |  | \ | 5.7

Topologically Perturbed Benchmark:  The shapes in TOPKIDS present various
adhesions – hand/arm adhered to abdomen/thigh (see Fig. 6 in the appendix).
Given the ground-truth maps from each topologically perturbed shape to the
reference shape, we evaluate all 25 maps (excluding the trivial one from
reference to itself) with models trained on FAUST_r and SCAPE_r respectively.
Since the orientation of the TOPKIDS dataset is not originally agreeable with
the aligned version of the two training sets, we align the input with our
orientation regressor and feed them into the baselines depending on extrinsic
information [16; 35; 24; 8]. Regarding the template shape, we adopt the
reference shape as our source mesh in registration. We report the quantitative
comparisons in Tab. 3. It is obvious that topological noise poses great
challenges for methods based on pure intrinsic information [13; 7; 29], while
the counterparts perform relatively well. Especially, among the point-based
methods, our method outperforms the latest SOTA method by a large margin,
namely, over 40% relative error reduction ($\mathbf{7.1}$ vs. $12.3$). We
refer readers to the appendix for a qualitative comparison, which agrees with
the quantitative results above.

Table 3: Mean geodesic errors (×100) on TOPKIDS which trained on FAUST_r and
SCAPE_r. The best is highlighted.

| GeomFMaps [13] | Deep Shells [16] | ConsistFMaps [7] | AttentiveFMaps [29] | DiffFMaps [35] | NIE [24] | SSMSM [8] | Ours
---|---|---|---|---|---|---|---|---
FAUST_r\TOPKIDS | 26.2 | 14.7 | 35.9 | 31.7 | 20.5 | 18.9 | 14.2 | 8.9
SCAPE_r\TOPKIDS | 21.7 | 15.3 | 33.1 | 39.4 | 18.0 | 16.2 | 12.3 | 7.1

###  4.2 Ablation Study

Table 4: Mean geodesic errors (×100) on different ablated settings, the models are all trained on SCAPE_r and test on SHREC’07. w/o Registration | w/o Stage I | w/o Stage II | w/o updating corre. | w/o cons. filter | Full
---|---|---|---|---|---
11.5 | 10.6 | 10.1 | 8.1 | 7.2 | 5.9

We report ablation studies in Tab. 4, in which we verify the effectiveness of
each core design formulated in Sec. 3. Throughout this part, we train on
SCAPE_r and test on SHREC07-H. In the registration stage, we use the SCAPE_r
template as the source mesh and deform it to each point cloud from SHREC07-H.
It is evident that each ablated module contributes to the final performance.
In particular, both stages of registration play an equally important role in
our framework.

Finally, to validate the superiority of our registration scheme, we report in
Tab. 5 the average errors of the initial maps computed by our point-based DFM,
and that of output maps of Ours, NDP [31], AMM [55], which are all based on
the former. It is evident that, across three different test sets, our method
consistently improves the initial maps, while NDP and AMM can even lead to
deteriorated maps than the initial input.

Table 5: Mean geodesic errors (×100) of Ours, NDP, AMM based on the same
initial maps.

|  SCAPE_r |  SHREC19_r |  SHREC07-H
---|---|---|---
Ini. |  5.5 |  8.1 |  11.5
NDP |  5.4 |  11.4 |  8.9
AMM |  11.4 |  10.7 |  8.8
Ours |  2.6 |  5.1 |  5.9

##  5 Conclusion and Limitation

In this paper, we propose a novel learning-based shape registration framework
without correspondence supervision. Our framework incorporates learning-based
shape matching and optimization-based shape registration. Based on the former,
our framework can perform registration between shapes undergoing significant
intrinsic deformations; On the other hand, thanks to the latter, our method
exhibit superior generalizability over the learning-based competitors. Apart
from several designs tailored for our intuitive pipeline, we also introduce a
data-driven solution to facilitate the burden of extrinsically aligning non-
rigid points. We verify our framework through a series of challenging tests,
in which it shows superior performance but also remarkable robustness.

Limitation & Future Work We identify two main limitations of our method: 1)
Our optimization procedure involves iteratively updating correspondences
between the intermediate point clouds and the target one, leaving sufficient
room for improvement for efficiency; 2) Our pipeline is designed for full
shapes, while it is applicable to partial targets, the registration quality is
sub-optimal. In the future, we also plan to incorporate more existing
techniques (_e.g._ , various regularization terms) from shape registration to
further enhance the overall performance.

Acknowledgement The authors thank anonymous reviewers for their constructive
comments and suggestions. This work was supported in part by the National
Natural Science Foundation of China under contract No. 62171256, 62331006, in
part by Shenzhen Key Laboratory of next-generation interactive media
innovative technology (No. ZDSYS20210623092001004).

## References

  * [1] Brian Amberg, Sami Romdhani, and Thomas Vetter.  Optimal step nonrigid icp algorithms for surface registration.  2007\.
  * [2] Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis.  SCAPE: Shape Completion and Animation of People.  2005\.
  * [3] Souhaib Attaiki and Maks Ovsjanikov.  Understanding and improving features learned in deep functional maps.  In CVPR, 2023.
  * [4] Federica Bogo, Javier Romero, Matthew Loper, and Michael J. Black.  FAUST: Dataset and evaluation for 3D mesh registration.  In CVPR, 2014.
  * [5] Aljaz Bozic, Pablo Palafox, Michael Zollhöfer, Angela Dai, Justus Thies, and Matthias Nießner.  Neural non-rigid tracking.  In NeurIPS, volume 33, pages 18727–18737, 2020.
  * [6] Aljaz Bozic, Michael Zollhofer, Christian Theobalt, and Matthias Nießner.  Deepdeform: Learning non-rigid rgb-d reconstruction with semi-supervised data.  In CVPR, pages 7002–7012, 2020.
  * [7] Dongliang Cao and Florian Bernard.  Unsupervised deep multi-shape matching.  In ECCV, 2022.
  * [8] Dongliang Cao and Florian Bernard.  Self-supervised learning for multimodal non-rigid shape matching.  In CVPR, 2023.
  * [9] Jiayi Chen, Yingda Yin, Tolga Birdal, Baoquan Chen, Leonidas J Guibas, and He Wang.  Projective manifold gradient layer for deep rotation regression.  In CVPR, 2022.
  * [10] Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas Guibas.  Vector neurons: a general framework for so(3)-equivariant networks.  In ICCV, 2021.
  * [11] Jiacheng Deng, Chuxin Wang, Jiahao Lu, Jianfeng He, Tianzhu Zhang, Jiyang Yu, and Zhe Zhang.  Se-ornet: Self-ensembling orientation-aware network for unsupervised point cloud shape correspondence.  arXiv preprint arXiv:2304.05395, 2023.
  * [12] Nicolas Donati, Etienne Corman, and Maks Ovsjanikov.  Deep orientation-aware functional maps: Tackling symmetry issues in shape matching.  In CVPR, pages 742–751, 2022.
  * [13] Nicolas Donati, Abhishek Sharma, and Maks Ovsjanikov.  Deep geometric functional maps: Robust feature learning for shape correspondence.  In CVPR, 2020.
  * [14] Marvin Eisenberger, Zorah Lahner, and Daniel Cremers.  Smooth shells: Multi-scale shape registration with functional maps.  In CVPR, 2020.
  * [15] Marvin Eisenberger, David Novotny, Gael Kerchenbaum, Patrick Labatut, Natalia Neverova, Daniel Cremers, and Andrea Vedaldi.  Neuromorph: Unsupervised shape interpolation and correspondence in one go.  In CVPR, 2021.
  * [16] Marvin Eisenberger, Aysim Toker, Laura Leal-Taixé, and Daniel Cremers.  Deep shells: Unsupervised shape correspondence with optimal transport.  In NeurIPS, 2020.
  * [17] Michael Garland and Paul S Heckbert.  Surface simplification using quadric error metrics.  In Proceedings of the 24th annual conference on Computer graphics and interactive techniques, pages 209–216, 1997.
  * [18] Daniela Giorgi, Silvia Biasotti, and Laura Paraboschi.  Shape retrieval contest 2007: Watertight models track.  SHREC competition, 2007.
  * [19] Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry.  3d-coded: 3d correspondences by deep deformation.  In ECCV, 2018.
  * [20] Chen Guo, Xu Chen, Jie Song, and Otmar Hilliges.  Human performance capture from monocular video in the wild.  In 3DV, 2021.
  * [21] Jiahui Huang, Tolga Birdal, Zan Gojcic, Leonidas J. Guibas, and Shi-Min Hu.  Multiway Non-rigid Point Cloud Registration via Learned Functional Map Synchronization.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2022.
  * [22] Ruqi Huang and Maks Ovsjanikov.  Adjoint map representation for shape analysis and matching.  In Computer Graphics Forum, volume 36, pages 151–163. Wiley Online Library, 2017.
  * [23] Ruqi Huang, Jing Ren, Peter Wonka, and Maks Ovsjanikov.  Consistent zoomout: Efficient spectral map synchronization.  In Computer Graphics Forum, volume 39, pages 265–278. Wiley Online Library, 2020.
  * [24] Puhua Jiang, Mingze Sun, and Ruqi Huang.  Neural intrinsic embedding for non-rigid point matching.  In CVPR, 2023.
  * [25] Zorah Lähner, Emanuele Rodolà, Michael M Bronstein, Daniel Cremers, Oliver Burghard, Luca Cosmo, Andreas Dieckmann, Reinhard Klein, and Yusuf Sahillioglu.  Shrec’16: Matching of deformable shapes with topological noise.  Proc. 3DOR, 2(10.2312), 2016.
  * [26] Itai Lang, Dvir Ginzburg, Shai Avidan, and Dan Raviv.  Dpc: Unsupervised deep point correspondence via cross and self construction.  In 2021 International Conference on 3D Vision (3DV), pages 1442–1451. IEEE, 2021.
  * [27] Zohar Levi and Craig Gotsman.  Smooth rotation enhanced as-rigid-as-possible mesh animation.  IEEE transactions on visualization and computer graphics, 21(2):264–277, 2014.
  * [28] Kun Li, Jingyu Yang, Yu-Kun Lai, and Daoliang Guo.  Robust non-rigid registration with reweighted position and transformation sparsity.  IEEE transactions on visualization and computer graphics, 25(6):2255–2269, 2018.
  * [29] Lei Li, Nicolas Donati, and Maks Ovsjanikov.  Learning multi-resolution functional maps with spectral attention for robust shape matching.  In NeurIPS, 2022.
  * [30] Yang Li and Tatsuya Harada.  Lepard: Learning partial point cloud matching in rigid and deformable scenes.  In CVPR, 2022.
  * [31] Yang Li and Tatsuya Harada.  Non-rigid point cloud registration with neural deformation pyramid.  In NeurIPS, 2022.
  * [32] Or Litany, Tal Remez, Emanuele Rodolà, Alexander M. Bronstein, and Michael M. Bronstein.  Deep functional maps: Structured prediction for dense shape correspondence.  In ICCV, 2017.
  * [33] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black.  SMPL: A skinned multi-person linear model.  TOG, 34(6):248:1–248:16, 2015.
  * [34] Robin Magnet, Jing Ren, Olga Sorkine-Hornung, and Maks Ovsjanikov.  Smooth non-rigid shape matching via effective dirichlet energy optimization.  In 3DV, 2022.
  * [35] Riccardo Marin, Marie-Julie Rakotosaona, Simone Melzi, and Maks Ovsjanikov.  Correspondence learning via linearly-invariant embedding.  Advances in Neural Information Processing Systems, 33:1608–1620, 2020.
  * [36] S. Melzi, R. Marin, E. Rodolà, U. Castellani, J. Ren, A. Poulenard, P. Wonka, and M. Ovsjanikov.  Matching Humans with Different Connectivity.  In Eurographics Workshop on 3D Object Retrieval, 2019.
  * [37] Simone Melzi, Jing Ren, Emanuele Rodolà, Peter Wonka, and Maks Ovsjanikov.  Zoomout: Spectral upsampling for efficient shape correspondence.  Proc. SIGGRAPH Asia, 2019.
  * [38] Dorian Nogneng and Maks Ovsjanikov.  Informative descriptor preservation via commutativity for shape matching.  Computer Graphics Forum, 36(2):259–267, 2017.
  * [39] Maks Ovsjanikov, Mirela Ben-Chen, Justin Solomon, Adrian Butscher, and Leonidas Guibas.  Functional Maps: A Flexible Representation of Maps Between Shapes.  ACM Transactions on Graphics (TOG), 31(4):30, 2012.
  * [40] Gianluca Paravati, Fabrizio Lamberti, Valentina Gatteschi, Claudio Demartini, and Paolo Montuschi.  Point cloud-based automatic assessment of 3d computer animation courseworks.  IEEE Transactions on Learning Technologies, 10(4):532–543, 2016.
  * [41] Adrien Poulenard, Marie-Julie Rakotosaona, Yann Ponty, and Maks Ovsjanikov.  Effective rotation-invariant point cnn with spherical harmonics kernels.  In 3DV, 2019.
  * [42] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In CVPR, 2017.
  * [43] Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  In NeurIPS, 2017.
  * [44] Jing Ren, Adrien Poulenard, Peter Wonka, and Maks Ovsjanikov.  Continuous and orientation-preserving correspondences via functional maps.  ACM Trans. Graph., 37(6):248:1–248:16, December 2018.
  * [45] Jean-Michel Roufosse, Abhishek Sharma, and Maks Ovsjanikov.  Unsupervised deep learning for structured shape matching.  In ICCV, 2019.
  * [46] Abhishek Sharma and Maks Ovsjanikov.  Weakly supervised deep functional maps for shape matching.  In NeurIPS, 2020.
  * [47] Nicholas Sharp, Souhaib Attaiki, Keenan Crane, and Maks Ovsjanikov.  Diffusionnet: Discretization agnostic learning on surfaces.  ACM Transactions on Graphics, 2022.
  * [48] Nicholas Sharp and Keenan Crane.  A laplacian for nonmanifold triangle meshes.  Computer Graphics Forum, 2020.
  * [49] Mingze Sun, Shiwei Mao, Puhua Jiang, Maks Ovsjanikov, and Ruqi Huang.  Spatially and spectrally consistent deep functional maps.  In ICCV, 2023.
  * [50] Carlos Sánchez-Belenguer, Simone Ceriani, Pierluigi Taddei, Erik Wolfart, and Vítor Sequeira.  Global matching of point clouds for scan registration and loop detection.  Robotics and Autonomous Systems, 123:103324, 2020.
  * [51] Giovanni Trappolini, Luca Cosmo, Luca Moschella, Riccardo Marin, Simone Melzi, and Emanuele Rodolà.  Shape registration in the time of transformers.  Advances in Neural Information Processing Systems, 34:5731–5744, 2021.
  * [52] Gül Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J. Black, Ivan Laptev, and Cordelia Schmid.  Learning from synthetic humans.  In CVPR, 2017.
  * [53] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon.  Dynamic graph cnn for learning on point clouds.  ACM Transactions on Graphics (TOG), 2019.
  * [54] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany.  Pointcontrast: Unsupervised pre-training for 3d point cloud understanding.  In ECCV, 2020.
  * [55] Yuxin Yao, Bailin Deng, Weiwei Xu, and Juyong Zhang.  Fast and robust non-rigid registration using accelerated majorization-minimization.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  * [56] Hao Yu, Ji Hou, Zheng Qin, Mahdi Saleh, Ivan Shugurov, Kai Wang, Benjamin Busam, and Slobodan Ilic.  Riga: Rotation-invariant and globally-aware descriptors for point cloud registration.  arXiv preprint arXiv:2209.13252, 2022.
  * [57] Hao Yu, Zheng Qin, Ji Hou, Mahdi Saleh, Dongsheng Li, Benjamin Busam, and Slobodan Ilic.  Rotation-invariant transformer for point cloud matching.  arXiv preprint arXiv:2303.08231, 2023.
  * [58] Tao Yu, Zerong Zheng, Kaiwen Guo, Jianhui Zhao, Qionghai Dai, Hao Li, Gerard Pons-Moll, and Yebin Liu.  Doublefusion: Real-time capture of human performances with inner body shapes from a single depth sensor.  In CVPR, 2018.
  * [59] Yiming Zeng, Yue Qian, Zhiyu Zhu, Junhui Hou, Hui Yuan, and Ying He.  Corrnet3d: Unsupervised end-to-end learning of dense correspondence for 3d point clouds.  In CVPR, pages 6052–6061, 2021.
  * [60] Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun.  Fast global registration.  ECCV, 2016.

In this appendix, we provide more technical details and experimental results.
In Sec. A, we provide 1) A detailed description of our orientation regressor
in Sec. A.1; 2) More details on the registration method presented in Alg. 1 of
the main text in Sec. A.3; 3) Implementation details of the whole pipeline in
Sec. A.4.

In the second part, we provide more experimental results. We first provide
more qualitative results in Sec. B.1, as well as quantitative results on
training on large datasets and on animal datasets in Sec. B.2; Then we
demonstrate the robustness of our method with respect to several perturbations
in Sec. B.3; In Sec. B.4, we showcase the scalability of our method in
matching real scans. Finally, we report a run-time decomposition and analysis
in Sec. B.5.

Note that we also include a short video showcasing the registration process of
our pipeline in the Supplemental Material.

##  Appendix A Technical Details

###  A.1 Orientation Regressor

Orientation plays an important role in point cloud processing. Classic point-
based feature extractors [42; 43; 53] are not rotation-invariant/equivariant
by design and, therefore are limited. A common practice is to leverage data
augmentation [42]. However, such can only robustify the model under small
perturbations, while falling short of handling arbitrary rotations in $SO(3)$.
On the other hand, recent advances in designing rotation-invariant/equivariant
feature extractor [41; 10] seem to shed light on alleviating alignment
requirements in non-rigid point matching. However, we empirically observe that
such a design generally degrades the expressive power of the feature
extractor, leading to inferior performance of the trained DFM.

This issue also attracts attention from the community of shape matching. In
[11], the rotation prediction, instead of directly estimating the
transformation, is formulated as a classification problem. However, this
method can only address the rotation around a fixed axis and exhibits
relatively low accuracy. Other works [56; 57] transform the input point cloud
into handcrafted features. However, it should be noted that this approach
often increases the computational complexity due to the additional feature
extraction and processing steps involved.

In this paper, we propose a data-driven solution to this problem, accompanied
by a registration module that is robust with the estimated alignment (see
Tab.3 in the main text). As mentioned in the main text, we adopt the regressor
model proposed in [9]. We use the synthetic SURREAL shapes [52], which are
implicitly aligned by the corresponding generative codes to train our
orientation regressor. We follow [9] and use PointNet++ [43] as the feature
extractor. As we assume to be given full shape, either mesh or point cloud,
for each input cloud, we translate it so its mass lies at the origin point.
Then we estimate the orientation deviation, $R$, of it from the orientation
implicitly defined by the SURREAL dataset. Finally, we apply the inverse
rotation $R^{T}$ on the input to obtain the aligned shape. We emphasize again
that all the shapes, either train or test, go through this alignment in our
pipeline.

###  A.2 Deep Functional Maps (DFM)

We illustrate the DFM, both for meshes and point clouds in Fig. 4.

Figure 4: (a) DFM architecture for pre-training on meshes; (b) Our proposed
architecture for training a point-based DFM, which is self-supervised by its
mesh-based counterpart.

Input: Source mesh $\mathcal{S}=\\{\mathcal{V},\mathcal{E}\\}$ and target
point cloud $\mathcal{T}$;Trained point feature extractor F

Output: $\textbf{X}^{*}$ converging to a local minimum of $E_{\mbox{total}}$;
Deformed source model $\\{\mathcal{V}^{*},\mathcal{E}\\}$; Correspondence
$\Pi_{\mathcal{S}\mathcal{T}}^{*},\Pi_{\mathcal{T}\mathcal{S}}^{*}$ between
$\mathcal{S}$ and $\mathcal{T}$.

1 Initialize deformation graph $\mathcal{DG}$ and $\textbf{X}^{(0)}$ with
identity transformations; $F_{\mathcal{T}}=\textbf{F}(\mathcal{T})$;k = 0;

2$eps=10^{-8}$; $count=0$;

3while  _True_ do

4 $\mathcal{V}^{k}=\mathcal{DG}(\textbf{X}^{k-1},\mathcal{V}^{k-1})$;

5 if  _$k\%100==0$ and Flag == Stage-I _ then

6 $F^{(k)}_{\mathcal{S}}=\textbf{F}(\mathcal{V}^{(k)})$;
$\Pi_{\mathcal{S}\mathcal{T}}^{(k)}=\textbf{NN}(F^{(k)}_{\mathcal{S}},F_{\mathcal{T}})$
;
$\Pi_{\mathcal{T}\mathcal{S}}^{(k)}=\textbf{NN}(F_{\mathcal{T}},F^{(k)}_{\mathcal{S}})$;

7

8 if  _$k\%100==0$ and Flag == Stage-II _ then

9
$\Pi_{\mathcal{S}\mathcal{T}}^{(k)}=\textbf{NN}(\mathcal{V}^{(k)},\mathcal{T})$
;
$\Pi_{\mathcal{T}\mathcal{S}}^{(k)}=\textbf{NN}(\mathcal{T},\mathcal{V}^{(k)})$;

10

11 Compute the set of filtered correspondences $\mathcal{C}^{k}$ ;

12
$E^{(k)}_{\mbox{total}}=\lambda_{\mbox{cd}}E_{\mbox{cd}}(\mathcal{V}^{(k)},\mathcal{T})+\lambda_{\mbox{corr}}E_{{\mbox{corr}}}(\mathcal{V}^{(k)}_{\mathcal{C}^{k}},\mathcal{T}_{\mathcal{C}^{k}})+\lambda_{\mbox{arap}}E_{\mbox{arap}}(\mathcal{DG})$;

13 $\textbf{X}^{(k)}=\arg\min{E^{(k)}_{\mbox{total}}}$;

14 if  _$(E^{(k)}_{\mbox{total}}-E^{(k-1)}_{\mbox{total}}) <eps$_ then

15

16 count = count + 1;

17 if  _count > 15_ then

18 if Flag == Stage-I then Flag = Stage-II; count = 0; eps = $10^{-7}$;

19 if Flag == Stage-II then return
$\Pi_{\mathcal{S}\mathcal{T}}^{*}$;$\Pi_{\mathcal{T}\mathcal{S}}^{*}$;
$\mathcal{V}^{*}$;

20

21

22 k = k + 1;

Algorithm 2 Shape registration pipeline.

###  A.3 Registration Algorithm

We provide a more detailed version of our registration algorithm in Alg. 2.
given a source mesh $\mathcal{S}$, target point cloud $\mathcal{T}$ and a
trained point feature extractor F, we first initialize the deformation graph
$\mathcal{DG}$ with QSlim method [17] and set $\textbf{X}^{(0)}$ with identity
transformations. After that, we compute the target point feature
$F_{\mathcal{T}}$. During registration, we first deform the source model with
a deformation graph, then we update the correspondence by computing the
nearest neighbor between the source feature and target feature in stage-I or
the source point cloud and target point cloud in stage-II. After we compute
the set of filtered correspondences $\mathcal{C}^{k}$, we update the total
cost function $E_{\mbox{total}}$ with the ADMM optimizer. Finally, we follow
[31] to implement an early stop strategy in both stages.

###  A.4 Implementation Details

We implement our framework in PyTorch. We use modified DGCNN [24] the backbone
of our feature extractor. The output feature dimension is set to 128. In the
training related to deep functional maps, we always use the first 50
eigenfunctions of Laplace-Beltrami operators. For training the DFM network, in
Eqn.(6) of the main text, we empirically set $\lambda_{\mbox{bij}}$ = 1.0,
$\lambda_{\mbox{orth}}$ = 1.0, $\lambda_{\mbox{align}}$ = 1e-4,
$\lambda_{\mbox{NCE}}$ = 1.0. We train our feature extractor with the Adam
optimizer with a learning rate equal to 2e-3. The batch size is chosen to be 4
for all datasets.

Regarding the registration optimization, in Eqn.(12) of the main text, we
empirically set $\lambda_{\mbox{cd}}$ = 0.01, $\lambda_{\mbox{corr}}$ = 1.0,
$\lambda_{\mbox{arap}}$ = 20 in Stage-I and $\lambda_{\mbox{cd}}$ = 1.0,
$\lambda_{\mbox{corr}}$ = 0.01, $\lambda_{\mbox{arap}}$ = 1 in Stage-II. For
$\alpha$ in Eqn.(9) of the main text, we set $\alpha=0.2$. Note that the roles
of $\lambda_{\mbox{cd}}$ and $\lambda_{\mbox{corr}}$ are reversed from Stage-I
to Stage-II. In fact, in Stage-I, the initial shapes are likely to differ by
large deformation, therefore it is more reliable to estimate correspondences
via the learned embeddings. Once Stage-I converges, we empirically observe
that the deformed source shape admits a similar pose with the target (see the
video in Supplemental Material), therefore it is reasonable to further deform
the source by extrinsic embeddings, _i.e._ , Chamfer distance on the
coordinates. We stress that both losses are critical in both stages and
perform the following ablation experiment on setting $\lambda_{\mbox{cd}}$ and
$\lambda_{\mbox{corr}}$ to be zero independently in the two stages. Tab. 6
reports the scores. We find that most of the time, turning off leads to worse
results, especially when the test shapes are heterogeneous (see, e.g.,
SHREC07-H).

Table 6: Mean geodesic errors (×100) of Ours on setting $\lambda_{cd}$ and
$\lambda_{corr}$ to be zero independently in the two stages

|  SCAPE_r |  SHREC19_r |  SHREC07-H
---|---|---|---
$\lambda_{cd}=0$ in Stage-I |  2.5 |  5.5 |  6.9
$\lambda_{corr}=0$ in Stage-II |  2.7 |  5.6 |  6.8
Ours |  2.6 |  5.1 |  5.9

##  Appendix B Experiments

###  B.1 Further Qualitative Results

Figure 5: More non-rigid shape registration results. The original source mesh
and its deformed counterpart are represented in blue, while the target point
cloud and the ground truth mesh are depicted in orange. Figure 6: We present
the qualitative results of testing the TOPKIDS trained on SCAPE_r. Our method
demonstrates robustness in regions of mesh topology with noise, maintaining
stable performance.  Figure 7: We showcase some examples of failed
registrations.

Visualization of Registration ResultsIn this section, we show more qualitative
results of registration regarding different templates and different test
shapes in Fig 5. Our method is capable of registering shapes with significant
deformations across different templates.

Qualitative Comparison in TOPKIDS:  In Fig. 6, we qualitatively visualize maps
obtained by different methods tested in the TOPKIDS benchmark. It is obvious
that our results outperform all the competing methods. Especially, a purely
intrinsic method, AttentiveFMaps seriously fails due to the topological
perturbation within the mesh structure.

Failure Cases: On the other hand, we also identify some failure cases, which
are shown in Fig. 7. The two cases both suffer prominent heterogeneity between
the template and the target. In the left case, where an ideal reference is
given, we adapt it in registration instead of enforcing the hard highly non-
isometric deformation with templates from SCAPE_r or FAUST_r (see Sec. B.3).
While in the right case, it is not obvious how to choose an easier template
for establishing correspondences between the two shapes, we, therefore, settle
down at the setting described in Sec.4.2. It is worth noting that, though, we
can easily identify that the problematic regions, _i.e.,_ head, and hands from
the visualization, which is unavailable for methods only return maps.

Templates: In the Fig. 8 we visualize the templates used in different
experiments. We select a pose close to the T-pose from the training sets of
SCAPE_r, DT4D-H, and FAUST_r, respectively, as our templates.

Figure 8:  We respectively use template from SCAPE (left), CRYPTO (middle),
and FAUST (right) in our experiments.

###  B.2 Further Quantitative Results

Trained on Large-Scale Dataset: We have followed the setting of SSMSM [8] to
train our feature extractor on the SURREAL 5k dataset [f] and to test it on
the FAUST_r, SCAPE_r, and SHREC19_r datasets. We select one training shape
from SURREAL 5k as the template in registration. As shown in Table 7, our
method outperforms the competing non-rigid point cloud matching methods by a
noticeable margin. We also refer the readers to Table 3 in [8] for a complete
table including methods utilizing mesh input, and we highlight that our method
is even comparable with the latter. In particular, ours vs. SOTA of mesh-based
techniques over the three test datasets: 3.2 vs. 2.3, 2.5 vs. 3.3, 4.6 vs.
4.7. That is, we achieved the best overall result in two of the three test
datasets.

Table 7: Mean geodesic errors (×100) trained on SURREAL

| Train | SURREAL |
---|---|---|---
Method | Test | FAUST_r | SCAPE_r | SHREC19_r | |Data|
DiffFMaps |  | 26.5 | 34.8 | 42.2 | 230k
DPC |  | 13.4 | 15.8 | 17.4 | 230k
SSMSM |  | 3.5 | 3.8 | 6.6 | 5k
Ours-SURREAL |  | 3.2 | 2.5 | 4.6 | 5k

Trained on Animal Dataset: We also perform experiments on the remeshed SMAL
dataset. We first randomly generate 5000 shapes with SMAL model to train the
alignment module. Then we train a DFM with the remeshed SMAL dataset. The
template shape is ‘dog_06’ from the training set. The quantitative results are
reported in Table 8. Remarkably, our method achieves more than a $40\%$
performance improvement than the second-best baselines.

Table 8: Mean geodesic errors (×100) on SMAL_r. | TransMatch | DPC | DiffFMaps | NIE | SSMSM | Ours
---|---|---|---|---|---|---
SMAL_r | 20.7 | 21 | 17.1 | 16.3 | 7.2 | 4.2

###  B.3 Robustness

Robustness with respect to Input Perturbation:  We evaluate the robustness of
different methods with respect to noise and orientation perturbations in Tab.
9. In the former, we perturb the input point clouds with Gaussian noise. In
the latter, we randomly rotate the input point clouds within $\pm 30$ degrees
along $x,y,z-$axis. Thanks to our optimization-based approach and orientation
regressor, in both tests we achieved the best accuracy, but also the lowest
standard deviation over three runs of experiments.

Table 9: Mean geodesic errors (×100) on under different perturbations. Noisy
PC means the input point clouds are perturbed by Gaussian noise. Rotated PC
means the input point clouds are randomly rotated within $\pm 30$ degrees
along $x,y,z-$axises respectively. The standard deviation value is shown in
parentheses.

|  Unperturbed PC |  Noisy PC |  Rotated PC
---|---|---|---
DiffFMaps [35] |  12.0 |  14.9(2.57) |  26.5(3.35)
NIE [24] |  11.0 |  11.5(0.32) |  19.9(1.29)
SSMSM [8] |  4.1 |  5.4(0.11) |  9.2(1.01)
Ours |  2.6 |  2.9(0.06) |  3.6(0.96)

Robustness with respect to Missing Parts: In Tab.1 of the main text, we report
shape matching results on SHREC19_r dataset, where the $40^{\mbox{th}}$ shape
is commonly excluded in the test phase of prior works [29; 8]. The main reason
is that this shape is incomplete, consisting of two main connected components
(see the abdomen of the source shape in Fig. 9). Therefore, for intrinsic
methods depending on mesh eigenbasis, such disconnectedness makes spectral
alignment extremely difficult, if not impossible. On the other hand, extrinsic
methods show a more robust outcome. In particular, we show in Fig. 9 that our
framework can reasonably deform the template mesh (from SCAPE_r) to the
incomplete point cloud with a significantly isolated portion, which leads to
superior matching results compared to the baselines.

Figure 9: A challenging case from SHREC19_r dataset, see the text for details.

Robustness of Orientation Regressor:  To demonstrate that our learned
orientation regressor (see Sec.3.1 of the main text), we visualize its outcome
on shapes from SHREC07-H, which manifests notable diversity in sampling
density, size, and initial orientation.

Figure 10: Bottom row: The input shapes of SHREC’07 exhibit various
orientations; Top row: The shapes aligned by our orientation regressor. Note
that there exists significant differences in sampling density.

###  B.4 Scalability

Our approach is scalable w.r.t input size. Thanks to the fact that we non-
rigidly align shapes in $\mathbb{R}^{3}$, we can in theory freely down- and
up-sample both the template mesh and the target point clouds. Note this is
non-trivial for methods based on mesh or graph Laplacian, as deducing dense
maps with landmark correspondences over graph structures is a difficult task
on its own. In Fig. 11, we show the matching results on the real scans from
the FAUST challenge, each of which consists of around 160,000 vertices. In
contrast, [8] can handle at most 50,000 vertices without running out of 32G
memory on a V100 GPU. We visualize its matching results on the subsampled (to
40,000 vertices) point clouds for comparison.

Figure 11: Qualitative results on matching MPI-FAUST raw scan data. Our method
can match the point clouds at the original resolution, while SSMSM [8] can
not.

###  B.5 Run-time Analysis

In this part, we analyze the time cost of our optimization-based registration
module. In particular, we separate it into two main parts, initialization, and
iterative optimization. For the former, we in general fix a source mesh as a
template, and then we construct a deformation graph on it as described in
Sec.3.2 in the main text. Since the deformation graph can be pre-computed and
cached for further use of the same template, we put more emphasis on the
latter.

In particular, we perform registration between the template shape from SCAPE_r
and the 20 test shapes therein and compute the average run-time of each core
step in the optimization. As illustrated in Tab. 10, our method achieves an
average optimization time of approximately 13.4 seconds per shape, with about
$90\%$ of the total time being dedicated to Stage-I. Analysis depicted in Fig.
12 reveals that the most time-consuming operation is the ADMM backward step,
while the combined time cost of our correspondence filtering and updating
operations accounts for less than $3\%$ of the overall execution time. Though
the reported run-time is not quite satisfying, we emphasize that to evaluate
maps across $n$ point clouds, we only need to register them to the same
template shape ($O(n)$), which is different from the pair-wise ($O(n^{2})$)
mapping and post-processing approaches [7].

Table 10: Average time and iteration cost for each shape. | Total | Stage-I | Stage-II
---|---|---|---
Time cost(s) | 13.4 | 12.0 | 1.4
Iterations | 1274. | 1130. | 144.
Figure 12: Run-time decomposition of our registration module.


# ragamAI: A Network Based Recommender System to Arrange a Indian Classical
Music Concert

Arunkumar Bagavathi1, Siddharth Krishnan2, Sanjay Subrahmanyan, S.L.
Narasimhan  Department of Computer Science12
Oklahoma State University1, University of North Carolina at Charlotte2
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

South Indian classical music (_Carnatic music_) is best consumed through live
concerts. A carnatic recital requires meticulous planning accounting for
several parameters like the performers’ repertoire, composition variety,
musical versatility, thematic structure, the recital’s arrangement, etc. to
ensure that the audience have a comprehensive listening experience. In this
work, we present _ragamAI_ a novel machine learning framework that utilizes
the tonic nuances and musical structures in the carnatic music to generate a
concert recital that melodically captures the entire range in an octave.
Utilizing the underlying idea of playlist and session-based recommender
models, the proposed model studies the mathematical structure present in past
concerts and recommends relevant items for the playlist/concert. _ragamAI_
ensembles recommendations given by multiple models to learn user idea and past
preference of sequences in concerts to extract recommendations. Our
experiments on a vast collection of concert show that our model performs
25%-50% better than baseline models. _ragamAI_ ’s applications are two-fold.
1) it will assist musicians to customize their performance with the necessary
variety required to sustain the interest of the audience for the entirety of
the concert 2) it will generate carefully curated lists of south Indian
classical music so that the listener can discover the wide range of melody
that the musical system can offer.

###### Index Terms:

recommender system, music information retrieval, tone embeddings

##  I Introduction

The analysis of classical music has become a mainstay in the field of music
information retrieval (MIR) over the last decade. Recent research has
investigated several nuances of classical music, be it in rhythm cycles, tempo
estimation, beat tracking, instrument classification, and melodic analysis,
using data-driven techniques. However, there is a paucity of work in bridging
these information retrieval techniques to applications like playlist
recommendations and automated concert planners. In this work, using the lens
of south Indian classical music, also known as _Carnatic_ music, we exploit
the grammatical structure and mathematical underpinnings of the music system
to develop a machine learning model that can be applied to build a concert
plan. While there are several recent works that model different aspects of
Indian classical music, to the best of our knowledge we are the first to use
the melodic structure, defined as a _ragam_ , present in the music system to
design a recommender system. Carnatic musicians have been relying on their
repertoire and memory to design song lists in the past. An automated system
will enhance the experience and make it more versatile in the development of a
concert song list. The proposed model can also be used as an ideal tool to
improve a repertoire given that it can draw on source material from different
types of databases that suit the case. Moreover, the model can also be used to
generate music playlists for streaming applications like Spotify, Pandora,
last.fm, etc. to provide an enriching experience that encompasses the musical
range offered by the carnatic system in a song sequence.

The efficacy of a session-based recommendation model is to provide desired
items for a user in their current session based on their past preferences. For
example, a video streaming service like YouTube predicting the user’s
preference to watch next. In the research literature, there are several
frameworks and methodologies for music and video playlist recommendations.
Given the mathematical details in south Indian classical music and the task of
arranging songs in a concert for a musician, we formulate this problem as
session (concert)-based playlist (song) recommendation and propose a network
based recommender model. We give a network/graph representation for sequences
in Indian classical music concerts with our proposed _Raaga network_. Along
with features extracted from the network, the proposed model capture co-
occurrences, melodic patterns, and musician’s preference of flow in concerts
as features. With experiments we show that inclusion of such features improve
the performance of recommender model in constructing Indian classical music
concerts.

In this work, we aim to solve following research questions by providing a
music recommender framework for Indian classical music concerts:

  * •

How the concerts can be structured to understand its organization? We propose
a network structure called _Raaga Network_ that captures order, co-occurrence,
and context of each item(song/raaga) in concerts.

  * •

Can historical data help musicians to build new repertoire? We postulate a
network-based machine learning model to recommend items(songs/raagas) based on
their underlying structure and mathematical constructs in Raaga Network. We
frame this model to extract a set of recommendations for a given sequence of
songs.

  * •

How useful are item(songs/raagas) recommendations given by the models? We
evaluate the usefulness of recommendations from the proposed framework using
offline methods like precision and discounted cumulative gain.

##  II Related Work

Innovations in _machine learning_ helped the scientific community to
contribute to large scale recommendation systems like Pinterest [1] and
Spotify [2]. All recommender systems fall in one of the three categories:
content-based (_domain dependent_), collaborative (_domain independent_), and
hybrid (_multi-model frameworks_) [3]. These algorithmically constructed
systems are evaluated using either offline measures like precision, recall,
mean average precision(MAP), Normalized Discounted Cumulative Gain(nDCG) or
online measures like A/B test and p-value [4].

Advancements in recommendation systems have been applied to music domain as
well, Spotify and Pandora for example. The most popular problem in music
recommendations is to select a set of songs as a playlist for a user based on
their mood and preferred genre and artist [5]. This problem has been answered
using multiple methods like: frequent pattern mining [6], collaborative
filtering [7], and hybrid models [8]. IN addition to the playlist
recommendation models are the session-based prediction models (i.e.), [9, 10,
11]. _Novelty_ and _diversity_ are considered to be important evaluation
measures for music recommendation models [12] to provide the user interesting
and unexplored suggestions. In this paper, we present a session-based
recommendation model that utilizes deep learning algorithms to give
recommendations.

Machine learning has been used in the Indian classical music over the past
decade to study multitude of concepts. For example, classifying recordings
from YouTube based on the swara using random forest algorithm [13], using
pitch information in music signals [14], and a vector based classification
model [15], similar to text classification model. Identifying tonic of a music
from multi-pitch analysis of the given audio was also given as a
classification problem [16]. Unlike the existing methods, we describe methods
through the lens of recommender systems. We incorporate the idea of embedding
possible features from historical concerts and frequent pattern mining methods
to make recommendations. Moreover, our model is designed as a human computer
interaction system to communicate recommendations.

##  III Background

Carnatic music is the classical music system of Southern India. The Carnatic
music tradition is built on the melodic foundation of the ragam or scale that
encompasses a collection of swaras (notes) in an octave. The scale follows a
12 note per octave system. This has been expanded into a 16 note system
through the 72 melakarta scheme. The ragam system over the last 3 or more
centuries has evolved into being categorized according to the mela janya
scheme where by ragams are either parent scales or derivatives (mela or
janya).

A melakarta by definition will have the same notes in both its ascending and
descending scales. The math in the organization of the melakarta scheme will
give an idea of the variety of ragams available for musicians to learn,
practice, and perform. A typical octave will have the 7 basic notes: S R G M P
D N. The S and the P are fixed and static notes. R, G, M, D, N will have
different variations lie R1, R2, R3, G1, G2, G3, M1, M2, D1, D2, D3, , N1, N2,
N3. If one looks at the frequency values of these notes, R3 corresponds to G1
and D3 corresponds to N1. The melakarta scheme by definition will have a set
of 7 notes with S and P constant. So the first ragam will be S R1, G1, M1, P,
D1, N1, . The scheme also follows the rule that there will only be one variety
of each note that is permissible. So there can be no occurrence of R1 and R2
in a melakarta. Another constraint is that since the frequency values of R3
and G1 are the same they will never occur together. So typically the scheme
follows the system of the ragams $1-36$ having S, M1 & P as constants and
$37-72$ having S, M2 & P as constants.

Subdividing the ragams further the first 6 will have S R1, G1, M1, P as
constants and the six variations will add the combinations D1/N1, D1/N2,
D1/N3, D2/N2, D2/N3 & D3/N3. The next six will have R1/G2 with the six
varities of D/N and so on to arrive at a total of 72 ragams.

The janya ragams are derivative scales from the parent. Janya ragams have no
rules. One can miss few notes in the scale and they can even have different
ascending and descending scales.

A typical Carnatic music concert consists of pieces that are performed over a
$60-240$ minute period depending on the artist and location. Each composition
in a concert can be of varying lengths and usually revolving around a central
piece (main ragam). Performers therefore have to train a huge repertoire of
compositions set to perform in a variety of ragams. These ragams offer melodic
variety in terms of sound, color, and aesthetics. This variety comes through a
process of inherent differences because of the specific notes occurring as
well as musical and aesthetic differences based on how they are handled by the
musician.

The challenge therefore to the performer is to come up with a list of pieces
that can offer variety musically, keep the listener engaged and remain fresh
so as to avoid being stale and monotonous. A primary constraint of any song
list would be non repetition of ragams. The Carnatic music system technically
has innumerable ragams, however the numbers in vogue and practiced on a
regular basis would probably be in the region of less than 1000. The second
aspect of choosing pieces will be compositions. The process involved is coming
up with an ideal list of pieces for a concert is taking into consideration
both compositions and ragas. All compositions are set to a particular ragam
and the entire rendition of the same will conform to that scale of notes only.
We plan to exploit composition based recommendations in our follow-up work.

##  IV Methodologies

Figure 1: Architecture of the ragamAI. The model composes 3 layers: 1. Network
embedding(skip-gram model) and Concert embedding(embed hand picked features),
2. Vector aggregation from two embedding, and 3. Recommendation

The proposed ragamAI framework relies on the influence of two separate models:
1. a deep attention model to capture the importance of sequence of ragams, and
2. an embedding model to capture the importance of hand picked features to
train. Unlike other methods, which predict next event or item in a given
sequence [6, 9, 17, 18, 19], the proposed model(s) predict all forthcoming
items (songs) in a given concert. All our models use both 1-to-1 mapping of
features and feature similarity to measure relevancy of recommendations. An
overview of the proposed model is represented in Figure 1

###  IV-A Raaga network

Figure 2: Sample raaga network constructed from concerts. Concerts C1,C2,C3,
and C4 are represented as an ordered list of ragams. Colors in the figure are
replaced with edge weights in all experiments. NOTE: Each song in a concert
consists of only one ragam

Networks or graphs have been considered as a promising framework to study
variety of applications like influence modeling [20], community detection
[21], and recommender systems [22]. Their organization of nodes and edges help
to study the structural organization and positional values of entities (nodes
and communities). We propose the _Raaga network_ as a systematic
representation for concerts, where each concert is an ordered transaction of
items (songs/ragams). Formally, we define the _Raaga network_ as
$\mathcal{G}=(\mathcal{R},\mathcal{E},\textit{w})$, where $\mathcal{R}$ is a
set of items/ragams, $\mathcal{E}$ is a set of edges representing the flow
from one ragam($\mathcal{R}_{m}$) to the next($\mathcal{R}_{n}$)
$\forall\mathcal{R}_{m},\mathcal{R}_{n}\in\mathcal{R}$, w is edge weight
representing the overall frequency of the sequence, and $|\mathcal{R}|=$ Total
number of unique items/ragams. Example raaga networks are given in Figures 1
and 2. We capture the order of ragams in concerts by maintaining a set of
attributes $\mathcal{A}_{i}={a_{1},a_{2}\dots a_{n}}$, where
$i=\mathcal{R}_{1},\mathcal{R}_{2}\dots\mathcal{R}_{|\mathcal{R}|}$. Example
attributes include concert id, ragam type, and tonic information of a ragam.
The proposed network gives intuition on the flow of each concert and gives an
intuition of a ragam’s context throughout all concerts. We use this network as
an underlying framework to produce recommendations in the proposed model.

###  IV-B Ragam representation learning using node2vec

The structural knowledge from the ragam network can help the recommender model
to give better results to the user based on their co-occurrence. We use a
network embedding model: node2vec [23] to extract features of nodes in the
raaga network. This model capture the node context with k-iterations of random
walks of length l along with the optimization function given in Equation 1

$\max_{f}\sum_{v\in\mathcal{R}}logP(N(v)|f(v))$ (1)

where $N(v)$ is the neighbor nodes of node $v$ and the likelihood of the
neighborhood of a node is modeled as a softmax function given in Equation 2

$P(N(v)|f(v))=\prod_{m\in
N(v)}\frac{exp(f(m).f(v))}{\sum_{w\in\mathcal{R}exp(f(w).f(v))}}$ (2)

Thus with node2vec, we collect feature representation of each ragam from all
the concerts. Since each data instance is given as a set of ragams in a
concert, we aggregate the ragam vectors. Like many other session based
applications, the number of songs in every music concerts also vary. Our model
handles such variable length inputs by padding zero vector(s). Since order in
the song sequence given in the feature vector may have different priorities,
we use an attention model to get the concert vector ($V_{c_{1}}$) (given in
Equation 3)

$V_{c_{1}}=\sum_{i=1}^{k}\sigma(W_{1}s_{i-1}+W_{2}s_{i}+c)s_{i}$ (3)

where $W_{1},W_{2}\in\mathbb{R}^{d*d}$, $k$ is the number of songs in the
concert, and $s$ is the vector representation of the given ragam $i$.

###  IV-C Representing the hand picked features

Instead of optimizing the learning model (like node2vec), we give a simple
strategy to embed a concert (or a set of songs/raaga) into a vector space. In
other words, we represent a concert a distribution of concert features
(audava,shadava, and sampoorna for example). All features considered for this
study is given in Table I. For a given concert, we create one-hot encoded
vectors for each ragam based on these hand picked features. We aggregate the
features with element-wise average on all the vectors. Thus we represent
entire concert into a vector space($V_{c_{2}}$).

###  IV-D RagamAI model

With concert vectors from the attention model ($V_{c_{1}}$) and the concert
embedding model ($V_{c_{2}}$), we perform a linear transformation after
concatenating the vectors to obtain the aggregate vector using the Equation 4

$V_{g}=W_{3}(V_{c_{1}}\oplus V_{c_{2}})$ (4)

We apply softmax function to get the $|\mathcal{R}|-dimensional$ output vector
($y^{\prime}$), where each element in $y^{\prime}$ vector represent a score
for each ragam for recommendations and $|\mathcal{R}|$ is total number of
nodes in the raaga network $\mathcal{G}$.

We use cross-entropy, given in Equation 5, as a loss function for training the
proposed model.

$\mathcal{L}(y^{\prime})=-\sum_{i=1}^{|\mathcal{R}|}y_{i}log(y_{i}^{\prime})+(1-y_{i})log(1-y_{i})$
(5)

##  V Experiments and Results

###  V-A Dataset

(a) Overall distribution of songs across concerts

(b) Distribution of janya and melakartha ragams across concerts

(c) Distribution of melakartha ragams across concerts

(d) Distribution of ragam features across concerts

Figure 3: Distribution of various aspects(features) of south Indian classical
music performed by one musician.

We use the real world data consisting of concerts performed by multiple Indian
classical musicians. The data comprises of 1664 concerts performed between
1984 and 2019 with more than 800 unique songs and 280 unique ragams. Each
concert is a set of songs, where each song can map to its corresponding ragam.
A performer’s creativity, impromptu improvisations, and different song
durations during the concert, the number of songs in each concert is a dynamic
value. Also, since all these concerts are performed in several locations
around the world, the songs and the concert style (organization of songs and
ragams) are unique. In Figure 3, we give basic data distribution available in
our data. As implied in Figure 3a, most concerts have in the range of 9-16
songs. Figure 3b,3c, and 3d give distribution of other features in concerts
and we give proportion of songs matching these features in concerts. For
example, Figure 3b gives the ratio of number of songs in a concert is janya or
melakartha ragam, and Figure 3d gives the ratio of types of ragams in
concerts(audava,sampoorna,vakra, and other ragams). Due to space constraints,
we neglected the distribution of rarely occurring type of ragam(shadava ragam)
from this figure. Our data also consists of metadata of all 280 ragams. We use
concert structure(organization of ragams in each concert) to build the raaga
network and we populate metadata as node features in the network. We further
use this network of ragam patterns in experiments for the proposed model to
recommend ragams.

TABLE I: Features of ragams in the RagaNetwork. Each node in the raga network depicts a ragam and thus each node holds the below features. Although position of ragams in a concert is important, it can be identified by traversing before and after nodes matching the concert id. Feature |  Description
---|---
Janya/Melakartha |  Boolean feature on weather a ragam is janya
Mela number |  Melakartha number of a ragam
Mela category |  Melakartha category of a ragam
Type |  Ragam type - Audava or Shadava or Sampoorna ragam
Combo |  Some ragams fall between 2 different ragam types
Vakram |  Boolean feature on whether a ragam is Vakra ragam
Concert |  List of concert ids that the ragam is present

###  V-B Baseline Models

Below are some of the popular baseline models that we used to compare our
methods

  * •

Item-kNN [24]: Item-kNN is similar to extracting the nearby recommendations.
In this method, we find co-occurring items normalized by popularity of all
items in the list.

  * •

FPMC [25]: Factorized Personalized Markov Chain is one of the popular models
to recommend sequential item in a list. FPMC models the user preference on a
list and the preference transitions to recommend next set of items.

  * •

SWIWO [17]: Session-based Wide In Wide Out(SWIWO) is a deep learning based
recommender model that uses a both user and item based feature set to predict
the next item. To adapt to our problem, we eliminate the user feature section
and add a softmax layer to predict the score of items for a given sequence.

We compare proposed model against existing methods in producing
recommendations for south Indian classical music concerts. To make fair
comparison, we use offline evaluation methods for all algorithms, similar to
comparisons made in existing works. In particular, we use following evaluation
methods:

  * •

Precision@k: We evaluate on number of recommendations ($k$) to make (i.e.) We
evaluate from $k=1$ (Recommending just one item) to $k=15$ (Recommending 15
items).

  * •

Normalized Discounted Cumulative Gain(nDCG): It is a measure to evaluate the
rank of relevant recommendations, calculated using the formula given in
Equation 6

$nDCG=\frac{\sum_{i=1}^{n}\frac{1}{log_{2}(i)}}{DCG_{id}}$ (6)

, where $n$ is the number of items in the list to predicted correctly, $i$ is
the position of the relevant item in the recommendation, $DCG_{id}$ is an
ideal score for the given test sample. Similar to Precision@k, we give results
of $nDCG@k$, where $k$ is the number of items to recommend.

###  V-C Results

Figure 4: Precision@k evaluation for different models Figure 5: nDCG@k
evaluation for different models

For all our experiments we split the available concerts into training (80%)
and test (20%) data, where the training data is used as historical data and
the test data is considered as new concerts. Since the goal is to recommend
ragams for incomplete concert list and number of ragams in a concert is
inconsistent, we conduct our experiments with an assumption that the user
starts with one ragam (first ragam from the test concert), the system gives
prediction and evaluate obtain results, then it considers first two ragams,
and the cycle continues. For training the model, we split each concert into
multiple sequences in such a way that all instances have at least one user
input ragam and one prediction. All results that we report here is an average
score of all measures.

Figure 4 gives precision@k scores and the Figure 5 gives nDCG@k scores for all
baseline and proposed models, where $k$ is the required number of
recommendations. We use the average number of songs in overall concerts as the
upper bound(15 recommendations), while maintaining a simple lower bound(1
recommendation). From these plots, we can evidence that the baseline models
give poor performance in giving recommendations for south Indian classical
music concerts. Importantly, we can study from Figure 5 that primarily the
proposed ragamAI model give more relevant predictions with even shorter list
of recommendations. It is also evident from these plots that structural
organization of ragams in Indian classical music concerts play a vital role in
building recommendations.

Table II gives the ablation study with multiple combinations in the proposed
model. It is evident from these results that each model gives only average
performance compared to the proposed model when used alone for the
recommendation task. When both features combined as given in the proposed
model, we get improved performance.

TABLE II: Ablation study Model |  Precision@15 |  nDCG@15
---|---|---
Attention W/O node2vec |  0.1524 |  0.0189
Attention |  0.2141 |  0.2167
Concert Embedding |  0.2594 |  0.5398
ragamAI |  0.4875 |  0.7561

##  VI Conclusion and Discussion

Session-based item recommendation systems have gained many innovations by
incorporating deep learning models for variety of applications. Session-based
systems can give a precise set of recommendations based on a user’s current
session activity and their past preferences. In this work we proposed such a
model to recommend a set of ragam(music scale in Indian classical music) to
perform in a concert, given a set of preferred ragams. Particularly, we
proposed a network-based deep learning approach to utilize both hand picked
features and structural features of ragam sequences in concerts for the
recommendation task. With the experiments, we show that our model can
outperform state-of-the-art methods in recommending songs for Indian classical
music concerts.

Although this is the first step to merge recommender system with classical
music concerts, the model can be extended in several ways in the future: 1.
Even though a ragam is an integral part of a song, adding the composer and
composition level features to recommend songs benefit and save time for
musicians to arrange the concert, 2. Generalize the architecture to support
multiple forms of classical music like Western classical music, and 3. Such an
extensive model can be adopted in music streaming services like Pandora and
Spotify to match the listener preference in the classical music.

## References

  * [1] J. You, Y. Wang, A. Pal, P. Eksombatchai, C. Rosenburg, and J. Leskovec, “Hierarchical temporal convolutional networks for dynamic recommender systems,” ser. WWW ’19, 2019, pp. 2236–2246.
  * [2] M. Pichl, E. Zangerle, and G. Specht, “Understanding user-curated playlists on spotify: A machine learning approach,” _Int. J. Multimed. Data Eng. Manag._ , vol. 8, no. 4, pp. 44–59, Oct. 2017.
  * [3] F. Isinkaye, Y. Folajimi, and B. Ojokoh, “Recommendation systems: Principles, methods and evaluation,” _Egyptian Informatics Journal_ , vol. 16, no. 3, pp. 261–273, 2015.
  * [4] A. Gunawardana and G. Shani, “Evaluating recommender systems,” in _Recommender systems handbook_. Springer, 2015, pp. 265–308.
  * [5] G. Bonnin and D. Jannach, “Automated generation of music playlists: Survey and experiments,” _ACM Comput. Surv._ , vol. 47, no. 2, pp. 26:1–26:35, Nov. 2014.
  * [6] S. Chen, J. L. Moore, D. Turnbull, and T. Joachims, “Playlist prediction via metric embedding,” in _ACM SIGKDD_ , 2012, pp. 714–722.
  * [7] Z. Cheng, J. Shen, L. Nie, T.-S. Chua, and M. Kankanhalli, “Exploring user-specific information in music retrieval,” in _ACM SIGIR_ , ser. SIGIR ’17, 2017, pp. 655–664.
  * [8] T. Hornung, C.-N. Ziegler, S. Franz, M. Przyjaciel-Zablocki, A. Schätzle, and G. Lausen, “Evaluating hybrid music recommender systems,” in _Proceedings of the IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT)-Volume 01_ , 2013, pp. 57–64.
  * [9] B. Hidasi, A. Karatzoglou, L. Baltrunas, and D. Tikk, “Session-based recommendations with recurrent neural networks,” _CoRR_ , vol. abs/1511.06939, 2016.
  * [10] D. Jannach and M. Ludewig, “When recurrent neural networks meet the neighborhood for session-based recommendation,” in _RecSys_ , ser. RecSys ’17, 2017, pp. 306–310.
  * [11] B. Hidasi and A. Karatzoglou, “Recurrent neural networks with top-k gains for session-based recommendations,” in _Proceedings of the 27th ACM International Conference on Information and Knowledge Management_ , 2018, pp. 843–852.
  * [12] M. Schedl and D. Hauger, “Tailoring music recommendations to users by considering diversity, mainstreaminess, and novelty,” in _ACM SIGIR_ , 2015, pp. 947–950.
  * [13] P. Dighe, H. Karnick, and B. Raj, “Swara histogram based structural analysis and identification of indian classical ragas.” in _ISMIR_ , 2013, pp. 35–40.
  * [14] V. Kumar, H. Pandya, and C. Jawahar, “Identifying ragas in indian music,” in _22nd International Conference on Pattern Recognition_. IEEE, 2014, pp. 767–772.
  * [15] S. Gulati, J. Serra, V. Ishwar, S. Sentürk, and X. Serra, “Phrase-based rāga recognition using vector space modeling,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2016, pp. 66–70.
  * [16] J. Salamon, S. Gulati, and X. Serra, “A multipitch approach to tonic identification in indian classical music,” in _ISMIR_ , 2012.
  * [17] L. Hu, L. Cao, S. Wang, G. Xu, J. Cao, and Z. Gu, “Diversifying personalized recommendation with user-session context.” in _IJCAI_ , 2017, pp. 1858–1864.
  * [18] S. Wang, L. Hu, L. Cao, X. Huang, D. Lian, and W. Liu, “Attention-based transactional context embedding for next-item recommendation,” in _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018.
  * [19] S. Wu, Y. Tang, Y. Zhu, L. Wang, X. Xie, and T. Tan, “Session-based recommendation with graph neural networks,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 346–353.
  * [20] C. Aslay, L. V. Lakshmanan, W. Lu, and X. Xiao, “Influence maximization in online social networks,” in _ACM WSDM_ , 2018, pp. 775–776.
  * [21] I. Chien, C.-Y. Lin, and I.-H. Wang, “Community detection in hypergraphs: Optimal statistical limit and efficient algorithms,” in _International Conference on Artificial Intelligence and Statistics_ , 2018, pp. 871–879.
  * [22] R. Ying, R. He, K. Chen, P. Eksombatchai, W. L. Hamilton, and J. Leskovec, “Graph convolutional neural networks for web-scale recommender systems,” in _ACM SIGKDD_ , 2018, pp. 974–983.
  * [23] A. Grover and J. Leskovec, “node2vec: Scalable feature learning for networks,” in _ACM SIGKDD_ , 2016, pp. 855–864.
  * [24] G. Linden, B. Smith, and J. York, “Amazon. com recommendations: Item-to-item collaborative filtering,” _IEEE Internet computing_ , no. 1, pp. 76–80, 2003\.
  * [25] S. Rendle, C. Freudenthaler, and L. Schmidt-Thieme, “Factorizing personalized markov chains for next-basket recommendation,” in _ACM WWW_ , 2010, pp. 811–820.