# CloudAAE: Learning 6D Object Pose Regression with On-line Data Synthesis on
Point Clouds

Ge Gao1, Mikko Lauri1, Xiaolin Hu2, Jianwei Zhang1 and Simone Frintrop1 *This
work is funded by the German Science Foundation (DFG) in project Crossmodal
Learning, TRR 1691Department of Informatics, University of Hamburg, Germany
{gao,lauri,zhang}<EMAIL_ADDRESS>hamburg.de2State Key Laboratory of Intelligent Technology and Systems,
Department of Computer Science and Technology, Tsinghua University, China
<EMAIL_ADDRESS>
###### Abstract

It is often desired to train 6D pose estimation systems on synthetic data
because manual annotation is expensive. However, due to the large domain gap
between the synthetic and real images, synthesizing color images is expensive.
In contrast, this domain gap is considerably smaller and easier to fill for
depth information. In this work, we present a system that regresses 6D object
pose from depth information represented by point clouds, and a lightweight
data synthesis pipeline that creates synthetic point cloud segments for
training. We use an augmented autoencoder (AAE) for learning a latent code
that encodes 6D object pose information for pose regression. The data
synthesis pipeline only requires texture-less 3D object models and desired
viewpoints, and it is cheap in terms of both time and hardware storage. Our
data synthesis process is up to three orders of magnitude faster than commonly
applied approaches that render RGB image data. We show the effectiveness of
our system on the LineMOD, LineMOD Occlusion, and YCB Video datasets. The
implementation of our system is available at:
https://github.com/GeeeG/CloudAAE.

##  I Introduction

A 6 degrees of freedom (6D) pose is a 3D rotation and 3D transformation
between a local object coordinate and a camera or robot coordinate. Knowing
the 6D poses of objects is important for applications such as robotic grasping
and manipulation [1]. Estimating 6D object pose from images is challenging due
to occlusion, background clutter, and different illumination conditions.
According to the recent benchmark challenge for 6D object pose estimation
(BOP) [2], the performance of deep learning methods has improved compared to
the previous year. Deep learning-based methods benefit from large amounts of
high quality training data. Although there are tools and approaches for
annotating the 6D object pose on real images [3, 4], the annotation for the 6D
object pose is expensive. Moreover, the accuracy of manual labels can not
always be guaranteed [4].

Creating synthetic data eliminates the need for manual labeling and can
guarantee the accuracy of labels. The difference in appearance between real
and synthetic data is called the domain gap. The domain gap between real and
synthetic RGB images is often large. In the most recent BOP challenge, the
main reason for the performance boost of deep learning-based methods is having
additional photo-realistic synthetic images for training [2]. These images are
created using a physically-based renderer (PBR) and present a much smaller
domain gap, compared to naive approaches such as “render & paste” [2]. Despite
being effective, PBR is expensive in the terms of time and hardware storage.
This high cost makes it challenging to scale PBR up to a large number of
objects, which is often desired in robotic applications. Methods using only
depth information are more robust in the presence of the domain gap [5]. This
indicates that if synthesizing data with only depth information, the domain
gap is potentially much smaller and easier to fill. This opens up the
possibility to create a lightweight data synthesis pipeline using only depth
information.

In this work, we propose a point cloud based 6D pose estimator and a
lightweight data synthesis pipeline. We propose to use an implicit approach
during the pose inference stage. We achieve this by adapting the augmented
autoencoder (AAE) proposed by Sundermeyer et al. [6] for a point cloud based
system. By using an AAE, we are able to control which properties the latent
code encodes and which properties are ignored [6]. This is achieved by
applying augmentations to the input, and the encoding becomes invariant to
those augmentations. We focus on learning a latent code that encodes the 6D
pose information. The task of the AAE is to reconstruct a point cloud segment
in the desired 6D pose. Moreover, this segment is noise and occlusion free. In
this way, the latent code contains the necessary information for regressing
the object pose. Using the latent code as the input, we use two separate
networks for regressing 3D rotation and 3D translation, as suggested in [7,
8]. The on-line data synthesis pipeline requires a texture-less 3D object
model and the desired viewpoint as the input. The computational cost is low.

Our contributions are:

  * •

We present a new framework for regressing the 6D object pose from point cloud
segments. An point cloud based augmented autoencoder is used to learn a latent
code that encodes object pose information. This code is used for regressing
the 6D object pose.

  * •

We present a point cloud based lightweight data synthesis pipeline for
generating training data. Compared to existing RGB based data synthesis
systems, the cost of ours is lower in the sense of time and hardware storage.

We show the effectiveness of the combination of our data synthesis pipeline
and pose estimation system on three datasets. When using only synthetic
training data, our model achieves state-of-the-art performance among other
synthetic trained methods on the LineMOD dataset [9]. Our cheap synthetic
point cloud data can replace costly render based synthetic data for training
systems using depth for pose inference.

##  II Related Work

We review deep learning 6D pose estimators on point clouds, unsupervised
learning of 6D pose, as well as how training data is synthesized.

Deep learning 6D pose estimation on point clouds. Frustum PointNets [10] uses
point cloud segments for pose estimation. They use PointNet [11] for
processing point clouds. The rotation estimation is formulated as a
classification problem, and the result for one angle is reported. Densefusion
[12] uses PointNet to extract feature vectors from point cloud segments, and a
convolutional neural network (CNN) to extract features from the corresponding
color information. Those features are combined for regressing the 6D object
pose. PVN3D [13] adapts the feature extraction pipeline from DenseFusion, and
uses a 3D Keypoint detection and Hough voting scheme for 6D pose estimation.
CloudPose [8] uses a PointNet like structure to extract features from point
clouds, and directly regress to 3D rotation and 3D translation with two
separate networks. Our method is the most similar to CloudPose, as we regress
to 6D object poses from point clouds. However, rather than directly regressing
to 6D pose from point clouds, we use the latent code from an AAE as the input
to pose regressors.

Unsupervised learning of 6D pose. The augmented autoencoder is proposed in
[6]. It is a variant of the Denoising Autoencoder [14]. The authors use 2D
bounding boxes for translation estimation, and use the AAE for 3D rotation
estimation. For 3D rotation, rather than explicitly mapping from an input
image to a pose label, the AAE learns implicit codes of object orientations in
a latent space. A codebook of latent codes is created off-line, and they use a
nearest neighbor search to compare a test code within the codebook. This
approach is improved by [15] by adding edge priors. Our 6D pose estimation
pipeline adapts the AAE concept [6] for point clouds. There are three main
differences between our approach and [6]. First, our approach does not require
creating off-line codebooks. Second, we use the latent code from the AAE for
both 3D translation and 3D rotation estimation. Third, they use 2D color image
as input while ours uses point clouds.

Data synthesis methods for 6D pose estimation. Existing data synthesis methods
outputs synthetic color images. To create synthetic data, the viewpoint and
the appearance from that viewpoint must be determined. Object appearance is
obtained from either textured 3D object models or real world training
datasets. Viewpoints are selected using some simple or dataset-based
heuristics. One common approach is to define spheres around the textured 3D
object model with fixed radii, and sample viewpoints from a hemisphere that
covers the upper part of the object model [9, 16], or from the full sphere [6,
15]. The object model is rendered in the target viewpoint onto a plain or
random background. The advantage of this method is that it provides a good
coverage for 3D rotation. The limitation is that using fixed radii puts
constraints on the coverage of 3D translation.

The object appearance and viewpoint can also be directly obtained by
segmenting foreground objects from an existing training set [17, 18, 19]. More
data augmentation, such as background noise, can be applied to the rendered
image for reducing the domain gap [20]. However, due to the domain gap, this
strategy tends to be insufficient by itself for the 6D object pose estimation
task and causes performance drop [5]. Moreover, this approach also relies on
having the annotated real training data.

Some approaches synthesize data with photo-realistic appearance and physically
plausible object poses [20, 21, 22]. This approach requires a render to create
object appearances with realistic lighting and reflection, and physics
simulators to provide physically plausible poses. Rendering is computationally
expensive, and the data created off-line requires large amounts of hardware
storage. Compared to existing approaches, ours is computationally lightweight,
and does not need textured 3D object models or segmented foreground objects
from real-world training data. Another difference is that the existing
approaches mostly conduct off-line data generation while ours can be used on-
line.

Figure 1: Left, Data synthesis pipeline. The input for our on-line training
pipeline is a 3D object model $P^{O}$ with class information, and the desired
3D rotation $R$ and translation $\mathbf{t}$. $P^{O}$ is at object coordinate
$O$. With $R$ and $\mathbf{t}$, the object model is first transformed to the
camera coordinate $C$. The transformed model is denoted by $P^{C}$. Spherical
occluders $S$ (denoted in blue) are added between $C$ and $P^{C}$. Hidden
point removal is applied to $P^{C}\cup S$ to remove points in $P^{C}$ and are
not visible from $C$. Zero-mean Gaussian noise is added to each remaining
point to introduce variance. The final point set is $P^{C}_{occ}$.
$P^{C}_{occ}$ is normalized into $P^{N}_{occ}$, by subtracting the mean
$\mathbf{{\mu}_{p}}$ of $P^{C}_{occ}$. Right, CloudAAE. The input is
$P^{N}_{occ}$ and the corresponding class information. The expected $\Delta
P^{C}_{recon}$ is a noise and occlusion free segment. By adding
$\mathbf{{\mu}_{p}}$ to $\Delta P^{C}_{recon}$, we obtain $P^{C}_{recon}$ at
the desired 6D pose. Meanwhile, the latent code is used with two separate
3-layer multi-layer perceptrons (MLP1 and MLP2) for regressing 3D rotation
$\hat{R}$ and 3D translation $\hat{\mathbf{t}}$.

##  III Augmented Autoencoder based 6D pose estimation

We introduce our data synthesis pipeline and augmented autoencoder (AAE) based
6D pose estimation system, referred to as CloudAAE. CloudAAE is a multi-class
system and we use the same system to predict poses for objects from different
classes. Figure 1 shows an overview of the proposed system. Given a known
object represented by a set of points in the camera coordinate $C$, the aim of
a 6D pose estimation system is to find the translation $\mathbf{t}$ and
rotation $R$ that describes the transformation from the object coordinate
system $O$ to $C$. The data synthesis pipeline creates a point cloud segment
$P^{C}_{occ}$, and $P^{C}_{occ}$ is normalized by removing its coordinate mean
$\mathbf{{\mu}_{p}}$ before further steps. We denote the normalized
$P^{C}_{occ}$ as $P^{N}_{occ}$.

By using the data synthesis pipeline, we apply random noise and occlusion to
the input point cloud segment to add variance in the training samples. The AAE
reconstructs a noise and occlusion free object segment at the desired 6D pose.
Hence, the latent vector encodes the 6D object pose information, and it is
used for 6D pose regression.

###  III-A Data synthesis with point clouds

The inputs for the data synthesis pipeline are a 3D object model and a 6D
pose. We use 3D object models represented by point clouds. A 3D rotation $R$
and a 3D translation $\mathbf{t}$ are drawn from a desired pose distribution.
The desired pose distribution can be the poses in a training set [17, 19]. It
can also be a pool of poses covering the same distribution as the poses in a
training set [23, 7, 24]. In this work, if the train dataset contains a large
amount of poses, we use them as the 6D pose for data synthesis. Otherwise, we
draw poses from the distribution of train poses.

Given an object model represented by the set of points
$P^{O}=\\{\mathbf{x}_{i}^{O}\in\mathbb{R}^{3}\mid i=1,2,\ldots,n\\}$, the
model is transformed from its object coordinate system $O$ to the camera
coordinate system $C$ by

$\mathbf{x}_{i}^{C}=R\mathbf{x}_{i}^{O}+\mathbf{t}.$ (1)

$P^{C}=\left\\{\mathbf{x}_{i}^{C}\in\mathbb{R}^{3}\middle|i=1,2,\ldots,n\right\\}$
is the transformed model, where $\mathbf{x}_{i}^{C}$ is the $i$th point. To
simulate occlusions, we randomly generate a set of points $S$ as the spherical
occluders (denoted in blue in Figure 1) between the camera origin $C$ and $P$.
We remove points in $P^{C}\cup S$ that are not visible from $C$ by applying
hidden point removal (HPR) [25]. From the set of visible points, we remove the
points belonging to $S$. To add variance to the training sample, we add zero-
mean Gaussian noise with standard deviation $\sigma$ to each remaining point.
In all of our experiments, we use $\sigma=1.3mm$. The final resulting object
segment is $P^{C}_{occ}=\left\\{\mathbf{x}_{i}\in\mathbb{R}^{3}\mid
i=1,\ldots,m\right\\}$.

###  III-B CloudAAE: network structure

CloudAAE contains an augmented autoencoder, and the 6D pose regressors. We
adapted the idea of AAE proposed in [6] for point clouds, and use an adapted
version of the dynamic graph CNN (DGCNN) [26] as the encoder. We additionally
add two networks for regressing the 6D poses.

Augmented autoencoder (AAE): Figure 2 illustrates the architecture of our
point cloud based AAE. The AAE consists of an encoder and a decoder. The 3D
coordinate of each point in $P^{N}_{occ}$ is concatenated with one-hot class
information. This is used as the input to the encoder. The encoder computes a
latent vector, and the desired output from the decoder is a noise and
occlusion free segment in the 6D pose defined by $R$ and $\mathbf{t}$. The
input is of dimension $n\times(3+c)$, in which $3$ represents 3D coordinates,
and $c$ is the total number of classes.

Our encoder is an adapted version of DGCNN, which is a deep network processing
unordered point sets. Assume $Q=\left\\{\mathbf{q}_{i}\in\mathbb{R}^{p}\mid
i=1,\ldots,m\right\\}$ is a point set. $\mathbb{R}^{p}$ can be a 3D space or
an arbitrary feature space. For each point $\mathbf{q}_{i}$, a $k$-nearest
neighbor graph is calculated. In all our experiments, we use $k=10$. The graph
contains directed edges $(i,j_{i1}),\dots,(i,j_{ik})$, in which
$\mathbf{q}_{j_{i1}},\dots,\mathbf{q}_{j_{ik}}$ are the $k$ closest points to
$\mathbf{q}_{i}$. For the edge $\mathbf{e}_{ij}$, an edge feature
$\begin{bmatrix}\mathbf{q}_{i},&(\mathbf{q}_{j}-\mathbf{q}_{i})\end{bmatrix}^{T}$
is calculated.

The edge features are processed by an edge convolution operation (EdgeConv),
which contains an edge function and an aggregation operation [26]. For the
edge function, we use an MLP layer with the shared weights for each edge
feature. We use average pooling as the aggregation operation. This EdgeConv is
repeated, calculating the nearest neighbor graph for the feature vectors of
the first shared MLP layer, as well as for the subsequent layers. Finally, the
edge features from each EdgeConv are concatenated and processed with an MLP
layer. This concatenation is illustrated with skip connections in Figure 2. A
1024-dimensional feature vector is learned for each point. These features are
average-pooled to obtain a global representation of the input point cloud
segment. We use four EdgeConv layers on edge features and one EdgeConv layer
on the concatenated edge feature. The encoder outputs a latent code with the
dimension of 1024. The decoder contains 3 fully connected layers with
dimensions of 1024, 1024 and $n\times 3$.

6D pose regressors. Since the decoder is able to reconstruct the segment in
the same 6D pose from the latent code, the latent code contains the object
pose information. Hence, the latent code is used as the input to two networks
for regressing 3D rotation and 3D translation, respectively. Each network
contains three MLP layers with dimensions 512, 256, and $3$, respectively.

Figure 2: Point cloud based Augmented Autoencoder. The numbers of neurons in
EC-MLP and MLP layers are indicated in parentheses. EC is short for EdgeConv.
Dimensions of intermediate features are indicated without parentheses. Skip
connections denote the concatenation of edge features. For the encoder, a
1024-dimensional feature vector for each point with its local neighbors is
learned with shared weights. An average pooling layer aggregates the
individual features into a 1024-dimensional latent code. Finally, three fully-
connected layers output a noise and occlusion free point cloud segment.

###  III-C CloudAAE: loss function

We normalize $P^{C}_{occ}$ by removing its mean $\mathbf{{\mu}_{p}}$ before
inputting to the AAE (Figure 1). The output of AAE is the residual 3D
coordinates of the reconstructed point cloud segment. The output of
translation prediction is the residual of translation. The full 3D coordinates
and 3D translation are obtained by adding back $\mathbf{{\mu}_{p}}$.

Augmented autoencoder (AAE). Assuming the AAE outputs residual 3D coordinates
$\Delta P^{C}_{recon}$, the full 3D coordinate is $P^{C}_{recon}=\Delta
P^{C}_{recon}+\mathbf{{\mu}_{p}}.$ The learning target $P^{C}_{vis}$ is
obtained by applying HPR to $P^{C}$ without introducing any occluders, as
illustrated in Figure 3LABEL:sub@fig:p_vis. Chamfer distance is used to
measure the difference between $P^{C}_{recon}$ and $P^{C}_{vis}$,

$\displaystyle
l_{CD}(P^{C}_{recon},P^{C}_{vis})=\frac{1}{m}\sum_{\mathbf{x}\in
P^{C}_{recon}}\min_{\mathbf{y}\in P^{C}_{vis}}\|\mathbf{x}-\mathbf{y}\|_{2}$
$\displaystyle+\frac{1}{n}\sum_{\mathbf{y}\in P^{C}_{vis}}\min_{\mathbf{x}\in
P^{C}_{recon}}\|\mathbf{y}-\mathbf{x}\|_{2}.$ (2)

The number of points in $P^{C}_{vis}$ and $P^{C}_{recon}$ are denoted by $m$
and $n$, respectively. The desired $P^{C}_{recon}$ is a denoised, unoccluded
object segment. Figure 3LABEL:sub@fig:AE_result shows examples of
$P^{C}_{recon}$ output by a trained AAE.

6D pose regressors. For rotation regression, the axis-angle representation is
used as the regression target. An axis-angle is a vector
$\mathbf{r}\in\mathbb{R}^{3}$ that represents a rotation of
$\theta=\left\lVert\mathbf{r}\right\rVert_{2}$ radians around the unit vector
$\frac{\mathbf{r}}{\left\lVert\mathbf{r}\right\rVert_{2}}$. Geodesic distance
is used as the loss function for measuring the distance between prediction
$\hat{\mathbf{r}}$ and ground truth $\mathbf{r}$ [27]. With $\hat{R}$ and $R$
being the corresponding rotation matrices for $\hat{\mathbf{r}}$ and
$\mathbf{r}$ respectively, the rotation loss function
$l_{r}(\hat{\mathbf{r}},\mathbf{r})$ is defined as

$l_{r}(\hat{\mathbf{r}},\mathbf{r})=\arccos\left(\frac{\mathrm{trace}(\hat{R}R^{T})-1}{2}\right).$
(3)

The regression target for translation is the residual of translation. Given
$\widehat{\Delta\mathbf{t}}$ as the translation residual, full translation
prediction is
$\hat{\mathbf{t}}=\widehat{\Delta\mathbf{t}}+\mathbf{{\mu}_{p}}$. With
$\mathbf{t}$ being the ground truth translation, the translation loss function
$l_{t}(\hat{\mathbf{t}},\mathbf{t})$ is measured with $L2$-norm
$l_{t}(\hat{\mathbf{t}},\mathbf{t})=\left\lVert\mathbf{t}-\hat{\mathbf{t}}\right\rVert_{2}.$

Total loss function. The total loss $l_{total}$ used for training the AAE and
pose regressors is the combination of the reconstruction and the pose loss:

$l_{total}=\alpha l_{CD}+\beta l_{t}+l_{r},$ (4)

where $\alpha$ and $\beta$ are scaling factors.

(a)

(b)

Figure 3: Illustration of the learning target and the output of AAE. (a) The
learning target $P^{C}_{vis}$. (b) Examples of AAE output for objects in the
YCB video dataset during testing. The input to the AAE is denoted in red, and
the reconstructed noise and occlusion free point cloud segment is denoted in
blue.

###  III-D CloudAAE: testing phase

In the testing phase, the trained CloudAAE is used for estimating 6D object
poses. If the input is an RGBD image, semantic segmentation is used to obtain
the object segment and the corresponding class information. After segmenting
the depth image, the point cloud is obtained using the depth value and the
camera intrinsic parameters. The point cloud segment is normalized and
concatenated with one-hot class information. This is used as the CloudAAE
input. As semantic segmentation is a well-studied topic [28], we assume the
object segment and class information is provided by an off-the-shelf method.
After obtaining the pose estimates, we further refine the poses with Iterative
Closest Point (ICP).

TABLE I: Percentage of correctly estimated poses on LineMOD and LineMOD
Occlusion. E.box and glue are reported with ADD-S, others with ADD. Among the
methods trained only on synthetic data, our method has the best results on
LineMOD.

| Synthetic | Synthetic+Real
---|---|---
Modality | RGB | RGBD | D | RGBD | D
Method |  | EEGP-AAE
---
+ICP [15]
| SSD-6D
---
+ICP [29]
| CloudPose
---
+ICP [8]
Ours |  | Ours
---
+ICP
PVN3D [13] | Ours |  | Ours
---
+ICP
ape | 87.4 | - | 58.3 | 74.5 | 92.5 | 97.3 | 80.2 | 92.5
bvise | 96.1 | - | 65.6 | 86.6 | 90.8 | 99.7 | 85.7 | 91.8
cam | 91 | - | 43 | 65.6 | 85.7 | 99.6 | 61 | 88.9
can | 89.5 | - | 84.7 | 90.2 | 95.1 | 99.5 | 93.1 | 96.4
cat | 96.6 | - | 84.6 | 90.7 | 96.8 | 99.8 | 94.4 | 97.5
driller | 78 | - | 83.3 | 97.3 | 98.7 | 99.3 | 98.2 | 99
duck | 69.4 | - | 43.2 | 50 | 84.4 | 98.2 | 62.6 | 92.7
e.box | 100 | - | 99.5 | 99.7 | 99.2 | 99.8 | 99.8 | 99.8
glue | 99 | - | 98.8 | 93.5 | 98.7 | 100.0 | 94.1 | 99
holep | 66.5 | - | 72.1 | 57.9 | 85.3 | 99.9 | 84.4 | 93.7
iron | 98.8 | - | 70.3 | 85 | 91.4 | 99.7 | 89.5 | 95.9
lamp | 94.4 | - | 93.2 | 82.1 | 86.5 | 99.8 | 91.6 | 96.6
phone | 93.5 | - | 81 | 94.4 | 97.4 | 99.5 | 93.5 | 97.4
avg | 89.2 | 90.9 | 75.2 | 82.1 | 92.5 | 99.4 | 86.8 | 95.5

(a) LineMOD.

Train | Mod. | Method | avg
---|---|---|---
Syn. | D |  | CloudPose
---
+ICP [8]
44.2
Ours | 57.1
| Ours
---
+ICP
63.2
Syn. + Real | RGB | PoseCNN [4] | 24.9
PVNet [30] | 47.25
RGBD |  | PoseCNN
---
+ICP [4]
78.0
D |  | PointVotNet
---
+ICP [31]
52.6
Ours | 58.9
| Ours
---
+ICP
66.1

(b) LineMOD Occlusion.

##  IV Experiments

We compare our results to the state-of-the-art methods on public datasets. We
also investigate the impact of the amount of synthetic data used for training,
as well as different sizes for the latent code in AAE.

###  IV-A Datasets and experiment setup

We consider three datasets, the LineMOD dataset (LM) [9], LineMOD Occlusion
(LMO) [32] and YCB video dataset (YCBV) [4]. Depending on the applicability of
the methods, for each dataset, we compare to a subset of state-of-the-art
methods SSD-6D [29], EEPG-AAE [15], CloudPose [8], PVNet [30], PoseCNN [4],
DenseFusion [12], PVN3D [13] and PointVoteNet [31].

LM [9] contains 13 objects from daily scenarios in 13 videos. Although the
target objects are not occluded, this dataset is very challenging for depth
based methods due to noise in the depth data [15]. LMO [32] contains 8 objects
from the LM dataset. The objects belong to one video sequence, and have a high
level of occlusion. YCBV [4] contains 21 objects from YCB object set [33] in
92 videos. It contains many objects with different degrees of rotational
symmetry, as well as occlusions. It provides both real and synthetic training
data. Compared to the other two datasets, the quality of its depth information
is good.

We use the 3D models in point clouds provided by the datasets for the on-line
data synthesis. Tests are conducted with real test images from the official
test split. LM provides approximately $200$ training poses for each class,
which is a small amount for generating synthetic data. To obtain more 6D pose
for training, we first calculate a kernel density estimate on the training set
poses. Then we draw $100k$ 6D poses from the distribution for each object
class for data synthesis. For YCBV, we use the $80k$ 6D poses from its
synthetic training set as the 6D poses for data synthesis.

For training, we use Adam optimizer with learning rate 0.0008. The batch size
is 128. The number of points of the input point cloud segments is $n=256$.
Batch normalization is applied to all layers and no dropout is used. For 6D
pose regression, we use $\alpha=1000$ and $\beta=10$, which is given by the
ratio between the expected errors of the reconstruction, translation and
rotation at the end of the training [8]. Standard derivation for zero-mean
Gaussian noise is $1.3mm$. For ICP refinement, we use the simple Point-to-
Point registration provided by Open3D [34] and refine for 10 iterations. The
initial search radius is 0.01 meter and it is reduced by $10\%$ after each
iteration. When training with synthetic data, both CloudPose [8] and ours use
the proposed data synthesis pipeline. For testing, our method, CloudPose and
DenseFusion [12] require an off-the-shelf semantic segmentation method. For LM
and LMO, both our method and CloudPose use the test object segmentation
provided by the corresponding dataset. For YCBV, ours, CloudPose and
DenseFusion use the object segmentation provided by PoseCNN [4]. When training
with additional real data, we first generate the synthetic segments
($P^{O}_{occ}$ in Fig. 1) off-line and use them with the real data for
training.

###  IV-B Evaluation Metrics

The average distance (ADD) of model points and the average distance for a
rotationally symmetric object (ADD-S) proposed in [9] are used as evaluation
metrics. With a set $M$ with $m$ points representing a 3D model, the ADD is
defined as:

$\displaystyle\mathrm{ADD}=\frac{1}{m}\displaystyle\sum_{\mathbf{x}\in\mathcal{M}}\left\lVert(R\mathbf{x}+\mathbf{t})-(\hat{R}\mathbf{x}+\hat{\mathbf{t}})\right\rVert_{2}.$
(5)

in which $R$ and $\mathbf{t}$ are the ground truth rotation and translation,
and $\hat{R}$ and $\mathbf{\hat{t}}$ are the estimated rotation and
translation. ADD-S is computed using closest point distance. It is a distance
measure that considers possible pose ambiguities caused by object rotational
symmetry:

$\displaystyle\mathrm{ADD{\text{-}}S}=\frac{1}{m}\\!\\!\displaystyle\sum_{\mathbf{x}_{1}\in\mathcal{M}}\min_{\mathbf{x}_{2}\in\mathcal{M}}\left\lVert(R\mathbf{x}_{1}+\mathbf{t})-(\hat{R}\mathbf{x}_{2}+\hat{\mathbf{t}})\right\rVert_{2}.$
(6)

For YCBV, a 6D pose estimate is considered to be correct if ADD and ADD-S are
smaller than a given threshold $0.1$m. The area under error threshold-accuracy
curve (AUC) for ADD and ADD-S is calculated with maximum threshold $0.1$m. For
LM and LMO Occlusion, a pose is considered correct if the error is less than
10% of the maximum diameter of the target object [9].

###  IV-C Comparison of prediction accuracy

We first discuss our system performance when using only synthetic data for
training on three datasets. We then briefly discuss the our system performance
when using additional real training data. We divide the result tables into two
main parts, separating methods depending on whether real data is used
additionally during the training stage. Furthermore, we also denote the data
modality that is used during the pose inference stage.

#### Synthetic data on LM

Table ILABEL:sub@tab:lm_add shows the evaluation results. We achieve state-of-
the-art performance using only depth for pose inference. This shows the
effectiveness of the combination of our data synthesis pipeline and CloudAAE.
Both ours and CloudPose use depth for pose inference, but our method
generalizes better to real test data. This shows learning a latent code that
encodes pose information can improve system robustness. Using depth for pose
inference, our method outperforms RGBD based method SSD-6D.

#### Synthetic data on LMO

Table ILABEL:sub@tab:lmo_adds shows the comparison results. Ours outperforms
CloudPose by a large margin. Among the methods without ICP, ours has better
performance compared to PVNet and PointVoteNet, which use real data for
training. This shows our system can also handle occlusion.

#### Synthetic data on YCBV

Table II shows the results. Our method is slightly better compared to
CloudPose. Moreover, although trained on synthetic data only, our method has
comparable performance to other methods trained on both real and synthetic
data. To the best of our knowledge, we are the first to report results using
only synthetic training data on this dataset.

#### Synthetic and real data

As shown in Table I and II, using additional real training data boosts the
performance of our system on all three datasets. On LM, ours with ICP has
close performance to RGBD based PVN3D. On LMO, among the methods without ICP,
ours has the best performance. With ICP, PoseCNN outperforms our method. One
possible reason is that PoseCNN uses a sophisticated ICP while we use a
standard point to point ICP. Moreover, table ILABEL:sub@tab:lmo_adds shows LMO
is very challenging for methods that rely on single modality for pose
inference, and combining both color and depth is very beneficial. Another
possible factor that is limiting our performance is that, there might be a
domain gap between our occlusion simulation and the real data. On YCBV, we
report the results when using real training data with either synthetic data
from YCBV (ycbv_syn) or from our pipeline (our_syn). Using real and the two
kinds of synthetic data achieves similar performances, and ours is comparable
with the RGBD methods [12, 13].

TABLE II: AUC of ADD-S metric on YCB-Video dataset [4].

| Synthetic | Synthetic+Real
---|---|---
Modality | D | RGBD | D
Method |  | CloudPose
---
+ICP [8]
| Ours
---
+ICP
| DenseFusion
---
Iterative [12]
| PVN3D
---
+ICP [13]
| Ours
---
+ICP
(ycbv_syn)
| Ours
---
+ICP
(our_syn)
avg | 93 | 93.5 | 93.2 | 96.1 | 94.0 | 93.6

###  IV-D Ablation study

We first study how the system performance is impacted by the amount of
synthetic training data. For each object class in LM, we generate either
$10k$, $100k$, $1000k$ training poses per class. For each pose data amount, we
train multiple networks with early stopping. If the training error of the
current epoch failed to improve more than $10\%$ of the previous lowest
training error, the training process is terminated. We conducted five trials
for $10k$, and three trials for $100k$ and $1000k$. We generate a set of
training data for each trial separately. The trained networks are evaluated on
the test set of LM. Figure 4LABEL:sub@fig:amount_vs_accuracy shows the test
accuracy (without ICP). $100k$ samples per class are sufficient for the
network to perform well on the test set, and using $1000k$ per class does not
provide further performance gain.

(a)

(b)

Figure 4: Illustrations of ablation studies. (a) Amount of synthetic training
data and performance accuracy (w/o ICP). (b) T-SNE visualization of latent
code for the LineMOD dataset.

We also investigate how the performance is impacted by latent code size. We
train networks with different sizes for the latent code. All the networks are
trained with the same synthetic data and trained until convergence. The
results without ICP are shown in Table III. The generalization ability of a
network is the best when the latent code is of size 1024 and 2048. We pick
1024 for our system. We show a T-SNE [35] visualization of the latent code for
the LineMOD dataset in Figure 4LABEL:sub@fig:tsne. The latent codes are well
clustered for each class.

TABLE III: Results (w/o ICP) with different latent code sizes on LM. Dimension | 512 | 1024 | 2048 | 4096
---|---|---|---|---
avg | 82.0 | 82.1 | 82.1 | 80.9

###  IV-E Runtime and hardware storage

We measure the time performance on a Nvidia Titan X GPU. Our system is
implemented with Tensorflow. For the on-line data synthesis, it takes under 30
milliseconds to generate 128 object segments. The rendering based approach
[36] takes 1-4 seconds to generate an image with 10-20 objects. Pose
estimation by a forward pass through our network takes $0.07$ seconds for a
single object. The 10 iterations of ICP refinement require an additional
$0.02$ seconds. In the LM experiments, the 3D object models and $13\times
100k$ training poses take $128\text{\,}\mathrm{MB}$ of storage. If to generate
PBR images [2] with similar amount of objects (e.g. 10 objects per image),
$13\times 10k$ images would take $63\text{\,}\mathrm{GB}$.

##  V Conclusion

We propose a point cloud based lightweight data synthesis pipeline and an
augmented autoencoder based system for accurate and fast 6D pose estimation of
known objects represented by point clouds. The data synthesis pipeline is low
cost in terms of both time and hardware storage. Using the synthetic data
created by our data synthesis pipeline, our pose estimation system achieves
state-of-the-art performance among other synthetic trained methods on a public
benchmark. Moreover, our cheap synthetic point cloud data can replace
expensive render based synthetic data for training systems using depth for
pose inference. Our lightweight data synthesis pipeline enables more agile
deployment of object pose estimation systems in robotic applications.

## References

  * [1] A. Zeng, K. Yu, S. Song, D. Suo, E. Walker, A. Rodriguez, and J. Xiao, “Multi-view self-supervised deep learning for 6D pose estimation in the amazon picking challenge,” in _ICRA_ , 2017.
  * [2] T. Hodaň, M. Sundermeyer, B. Drost, Y. Labbe, E. Brachmann, F. Michel, C. Rother, J. Matas, and C. Rother, “BOP challenge 2020 on 6D object localization,” in _ECCV Workshop on Recovering 6D Object Pose_ , 2020.
  * [3] P. Marion, P. R. Florence, L. Manuelli, and R. Tedrake, “LabelFusion: A pipeline for generating ground truth labels for real RGBD data of cluttered scenes,” in _ICRA_ , 2018.
  * [4] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox, “PoseCNN: A convolutional neural network for 6D object pose estimation in cluttered scenes,” in _RSS_ , 2018.
  * [5] T. Hodaň, F. Michel, E. Brachmann, W. Kehl, A. Glent Buch, D. Kraft, B. Drost, J. Vidal, S. Ihrke, X. Zabulis, C. Sahin, F. Manhardt, F. Tombari, T.-K. Kim, J. Matas, and C. Rother, “BOP: Benchmark for 6D object pose estimation,” in _ECCV_ , 2018.
  * [6] M. Sundermeyer, Z.-C. Marton, M. Durner, M. Brucker, and R. Triebel, “Implicit 3D orientation learning for 6D object detection from RGB images,” in _ECCV_ , 2018.
  * [7] J. Wu, L. Ma, and X. Hu, “Delving deeper into convolutional neural networks for camera relocalization,” in _ICRA_ , 2017.
  * [8] G. Gao, M. Lauri, Y. Wang, X. Hu, J. Zhang, and S. Frintrop, “6D object pose regression via supervised learning on point clouds,” in _ICRA_ , 2020.
  * [9] S. Hinterstoisser, V. Lepetit, S. Ilic, S. Holzer, G. Bradski, K. Konolige, and N. Navab, “Model based training, detection and pose estimation of texture-less 3D objects in heavily cluttered scenes,” in _ACCV_ , 2012\.
  * [10] C. R. Qi, W. Liu, C. Wu, H. Su, and L. J. Guibas, “Frustum PointNets for 3D object detection from RGB-D data,” in _CVPR_ , 2018.
  * [11] C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “PointNet: Deep learning on point sets for 3D classification and segmentation,” in _CVPR_ , 2017.
  * [12] C. Wang, D. Xu, Y. Zhu, R. Martín-Martín, C. Lu, L. Fei-Fei, and S. Savarese, “DenseFusion: 6D object pose estimation by iterative dense fusion,” in _CVPR_ , 2019.
  * [13] Y. He, W. Sun, H. Huang, J. Liu, H. Fan, and J. Sun, “PVN3D: A deep point-wise 3D keypoints voting network for 6DoF pose estimation,” in _CVPR_ , 2020.
  * [14] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, P.-A. Manzagol, and L. Bottou, “Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.” _Journal of machine learning research_ , vol. 11, no. 12, 2010.
  * [15] Y. Wen, H. Pan, L. Yang, and W. Wang, “Edge enhanced implicit orientation learning with geometric prior for 6D pose estimation,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 3, pp. 4931–4938, 2020\.
  * [16] S. Zakharov, I. Shugurov, and S. Ilic, “DPOD: 6D pose object detector and refiner,” in _ICCV_ , 2019.
  * [17] B. Tekin, S. N. Sinha, and P. Fua, “Real-time seamless single shot 6D object pose prediction,” in _CVPR_ , 2018.
  * [18] M. Rad and V. Lepetit, “BB8: A scalable, accurate, robust to partial occlusion method for predicting the 3D poses of challenging objects without using depth,” in _ICCV_ , 2017.
  * [19] K. Park, T. Patten, and M. Vincze, “Pix2Pose: Pixel-wise coordinate regression of objects for 6D pose estimation,” in _ICCV_ , 2019.
  * [20] J. Tremblay, T. To, B. Sundaralingam, Y. Xiang, D. Fox, and S. Birchfield, “Deep object pose estimation for semantic robotic grasping of household objects,” in _CoRL_ , 2018.
  * [21] T. Hodaň, V. Vineet, R. Gal, E. Shalev, J. Hanzelka, T. Connell, P. Urbina, S. Sinha, and B. Guenter, “Photorealistic image synthesis for object instance detection,” in _ICIP_ , 2019.
  * [22] C. Mitash, K. E. Bekris, and A. Boularias, “A self-supervised learning system for object detection using physics simulation and multi-view pose estimation,” in _IROS_ , 2017.
  * [23] H. Su, C. R. Qi, Y. Li, and L. J. Guibas, “Render for CNN: Viewpoint estimation in images using CNNs trained with rendered 3D model views,” in _ICCV_ , 2015.
  * [24] F. Manhardt, W. Kehl, and A. Gaidon, “ROI-10D: monocular lifting of 2D detection to 6D pose and metric shape,” in _CVPR_ , 2019.
  * [25] S. Katz, A. Tal, and R. Basri, “Direct visibility of point sets,” in _SIGGRAPH_ , 2007.
  * [26] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon, “Dynamic graph CNN for learning on point clouds,” _ACM Transactions on Graphics (TOG)_ , 2019.
  * [27] G. Gao, M. Lauri, J. Zhang, and S. Frintrop, “Occlusion resistant object rotation regression from point cloud segments,” in _ECCV 4th International Workshop on Recovering 6D Object Pose_ , 2018.
  * [28] V. Badrinarayanan, A. Kendall, and R. Cipolla, “SegNet: A deep convolutional encoder-decoder architecture for image segmentation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 39, no. 12, pp. 2481–2495, 2017.
  * [29] W. Kehl, F. Manhardt, F. Tombari, S. Ilic, and N. Navab, “SSD-6D: Making RGB-based 3D detection and 6D pose estimation great again,” in _ICCV_ , 2017.
  * [30] S. Peng, Y. Liu, Q. Huang, X. Zhou, and H. Bao, “PVNet: Pixel-wise voting network for 6dof pose estimation,” in _CVPR_ , 2019.
  * [31] F. Hagelskjær and A. Buch, “PointVoteNet: Accurate object detection and 6 DoF pose estimation in point clouds,” in _ICIP_ , 2020.
  * [32] E. Brachmann, A. Krull, F. Michel, J. S. S. Gumhold, and C. Rother, “Learning 6D object pose estimation using 3D object coordinates,” in _ECCV_ , 2014\.
  * [33] B. Calli, A. Walsman, A. Singh, S. Srinivasa, and P. Abbeel, “Benchmarking in manipulation research using the Yale-CMU-Berkeley object and model set,” _Robotics & Automation Magazine, IEEE_, vol. 22, no. 3, pp. 36–52, 2015\.
  * [34] Q.-Y. Zhou, J. Park, and V. Koltun, “Open3D: A modern library for 3D data processing,” _arXiv:1801.09847_ , 2018.
  * [35] L. van der Maaten and G. Hinton, “Visualizing data using t-sne,” _Journal of Machine Learning Research_ , vol. 9, no. 86, pp. 2579–2605, 2008\.
  * [36] M. Denninger, M. Sundermeyer, D. Winkelbauer, Y. Zidan, D. Olefir, M. Elbadrawy, A. Lodhi, and H. Katam, “BlenderProc,” _arXiv preprint arXiv:1911.01911_ , 2019.


Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0)

The 17th International Workshop on Ontology Matching, The 21st International
Semantic Web Conference (ISWC) 2022, 23 October 2022, Hangzhou, China

[orcid=<EMAIL_ADDRESS>]

# GraphMatcher: A Graph Representation Learning Approach for Ontology Matching

Sefika Efeoglu  Free University of Berlin, Department of Computer Science,
Takustraße 9, 14195 Berlin, Germany

(2022)

###### Abstract

Ontology matching is defined as finding a relationship or correspondence
between two or more entities in two or more ontologies. To solve the
interoperability problem of the domain ontologies, semantically similar
entities in these ontologies must be found and aligned before merging them.
GraphMatcher, developed in this study, is an ontology matching system using a
graph attention approach to compute higher-level representation of a class
together with its surrounding terms. The GraphMatcher has obtained remarkable
results in in the Ontology Alignment Evaluation Initiative (OAEI) 2022
conference track. Its codes are available at
https://github.com/sefeoglu/gat_ontology_matching.

###### keywords:

graph attentiongraph representationontology matching

##  1 Presentation of the system

GraphMatcher is a new ontology matching system based on graph representation
learning using a graph attention [1] together with a new neighbourhood
aggregation approach. The graph representation learning approach has leveraged
the graph attention and introduces a new neighbourhood aggregation algorithm
that increases the contextual information of the centre class and property.

###  1.1 Proposal and general statement

Ontology matching is to find a relationship or correspondence between two or
more entities in two or more independent ontologies. The alignments of two
ontologies are classified as two different cases: (i) simple alignment and
(ii) complex alignment [2]. The simple alignment is defined as the mapping of
the class names according to word-based similarity, while complex alignment
considers the meaning of two classes to decide whether they are similar [2].
To understand the meaning of a class (sequence), the contextual information of
the class or property is needed. In this case, we must decide which neighbour
contributes to their contextual information.

Many logic- and algorithm-based ontology matching tools, such as LogMAP [3]
and AML [4], solve this interoperability problem of domain ontologies by using
algorithms and logic-based approaches. In addition to these approaches,
DeepAlignment [5], VeeAlign [6], and Convolutional Networks of Bento et al.
(2020) [7] apply machine learning (ML) for matching. Nevertheless, according
to OAEI’s 111Ontology Alignment Evaluation Initiative (OAEI):
http://oaei.ontologymatching.org/ conference track results, these ML
approaches cannot achieve a better performance than traditional tools such as
AML and LogMAP. The weakness of these ML approaches might be due to the lack
of contextual information about the property and class. Another limitation is
in how to represent the ontology’s data as a convolutional graph - such as an
image in which each pixel in the image data has the same number as its
neighbouring pixels - whereas each class in ontology has a different number of
its neighboring terms like an arbitrary graph [1]. The most appropriate way of
representing the data in the ontology is the arbitrary graph.

Since the ontology represents the data with the arbitrary graph, we aim to
develop a graph representation learning model based on a graph attention
mechanism [1] using Siamese networks [7, 8, 6] to find the semantically
similar concepts within the ontologies. The graph attention mechanism computes
the higher-level representation of a concept and its surrounding concepts
(features). The model then finds similarity scores between the concept pairs
among the aligned ontology pairs and determines the concept alignments.

###  1.2 Specific techniques used

GraphMatcher utilises a graph representation learning approach that uses the
graph attention [1]. Its network consists of five layers. The main
contribution is the adaptation of the graph attention to the Siamese network
in the third layer.

Figure 1: The proposed network 222The orders of the layers in the network is
similar to the VeeAlign [6], since this work’s extended version has also
increased the performance of the VeeAlign with our neighborhood aggregation
algorithm in “ Efeoglu, S. (2021). A Deep Learning Approach for Domain-
Specific Ontology Construction. University of Potsdam. [Master’s Thesis]”.is
an application of heterogeneous graph attention on Siamese networks to find
the similar classes.

####  1.2.1 Preprocessing

Data preprocessing is one of the most significant parts of developing a ML
model and is required to explain the variability of features in a sample. In
this study, we have handled data preprocessing of an ontology in six steps:
(i) an ontology parsing, (ii) tokenization, (iii) finding the abbreviations,
(iv) cleaning from stop words, (v) neighbourhood aggregation for creating the
context, and (vi) finding the embedding of the terms.

####  1.2.2 Embedding layer

We used the pre-trained Universal Sentence Encoder (USE) to obtain the word
embedding vector of the class and property. According to benchmark results
[9], the USE outperforms the BERT encoder in semantic sentence encoding.
Therefore, we chose the USE for word embeddings.

####  1.2.3 Heterogeneous graph attention layer

The graph attention introduced in graph attention networks is used to cluster
and classify the citation graphs [1]. It provides inductive and transactive
learning approaches. Its inductive learning approach computes contextual
information by applying an attention centred on the centre class in the
neighbourhood graph in our system. The centre class has five homogeneous
graphs consisting of its neighbouring terms, and these five graphs refer to
one heterogeneous graph.

Figure 2: This figure shows how graph attention is applied to a homogeneous
graph. The relationship between the centre class and its neighbouring classes
is the same in the homogeneous graphs such as ‘subClassOf’. In this graph, the
‘ObjectProperty’ node represents the restrictions belonging to the definition
of a child or parent class. Bag-Of-Word is used to represent the centre class
and its neighbouring classes. $h_{1}^{{}^{\prime}}$ shows the contextual
information of $h_{1}$ in terms of this ‘subClassOf’ relation after applying
graph attention. In addition, aij denotes $\alpha_{ij}$ in Equation 5.

The relation between the centre class and other terms in each homogeneous
graph is the same. For example, the relation between the centre node and other
terms in Figure 2 is ‘subClassOf’. However, some homogeneous graphs having
‘equivalentClass’ or ‘subClassOf’ relationships might contain some terms
regarding restrictions defined with ‘ObjectProperty’ in the neighbourhood
aggregation. The figure explains how the centre class’ contextual information
is computed by the graph attention in one of its homogeneous neighbourhood
graphs. The main difference between the original graph attention approach and
our graph attention is that this attention mechanism is applied to the
heterogeneous graph containing five homogeneous subgraphs. The attention in
this layer runs the following various attention mechanisms [1].

A set of features (the centre class and its neighbouring terms), which is
inputs of the graph attention layer, is denoted in Eq. 1 where
$\vec{h_{i}}\epsilon\mathbb{R}^{F}$.

$h=\left\\{\vec{h_{1}},\vec{h_{2}},\vec{h_{3}}......\vec{h_{N}},\right\\}$ (1)

The layer converts the input features to the new higher-level representation
of the feature list like defined in Eq. 2 where
$\vec{h_{i}^{{}^{\prime}}}\epsilon\mathbb{R}^{F^{{}^{\prime}}}$

$h^{{}^{\prime}}=\left\\{\vec{h_{1}^{{}^{\prime}}},\vec{h_{2}^{{}^{\prime}}},\vec{h_{3}^{{}^{\prime}}}......\vec{h_{N}^{{}^{\prime}}},\right\\}$
(2)

Using features in Eq. 1, we have obtained the higher-level representation of
the class’ neighbours, namely its contextual information in this layer. The
new features are computed by Eq. 3, and $\sigma$ indicates the linear
activation function like sigmoid or softmax in the Eq. 3. To indicate the
higher-level representation of the set of features,
$W\epsilon\mathbb{R}^{F^{\prime}XF}$ is used as a learnable parameter, and
shared linear transformation is applied to each feature. “K” is the number of
independent heads, and “K” equals five in our system.

$h^{\prime}=||_{k=1}^{K}\sigma\left(\sum\alpha^{k}_{ij}W^{k}\vec{h_{j}}\right)$
(3)

$\alpha$ $\epsilon$ $\mathbb{R}^{F}_{{}^{\prime}}$ X
$\mathbb{R}^{F}_{{}^{\prime}}$ denotes a shared attention mechanism and a
layer using the self-attention. The following equation computes attention
coefficients ($e_{ij}$):

$e_{ij}=\alpha(W\vec{h}_{i},W\vec{h}_{j})$ (4)

To compute the $\alpha_{ij}$ for $h^{\prime}$ in the Eq. 2, this coefficient
attention mechanism is applied in Eq. 5 where
$\vec{a}\epsilon\mathbb{R}^{2F}$, and $\alpha_{ij}$ is attention mechanism
parameterized by $\vec{a}$ weight vector.

$\alpha_{ij}=softmax(e_{ij})=\frac{\exp(LeakyReLU(\vec{a}{}^{T}[W\vec{h_{i}}||W\vec{h_{j}}]))}{\sum_{k\epsilon
N_{i}}\exp(LeakyReLU(\vec{a}{}^{T}[W\vec{h_{i}}||W\vec{h_{k}}]))}$ (5)

Using the formula in the Eq. 3 in this layer, we have obtained the higher-
level representation of the class’ neighbours, namely its contextual
information in this layer.

####  1.2.4 Output layer

The output layer provides down sampling (dimensional reduction) of the
contextual information, which is the concatenation of the class embedding and
higher representations of the class’ neighbours.

####  1.2.5 Cosine similarity layer

The cosine similarity layer measures the cosine similarity of output in the
previous layer.

###  1.3 Adaptations made for the evaluation

The GraphMatcher’s framework has been developed in Python with PyTorch and
Ontospy, and is packed by SEALS using MELT [10].

###  1.4 Parameter settings

The model’s parameters 333These parameters do not give the optimum model. The
best model has different parameters, but we mistakenly submitted the model
using these parameters to the conference track challenge. are 0.01 of learning
rate, 5 epochs, 0.001 of weight decay and 16 of batch size. The threshold is
computed from false positive alignments in the validation data as how the
VeeAlign [6] system proposes 444The project uses VeeAlign’s approach directly
to compute the threshold with the permission of the first author..

##  2 Results

The conference track consists of sixteen ontologies, but only seven ontologies
have (twenty-one) reference alignment cases. These reference alignments have
been utilized as ground truths to use true positive alignments. Besides,
negative alignment cases have been computed by oversampling from all the
possible class and property alignments. Therefore, once we apply a supervised
machine learning approach, we can use only these seven ontologies from this
dataset.

Table 1: The table shows GraphMatcher results in the conference track. Evaluation | Precision | F.5-measure | F1-measure | F2-measure | Recall
---|---|---|---|---|---
ra1-M1 | 0.82 | 0.77 | 0.71 | 0.65 | 0.62
ra1-M2 | 0.65 | 0.51 | 0.39 | 0.32 | 0.28
ra1-M3 | 0.8 | 0.74 | 0.67 | 0.6 | 0.57
ra2-M1 | 0.78 | 0.73 | 0.66 | 0.6 | 0.57
ra2-M2 | 0.65 | 0.5 | 0.39 | 0.32 | 0.28
ra2-M3 | 0.76 | 0.7 | 0.62 | 0.56 | 0.53
rar2-M1 | 0.77 | 0.73 | 0.67 | 0.62 | 0.59
rar2-M2 | 0.65 | 0.52 | 0.4 | 0.33 | 0.29
rar2-M3 | 0.75 | 0.7 | 0.63 | 0.58 | 0.55

The Table 1 shows the GraphMatcher results in the conference track. The
performance of the GraphMatcher is better in M1 variants than in M2 in terms
of F1-measure. Therefore, its F1-measure has been decreased in M3 variants. As
a result, the model has a weakness in the M2 variants, namely property
alignments.

##  3 General Comments

###  3.1 Comments on the results

The GraphMatcher is the new ontology matching system participating in OAEI
2022 and is evaluated in the conference track. The GraphMatcher demonstrates
remarkable performance in the M1 and M3 evaluation variants in terms of
F1-measure, even though it does not have high performance in the M2 evaluation
variant. However, all other matchers do not show remarkable results in this M2
evaluation variant.

On the other hand, it is also evaluated in the uncertain reference alignments
in OAEI 2022 conference track. It has the highest F1-measure (72% in both of
them) in the discrete and continuous metrics 555The evaluation based on the
uncertain reference alignments:
http://oaei.ontologymatching.org/2022/results/conference/index.html among all
other matchers evaluated in this track. This means that the GraphMatcher’s
confidence is higher than the other matchers evaluated in the OAEI 2022
conference track.

Table 2: The table shows the matchers’ performances on the rar2-M3 reference alignments. Matcher | Precision | F.5-measure | F1-measure | F2-measure | Recall
---|---|---|---|---|---
LogMap | 0.76 | 0.71 | 0.64 | 0.59 | 0.56
GraphMatcher | 0.75 | 0.7 | 0.63 | 0.58 | 0.55
SEBMatcher | 0.79 | 0.7 | 0.6 | 0.52 | 0.48
ATMatcher | 0.69 | 0.64 | 0.59 | 0.54 | 0.51
ALIN | 0.82 | 0.7 | 0.57 | 0.48 | 0.44
LogMapLt | 0.68 | 0.62 | 0.56 | 0.5 | 0.47
LSMatch | 0.83 | 0.69 | 0.55 | 0.46 | 0.41
AMD | 0.82 | 0.68 | 0.55 | 0.46 | 0.41
KGMatcher+ | 0.83 | 0.67 | 0.52 | 0.43 | 0.38
ALIOn | 0.66 | 0.44 | 0.3 | 0.22 | 0.19
Matcha | 0.37 | 0.2 | 0.12 | 0.08 | 0.07

###  3.2 Improvements

The GraphMatcher should be improved to match the properties, since it does not
perform well in the M2 evaluation variants. Its current version does not apply
the graph attention to align the properties because of the lack of properties’
neighbours, especially datatype properties. These object type and data type
properties might not have enough neighbouring terms to construct contextual
information in the ontology. In this case, the property’s context might be
improved with the external information in its next version, and the graph
attention can also be applied to align the properties. In addition to these
improvements, we will train the model with its optimum parameter settings in
its further version.

##  4 Conclusion

In this study, we have introduced the new ontology-matching system called
GraphMatcher. The GraphMatcher adapted the graph attention to the homogeneous
subgraphs of the centre class’ neighbours to obtain the contextual information
about the centre class. The graph attention has computed the higher-level
representation of each class and its surrounding classes and properties. The
results demonstrate promising performances in M1 and M3 evaluation variants.
The future work of this study will be to increase its performance in M2
evaluation variants.

## References

  * Veličković et al. [2018] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, Y. Bengio,  Graph attention networks (2018). arXiv:1710.10903.
  * Thiéblin et al. [2020] É. Thiéblin, O. Haemmerlé, N. Hernandez, C. Trojahn,  Survey on complex ontology matching,  Semantic Web 11 (2020) 689–727.
  * Jiménez-Ruiz and Cuenca Grau [2011] E. Jiménez-Ruiz, B. Cuenca Grau,  Logmap: Logic-based and scalable ontology matching,  in: L. Aroyo, C. Welty, H. Alani, J. Taylor, A. Bernstein, L. Kagal, N. Noy, E. Blomqvist (Eds.), The Semantic Web – ISWC 2011, SBH, Berlin, Heidelberg, 2011, pp. 273–288.
  * Faria et al. [2013] D. Faria, C. Pesquita, E. Santos, M. Palmonari, I. F. Cruz, F. M. Couto,  The agreementmakerlight ontology matching system,  in: R. Meersman, H. Panetto, T. Dillon, J. Eder, Z. Bellahsene, N. Ritter, P. De Leenheer, D. Dou (Eds.), On the Move to Meaningful Internet Systems: OTM 2013 Conferences, SBH, 2013, pp. 527–541.
  * Kolyvakis et al. [2018] P. Kolyvakis, A. Kalousis, D. Kiritsis,  DeepAlignment: Unsupervised ontology matching with refined word vectors,  in: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), ACL, 2018, pp. 787–798.
  * Iyer1 et al. [2020] V. Iyer1, A. Agarwal, H. Kumar,  Veealign: A supervised deep learning approach to ontology alignment (2020).
  * Bento et al. [2020] A. Bento, A. Zouaq, M. Gagnon,  Ontology matching using convolutional neural networks,  in: Proceedings of the 12th Language Resources and Evaluation Conference, ELRA, Marseille, France, 2020, pp. 5648–5653. URL: https://www.aclweb.org/anthology/2020.lrec-1.693.
  * Chen and Gu [2021] J. Chen, J. Gu,  Adol: a novel framework for automatic domain ontology learning,  Super Computing (2021).
  * Hassan et al. [2019] H. Hassan, G. Sansonetti, F. Gasparetti, A. Micarelli, J. Beel,  Bert, elmo, use and infersent sentence encoders: The panacea for research-paper recommendation?,  in: RecSys, WS, 2019\.
  * Hertling et al. [2019] S. Hertling, J. Portisch, H. Paulheim,  Melt - matching evaluation toolkit,  in: M. Acosta, P. Cudré-Mauroux, M. Maleshkova, T. Pellegrini, H. Sack, Y. Sure-Vetter (Eds.), Semantic Systems. The Power of AI and Knowledge Graphs, Springer International Publishing, Cham, 2019, pp. 231–245.


#  ToxVis: Enabling Interpretability of Implicit vs. Explicit Toxicity
Detection Models with Interactive Visualization

Uma Gunturi  Department of Computer Science, Virginia TechBlacksburgUSA
<EMAIL_ADDRESS>,  Xiaohan Ding  Department of Computer Science, Virginia
TechBlacksburgUSA<EMAIL_ADDRESS>and  Eugenia H. Rho  Department of Computer
Science, Virginia TechBlacksburgUSA<EMAIL_ADDRESS>
WARNING: This paper contains some content which is offensive in nature.

##  1\. Introduction

People’s judgment of hateful online content is inherently subjective and
multi-faceted as interpretations around what is and is not hateful can depend
on personal values, social identities, and culture (Nockleby, 2000). Hence, as
opposed to explicit hate-speech or profanity-laden attacks where the
offender’s mal intent is expressed in black and white language, it is more
challenging to effectively respond to, or mitigate harms caused by implicit
hatespeech. This is because implicit toxicity often takes form as humour
(Fortuna and Nunes, 2018), insider expressions, neologisms (Borsotti and
Bjorn, 2022), and microaggressions, (Espaillat et al., 2019), (e.g. ”Black
people seem to think everything revolves around them being royalty” (from
r/unpopularopinion). Hence, human moderators and content moderation systems
have difficulty understanding, recognizing, and responding to implicit
hatespeech (Wright et al., 2021), resulting in moderation failures where false
negatives (harmful, yet undetected texts) are left unmoderated (Haimson et
al., 2021). Furthermore, most hate-classification research, including those
that examine implicit hatespeech, often use black-box deep learning (DL)
models that lack interpretability, failing to explain why a content was
classified as hateful. In this work, we aim to address these challenges by
developing a visually interactive and explainable tool called ToxVis (Fig.1).
We built ToxVis by using RoBERTa [cite], XLNET [cite], and GPT-3 [cite] and
fine-tuning two transformer-based models to classify hatespeech into three
categories of hatespeech: implicit, explicit, and non-hateful. We then used DL
interpretation techniques to make the classification results explainable.
Through ToxVis, users can (1) type in a potentially hateful text into the
system. The system will then (2) classify the input text into one of the three
categories of hatespeech (implicit, explicit, or non-hateful). The user can
then click on the classification results (3) to see which words from the input
text contributed most to the classification decision, as well as the model’s
prediction confidence score. Live demo of ToxVis is available here.

##  2\. ToxVis Interactive System

Data collection and pre-processing. We collected existing hate speech datasets
from multiple sources: ETHOS (Mollas et al., 2020), AbuseEval (Caselli et al.,
2020), and a microaggressions dataset from Tumblr (Breitfeller et al., 2019).
The combined corpora had multiple labeling structures (implicit vs. explicit;
hate vs. non-hate). Hence, we limited our sample to those that contained
explicit, implicit, and non-hate labels. Three researchers then manually
verified a randomized sample into three categories: explicit hate, implicit
hate, and non-hate. Through this process, we eliminated previously mislabeled
instances. We further eliminated instances with urls, and special characters,
leaving us with 14,473 posts.

Training and interpretation. We fine-tuned transformer-based models (BERT,
RoBERTa, XLNET, GPT-3) on our data (train:dev:test split of 33:33:33) for a
multi-class classification task (label 1: explicit hate, label 2: implicit
hate, label 3: non-hate). We implemented the models using the HuggingFace
Transformers (Wolf et al., 7 13) and PyTorch libraries (Paszke et al., 2019).
We then evaluated the fine-tuned models on a held-out test set using standard
evaluation metrics (accuracy, precision, recall, and F1). We further improved
model performance through hyperparameter tuning with multiple learning rates
($lr=$ 1e-5, 2e-5 or 3e-5), batch sizes ($n_{bs}=$ 16, 32, or 64), and the
number of epochs ($n_{epoch}=$ $3$, $4$, or $5$).

Figure 1. The ToxVis user interface. The system leverages three different
Transformer-based models A. XLNet B. RoBERTa C. OpenAI’s GPT-3. For A and B,
the system displays (1) the model’s predicted label (explicit, implicit, non-
hateful), (2) the contribution of each input word to the predicted label, and
(3) the model’s confidence score of the predicted label. For C, the system
displays (1) the classification label, (2) the prediction confidence score,
and (3) the predictive tokens.

To make the classification results explainable, we used Integrated Gradients
(IG) (Sundararajan et al., 6 12), a post-hoc model interpretability technique,
to investigate the contribution of individual tokens to the classified label.
In other words, we wanted to understand how much a given word from the input
text contributed to whether the text was classified as 1. explicitly hateful,
2. implicitly hateful, or 3. non-hateful. For this task, we used IG to
calculate the gradient of each input token to analyze the attribution score
(ranging from -1 to 1) on the final label. Positive attribution scores
indicate a given token’s positive contribution towards the predicted label,
whereas negative attribution values of a given token do not contribute to the
predicted label.

Table 1. Comparison of F1 score, Precision, Recall, and Accuracy among
different language models.

$\begin{gathered}\text{ Model }\text{ Name }\end{gathered}$ | F1 Score | Precision | Recall | Accuracy
---|---|---|---|---
BERT | $0.750$ | $0.724$ | $0.778$ | $0.755$
RoBERTa | $0.848$ | $0.829$ | $0.869$ | $0.820$
XLNet | $0.804$ | $0.745$ | $0.873$ | $0.791$
$\begin{gathered}\text{ GPT-3 }\text{ (davinci) }\end{gathered}$ | $0.976$ | $0.981$ | $0.971$ | $0.977$

Interactive System. After completing the training and evaluation processes, we
deployed our explainable multi-label classification system on a cloud server
utilizing Python Flask. The user-interface design goals of ToxVis (Fig.1) was
to empower users by allowing them to interactively compare the classificaiton
results of three different linguistic DL classifiers. We also visualized the
input tokens by different colors according to their contribution to the output
prediction: blue indicating negative and green indicating positive
contribution to the predicted labels. We instructed GPT-3 to identify and list
words from the input sentence that contributed most (in descending order of
importance) in conveying the harmful aspect of the input sentence.

## References

  * (1)
  * Borsotti and Bjorn (2022) Valeria Borsotti and Pernille Bjorn. 2022.  Humor and Stereotypes in Computing: An Equity-focused Approach to Institutional Accountability.  _Computer Supported Cooperative Work (CSCW)_ 31 (07 2022).  https://doi.org/10.1007/s10606-022-09440-9
  * Breitfeller et al. (2019) Luke Breitfeller, Emily Ahn, David Jurgens, and Yulia Tsvetkov. 2019.  Finding Microaggressions in the Wild: A Case for Locating Elusive Phenomena in Social Media Posts. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics, Hong Kong, China, 1664–1674.
  * Caselli et al. (2020) Tommaso Caselli, Valerio Basile, Jelena Mitrović, Inga Kartoziya, and Michael Granitzer. 2020\.  I Feel Offended, Don’t Be Abusive! Implicit/Explicit Messages in Offensive and Abusive Language. In _Proceedings of the Twelfth Language Resources and Evaluation Conference_. European Language Resources Association, Marseille, France, 6193–6202.  https://aclanthology.org/2020.lrec-1.760
  * Espaillat et al. (2019) Andre Espaillat, Danielle Panna, Dianne L. Goede, Matthew J. Gurka, Maureen A Novak, and Zareen Zaidi. 2019.  An exploratory study on microaggressions in medical school: What are they and why should we care?  _Perspectives on Medical Education_ 8 (2019), 143 – 151.
  * Fortuna and Nunes (2018) Paula Fortuna and Sérgio Nunes. 2018.  A survey on automatic detection of hate speech in text.  _ACM Computing Surveys (CSUR)_ 51, 4 (2018), 1–30.
  * Haimson et al. (2021) Oliver L. Haimson, Daniel Delmonaco, Peipei Nie, and Andrea Wegner. 2021.  Disproportionate Removals and Differing Content Moderation Experiences for Conservative, Transgender, and Black Social Media Users: Marginalization and Moderation Gray Areas.  _Proc. ACM Hum.-Comput. Interact._ 5, CSCW2, Article 466 (oct 2021), 35 pages.
  * Mollas et al. (2020) Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas. 2020.  ETHOS: an Online Hate Speech Detection Dataset.  arXiv:2006.08328 [cs.CL]
  * Nockleby (2000) John T Nockleby. 2000\.  Hate speech.  _Encyclopedia of the American constitution_ 3, 2 (2000), 1277–1279.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In _Advances in Neural Information Processing Systems 32_. Curran Associates, Inc., 8024–8035.  http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  * Sundararajan et al. (6 12) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017-06-12.  Axiomatic Attribution for Deep Networks.  arXiv:1703.01365 [cs]
  * Wolf et al. (7 13) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020-07-13.  HuggingFace’s Transformers: State-of-the-art Natural Language Processing.  arXiv:1910.03771 [cs] http://arxiv.org/abs/1910.03771
  * Wright et al. (2021) Austin P Wright, Omar Shaikh, Haekyu Park, Will Epperson, Muhammed Ahmed, Stephane Pinel, Duen Horng Chau, and Diyi Yang. 2021\.  Recast: Enabling user recourse and interpretability of toxicity detection models with interactive visualization.  _Proceedings of the ACM on Human-Computer Interaction_ 5, CSCW1 (2021), 1–26.