# Knowledge Graph Embedding with 3D Compound Geometric Transformations

Xiou Ge, Yun-Cheng Wang, Bin Wang,
C.-C. Jay Kuo Xiou Ge, Yun-Cheng Wang, and C.-C. Jay Kuo are with the Ming
Hsieh Department of Electrical and Computer Engineering, University of
Southern California, Los Angeles, CA, 90089, USA.
Bin Wang is with National University of Singapore, Singapore.

###### Abstract

The cascade of 2D geometric transformations were exploited to model relations
between entities in a knowledge graph (KG), leading to an effective KG
embedding (KGE) model, CompoundE. Furthermore, the rotation in the 3D space
was proposed as a new KGE model, Rotate3D, by leveraging its non-commutative
property. Inspired by CompoundE and Rotate3D, we leverage 3D compound
geometric transformations, including translation, rotation, scaling,
reflection, and shear and propose a family of KGE models, named CompoundE3D,
in this work. CompoundE3D allows multiple design variants to match rich
underlying characteristics of a KG. Since each variant has its own advantages
on a subset of relations, an ensemble of multiple variants can yield superior
performance. The effectiveness and flexibility of CompoundE3D are
experimentally verified on four popular link prediction datasets.

###### Index Terms:

Knowledge graph embedding, link prediction, geometric transformation.

##  I Introduction

Knowledge graphs (KGs) find rich applications in knowledge management and
discovery [1, 2, 3], recommendation systems [4, 5], fraud detection [6, 7],
chatbots [8, 9], etc. KGs are directed relational graphs. They are formed by a
collection of triples in form of $(h,r,t)$, where $h$, $r$, and $t$ denote
head, relation, and tail, respectively. Heads and tails are called entities
and represented by nodes while relations are links in KGs. KGs are often
incomplete. One critical task in knowledge graph (KG) management is “missing
link prediction”. Knowledge graph embedding (KGE) methods have received a lot
of attention in recent years due to their effectiveness in missing link
prediction. Many KGs such as DBpedia [10], YAGO [11], Freebase [12], NELL
[13], Wikidata [14], and ConceptNet [15] have been created and made publicly
available for KGE model development and evaluation.

One family of KGE models builds a high-dimensional embedding space, where each
entity is a vector. The relation is modeled by a certain geometric
manipulation such as translation and rotation. To evaluate the likelihood of a
candidate triple, the geometric manipulation associated with the relation is
applied to the head entity and then the distance between the manipulated head
and the tail is measured. The shorter the distance, the higher likelihood of
the triple. To this end, these KGE models are called distance-based KGEs.
Examples of distance-based KGEs include TransE [16], RotatE [17], and PairRE
[18]. Each of them uses a single geometric transformation to represent
relations between entities. Specifically, translation, rotation, and scaling
operations are adopted by TransE, RotatE, and PairRE, respectively.

The above-mentioned KGE models achieve reasonably good performance in link
prediction with only a single geometric transformation. The cascade of
multiple 2D geometric transformations offers a powerful tool in image
manipulation [19]. This idea was exploited to develop a new KGE model, called
CompoundE, in [20]. TransE, RotatE and PairRE are all degenerate cases of
CompoundE. Thus, CompoundE outperforms them in link prediction performance.
CompoundE unifies translation, rotation, and scaling operations under one
common framework. It has several mathematically provable properties that
facilitate the modeling of different complex relation types in KGE. The
effectiveness of these composite operators has been successfully demonstrated
through extensive experiments and applications in downstream tasks such as
entity typing and multihop query answering in [20]. Furthermore, borrowed from
the concept of rotation in the 3D space, Rotate3D [21, 22, 23] achieves more
effective parameterization and endows a model with greater modeling power than
RotatE based on 2D rotation. That is, Rotate3D can model non-commutative
relations better than RotatE.

Inspired by the success of CompoundE and Rotate3D, we wonder whether it would
be beneficial to look for compound geometric transformations in the 3D space
in the KGE model design. Here, we extend the CompoundE work in [20] along
three directions. First, we include more affine operations beyond translation,
rotation, and scaling such as reflection and shear. Second, we extend these
geometric transformations from the 2D space to the 3D space and propose a
family of KGE models, CompoundE3D. Third, CompoundE3D allows multiple design
variants to match rich underlying characteristics of a KG. Since each variant
has its own advantages on a subset of relations, an ensemble of multiple
variants can yield superior performance. The effectiveness of CompoundE3D is
experimentally verified on four popular link prediction datasets.

It is worthwhile to emphasize that we enhance CompoundE by addressing two
critical issues. First, compound operations lead to numerous model variants,
and it is unclear how to determine a scoring function that performs the best
for a given dataset. Here, we propose an adapted beam search algorithm that
builds more complex scoring functions from simple but effective ones
gradually. Second, although ensemble learning is a popular strategy, it
remains under-explored when it comes to building KGE models. In this work, we
explore two ensemble strategies that potentially boost link prediction
performance and allow different CompoundE3D variants to work together and
complement each other. First, we implement a weighted sum of different scoring
functions for link prediction. Second, we apply unsupervised rank aggregation
functions to unify rank predictions from individual model variants. Both
strategies help boost the ranking of valid candidate entities and reduce the
impact of outliers.

The major contributions of this work are summarized below.

  * •

We examine affine operations in the 3D space, instead of the 2D space, to
allow more versatile relation representations. Besides translation, rotation,
and scaling used in CompoundE, we include reflection and shear transformations
which allow an even larger design space.

  * •

We propose an adapted beam search algorithm to discover better model variants.
Such a procedure avoids unnecessary exploration of poor variants but zooms
into more effective ones to strike a good balance between model complexity and
prediction performance.

  * •

We analyze the properties of each operation and its advantage in modeling
different relations. Our analysis is backed by empirical results on four
datasets.

  * •

To reduce errors of an individual model variant and boost the overall link
prediction performance, we aggregate decisions from different variants with
two approaches; namely, the sum of weighted distances and rank fusion.

The rest of this paper is organized as follows. A brief review of related work
is provided in Sec. II. The design methodology of CompoundE3D is explained and
the decision ensemble of multiple model variants is elaborated in Sec. III.
Experimental results and performance benchmarking with previous work are
presented in Sec. IV. Finally, concluding remarks and future research
directions are given in Sec. V.

##  II Related Work

###  II-A Knowledge Graph Embedding (KGE) Models

A large number of distance-based KGE models are derived by treating relations
as certain transformations. They are briefly reviewed below.

####  II-A1 2D Geometric Transformations

Quite a few KGE models are inspired by 2D geometric transformations such as
translation, rotation, and scaling in the 2D plane. TransE [16] models the
relation as a translation between head and tail entities. This simple model is
not able to model symmetric relations effectively. RotatE [17] treats
relations as certain rotations in the complex space, which works well for
symmetric relations. Furthermore, RotatE introduces a self-adversarial
negative sampling loss that improves distance-based KGE model performance
significantly. PairRE [18] models relations with the scaling operation to
allow variable margins. This is helpful in encoding complex relations. The
unitary constraint on entity embedding in PairRE is also effective in
practice. CompoundE [20] adopts compound geometric transformations, including
translation, rotation, and scaling, to model different relations. It offers a
superior KGE model without increasing the overall complexity much.

####  II-A2 Advanced Transformations

NagE [22] introduces generic group theory to the design of KGE models and
gives a generic recipe for their construction. QuatE [24] extends the KGE
design to the Quaternion space which enables more compact interactions between
entities and relations while introducing more degree of freedom. To model non-
commutativeness in relation composition more effectively, both RotatE3D [21]
and DensE [23] leverage quaternion rotations but in different forms. ROTH [25]
adopts the hyperbolic curvature to capture the hierarchical structure in KGs.
On the other hand, it is questioned in [26] whether the introduction of
hyperbolic geometry in KGE is necessary.

###  II-B Classification-based Models

Another family of models is built by classifying an unseen triple into “valid”
(or positive) and “invalid” (or negative) two classes and then using the soft
decision to measure the likelihood of the triple.

####  II-B1 Simple Neural Networks

A multilayer perceptron (MLP) network [27] is used to measure the likelihood
of unseen triples for link prediction. The neural tensor network (NTN) [28]
adopts a bilinear tensor neural layer to model interactions between entities
and relations of triples. ConvE [29] stacks head entities and relations,
reshapes them to 2D arrays, and uses the convolutional neural network (CNN) to
extract the information from them. The resulting feature map interacts with
tail entities through dot products. R-GCN [30] uses the graph convolutional
network (GCN) with relation-specific weights to obtain entity representations,
which are subsequently fed to DistMult [31] for link prediction. Despite its
potential of handling the inductive setting, its performance is not on par
with the embedding based approach.

####  II-B2 Advanced Neural Networks

KG-BERT [32] uses the pretrained language model, BERT [33], to obtain the
entity representation from textual descriptions (rather than from KG links).
However, its inference time is much longer compared to embedding-based models.
SimKGC [34] improves transformer-based classification methods by constructing
contrastive pairs. It uses BERT to estimate the semantic similarity and treats
triples of higher similarity score as positive sample pairs, and vice versa.
However, its performance is sensitive to the language model quality, and its
required computational resource is high.

####  II-B3 Lightweight Classification Model

KGBoost [35] proposes a novel negative sampling scheme, and uses the XGBoost
[36] classifier for link prediction. Inspired by the Discriminant Feature
Learning (DFT) [37, 38] that extracts most discriminative features from
trained embeddings, GreenKGC [39] is a lightweight and modularized
classification method that trains a binary classifier to classify unseen
triples.

###  II-C Advanced Relation Modeling

Special techniques have been developed to model complex relations. For
example, to model relations such as 1-to-N, N-to-1, and N-to-N effectively,
TransH [40] projects the embedded entity space into relation-specific hyper-
planes. TransR [41] learns a relation-specific projection that maps entity
vectors to a certain relation space. TransD [42] derives dynamic mapping based
on relation and entity projection vectors. TranSparse [43] enforces the
relation projection matrix to be sparse. Recently, many KGE models including
X+AT [44], SFBR [45], and STaR [46] apply translation and scaling operations
to both distance-based and semantic-matching-based [47] models to improve the
performance gain. The inclusion of translation is proven to be effective in
improving KGEs in the Quaternion space such as DualE [48], BiQUE [49].
ReflectE [50] models each relation as a normal vector of a hyper-plane that
reflects entity vectors. It can be used to model symmetric and inverse
relations well. So far, the cascade of various affine operations is a natural
yet unexplored idea to pursue.

###  II-D Model Ensembles

Although ensemble learning is a prevailing strategy in machine learning, it
remains under-explored for knowledge graph completion. Link prediction
evaluation is essentially a ranking problem. It is desired to optimize an
ensemble decision so that valid triples get ranked higher than invalid ones
among all candidates. Rank aggregation is a classical problem in information
retrieval. Both supervised methods [51, 36] and unsupervised methods [52, 53]
have been studied. Since the ground truth ranking in KG’s link prediction is
not available (except the top-1 triple), the unsupervised setting is more
relevant. Yet, the use of rank aggregation to boost link prediction
performance has received limited attention. Several examples are given below.
KEnS [54] performs ensemble inference to combine predictions from multiple
language-based KGEs for multilingual knowledge graph completion. AutoSF [55]
develops an algorithm to search for the best scoring functions from multiple
semantic matching models. The ensemble of multiple identical low-dimensional
KGE models is adopted in [56] to boost the link prediction performance.
Recently, DuEL [57] treats link prediction as a classification problem and
aggregates binary decisions from several different classifiers using
unsupervised techniques.

(a) Translation

(b) Scaling

(c) Rotation

(d) Reflection

(e) Shear

(f) Compound

Figure 1: Composing different geometric operations in the 3D subspace.

##  III Proposed Method

###  III-A CompoundE3D

In this work, we use 3D affine transformations, including Translation,
Scaling, Rotation, Reflection, and Shear as illustrated in 1, to model
different relations in KGs. This large set of transformation operators offer
immense flexibility in the KGE design against different characteristics of KG
datasets. Below, we formally define each of the 3D affine operators in
homogeneous coordinates.

####  III-A1 Translation

Component $\mathbf{T}\in\mathbf{SE}(3)$, illustrated by Fig. 1a, is defined as

$\mathbf{T}=\begin{bmatrix}1&0&0&v_{x}\\\ 0&1&0&v_{y}\\\ 0&0&1&v_{z}\\\
0&0&0&1\end{bmatrix},$ (1)

####  III-A2 Scaling

Component $\mathbf{S}\in\mathbf{Aff}(3)$, illustrated by Fig. 1b, is defined
as

$\mathbf{S}=\begin{bmatrix}s_{x}&0&0&0\\\ 0&s_{y}&0&0\\\ 0&0&s_{z}&0\\\
0&0&0&1\end{bmatrix},$ (2)

####  III-A3 Rotation

Component $\mathbf{R}\in\mathbf{SO}(3)$, illustrated by Fig. 1c, is defined as

$\mathbf{R}=\mathbf{R}_{z}(\alpha)\mathbf{R}_{y}(\beta)\mathbf{R}_{x}(\gamma)=\begin{bmatrix}a&b&c&0\\\
d&e&f&0\\\ g&h&i&0\\\ 0&0&0&1\end{bmatrix},$ (3)

where

$\displaystyle a$ $\displaystyle=\cos(\alpha)\cos(\beta),$ (4) $\displaystyle
b$
$\displaystyle=\cos(\alpha)\sin(\beta)\sin(\gamma)-\sin(\alpha)\cos(\gamma),$
$\displaystyle c$
$\displaystyle=\cos(\alpha)\sin(\beta)\cos(\gamma)+\sin(\alpha)\sin(\gamma),$
$\displaystyle d$ $\displaystyle=\sin(\alpha)\cos(\beta),$ $\displaystyle e$
$\displaystyle=\sin(\alpha)\sin(\beta)\sin(\gamma)+\cos(\alpha)\cos(\gamma),$
$\displaystyle f$
$\displaystyle=\sin(\alpha)\sin(\beta)\cos(\gamma)-\cos(\alpha)\sin(\gamma),$
$\displaystyle g$ $\displaystyle=-\sin(\beta),$ $\displaystyle h$
$\displaystyle=\cos(\beta)\sin(\gamma),$ $\displaystyle i$
$\displaystyle=\cos(\beta)\cos(\gamma).$

This general 3D rotation operator is the result of compounding yaw, pitch, and
roll rotations. They are, respectively, defined as

  * •

Yaw rotation component:

$\mathbf{R}_{z}(\alpha)=\begin{bmatrix}\cos(\alpha)&-\sin(\alpha)&0&0\\\
\sin(\alpha)&\cos(\alpha)&0&0\\\ 0&0&1&0\\\ 0&0&0&1\end{bmatrix},$ (5)

  * •

Pitch rotation component:

$\mathbf{R}_{y}(\beta)=\begin{bmatrix}\cos(\beta)&0&-\sin(\beta)&0\\\
0&1&0&0\\\ \sin(\beta)&0&\cos(\beta)&0\\\ 0&0&0&1\end{bmatrix},$ (6)

  * •

Roll rotation component:

$\mathbf{R}_{x}(\gamma)=\begin{bmatrix}1&0&0&0\\\
0&\cos(\gamma)&-\sin(\gamma)&0\\\ 0&\sin(\gamma)&\cos(\gamma)&0\\\
0&0&0&1\end{bmatrix}.$ (7)

####  III-A4 Reflection

Component $\mathbf{F}\in\mathbf{SO}(3)$, illustrated by Fig. 1d, is defined as

$\mathbf{F}=\begin{bmatrix}1-2n_{x}^{2}&-2n_{x}n_{y}&-2n_{x}n_{z}&0\\\
-2n_{x}n_{y}&1-2n_{y}^{2}&-2n_{y}n_{z}&0\\\
-2n_{x}n_{z}&-2n_{y}n_{z}&1-2n_{z}^{2}&0\\\ 0&0&0&1\end{bmatrix}.$ (8)

The above expression is derive from the Householder reflection,
$\mathbf{F=I-2nn^{T}}$. In the 3D space, $\mathbf{n}$ is a 3-D unit vector
that is perpendicular to the reflecting hyper-plane,
$\mathbf{n}=[n_{x},n_{y},n_{z}]$.

####  III-A5 Shear

Component $\mathbf{H}\in\mathbf{Aff}(3)$, illustrated by Fig. 1e, is defined
as

$\mathbf{H}=\mathbf{H}_{yz}\mathbf{H}_{xz}\mathbf{H}_{xy}=\begin{bmatrix}1&\text{Sh}^{y}_{x}&\text{Sh}^{z}_{x}&0\\\
\text{Sh}^{x}_{y}&1&\text{Sh}^{z}_{y}&0\\\
\text{Sh}^{x}_{z}&\text{Sh}^{y}_{z}&1&0\\\ 0&0&0&1\end{bmatrix}.$ (9)

The shear operator is the result of compounding 3 operators:
$\mathbf{H}_{yz}$, $\mathbf{H}_{xz}$, and $\mathbf{H}_{xy}$ They are
mathematically defined as

$\displaystyle\mathbf{H}_{yz}$ $\displaystyle=$
$\displaystyle\begin{bmatrix}1&0&0&0\\\ \text{Sh}^{x}_{y}&1&0&0\\\
\text{Sh}^{x}_{z}&0&1&0\\\ 0&0&0&1\\\ \end{bmatrix},$ (10)
$\displaystyle\mathbf{H}_{xz}$ $\displaystyle=$
$\displaystyle\begin{bmatrix}1&\text{Sh}^{y}_{x}&0&0\\\ 0&1&0&0\\\
0&\text{Sh}^{y}_{z}&1&0\\\ 0&0&0&1\\\ \end{bmatrix},$ (11)
$\displaystyle\mathbf{H}_{xy}$ $\displaystyle=$
$\displaystyle\begin{bmatrix}1&0&\text{Sh}^{z}_{x}&0\\\
0&1&\text{Sh}^{z}_{y}&0\\\ 0&0&1&0\\\ 0&0&0&1\\\ \end{bmatrix}.$ (12)

Matrix $\mathbf{H}_{yz}$ has a physical meaning - the shear transformation
that shifts the $y$\- and $z$\- components by a factor of the $x$ component.
Similar physical interpretations are applied to $\mathbf{H}_{xz}$ and
$\mathbf{H}_{xy}$.

The above transformations can be cascaded to yield a compound operator; e.g.,

$\mathbf{O=T\cdot S\cdot R\cdot F\cdot H},$ (13)

In the actual implementation, we use the operator’s representation in regular
Cartesian coordinate instead of the homogeneous coordinate. Furthermore, a
high-dimensional relation operator can be represented as a block diagonal
matrix in the form of

$\mathbf{M_{r}=\textbf{diag}(O_{r,1},O_{r,2},\dots,O_{r,n})},$ (14)

where $\mathbf{O_{r,i}}$ is the compound operator at the $i$-th stage.

We can define the following three scoring functions for CompoundE3D:

  * •

CompoundE3D-Head

$f_{r}^{(h)}(h,t)=\|\mathbf{M_{r}\cdot h-t}\|,$ (15)

  * •

CompoundE3D-Tail

$f_{r}^{(t)}(h,t)=\|\mathbf{h-\hat{M}_{r}\cdot t}\|,$ (16)

  * •

CompoundE3D-Complete

$f_{r}^{(h,t)}(h,t)=\|\mathbf{M_{r}\cdot h-\hat{M}_{r}\cdot t}\|,$ (17)

where $\mathbf{h}$ and $\mathbf{t}$ denote head and tail entity embeddings,
and $\mathbf{M_{r}}$ and $\mathbf{\hat{M}_{r}}$ denote the relation-specific
operators that operate on head and tail entities, respectively.

Generally speaking, we have five different affine operations available to use,
i.e. translation, scaling, rotation, reflection, and shear. Each operator can
be applied to 1) head entity, 2) tail entity, or 3) both head and tail. Hence,
we have in total 15 different ways of applying operators at each stage. All
these possible choices are called CompoundE3D variants. For a given KG
dataset, there is a huge search space in finding the optimal CompoundE3D
variant. It is essential to develop a simple yet effective mechanism to find a
variant that gives the best performance under a certain complexity constraint.

###  III-B Beam Search for Best CompoundE3D Variant

In this subsection, we present a beam search algorithm to find the optimal
CompoundE3D variant. For the $i$-th stage, the set of all operator pairs that
can be applied at a certain step is

$\displaystyle\mathbf{P}\in\\{$
$\displaystyle\mathbf{(T,I),(S,I),(R,I),(F,I),(H,I),}$ (18)
$\displaystyle\mathbf{(I,\hat{T}),(I,\hat{S}),(I,\hat{R}),(I,\hat{F}),(I,\hat{H}),}$
$\displaystyle\mathbf{(T,\hat{T}),(S,\hat{S}),(R,\hat{R}),(F,\hat{F}),(H,\hat{H})}\\},$

where $\mathbf{I}$ is the identity operator. First, we apply all operator
pairs in $\mathbf{P}$ and calculate scoring functions for all intermediate
variants. Each variant is optimized with $l$ iterations using the training set
and its performance is evaluated on the validation dataset. Then, we choose
the top-$k$ best-performing variants as starting points for further
exploration in the next step. The same process is repeated until one of
terminating conditions is triggered. Afterward, we proceed to the $(i+1)$-th
stage. The whole search is completed after the final stage is reached. The
total number of stages is a user selected hyper-parameter.

The beam search process in building more complex KGE models from simpler ones
is described in Algorithm 1. Additional comments are given below.

  * •

We initialize the algorithm by setting up a loop to iterate over the set,
$\mathbf{P}$, of all possible operator combinations to train and evaluate them
and find the top-$k$ variants as starting points.

  * •

In the next loop, we have two stopping criteria to terminate the beam search:
1) $\\#$ operators $>\lambda$, meaning that we stop the search when the number
of operators exceeds the upper bound $\lambda$; 2)
$\frac{\Delta\text{MRR}}{\Delta\text{Param}}<\gamma$, meaning that the ratio
of increase in MRR versus the increase in free parameters fall below the
threshold $\gamma$, and it is no longer worthwhile to increase the model
complexity for the marginal gain in model performance.

  * •

$\mathbf{P}\times\mathbf{W}$ denotes the Cartesian product between the
operator pairs set $\mathbf{P}$ and top-$k$ variants set $\mathbf{W}$ from the
last step while
$\dot{f}^{i-1}_{r}(h,t)\lhd(\mathbf{M}^{i},\mathbf{\hat{M}}^{i})$ denotes
applying the operator pair $(\mathbf{M}^{i},\mathbf{\hat{M}}^{i})$ to previous
optimal scoring function $\dot{f}^{i-1}_{r}(h,t)$.

  * •

For example, if $\dot{f}^{i-1}_{r}(h,t)=\|\mathbf{R\cdot h-t}\|$ and
$(\mathbf{M}^{i},\mathbf{\hat{M}}^{i})=(\mathbf{S},\mathbf{\hat{S}})$, then
$\tilde{f}_{r}^{i}(h,t)=\|\mathbf{S\cdot R\cdot h-\hat{S}\cdot t}\|$.

  * •

After the loop terminates due to any terminating condition is triggered, we
select the top-1 performing variant from the explored variants set,
$\mathbf{W}$, as the best choice.

Algorithm 1 Beam Search for Best CompoundE3D Variant

initialize $i\leftarrow 1,\mathbf{U}\leftarrow\\{\\}$

for $(\mathbf{M}^{i},\mathbf{\hat{M}}^{i})\in\mathbf{P}$ do

$\tilde{f}_{r}^{i}(h,t)\leftarrow\|\mathbf{M}^{i}\cdot\mathbf{h}-\mathbf{\hat{M}}^{i}\cdot\mathbf{t}\|$;

train $\tilde{f}_{r}(h,t)$ for $l$ iterations;

MRR $\leftarrow$ evaluate $\tilde{f}_{r}^{i}(h,t)$ with valid set;

$\mathbf{U}$.insert($\\{\text{MRR},\tilde{f}_{r}^{i}(h,t)\\}$);

end for

$\mathbf{W}\leftarrow$ top-$k$ variants from $\mathbf{U}$

$i\leftarrow i+1$

$\Delta\text{MRR}\leftarrow\gamma$, $\Delta\text{Param}\leftarrow 1$

while $\\#$ operators $<\lambda$ and
$\max\frac{\Delta\text{MRR}}{\Delta\text{Param}}\geq\gamma$ do

initialize $\mathbf{V}\leftarrow\\{\\}$

for
{$(\mathbf{M}^{i},\mathbf{\hat{M}}^{i}),\dot{f}^{i-1}_{r}(h,t)\\}\in\mathbf{P}\times\mathbf{W}$
do

$\tilde{f}_{r}^{i}(h,t)\leftarrow\dot{f}^{i-1}_{r}(h,t)\lhd(\mathbf{M}^{i},\mathbf{\hat{M}}^{i})$;

train $\tilde{f}_{r}^{i}(h,t)$ for $l$ iterations;

evaluate $\tilde{f}_{r}^{i}(h,t)$ with valid set;

$\Delta$MRR $\leftarrow$ $\tilde{f}_{r}^{i}(h,t)$ MRR$-\dot{f}_{r}^{i-1}(h,t)$
MRR;

$\Delta$Param $\leftarrow$ $\tilde{f}_{r}^{i}(h,t)$
Param$-\dot{f}_{r}^{i-1}(h,t)$ Param;

$\mathbf{V}$.insert(MRR, $\Delta$MRR, $\Delta$Param,
$\tilde{f}_{r}^{i}(h,t)$);

end for

$\mathbf{W}\leftarrow$ top-$k$ variants from $\mathbf{V}$;

end while

${f}^{*}_{r}(h,t)\leftarrow$ best variant from $\mathbf{W}$;

Figure 2: The ensemble of multiple CompoundE3D variants.

###  III-C Model Ensembles

####  III-C1 Weighted-Distances-Sum (WDS) Strategy

We choose the top-$k$ performing CompoundE3D variants and conduct a weighted
average of their predicted scores. The following three weighting schemes are
considered.

  * •

Uniform Weights. This scheme takes an equal weight of selected $k$ variants as

$\hat{f}_{r}(h,t)=\frac{1}{k}\sum_{i=1}^{k}f_{r}^{i}(h,t),$ (19)

where $f_{r}^{i}(h,t)$ is the scoring function for the $i$-th variant.

  * •

Geometric Weights. This scheme sorts the variants based on their MRR
performance on the validation dataset in a descending order and assign weight
$\lambda^{k}$, $0<\lambda<1$, to the $k$-th variant. That is,

$\hat{f}_{r}(h,t)=\frac{1}{\sum_{i=1}^{k}\lambda^{k}}\sum_{i=1}^{k}\lambda^{k}f_{r}^{i}(h,t).$
(20)

Since $\lambda^{k}>\lambda^{k+1}$, we assign a higher weight to a better
performer in computing the aggregated distance.

  * •

Learnable Weights. This scheme trains a set of learnable weights, $w_{i}>0$,
based on the training dataset to minimize the following weighted score:

$\hat{f}_{r}(h,t)=\frac{1}{\sum_{i=1}^{k}w_{i}}\sum_{i=1}^{k}w_{i}f_{r}^{i}(h,t).$
(21)

The learnable weights are implemented as parameters in the optimization
process under the same learning rate and the optimizer in finding the best
variants.

For each relation, we compare the three weight schemes and choose the one that
offers the best performance.

####  III-C2 Rank Fusion Strategy

Link prediction is a list ranking problem. Rank fusion can be exploited to
boost the performance. A few simple rank fusion methods can be applied to
score-based KGE methods. For example, we can take the maximum, minimum,
median, sum, and L2 distance of candidates’ ranks. They are denoted by
CombMAX, CombMIN, CombMEDIAN, CombSUM, and Euclidean in Table I, respectively.
Three advanced rank fusion methods are also considered and included in the
table. They are: Borda Count [58], Reciprocal Rank Fusion (RRF) [53], and RBC
(Rank Biased Centroid) [59]. Borda Count awards points to candidates based on
their positions in an individual preference list, where the top candidate gets
the most points and the last candidate gets the least points. RRF aggregates
the reciprocal rank to discount the importance of lower-ranked candidates. The
factor $k$ in the table mitigates the impact of high rankings by outliers. RBC
discounts the weights of lower-ranked candidates using a geometric
distribution. The mathematical formulas of all rank fusion functions are given
in the second column of Table I, where $R_{i}$ is the rank of the $i$-th base
model (or variant), $1\leq i\leq n$, $e\in E$ represents an entity in the
entity set, and $k$ and $\phi$ are hyper-parameters.

TABLE I: A list of rank fusion functions under consideration.

Name Function CombMAX ${\max\\{R_{1}(e),\cdots,R_{n}(e)\\}}$ CombMIN
${\min\\{R_{1}(e),\cdots,R_{n}(e)\\}}$ CombMEDIAN
${\text{median}\\{R_{1}(e),\cdots,R_{n}(e)\\}}$ CombSUM
${\sum}_{i=1}^{n}R_{i}(e)$ Euclidean $\sqrt{R_{1}(e)^{2}+\cdots+R_{n}(e)^{2}}$
Borda Count ${\sum_{i=1}^{n}\frac{|E|-R_{i}(e)+1}{|E|}}$ RRF [53]
${\sum_{i=1}^{n}\frac{1}{k+R_{i}(e)}}$ RBC [59]
${\sum}_{i=1}^{n}(1-\phi)\phi^{R_{i}(e)-1}$

###  III-D Optimization

By following RotatE’s negative sampling loss and the self-adversarial training
strategy, we choose the following loss function of CompoundE3D

$\displaystyle L_{\textnormal{KGE}}=$
$\displaystyle-\log\sigma(\zeta_{1}-f_{r}(h,t))$
$\displaystyle-\sum_{i=1}^{n}p(h^{\prime}_{i},r,t^{\prime}_{i})\log\sigma(f_{r}(h^{\prime}_{i},t^{\prime}_{i})-\zeta_{1}),$

where $\sigma$ is the sigmoid function, $\zeta_{1}$ is a preset margin hyper-
parameter, $(h^{\prime}_{i},r,t^{\prime}_{i})$ is the $i$-th negative triple,
and $p(h^{\prime}_{i},r,t^{\prime}_{i})$ is the probability of drawing
negative triple $(h^{\prime}_{i},r,t^{\prime}_{i})$. Given a positive triple,
$(h_{i},r,t_{i})$, the negative sampling distribution is

$p(h^{\prime}_{j},r,t^{\prime}_{j}|\\{(h_{i},r,t_{i})\\})=\frac{\exp\alpha_{1}f_{r}(h^{\prime}_{j},t^{\prime}_{j})}{\sum_{i}\exp\alpha_{1}f_{r}(h^{\prime}_{i},t^{\prime}_{i})},$
(23)

where $\alpha_{1}$ is the temperature in the softmax function.

##  IV Experiments

TABLE II: Statistics of four link prediction datasets.

Dataset #Entities #Relations #Training #Validation #Test Ave. Degree DB100K
99,604 470 597,572 50,000 50,000 12 ogbl-wikikg2 2,500,604 535 16,109,182
429,456 598,543 12.2 YAGO3-10 123,182 37 1,079,040 5,000 5,000 9.6 WN18RR
40,943 11 86,835 3,034 3,134 2.19

TABLE III: The search space of six hyper-parameters. Dataset | DB100K | ogbl-wikikg2 | YAGO3-10 | WN18RR
---|---|---|---|---
Dim | $\\{150,300,450,600\\}$ | $\\{90,150,180,240,300\\}$ | $\\{450,600,750,900\\}$ | $\\{180,240,360,480,600\\}$
lr | $\\{2,3,4,5,6,7,8,9\\}\times 10^{-5}$ | $\\{0.0005,0.001,0.005,0.01\\}$ | $\\{3,4,5,6,7\\}\times 10^{-4}$ | $\\{4,5,6,7,8\\}\times 10^{-4}$
B | $\\{256,512,1024,2048\\}$ | $\\{2048,4096,8192\\}$ | $\\{512,1024,2048,4096\\}$ | $\\{512,1024,2048,4096\\}$
N | $\\{256,512,1024,2048\\}$ | $\\{125,250,500\\}$ | $\\{256,512,1024,2048\\}$ | $\\{256,512,1024,2048\\}$
$\boldsymbol{\zeta}$ | $\\{4,5,6,7,8,9,10,11,12,13\\}$ | $\\{5,6,7,8,9\\}$ | $\\{11,12,13,13.1,13.3,13.5\\}$ | $\\{5,6,7,8,9\\}$
$\boldsymbol{\alpha}$ | $\\{0.5,0.7,0.9,1.0,1.2\\}$ | $\\{0.5,1.0\\}$ | $\\{0.8,0.9,1.0,1.1,1.2\\}$ | $\\{0.5,0.7,0.9,1.0,1.2\\}$
TABLE IV: Optimal configurations for link prediction tasks, where B and N
denote the batch size and the negative sample size, respectively.

Dataset CompoundE3D Variant #Dim lr B N $\boldsymbol{\zeta}$
$\boldsymbol{\alpha}$ DB100K $\|\mathbf{S\cdot
h-\hat{T}\cdot\hat{R}\cdot\hat{S}\cdot t}\|$ 600 0.00005 1024 512 9 1 ogbl-
wikikg2 $\|\mathbf{T\cdot h-\hat{H}\cdot t}\|$ 300 0.001 8192 125 8 1 YAGO3-10
$\left\|\mathbf{T\cdot S\cdot R\cdot h-t}\right\|$ 600 0.0005 1024 1024 13.3
1.1 WN18RR $\|\mathbf{R\cdot S\cdot T\cdot h-t}\|$ 480 0.00005 512 256 6 1

###  IV-A Experimental Setup

####  IV-A1 Datasets

We evaluate the link prediction performance of CompoundE3D and compare it with
several benchmarking methods on the following four KG datasets.

  * •

DB100K [60]. It is a subset of the DBpedia KG. The dataset contains
information related to music content such as genre, band, and musical
artisits. It is a relatively dense KG since each entity appears in at least 20
different relations.

  * •

YAGO3-10 [61]. It is a subset of YAGO3, which describes citizenship, gender,
and profession of people. YGGO3-10 contains entities associated with at least
10 different relations.

  * •

WN18RR [16, 29]. It is a subset of the WordNet lexical database. The inverse
relation is removed from WN18RR to avoid test leakage.

  * •

Ogbl-Wikikg2 [62]. It is extracted from Wikipedia. It contains 2.5M entities
and is the largest one among the four selected datasets.

The statistics of the four KG datasets are given in Table II.

####  IV-A2 Evaluation Protocol

The commonly used evaluation protocol for the link prediction task is
explained below. For every triple $(h,r,t)$ in the test set, we corrupt either
the head entity $h$ or tail entity $t$ to generate test examples $(?,r,t)$ and
$(h,r,?)$. Then, for every head candidate that forms triple $(\hat{h},r,t)$
and tail candidate that forms triple $(h,r,\hat{t})$, we compute distance-
based scoring functions $f_{r}(\hat{h},t)$ and $f_{r}(h,\hat{t})$,
respectively. The lower score value indicates that the generated triple is
more likely to be true. Then, we sort scores of all candidate triples in
ascending order and locate the rank of the ground truth triple. Furthermore,
we evaluate the link prediction performance under the filtered rank setting
[16] that gives salience to unseen triple predictions since embedding models
tend to give observed triples better ranks. We adopt the Hits@$k$ and the mean
reciprocal rank (MRR) as evaluation metrics to compare the quality of KGE
models.

####  IV-A3 Hyper-parameter Search

We perform an extensive search on six hyper-parameters of CompoundE3D with
respect to different KG datasets. They are: 1) the dimension of the embedding
space (Dim), 2) the learning rate (lr), 3) the batch size (B), 4) the negative
sample size (N), 5) the margin hyper-parameter ($\boldsymbol{\zeta}$), and 6)
the sampling temperature ($\boldsymbol{\alpha}$). Their search values are
listed in Table III.

In the search process, we first compute scoring functions with a certain
hyper-parameter setting that allows a few variants to have decent performance,
where the number of training iterations for each variant is set to $l=30000$.
After locating the optimal variant, we finetune hyper-parameters under the
optimal variant. The optimal configurations are shown in Table IV. The Adam
optimizer [63] is employed for all parameter tuning. For ensemble experiments,
we adopt the same optimal configuration for each base variant model.

####  IV-A4 Other Implementation Details

We run experiments and perform hyper-parameter tuning on a variety of GPUs,
including Nvidia P100 (16G), V100 (32G), A100 (40G) and A40 (48G), depending
on the GPU memory requirement of a job. Typically, we request 8 CPU cores with
less than 70G RAM for each job. Results of each optimal configuration in Table
IV can be reproduced on one single V100 for all datasets. For the WN18RR
dataset, we adopt the rotation implementation from Rotate3D [21].

TABLE V: Comparison of the link prediction performance under the filtered rank
setting for DB100k.

Datasets DB100K Model MRR H@1 H@3 H@10 TransE [16] 0.111 0.016 0.164 0.27
DistMult [31] 0.233 0.115 0.301 0.448 HolE [64] 0.26 0.182 0.309 0.411 ComplEx
[65] 0.242 0.126 0.312 0.44 Analogy [66] 0.252 0.142 0.323 0.427 RUGE [67]
0.246 0.129 0.325 0.433 ComplEx-NNE+AER [60] 0.306 0.244 0.334 0.418 SEEK [68]
0.338 0.268 0.37 0.467 AcrE (Parallel) [69] 0.413 0.314 0.472 0.588 PairRE
[18] 0.412 0.309 0.472 0.600 TransSHER [70] 0.431 0.345 0.476 0.589 CompoundE
[20] 0.405 0.306 0.461 0.588 CompoundE3D 0.450 0.373 0.488 0.594 CompoundE3D
RRF 0.457 0.376 0.497 0.607 CompoundE3D WDS 0.462 0.378 0.506 0.616

TABLE VI: Comparison of the link prediction performance under the filtered
rank setting for ogbl-wikikg2.

Datasets ogbl-wikikg2 Metrics Dim Valid Test MRR MRR AutoSF+NodePiece 100
0.5806 0.5703 ComplEx-N3-RP 100 0.6701 0.6481 TransE [16] 500 0.4272 0.4256
DistMult [31] 500 0.3506 0.3729 ComplEx [65] 250 0.3759 0.4027 RotatE [17] 250
0.4353 0.4353 Rotate3D [21] 100 0.5685 0.5568 PairRE [18] 200 0.5423 0.5208
TripleRE [71] 200 0.6045 0.5794 CompoundE [20] 100 0.6704 0.6515 CompoundE3D
90 0.6994 0.6826 180 0.7146 0.6962 300 0.7175 0.7006

TABLE VII: Comparison of the link prediction performance under the filtered
rank setting for YAGO3-10.

Datasets YAGO3-10 Metrics MRR Hit@1 Hit@3 Hit@10 DistMult [31] 0.34 0.24 0.38
0.54 ComplEx [65] 0.36 0.26 0.4 0.55 DihEdral [72] 0.472 0.381 0.523 0.643
ConvE [29] 0.44 0.35 0.49 0.62 RotatE [17] 0.495 0.402 0.55 0.67 InteractE
[73] 0.541 0.462 - 0.687 HAKE [74] 0.545 0.462 0.596 0.694 DensE [23] 0.541
0.465 0.585 0.678 Rot-Pro [75] 0.542 0.443 0.596 0.699 CompoundE [20] 0.477
0.376 0.538 0.664 CompoundE3D 0.542 0.450 0.602 0.701 CompoundE3D RRF 0.541
0.446 0.607 0.707 CompoundE3D WDS 0.551 0.463 0.608 0.703

TABLE VIII: Comparison of different weighted-distances-sum (WDS) strategies
for DB100K and YAGO3-10.

Datasets DB100K YAGO3-10 Strategies MRR H@1 H@3 H@10 MRR H@1 H@3 H@10
Learnable Weights 0.462 0.378 0.506 0.616 0.545 0.451 0.586 0.696 Uniform
Weights 0.460 0.376 0.503 0.614 0.551 0.463 0.608 0.703 Geometric Weights
0.446 0.348 0.503 0.618 0.531 0.439 0.580 0.691

TABLE IX: Performance comparison of different rank fusion methods for DB100K
and YAGO3-10.

Datasets DB100K YAGO3-10 Aggregation Function MRR Hit@1 Hit@3 Hit@10 MRR Hit@1
Hit@3 Hit@10 CombMAX 0.455 0.375 0.496 0.603 0.536 0.440 0.600 0.701 CombMIN
0.452 0.369 0.4955 0.606 0.527 0.427 0.597 0.702 CombMEDIAN 0.456 0.376 0.497
0.606 0.541 0.445 0.605 0.705 CombSUM 0.456 0.376 0.4969 0.6060 0.540 0.446
0.606 0.704 Euclidean 0.455 0.375 0.496 0.605 0.540 0.445 0.603 0.702 Borda
0.456 0.376 0.497 0.606 0.540 0.446 0.606 0.704 RRF 0.457 0.376 0.497 0.607
0.541 0.446 0.607 0.707 RBC 0.456 0.376 0.497 0.606 0.540 0.445 0.604 0.703

TABLE X: Ablation study on CompoundE3D for DB100K.

Datasets DB100K Variant MRR Hit@1 Hit@3 Hit@10 $\|\mathbf{S\cdot
h-\hat{S}\cdot t}\|$ 0.417 0.323 0.471 0.590 $\|\mathbf{S\cdot
h-\hat{R}\cdot\hat{S}\cdot t}\|$ 0.447 0.364 0.492 0.600 $\|\mathbf{S\cdot
h-\hat{T}\cdot\hat{R}\cdot\hat{S}\cdot t}\|$ 0.450 0.373 0.488 0.594

TABLE XI: Ablation study on CompoundE3D for YAGO3-10.

Datasets YAGO3-10 Metrics MRR Hit@1 Hit@3 Hit@10 $\left\|\mathbf{R\cdot
h-t}\right\|$ 0.496 0.402 0.547 0.676 $\left\|\mathbf{S\cdot R\cdot
h-t}\right\|$ 0.501 0.404 0.554 0.680 $\left\|\mathbf{T\cdot S\cdot R\cdot
h-t}\right\|$ 0.542 0.450 0.602 0.701

###  IV-B Experimental Results

####  IV-B1 Performance Evalution

We compare the link prediction performance of a few benchmarking KGE methods
with that of CompoundE3D using the optimal configuration given in Table IV.
The performance benchmarking results for DB100K, ogbl-wikikg2, and YAGO3-10
datasets are shown, respectively, in Table V, Table VI, and Table VII.
Furthermore, the best and the second-best results in each column are indicated
by the boldface font and with an underline, respectively. CompoundE3D has
significant performance improvement over CompoundE and other recent models. We
see a clear advantage of CompoundE3D by including more affine operators and
extending affine transformations from 2D to 3D in the new framework,

To verify the effectiveness of model ensembles, we examine two different
ensemble strategies for DB100K and YAGO3-10 datasets.

  * •

For the DB100K dataset, we select the best two performing variants. They are
$\|\mathbf{S\cdot h-\hat{R}\cdot\hat{S}\cdot t}\|$ and $\|\mathbf{S\cdot
h-\hat{T}\cdot\hat{R}\cdot\hat{S}\cdot t}\|$

  * •

For the YAGO3-10 dataset, we select the best three performing variants. They
are $\left\|\mathbf{T\cdot S\cdot R\cdot h-t}\right\|$, $\left\|\mathbf{S\cdot
R\cdot T\cdot h-t}\right\|$, and $\left\|\mathbf{S\cdot T\cdot R\cdot
h-t}\right\|$.

####  IV-B2 Model Ensembles

As discussed in Sec. III-C, we have two strategies to conduct model ensembles:
weighted-distances-sum (WDS) and rank fusion. Among the three WDS strategies,
the learnable weight strategy is the most effective one for DB100K while the
uniform weight performs the best for YAGO3-10. We use CompoundE3D WDS to
denote the best WDS scheme in Tables V and VII and document the performance of
other weighting strategies in Table VIII. Among all eight rank fusion
strategies, we observe that reciprocal rank fusion (RRF) is the most effective
one for both DB100K and YAGO3-10. Thus, we use CompoundE3D RRF to denote the
best rank fustion scheme in Tables V and VII, and document the performance of
other rank fushion strategies in Table IX.

####  IV-B3 Effectiveness of Beam Search

Figure 3: The distribution of the MRR performance versus the operator number
of various model variants for the DB100K dataset. Figure 4: The distribution
of the MRR performance versus the operator number of various model variants
for the YAGO3-10 dataset.

We conduct ablation studies on DB100K and YAGO3-10 datasets to shed light on
the effects of different transformation operators on model performance. We
begin with the variant of the simplest configuration and add additional
operators at each stage. Good simple models that lead to optimal variants and
their performance numbers are reported in Tables X and XI. Furthermore, we
visualize the distribution of the MRR performance as more operators are added
with respect to DB100K and YAGO-3 in Figs. 3 and 4, respectively. To interpret
box plots, yellow bar represents the median, box represents the interquantile
range, two end-bars denote the lower and upper whiskers, and lastly dots are
outliers. They both show the effectiveness of the proposed beam search
algorithm.

Figure 5: Effects of rotation and reflection operators on symmetric relations.
Figure 6: Illustration of CompoundE3D’s capability in multiplicity modeling.
Figure 7: Comparing different model’s MRR performance metric across different
dimensions.

TABLE XII: Comparison of filtered MRR performance on each relation type of
WN18RR.

Relations Types $\text{Khs}_{G_{r}}$ $\xi_{G_{r}}$ TransE RotatE CompoundE
CompoundE3D similar to 1-to-1 0.07 -1.00 0.294 1.000 1.000 1.000 verb group
1-to-1 0.07 -0.50 0.363 0.961 0.974 0.898 member meronym 1-to-N 1.00 -0.50
0.179 0.259 0.230 0.246 has part 1-to-N 1.00 -1.43 0.117 0.200 0.190 0.202
member of domain usage 1-to-N 1.00 -0.74 0.113 0.297 0.332 0.378 member of
domain region 1-to-N 1.00 -0.78 0.114 0.217 0.280 0.413 hypernym N-to-1 1.00
-2.64 0.059 0.156 0.155 0.182 instance hypernym N-to-1 1.00 -0.82 0.289 0.322
0.337 0.356 synset domain topic of N-to-1 0.99 -0.69 0.149 0.339 0.367 0.396
also see N-to-N 0.36 -2.09 0.227 0.625 0.629 0.622 derivationally related form
N-to-N 0.07 -3.84 0.440 0.957 0.956 0.959

TABLE XIII: Complexity comparison of KGE models on ogbl-wikikg2 under a
similar testing MRR.

Models No. of Parameters TransE [16] 1,251M DistMult [31] 1,251M ComplEx [65]
1,251M ComplEx-RP [76] 250.1M RotatE [17] 500M RotatE3D [21] 750.4M PairRE
[18] 500M CompoundE [20] 250.1M CompoundE3D 225.2M

####  IV-B4 Modeling of Symmetric Relations

Rotation and reflection are isometric operations. As stated in [17, 50], their
2D versions can handle symmetric relations well in some cases. It is our
conjecture that the same property holds for their corresponding 3D operators.
To check it, we perform ablation studies and evaluate the base scoring
functions of those with only translation and scaling versus those with
rotation and reflection as well. The MRR performance numbers of different
model variants for symmetric and asymmetric relations in DB100K are compared
in Fig. 5. In this figure, we choose the most frequently observed relation
types for meaningful comparison. As expected, rotation and reflection
operators indeed bring more significant performance improvement on symmetric
relations than asymmetric relations. This supports our conjecture that
rotation and reflection operators are intrinsically advantageous for the
modeling of symmetric relations.

####  IV-B5 Modeling of Multiplicity

Multiplicity is the scenario where multiple relations co-exist between two
entities; namely, triples $(h,r_{1},t),\dots,(h,r_{n},t)$ hold simultaneously.
Generally, it is challenging to model multiplicity in traditional KGE models
due to their limited power in relational modeling. In contrast, CompoundE3D is
capable of modeling multiplicity relation patterns well since it can use
multiple distinct sets of transformations that map from the head to the tail.
We present two examples to illustrate CompoundE3D’s capability in modeling
multiplicity relations in Fig. 6. They are taken from the actual link
prediction examples in DB100K. There are three different relations held for a
fixed (head, tail) pair. The top three tail predictions for each relation in
the two examples are shown in the figure. We see that CompoundE3D can handle
multiplicity well due to its rich set of variants.

####  IV-B6 Modeling of Hierarchical Relations

We would like to investigate CompoundE3D’s capability in modeling hierarchical
relations. WN18RR offers a representative dataset containing hierarchical
relations. Two metrics can be used to measure the hierarchical behavior of
relations [25]: 1) the Krackhardt score denoted by $\text{Khs}_{G_{r}}$, and
2) the curvature estimate denoted by $\xi_{G_{r}}$. If relation $r$ has a high
$\text{Khs}_{G_{r}}$ score and a low $\xi_{G_{r}}$ score, then it has a
stronger hierarchical behavior, and vice versa. We compare the filtered MRR
performance of different baseline models, such as TransE, RotatE, CompoundE
(2D version), and CompoundE3D in Table XII. In the same table, we also list
the $\text{Khs}_{G_{r}}$ and $\xi_{G_{r}}$ values for each relation to see
whether it has a stronger hierarchical behavior. We see from the table that
CompoundE and CompoundE3D have better performance than TransE and RotatE in
almost all relations. Furthermore, CompoundE3D outperforms CompoundE in all
hierarchical relations except “member meronym”. This result indicates that
CompoundE3D can model hierarchical relations more effectively than CompoundE
by including more diverse 3D transformations.

####  IV-B7 Model Efficiency

It is important to investigate the relationship between the model performance
and the model dimension. The model dimension reflects memory and computational
complexities. To illustrate the advantage of CompoundE3D over prior models
across a wide range of embedding dimensions, we plot the MRR performance of
link prediction on the Wikikg2 dataset in Fig. 7, where the dimension values
are set to 12, 24, 48, 102, 150, 198, 252, and 300. We see from the figure
that CompoundE3D consistently outperforms all benchmarking models in all
dimensions. Furthermore, we analyze the complexity of different KGE models in
terms of the number of free parameters. Table XIII compares the number of free
parameters of different KGE models for the ogbl-wikikg2 dataset.

We refer to the ogbl-wikikg2 leaderboard when reporting the number of free
parameters used by baseline models. The reported number of parameters for
CompoundE3D is when the embedding dimension is set to 90. As shown in Fig. 7
and Table XIII, CompoundE3D offers the best performance among all benchmarking
models while having the smallest number of free parameters.

##  V Conclusion and Future Work

A novel and effective KGE model based on composite affine transformations in
the 3D space, named CompoundE3D, was proposed in this work. A beam search
procedure was devised to build a desired KGE from the simplest configuration
to more complicated ones. The ensemble of the top-$k$ model variants was also
explored to further boost link prediction performance. Extensive experimental
results were provided to demonstrate the superior performance of CompoundE3D.
We conducted ablation studies to assess the effect of each operator and
performed case studies to shed light on the modeling power of CompoundE3D for
several relation types such as multiplicity, symmetric relations, and
hierarchical relations.

As to future research directions, it will be interesting to explore the
effectiveness of CompoundE3D in other important KG problems such as entity
typing [77] and entity alignment [78]. Besides, research on performance
boosting in low-dimensional embedding space is valuable in practical real-
world applications and worth further investigation.

## Acknowledgment

The authors acknowledge the Center for Advanced Research Computing (CARC) at
the University of Southern California for providing computing resources that
have contributed to the research results reported within this publication.
URL: https://carc.usc.edu.

## References

  * [1] T. Yu, J. Li, Q. Yu, Y. Tian, X. Shun, L. Xu, L. Zhu, and H. Gao, “Knowledge graph for tcm health preservation: Design, construction, and applications,” _Artif. Intell. Med._ , vol. 77, pp. 48–52, 2017.
  * [2] S. Sang, Z. Yang, L. Wang, X. Liu, H. Lin, and J. Wang, “Sematyp: a knowledge graph based literature mining method for drug discovery,” _BMC Bioinform._ , vol. 19, pp. 1–11, 2018.
  * [3] X. Zeng, X. Tu, Y. Liu, X. Fu, and Y. Su, “Toward better drug discovery with knowledge graph,” _Curr. Opin. Struct. Biol._ , vol. 72, pp. 114–126, 2022\.
  * [4] H. Wang, F. Zhang, J. Wang, M. Zhao, W. Li, X. Xie, and M. Guo, “Ripplenet: Propagating user preferences on the knowledge graph for recommender systems,” in _Proc. 27th ACM Int. Conf. Inf. Knowl. Manage. (CIKM’18)_ , 2018, pp. 417–426.
  * [5] K. Zhou, W. X. Zhao, S. Bian, Y. Zhou, J.-R. Wen, and J. Yu, “Improving conversational recommender systems via knowledge graph based semantic fusion,” in _Proc. 26th ACM SIGKDD Int. Conf. Knowl. Discov. Data Min. (KDD ’20)_ , 2020, pp. 1006–1014.
  * [6] Q. Zhan and H. Yin, “A loan application fraud detection method based on knowledge graph and neural network,” in _Proc. 2nd Int. Conf. Innov. Artif. Intell._ , 2018, pp. 111–115.
  * [7] X. Zhu, X. Ao, Z. Qin, Y. Chang, Y. Liu, Q. He, and J. Li, “Intelligent financial fraud detection practices in post-pandemic era,” _The Innov._ , vol. 2, no. 4, p. 100176, 2021.
  * [8] H. He, A. Balakrishnan, M. Eric, and P. Liang, “Learning symmetric collaborative dialogue agents with dynamic knowledge graph embeddings,” in _Proc. 55th Annu. Meet. Assoc. Comput. Linguist. (ACL 2017)_ , 2017, pp. 1766–1776.
  * [9] A. Ait-Mlouk and L. Jiang, “Kbot: A knowledge graph based chatbot for natural language understanding over linked data,” _IEEE Access_ , vol. 8, pp. 149 220–149 230, 2020.
  * [10] S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, and Z. Ives, “DBpedia: A nucleus for a web of open data,” in _The Semantic Web_. Springer, 2007, pp. 722–735.
  * [11] F. M. Suchanek, G. Kasneci, and G. Weikum, “YAGO: A core of semantic knowledge,” in _Proc. 16th Int. Conf. World Wide Web (WWW’07)_ , 2007, pp. 697–706.
  * [12] K. Bollacker, C. Evans, P. Paritosh, T. Sturge, and J. Taylor, “Freebase: a collaboratively created graph database for structuring human knowledge,” in _Proc. 2008 ACM SIGMOD Int. Conf. Manage. Data (SIGMOD ’08)_ , 2008, pp. 1247–1250.
  * [13] A. Carlson, J. Betteridge, B. Kisiel, B. Settles, E. R. Hruschka, and T. M. Mitchell, “Toward an architecture for never-ending language learning,” in _Proc. 24th AAAI Conf. Artif. Intell._ , 2010, p. 1306–1313.
  * [14] D. Vrandečić and M. Krötzsch, “Wikidata: a free collaborative knowledge base,” _Commun. ACM_ , vol. 57, no. 10, pp. 78–85, 2014.
  * [15] R. Speer, J. Chin, and C. Havasi, “ConceptNet 5.5: An open multilingual graph of general knowledge,” in _Proc. 31st AAAI Conf. Artif. Intell._ , 2017, p. 4444–4451.
  * [16] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, and O. Yakhnenko, “Translating embeddings for modeling multi-relational data,” _Adv. Neural Info. Process. Syst. 26 (NeurIPS 2013)_ , p. 2787–2795, 2013.
  * [17] Z. Sun, Z.-H. Deng, J.-Y. Nie, and J. Tang, “RotatE: Knowledge graph embedding by relational rotation in complex space,” in _Proc. 8th Int. Conf. Learn. Represent. (ICLR)_ , 2019, p. 1–18.
  * [18] L. Chao, J. He, T. Wang, and W. Chu, “PairRE: Knowledge graph embeddings via paired relation vectors,” in _Proc. 59th Annu. Meet. Assoc. Comput. Linguist. (ACL 2021)_ , 2021, pp. 4360–4369.
  * [19] W. K. Pratt, _Introduction to digital image processing_. CRC press, 2013.
  * [20] X. Ge, Y.-C. Wang, B. Wang, and C.-C. J. Kuo, “CompoundE: Knowledge graph embedding with translation, rotation and scaling compound operations,” _arXiv preprint arXiv:2207.05324_ , 2022.
  * [21] C. Gao, C. Sun, L. Shan, L. Lin, and M. Wang, “Rotate3D: Representing relations as rotations in three-dimensional space for knowledge graph embedding,” in _Proc. 29th ACM Int. Conf. Inf. Knowl. Manage. (CIKM’20)_ , 2020, pp. 385–394.
  * [22] T. Yang, L. Sha, and P. Hong, “Nage: Non-Abelian group embedding for knowledge graphs,” in _Proc. 29th ACM Int. Conf. Inf. Knowl. Manage. (CIKM’20)_ , 2020, pp. 1735–1742.
  * [23] H. Lu, H. Hu, and X. Lin, “DensE: An enhanced non-commutative representation for knowledge graph embedding with adaptive semantic hierarchy,” _Neurocomputing_ , vol. 476, pp. 115–125, 2022.
  * [24] S. Zhang, Y. Tay, L. Yao, and Q. Liu, “Quaternion knowledge graph embedding,” in _Adv. Neural Info. Process. Syst. 32 (NeurIPS 2019)_ , 2019, pp. 2731–2741.
  * [25] I. Chami, A. Wolf, D.-C. Juan, F. Sala, S. Ravi, and C. Ré, “Low-dimensional hyperbolic knowledge graph embeddings,” in _Proc. 58th Annu. Meet. Assoc. Comput. Linguist. (ACL 2020)_ , 2020, pp. 6901–6914.
  * [26] K. Wang, Y. Liu, D. Lin, and Q. Z. Sheng, “Hyperbolic geometry is not necessary: Lightweight euclidean-based models for low-dimensional knowledge graph embeddings,” _arXiv preprint arXiv:2103.14930_ , 2021.
  * [27] X. Dong, E. Gabrilovich, G. Heitz, W. Horn, N. Lao, K. Murphy, T. Strohmann, S. Sun, and W. Zhang, “Knowledge vault: A web-scale approach to probabilistic knowledge fusion,” in _Proc. 20th ACM SIGKDD Int. Conf. Knowl. Discov. Data Min. (KDD ’14)_ , 2014, pp. 601–610.
  * [28] R. Socher, D. Chen, C. D. Manning, and A. Ng, “Reasoning with neural tensor networks for knowledge base completion,” _Adv. Neural Info. Process. Syst. 26 (NeurIPS 2013)_ , vol. 26, 2013.
  * [29] T. Dettmers, P. Minervini, P. Stenetorp, and S. Riedel, “Convolutional 2d knowledge graph embeddings,” in _Proc. 32nd AAAI Conf. Artif. Intell._ , 2018, pp. 1811–1818.
  * [30] M. Schlichtkrull, T. N. Kipf, P. Bloem, R. Van Den Berg, I. Titov, and M. Welling, “Modeling relational data with graph convolutional networks,” in _Proc. 15th Extended Semantic Web Conf. (ESWC’18)_. Springer, 2018, pp. 593–607.
  * [31] B. Yang, W.-t. Yih, X. He, J. Gao, and L. Deng, “Embedding entities and relations for learning and inference in knowledge bases,” in _Proc. 4th Int. Conf. Learn. Represent. (ICLR)_ , 2015, pp. 1–13.
  * [32] L. Yao, C. Mao, and Y. Luo, “KG-BERT: Bert for knowledge graph completion,” _arXiv preprint arXiv:1909.03193_ , 2019.
  * [33] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [34] L. Wang, W. Zhao, Z. Wei, and J. Liu, “SimKGC: Simple contrastive knowledge graph completion with pre-trained language models,” in _Proc. 60th Annu. Meet. Assoc. Comput. Linguist. (ACL 2022)_ , 2022, pp. 4281–4294.
  * [35] Y.-C. Wang, X. Ge, B. Wang, and C.-C. J. Kuo, “KGBoost: A classification-based knowledge base completion method with negative sampling,” _Pattern Recognit. Lett._ , vol. 157, pp. 104–111, 2022.
  * [36] T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” in _Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discov. Data Min. (KDD ’16)_ , 2016, pp. 785–794.
  * [37] Y. Yang, W. Wang, H. Fu, C.-C. J. Kuo _et al._ , “On supervised feature selection from high dimensional feature spaces,” _APSIPA Trans. Signal Inf. Process._ , vol. 11, no. 1, 2022.
  * [38] C.-C. J. Kuo and A. M. Madni, “Green learning: Introduction, examples and outlook,” _J. Vis. Commun. Image. Represent._ , p. 103685, 2022.
  * [39] Y.-C. Wang, X. Ge, B. Wang, and C.-C. J. Kuo, “Greenkgc: A lightweight knowledge graph completion method,” _arXiv preprint arXiv:2208.09137_ , 2022\.
  * [40] Z. Wang, J. Zhang, J. Feng, and Z. Chen, “Knowledge graph embedding by translating on hyperplanes,” in _Proc. 28th AAAI Conf. Artif. Intell._ , 2014, pp. 1112–1119.
  * [41] Y. Lin, Z. Liu, M. Sun, Y. Liu, and X. Zhu, “Learning entity and relation embeddings for knowledge graph completion,” in _Proc. 29th AAAI Conf. Artif. Intell._ , 2015, pp. 2181–2187.
  * [42] G. Ji, S. He, L. Xu, K. Liu, and J. Zhao, “Knowledge graph embedding via dynamic mapping matrix,” in _Proc. 53rd Annu. Meet. Assoc. Comput. Linguist. (ACL 2015)_ , vol. 1, 2015, pp. 687–696.
  * [43] G. Ji, K. Liu, S. He, and J. Zhao, “Knowledge graph completion with adaptive sparse transfer matrix,” in _Proc. 30st AAAI Conf. Artif. Intell._ , 2016, pp. 985–991.
  * [44] J. Yang, Y. Shi, X. Tong, R. Wang, T. Chen, and X. Ying, “Improving knowledge graph embedding using affine transformations of entities corresponding to each relation,” in _Proc. Conf. Empirical Meth. Natural Language Process. Find. (EMNLP’21 Findings)_ , 2021, pp. 508–517.
  * [45] Z. Liang, J. Yang, H. Liu, and K. Huang, “A semantic filter based on relations for knowledge graph completion,” in _Proc. Conf. Empirical Meth. Natural Language Process. (EMNLP’21)_ , 2021, pp. 7920–7929.
  * [46] J. Li and Y. Yang, “STaR: Knowledge graph embedding by scaling, translation and rotation,” in _Proc. 11th Artif. Intell. Mob. Serv. (AIMS 2022)_. Springer, 2022, pp. 31–45.
  * [47] Q. Wang, Z. Mao, B. Wang, and L. Guo, “Knowledge graph embedding: A survey of approaches and applications,” _IEEE Trans. Knowl. Data Eng._ , vol. 29, no. 12, pp. 2724–2743, 2017.
  * [48] Z. Cao, Q. Xu, Z. Yang, X. Cao, and Q. Huang, “Dual quaternion knowledge graph embeddings,” in _Proc. 35th AAAI Conf. Artif. Intell._ , vol. 35, no. 8, 2021, pp. 6894–6902.
  * [49] J. Guo and S. Kok, “Bique: Biquaternionic embeddings of knowledge graphs,” in _Proc. Conf. Empirical Meth. Natural Language Process. (EMNLP’21)_ , 2021, pp. 8338–8351.
  * [50] Q. Zhang, R. Wang, J. Yang, and L. Xue, “Knowledge graph embedding by reflection transformation,” _Knowl.-Based Syst._ , vol. 238, p. 107861, 2022\.
  * [51] Z. Cao, T. Qin, T.-Y. Liu, M.-F. Tsai, and H. Li, “Learning to rank: from pairwise approach to listwise approach,” in _Proc. 24th Int. Conf. Mach. Learn. (ICML 2007)_ , 2007, pp. 129–136.
  * [52] A. Klementiev, D. Roth, and K. Small, “Unsupervised rank aggregation with distance-based models,” in _Proc. 25th Int. Conf. Mach. Learn. (ICML 2008)_ , 2008, pp. 472–479.
  * [53] G. V. Cormack, C. L. Clarke, and S. Buettcher, “Reciprocal rank fusion outperforms condorcet and individual rank learning methods,” in _Proc. 32nd Annu. Int. ACM SIGIR Conf. Res. Dev. Inf. Retrieval (SIGIR ’09)_ , 2009, pp. 758–759.
  * [54] X. Chen, M. Chen, C. Fan, A. Uppunda, Y. Sun, and C. Zaniolo, “Multilingual knowledge graph completion via ensemble knowledge transfer,” in _Proc. Conf. Empirical Meth. Natural Language Process. Find. (EMNLP’20 Findings)_ , 2020, pp. 3227–3238.
  * [55] Y. Zhang, Q. Yao, W. Dai, and L. Chen, “Autosf: Searching scoring functions for knowledge graph embedding,” in _Proc. 36th IEEE Int. Conf. Data Eng. (ICDE 2020)_ , 2020, pp. 433–444.
  * [56] C. Xu, M. Nayyeri, S. Vahdati, and J. Lehmann, “Multiple run ensemble learning with low-dimensional knowledge graph embeddings,” in _Proc. 2021 Int. Joint Conf. Neural Netw. (IJCNN)_. IEEE, 2021, pp. 1–8.
  * [57] U. Joshi and J. Urbani, “Ensemble-based fact classification with knowledge graph embeddings,” in _Proc. 19th Extended Semantic Web Conf. (ESWC’22)_. Springer, 2022, pp. 147–164.
  * [58] M. Van Erp and L. Schomaker, “Variants of the borda count method for combining ranked classifier hypotheses,” in _7th International Workshop on frontiers in handwriting recognition_. International Unipen Foundation, 2000, pp. 443–452.
  * [59] P. Bailey, A. Moffat, F. Scholer, and P. Thomas, “Retrieval consistency in the presence of query variations,” in _Proc. 40th Annu. Int. ACM SIGIR Conf. Res. Dev. Inf. Retrieval (SIGIR ’17)_ , 2017, pp. 395–404.
  * [60] B. Ding, Q. Wang, B. Wang, and L. Guo, “Improving knowledge graph embedding using simple constraints,” _arXiv preprint arXiv:1805.02408_ , 2018.
  * [61] F. Mahdisoltani, J. Biega, and F. Suchanek, “Yago3: A knowledge base from multilingual wikipedias,” in _Proc. 7th Biennial Conf. Innov. Data Syst. Research_. CIDR Conference, 2014\.
  * [62] W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec, “Open graph benchmark: Datasets for machine learning on graphs,” _arXiv preprint arXiv:2005.00687_ , 2020.
  * [63] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [64] Nickel, M. others, L. Rosasco, and T. Poggio, “Holographic embeddings of knowledge graphs,” in _Proc. 30st AAAI Conf. Artif. Intell._ , 2016.
  * [65] T. Trouillon, J. Welbl, S. Riedel, É. Gaussier, and G. Bouchard, “Complex embeddings for simple link prediction,” in _Proc. 33rd Int. Conf. Mach. Learn. (ICML 2016)_ , 2016, pp. 2071–2080.
  * [66] H. Liu, Y. Wu, and Y. Yang, “Analogical inference for multi-relational embeddings,” in _Proc. 34th Int. Conf. Mach. Learn. (ICML 2017)_ , 2017, pp. 2168–2178.
  * [67] S. Guo, Q. Wang, L. Wang, B. Wang, and L. Guo, “Knowledge graph embedding with iterative guidance from soft rules,” in _Proc. 32nd AAAI Conf. Artif. Intell._ , vol. 32, no. 1, 2018.
  * [68] W. Xu, S. Zheng, L. He, B. Shao, J. Yin, and T.-Y. Liu, “SEEK: Segmented embedding of knowledge graphs,” _arXiv preprint arXiv:2005.00856_ , 2020\.
  * [69] F. Ren, J. Li, H. Zhang, S. Liu, B. Li, R. Ming, and Y. Bai, “Knowledge graph embedding with atrous convolution and residual learning,” in _Proc. 58th Annu. Meet. Assoc. Comput. Linguist. (ACL 2020)_ , 2020, pp. 1532–1543.
  * [70] Y. Li, W. Fan, C. Liu, C. Lin, and J. Qian, “TranSHER: Translating knowledge graph embedding with hyper-ellipsoidal restriction,” in _Proc. 60th Annu. Meet. Assoc. Comput. Linguist. (ACL 2022)_ , Dec. 2022, pp. 8517–8528.
  * [71] L. Yu, Z. Luo, H. Liu, D. Lin, H. Li, and Y. Deng, “Triplere: Knowledge graph embeddings via tripled relation vectors,” _arXiv preprint arXiv:2209.08271_ , 2022.
  * [72] C. Xu and R. Li, “Relation embedding with dihedral group in knowledge graph,” in _Proc. 57th Annu. Meet. Assoc. Comput. Linguist. (ACL 2019)_ , 2019, pp. 263–272.
  * [73] S. Vashishth, S. Sanyal, V. Nitin, N. Agrawal, and P. Talukdar, “Interacte: Improving convolution-based knowledge graph embeddings by increasing feature interactions,” in _Proc. 34th AAAI Conf. Artif. Intell._ , vol. 34, 2020, pp. 3009–3016.
  * [74] Z. Zhang, J. Cai, Y. Zhang, and J. Wang, “Learning hierarchy-aware knowledge graph embeddings for link prediction,” in _Proc. 34th AAAI Conf. Artif. Intell._ , vol. 34, 2020, pp. 3065–3072.
  * [75] T. Song, J. Luo, and L. Huang, “Rot-pro: Modeling transitivity by projection in knowledge graph embedding,” _Adv. Neural Info. Process. Syst. 34 (NeurIPS 2021)_ , vol. 34, pp. 24 695–24 706, 2021.
  * [76] Y. Chen, P. Minervini, S. Riedel, and P. Stenetorp, “Relation prediction as an auxiliary training objective for improving multi-relational graph representations,” in _Proc. 3rd Conf. Autom. Knowl. Base Constr. (AKBC2021)_ , 2021.
  * [77] X. Ge, Y.-C. Wang, B. Wang, and C. J. Kuo, “CORE: A knowledge graph entity type prediction method via complex space regression and embedding,” _Pattern Recognit. Lett._ , vol. 157, pp. 97–103, 2022.
  * [78] X. Ge, Y. C. Wang, B. Wang, C.-C. J. Kuo _et al._ , “Typeea: Type-associated embedding for knowledge graph entity alignment,” _APSIPA Trans. Signal Inf. Process._ , vol. 12, no. 1, 2023.

|  Xiou Ge (S’14) received the B.S. and M.S. degree in electrical and computer
engineering from University of Illinois, Urbana–Champaign, Urbana, Illinois,
in 2016 and 2018 respectively. He is working towards Ph.D. degree in
electrical and computer engineering with the Ming-Hsieh Department of
Electrical and Computer Engineering, University of Southern California. His
research interest includes knowledge graph embedding, node classification,
graph alignment, natural language processing.
---|---
|  Yun-Cheng Wang (S’19) received the B.S. degree in electrical engineering
from National Taiwan University, Taipei, Taiwan, in 2018. He received his M.S.
degree in 2020 and is working towards Ph.D. degree in electrical and computer
engineering with the Ming-Hsieh Department of Electrical and Computer
Engineering, University of Southern California. His research interest includes
knowledge graph embedding, natural language processing.
---|---
|  Bin Wang (S’19-M’22) received his B.Eng. degree in electronic information
engineering from the University of Electronic Science and Technology of China
(UESTC), Chengdu, China, in June 2017, and his Ph.D. degree in electrical
engineering from the University of Southern California (USC), Los Angeles, CA,
USA, in May 2021. He is currently a Research Fellow with the National
University of Singapore (NUS), Singapore. His research interests include
representation learning, graph learning, and dialogue summarization.
---|---
|  C.-C. Jay Kuo (F’99) received the B.S. degree in electrical engineering
from National Taiwan University, Taipei, Taiwan, in 1980, and the M.S. and
Ph.D. degrees in electrical engineering from the Massachusetts Institute of
Technology, Cambridge, in 1985 and 1987, respectively. He is the holder of the
William M. Hogue Professorship in Electrical and Computer Engineering, a
Distinguished Professor of Electrical and Computer Engineering and Computer
Science, and the Director of the USC Multimedia Communication Laboratory (MCL)
at the University of Southern California. He is the co-author of about 340
journal papers, 1000 conference papers, and 15 books. His research interests
include multimedia and green computing. He is a fellow of the National Academy
of Inventors, the American Association for the Advancement of Science, and the
International Society for Optical Engineers, and the Association for Computing
Machinery.
---|---