#  Universal Multi-modal Entity Alignment via Iteratively Fusing Modality
Similarity Paths

Bolin Zhu1, Xiaoze Liu2, Xin Mao3, Zhuo Chen4, Lingbing Guo4, Tao Gui1, Qi
Zhang1 1Fudan University2Purdue University3Nanyang Technological
University4Zhejiang University blzhu20, tgui, qzfudan.edu.cn,
<EMAIL_ADDRESS><EMAIL_ADDRESS>zhuo.chen<EMAIL_ADDRESS>
(2023)

###### Abstract.

The objective of Entity Alignment (EA) is to identify equivalent entity pairs
from multiple Knowledge Graphs (KGs) and create a more comprehensive and
unified KG. The majority of EA methods have primarily focused on the
structural modality of KGs, lacking exploration of multi-modal information. A
few multi-modal EA methods have made good attempts in this field. Still, they
have two shortcomings: (1) inconsistent and inefficient modality modeling that
designs complex and distinct models for each modality; (2) ineffective
modality fusion due to the heterogeneous nature of modalities in EA.

To tackle these challenges, we propose PathFusion, consisting of two main
components: (1) MSP, a unified modeling approach that simplifies the alignment
process by constructing paths connecting entities and modality nodes to
represent multiple modalities; (2) IRF, an iterative fusion method that
effectively combines information from different modalities using the path as
an information carrier. Experimental results on real-world datasets
demonstrate the superiority of PathFusion over state-of-the-art methods, with
$22.4\%-28.9\%$ absolute improvement on Hits@1, and $0.194-0.245$ absolute
improvement on MRR.

Entity Alignment, Knowledge Graphs, Unsupervised Learning

††copyright: acmcopyright††journalyear: 2023††copyright: acmlicensed††ccs:
Computing methodologies Knowledge representation and reasoning††ccs: Computing
methodologies Semantic networks

##  1\. Introduction

Knowledge graphs (KGs) are valuable representations of structured knowledge
pertaining to real-world objects, and they play a crucial role in various
practical applications, including question answering (Cao et al., 2023),
entity (Zhang et al., 2021; Jiang et al., 2022) and relation extraction (Hu et
al., 2021). However, existing KGs often suffer from significant incompleteness
(Sun et al., 2020c) due to their construction from diverse data sources,
resulting in overlapping entities across different KGs. This situation
presents an opportunity to integrate multiple KGs by considering the
overlapped entities.

A commonly employed strategy for KG integration is known as Entity Alignment
(EA)(Sun et al., 2020c), which aims to align entities from different KGs that
refer to the same real-world objects. Given two KGs and a small set of pre-
aligned entities (referred to as _seed alignment_), EA identifies all possible
alignments between the KGs. Recent studies based on embedding techniques(Li et
al., 2019; Mao et al., 2020b; Sun et al., 2020b, a; Mao et al., 2021a) have
demonstrated high effectiveness in performing EA, primarily focusing on the
_relational modality_ of KGs. These approaches operate under the assumption
that the neighbors of two equivalent entities in separate KGs are also
equivalent (Liu et al., 2020). By leveraging this assumption, these methods
align entities by training an EA model to learn representations of KGs.

Figure 1. An example of multi-modal entity alignment.

Many existing studies on Entity Alignment (EA) face issues with the geometric
embedding space, limiting their real-world applicability (Sun et al., 2020c;
Gao et al., 2022). To address these problems, incorporating information from
”side modalities” of knowledge graphs has been proposed to enhance the
robustness and effectiveness of EA (Liu et al., 2020, 2021a; Trisedya et al.,
2019; Xu et al., 2021; Liu et al., 2023; Lin et al., 2022; Chen et al.,
2022b). Textual information is commonly used, but recent studies have
highlighted potential test data leakage issues when incorporating textual
information (Liu et al., 2020; Sun et al., 2021). As a result, researchers
have explored alternative modalities like visual, attribute, and temporal
modalities to improve EA performance (Liu et al., 2021a; Sun et al., 2017; Xu
et al., 2021).

The field of multi-modal learning has inspired researchers to investigate
multi-modal EA, where multiple side modalities are jointly modeled for EA
(Chen et al., 2020; Li et al., 2023b; Lin et al., 2022; Chen et al., 2022b).
However, these methods have certain limitations:

  * •

Inconsistent and inefficient modality modeling. Different models are used for
different modalities, leading to time-consuming and non-generalizable
approaches. Additionally, these methods mainly focus on learning vectorized
representations and require transformation into similarity scores between
entities.

  * •

Ineffective modality fusion. Complex learning procedures are employed to
integrate information from different modalities. However, the heterogeneous
nature of modalities in knowledge graphs poses challenges to effective
integration. An example can illustrate this point.

###### Example 1.1.

In classic multi-modal (MM) learning scenarios, different modalities are
typically homogeneous. However, in the context of Multi-Modal Entity Alignment
(MMEA), different modalities are heterogeneous. As shown in Fig. 2, the texts
provide explicit descriptions of the images. However, in knowledge graphs
(KGs), the relations, attributes, and images exhibit heterogeneity. For
instance, in the MMKG example depicting Michael Jordan, we cannot infer his
height, weight, or birth date from the image alone, nor can we infer the image
from the attributes. Consequently, modeling interactions between different
modalities becomes challenging and unnecessary.

To address these challenges, we propose a novel method called PathFusion for
multi-modal EA. We introduce the _Modality Similarity Path_ (MSP) approach to
model different modalities in a unified manner. Instead of embedding modality
information, MSP constructs a _path_ between entities by connecting bipartite
graphs, simplifying the alignment process and enabling direct transformation
of information into alignment results. We also propose the _Iterative
Refinement Fusion_ (IRF) method to effectively fuse information from different
modalities. Unlike existing methods, IRF iteratively fuses information into a
unified _path_ to make it more discriminative, rather than forcing
interactions between modalities. To summarize, our contributions are as
follows:

Figure 2. Comparison between multi-modal entity alignment and the classic
multi-modal learning tasks.

  * •

Model. We propose the PathFusion111https://github.com/blzhu0823/PathFusion
framework for multi-modal EA, which models different modalities in a unified
way and effectively fuses their information.

  * •

Path. We introduce the MSP approach to model different modalities in a unified
manner, allowing for direct alignment results without information loss.

  * •

Fuse. We propose the IRF method to iteratively fuse information from different
modalities using the _path_ as the information carrier.

  * •

Experiments. We conduct extensive experiments on real-world datasets,
demonstrating that PathFusion outperforms state-of-the-art methods by a large
margin.

##  2\. Related Work

In this section, we review the related work of EA, including structure-based
EA and multi-modal EA.

###  2.1. Structure-based Entity Alignment

The foundation of EA is to learn the graph relation structures of KGs.
Existing structure-based EA models can be divided into two categories: _KGE_
-_based methods_(Chen et al., 2017; Zhu et al., 2017; Sun et al., 2018;
Trisedya et al., 2019) and _GNN-based methods_(Wang et al., 2018; Li et al.,
2019; Mao et al., 2020a; Sun et al., 2020b, a; Mao et al., 2021a). The former
uses the KG embedding models (e.g., TransE (Bordes et al., 2013)) to learn
entity embeddings, while the latter uses GNNs (Kipf and Welling, 2017; Yin et
al., 2022). PathFusion is designed for multi-modal EA, and thus can
incorporate any structure-based EA model as the backbone.

###  2.2. Multi-Modal Entity Alignment

All existing proposals of multi-modal EA incorporate _side modalities_ of KGs,
including _textual modality_ of entity names and descriptions (Ge et al.,
2021; Mao et al., 2021b; Li et al., 2023a), _attribute modality_ that use
attributes of entities (Trisedya et al., 2019; Liu et al., 2020; Li et al.,
2023b), _temporal modality_ that uses timestamps of relations (Xu et al.,
2021, 2022; Liu et al., 2023), and _visual modality_ (Liu et al., 2021a; Chen
et al., 2022b; Lin et al., 2022) that use images of entities. Some of such
proposals are able to perform EA even without seed alignment (Ge et al., 2021;
Mao et al., 2021b). However, models that incorporate textual modality may be
overestimated due to potential test data leakage (Chen et al., 2021; Liu et
al., 2020, 2021a; Sun et al., 2021). Thus, other modalities are more reliable
for EA. Recently, some multi-modal EA models have been proposed to incorporate
multiple modalities (Chen et al., 2020; Li et al., 2023b; Chen et al., 2022b;
Lin et al., 2022). However, they face limitations in incorporating all
modalities due to the lack of a unified framework. Also, their performance is
restricted by the absence of an effective mechanism to fuse multiple
modalities. In contrast, our proposed method, PathFusion, stands out as the
first approach to successfully incorporate all modalities and achieve state-
of-the-art performance.

Method | Relation | Visual | Attribute | Temporal
---|---|---|---|---
MTransE (Chen et al., 2017) | ✓ | ✗ | ✗ | ✗
IPTransE (Zhu et al., 2017) | ✓ | ✗ | ✗ | ✗
TransEdge (Sun et al., 2019) | ✓ | ✗ | ✗ | ✗
MRAEA (Mao et al., 2020a) | ✓ | ✗ | ✗ | ✗
RREA (Mao et al., 2020b) | ✓ | ✗ | ✗ | ✗
DualAMN (Mao et al., 2021a) | ✓ | ✗ | ✗ | ✗
JAPE (Sun et al., 2017) | ✓ | ✗ | ✓ | ✗
GCN-Align (Wang et al., 2018) | ✓ | ✗ | ✓ | ✗
AttrE (Trisedya et al., 2019) | ✓ | ✗ | ✓ | ✗
AttrGNN (Liu et al., 2020) | ✓ | ✗ | ✓ | ✗
TEA-GNN (Xu et al., 2021) | ✓ | ✗ | ✗ | ✓
TREA (Xu et al., 2022) | ✓ | ✗ | ✗ | ✓
STEA (Cai et al., 2022) | ✓ | ✗ | ✗ | ✓
DualMatch (Liu et al., 2023) | ✓ | ✗ | ✗ | ✓
MMEA (Chen et al., 2020) | ✓ | ✓ | ✓ | ✗
EVA (Liu et al., 2021a) | ✓ | ✓ | ✓ | ✗
MSNEA (Chen et al., 2022b) | ✓ | ✓ | ✓ | ✗
MCLEA (Lin et al., 2022) | ✓ | ✓ | ✓ | ✗
ACK-MMEA (Li et al., 2023b) | ✓ | ✓ | ✓ | ✗
MEAformer (Chen et al., 2022a) | ✓ | ✓ | ✓ | ✗
Ours | ✓ | ✓ | ✓ | ✓
Table 1. A table of methods and their supported modalities. The four
modalities are relation, visual, attribute, and temporal.

We report the supported modalities of PathFusion with the compared methods in
Table 1. Our method is the only one that can support all modalities.

##  3\. Problem definition

In this section, we present our problem definition.

A multi-modal knowledge graph (MMKG) can be denoted as
$G=(E,R,Q,T,I,P,K,V,A)$, where $E$ is the set of entities, $R$ is the set of
relations. $T,I,P,K,V$ are sets of items in the side modalities including the
temporal, attribute and visual modalities, and can be optional if the dataset
do not hold information of such modality. $T$ is a set of timestamps that
represent the information of the temporal modality. $I$ is the set of images
and $P=\\{e,i\\}~{}|~{}e\in E,i\in I$ is the set of entity-image pairs,
representing that $i$ is an image of the entity $e$. $K,V,A$ are the set of
attribute names, values and entity-attribute triples. $A$ is represented as
$\\{(e,k,v)~{}|~{}e\in E,k\in K,v\in V\\}$. $Q=\\{(h,r,t,\tau)~{}|~{}h,t\in
E,r\in R,\tau\in T\\}$ is the set of quadruples, each of which represents that
the subject entity $h$ has the relation $r$ with the object entity $t$ during
the time interval $\tau$. $\tau$ here is optional, if not given, the tuple
$(h,r,t)$ is often called a triple. For simplicity, we use the term “triple”
with or without the time interval $\tau$ to represent the quadruple
$(h,r,t,\tau)$.

Multi-modal entity alignment. (MMEA) is the process of finding a 1-to-1
mapping of entities $\phi$ from a source MMKG
$G_{s}=(E_{s},R_{s},Q_{s},T_{s},I_{s},P_{s},K_{s},V_{s},A_{s})$ to a target
MMKG $G_{t}=(E_{t},R_{t},Q_{t},T_{t},$ $I_{t},P_{t},K_{t},V_{t},A_{t})$.
$\phi=\\{(e_{s},e_{t})\in E_{s}\times E_{t}~{}|~{}e_{s}\equiv e_{t}\\}$ ,
where $e_{s}\in E_{s}$, $e_{t}\in E_{t}$, and $\equiv$ is an equivalence
relation between two entities. Here, since the timestamps (if available) are
overlapped between knowledge graphs, we have $T_{s}=T_{t}$. A small set of
equivalent entities $\phi^{\prime}\subset\phi$ is known beforehand, and is
used as training seed alignment.

##  4\. Methodology

In this section, we present the proposed PathFusion for EA on multi-modal KGs.
First, we give an overview of the proposed PathFusion

###  4.1. Overview

Fig. 3 shows the overview of the proposed PathFusion. PathFusion consists of
two parts: _Universal Modality Modeling_ with Modality Similarity Path and
_Iterative Refinement Fusion_. PathFusion first aggregates the information of
modalities into entity representations by MSP, and then iteratively fuse the
entity representations from the source and target KGs to obtain the mapping
matrix with IRF. We detail the two parts in the following subsections.

###  4.2. Universal Modality Modeling with Modality Similarity Path

Figure 3. The proposed PathFusion framework.

Previous research has focused on learning vectorized representations from side
modalities to enhance the performance of entity alignment (EA). Side
modalities provide additional information, as latent relations exist between
items of side modalities in two knowledge graphs. For example, in the visual
modality, latent relations involve visual similarities between entity images
(e.g., two images of the same person are more similar than two images of
different people). In the attribute modality, latent relations consist of
attribute name and value similarities (e.g., two people with the same height
and birth date are more likely to be the same person than two people with
different heights and birth dates).

Multi-modal learners incorporate modalities into entity embeddings, which
serve as intermediate representations in EA models. Typically, embedding
learners aggregate side modality information into entity representations,
leveraging explicitly defined entity-side modality relations. For instance, in
the visual modality, entity-image pairs $P$ indicate image-entity
associations, allowing the concatenation of visual features of entity images
to obtain entity embeddings (Liu et al., 2021a).

The final output of an EA model is a mapping matrix
$M\in\mathbb{R}^{E_{s}\times E_{t}}$ representing entity pairs between two
knowledge graphs. This matrix is obtained by taking the dot product of entity
embeddings from the source and target KGs, formulated as
$M=\mathbf{H}_{s}\mathbf{H}_{t}^{T}$, where $\cdot^{T}$ denotes the transpose
operation. Here, $\mathbf{H}_{s}\in\mathbb{R}^{E_{s}\times d}$ and
$\mathbf{H}_{t}\in\mathbb{R}^{E_{t}\times d}$ are the entity embeddings of the
source and target KGs, respectively.

Instead of learning entity embeddings, we propose a more direct and efficient
approach called Modality Similarity Path (MSP) to model side modalities. MSP
represents the relations between entities in the source and target KGs by
using side modalities as bridges. Each side modality in EA has two sets of
items representing the source and target graphs. For example, in the temporal
modality, there are two sets of timestamps $T_{s}$ and $T_{t}$ representing
temporal information in the source and target KGs, respectively. Hence, we
first define the similarity between these two sets as a cross-KG bridge.

Assuming we have an arbitrary modality $X$, MSP collects the _cross-KG intra-
modality similarity_ $M_{X}^{x}\in\mathbb{R}^{|X_{s}\times X_{t}|}$ between
items in the source and target KGs for the $X$ modality. Then, using the
_intra-KG cross-modality similarity_ matrices
$M_{X}^{s}\in\mathbb{R}^{|E_{s}\times X_{s}|}$ and
$M_{X}^{t}\in\mathbb{R}^{|E_{t}\times X_{t}|}$, MSP defines the relations
between entities and modality items in the source and target KGs. Finally, MSP
derives the mapping matrix $M_{X}\in\mathbb{R}^{|E_{s}\times E_{t}|}$ by
applying a matrix-multiplication-like operator on the similarity matrices. For
each $m_{i,j}\in M_{X}$, we can express MSP as:

(1) $m_{i,j}=\underset{x_{s}\in X_{s},x_{t}\in
X_{t}}{\Psi}\left(M_{X_{i,x_{s}}}^{s}M_{X_{x_{s},x_{t}}}^{x}M_{X_{j,x_{t}}}^{t}\right)$

The aggregation function $\Psi$ in MSP combines similarity matrices, which can
be either the max or sum function. If $\Psi$ is the sum function, MSP is
equivalent to the matrix multiplication operator, defined as
$M_{X}=M_{X}^{s}M_{X}^{x}(M_{X}^{t})^{T}$. If $\Psi$ is the max function, MSP
behaves like a max-pooling operator.

It is evident that MSP is a versatile operator applicable to any modality. We
only need to specify the cross-KG intra-modality similarity and intra-KG
cross-modality similarity for each modality. Subsequent sections provide a
comprehensive description of MSP for each specific modality.

Relational modality. In the relational modality, our PathFusion framework
incorporates existing relational learning methods (Mao et al., 2021a).
PathFusion adopts the state-of-the-art EA method (Mao et al., 2021a) to build
a $L$-layer GNN model for learning the representation of the knowledge graph
structure. We train a relational learning model on the two knowledge graphs
using the seed alignment as the training set. This yields vectorized entity
representation $\mathbf{H}\in\mathbb{R}^{|E|\times d}$, where $d$ is the
embedding space dimension. As the modality nodes are entities themselves, the
cross-KG intra-modality similarity is the identity matrix $I_{d}$. The intra-
KG cross-modality similarity consists of entity embeddings of the source and
target KGs. The mapping matrix $M_{R}$ is obtained by taking the dot product
of entity embeddings from both KGs:
$M_{R}=\mathbf{H}_{s}I_{d}(\mathbf{H}_{t})^{T}$.

Visual Modality. In MMKG, entities have visual representations as images. To
measure the visual cross-KG similarity, we utilize the visual features
extracted from the images using SwinTransformer (Liu et al., 2021b). Let
$\mathbf{I}\in\mathbb{R}^{|I|\times d_{v}}$ represent the visual features,
where $d_{v}$ is the dimension of the visual feature space. The visual cross-
KG similarity is computed as $M_{I}^{x}=\mathbf{I}_{s}(\mathbf{I}_{t})^{T}$.
The visual intra-KG similarity is derived from the entity-image pairs $P$,
where $m_{e_{i},j}\in M_{I}$ between an entity $e_{i}\in E$ and an image $j\in
I$ would be

(2) $m_{e_{i},j}=\begin{cases}1,&\text{if }(e_{i},j)\in P\\\
0,&\text{otherwise}\end{cases}$

Finally, MSP uses the max operator to obtain the mapping matrix $M_{I}$.

Attribute Modality. In the attribute modality, the cross-KG intra-modality
similarity is obtained by combining attribute name and attribute value
similarities. We define the attribute item set $S$ as the set of distinct
attribute name-value pairs. The attribute name similarity $M_{K}$ is computed
using a pre-trained language model, where we use the representation of the
[CLS] token as the attribute name representation. The attribute value
similarity $M_{V}$ is calculated as the reciprocal of the absolute difference
between attribute values. For each $m_{i,j}\in M_{V}$, we have
$m_{i,j}=\frac{1}{|v_{i}-v_{j}|}$, where $v_{i}\in V_{s}$ and $v_{j}\in V_{t}$
are the attribute values of the $i$-th and $j$-th items, respectively. The
attribute similarity $M_{A}^{x}$ is obtained by multiplying the attribute name
similarity and value similarity. The attribute intra-KG similarity is derived
from the entity-attribute triples $A$ using the same approach as in the visual
modality. The mapping matrix $M_{A}$ is computed by summing the fused
similarities using the sum operator.

Temporal modality. Temporal modality is represented as timestamps defined on
the relations between entities. Those timestamps are universal across all KGs,
which means that for any two MMKGs with temporal information $G_{s}$ and
$G_{t}$, we have $T_{s}=T_{t}$. Therefore, the cross-KG intra-modality
similarity is the identity matrix $I_{|T|}$. We can follow (Liu et al., 2023)
to obtain the intra-KG similarity. Specifically, we rely on two adjacency
matrices extracted from the MMKGs: (i) entity-entity matrix $\bm{A}$, built
following (Mao et al., 2022b); and (ii) entity-timestamp matrix $\bm{A^{t}}$,
built by counting the number of relations that one entity $e$ and one
timestamp $\tau$ appear together. We follow (Mao et al., 2022a, b; Liu et al.,
2023) to conduct an $L$-layer graph convolution-like forward pass to generate
an aggregated feature matrix:
$\bm{\Theta}=\left[\bm{A}\bm{A^{t}}||\bm{A}^{2}\bm{A^{t}}||\ldots||\bm{A}^{L}\bm{A^{t}}\right]$.
Here, $L$ serves as a hyper-parameter that specifies the number of layers for
the graph convolution-like forward pass, and
$\bm{A}^{l}=\prod_{i=1}^{l}\bm{A}$ represents the $l$-hop adjacency matrix of
the graph $G$. Finally, we use the sum operator to obtain the mapping matrix
$M_{T}=\bm{\Theta}_{s}I_{|T|}\bm{\Theta}_{t}^{T}.$ Note that in many Knowledge
Graphs, attributes may include temporal information. For instance, certain
attribute pairs can take the form of (“United States,” “was founded,” “July 4,
1776”). However, it’s important to emphasize that this temporal information is
primarily handled within the attribute modality encoders. Reanalyzing it would
be duplicative. Consequently, our focus on temporal information is exclusively
centered on the relational facts within the KG.

Overall, MSP provides a more direct and efficient way to model the side
modalities, allowing for effective entity alignment in MMKG.

###  4.3. Iterative Fusing of Multiple Modalities

Directly fusing the information of side modalities to produce the final output
is challenging. Some literature (Chen et al., 2022b) suggests letting
different modalities interact with each other to strengthen ”weak” modalities.
However, due to the heterogeneous nature of side modalities in EA, learning
the interactions between them is inefficient. Instead, the goal should be to
make entities more discriminative by leveraging multiple modalities.

Fusing the Modality Paths. A practical approach is to fuse the information of
multiple modalities by summing them up (Zeng et al., 2020). To enhance
discriminability, we adopt the optimal transport (OT) plan (Cuturi, 2013)
instead of a similarity matrix to represent the final alignment matrix. In
MSP, we have obtained mapping matrices for each modality. We directly sum them
to obtain the final mapping matrix $M$ using the following equation:

(3) $M=\operatorname{Sinkhorn}(\sum_{X\in\mathcal{X}}M_{X})$

where $\mathcal{X}$ represents the collection of all modalities. We use the
Sinkhorn algorithm (Cuturi, 2013) to obtain the optimal transport plan by
iterating $k$ steps to scale the similarity/cost matrix. The Sinkhorn
algorithm (Cuturi, 2013) is defined as follows:

(4) $\displaystyle\qquad S^{0}(\bm{X})=\exp(\bm{X})$ $\displaystyle\qquad
S^{k}(\bm{X})=\mathcal{N}_{c}\left(\mathcal{N}_{r}\left(S^{k-1}(\bm{X})\right)\right)$
$\displaystyle\operatorname{Sinkhorn}(\bm{X})=\lim_{k\rightarrow\infty}S^{k}(\bm{X}),$

where
$\mathcal{N}_{r}(\bm{X})=\bm{X}\oslash\left(\bm{X}\mathbf{1}_{N}\mathbf{1}_{N}^{T}\right)$
and $\mathcal{N}_{c}=\bm{X}\oslash$
$\left(\mathbf{1}_{N}\mathbf{1}_{N}^{T}\bm{X}\right)$ are the row and column-
wise normalization operators of a matrix, $\oslash$ represents the element-
wise division, and $\mathbf{1}_{N}$ is a column vector of ones. In reality, we
set the iteration step $k$ to a constant instead of $+\infty$.

Iterative Refinement of Relational Modality. In the relational modality, we
trained a model to obtain the similarity matrix $M_{R}$. However, this process
only uses a small fraction of the total alignment pairs as training seeds
(usually around $10\%-30\%$), which may underutilize the information of the
relational modality. To address this, we propose to refine the relational
modality using the mapping matrix $M$ obtained from the previous step,
following the bi-directional strategy (Mao et al., 2020a, b). In each
iteration, we derive the bi-directional alignment matrices $M$ and
$M^{\prime}$ by applying Eq. 3 in both directions. Then, we extract pseudo-
seeds from these matrices. Specifically, we consider $(e_{i},e_{j})$ as a
valid seed pair if $e_{i}\in E_{s}$, $e_{j}\in E_{t}$,
$\arg\max_{j^{\prime}}{M_{ij^{\prime}}}=j$, and
$\arg\max_{i^{\prime}}{M^{\prime}_{ji^{\prime}}}=i$. Finally, we fine-tune the
relational model using the generated pseudo-seed alignment. This refinement
process also enables unsupervised relational learning by using the mapping
matrix (fused without $M_{R}$) as the pseudo labels.

Discussions. This refinement process differs from the aforementioned
”interactions” between modalities because ”weak” modalities lack sufficient
discriminative information. On the other hand, knowledge graphs possess strong
graph structure information and are not considered ”weak.” The refinement
process aims to complete the seed alignment and allows the relational modality
to better express its information.

##  5\. Experiments

Table 2. Statistics of the multi-modal EA datasets. Dataset | Entity | $\begin{array}[]{c}\text{ Relation }\\\ \text{ Triple }\end{array}$ | $\begin{array}[]{c}\text{ Attribute }\\\ \text{ Triple }\end{array}$ | Image | Seed
---|---|---|---|---|---
FB15K | 14,951 | 592,213 | 29,395 | 13,444 | -
DB15K | 12,842 | 89,197 | 48,080 | 12,837 | 12,846
YG15K | 15,404 | 122,886 | 23,532 | 11,194 | 11,199
|  |  |  |  |
Table 3. Statistics of DICEWS and WY50K. Dataset | Entity | Rels | Time | Triples
---|---|---|---|---
DICEWS | 9,517-9,537 | 247-246 | 4,017 | 307,552-307,553
WY50K | 49,629-49,222 | 11-30 | 245 | 221,050-317,814
|  |  |  |

We report on extensive experiments aiming at evaluating the performance of
PathFusion.

Table 4. The results of PathFusion and the baselines on the FB15K-DB15K and FB15K-YG15K datasets. Method | FB15K-DB15K |  FB15K-YG15K
---|---|---
|  H@1$\uparrow$ |  H@5$\uparrow$ |  H@10$\uparrow$ |  MR$\downarrow$ |  MRR$\uparrow$ |  H@1$\uparrow$ |  H@5$\uparrow$ |  H@10$\uparrow$ |  MR$\downarrow$ |  MRR$\uparrow$
MTransE (Chen et al., 2017) | 0.4 | 1.4 | 2.5 | 1239.5 | 0.014 | 0.3 | 1.0 | 1.8 | 1183.3 | 0.011
IPTransE (Zhu et al., 2017) | 4.0 | 11.2 | 17.3 | 387.5 | 0.086 | 3.1 | 9.5 | 14.4 | 522.2 | 0.070
GCN-Align* (Wang et al., 2018) | 7.1 | 16.5 | 22.4 | 304.5 | 0.106 | 5.0 | 12.9 | 18.1 | 478.2 | 0.094
BootEA (Sun et al., 2018) | 32.3 | 49.9 | 57.9 | 205.5 | 0.410 | 23.4 | 37.4 | 44.5 | 272.1 | 0.307
SEA (Pei et al., 2019) | 17.0 | 33.5 | 42.5 | 191.9 | 0.255 | 14.1 | 28.7 | 37.1 | 207.2 | 0.218
IMUSE (He et al., 2019) | 17.6 | 34.7 | 43.5 | 182.8 | 0.264 | 8.1 | 19.2 | 25.7 | 397.6 | 0.142
HyperKA (Sun et al., 2020a) | 13.7 | 24.9 | 30.7 | 712.2 | 0.195 | 16.9 | 29.5 | 34.9 | 738.0 | 0.232
DualAMN* (Mao et al., 2021a) | 55.0 | 71.4 | 77.1 | 166.6 | 0.627 | 51.6 | 68.3 | 74.4 | 115.1 | 0.594
PoE (Liu et al., 2019a) | 12.0 | - | 25.6 | - | 0.167 | 10.9 | - | 24.1 | - | 0.154
MMEA (Chen et al., 2020) | 26.5 | 45.1 | 54.1 | 124.8 | 0.357 | 23.4 | 39.8 | 48.0 | 147.4 | 0.317
ACK-MMEA$\bullet$ (Li et al., 2023b) | 30.4 | - | 54.9 | - | 0.387 | 28.9 | - | 49.6 | - | 0.360
MCLEA$\bullet$ (Lin et al., 2022) | 44.5 | - | 70.5 | - | 0.534 | 38.8 | - | 64.1 | - | 0.474
EVA (Liu et al., 2021a) | 55.6 | 66.6 | 71.6 | 140.0 | 0.609 | 10.3 | 21.7 | 27.8 | 616.8 | 0.164
MEAformer$\bullet$ (Chen et al., 2022a) | 57.8 | - | 81.2 | - | 0.661 | 44.4 | - | 69.2 | - | 0.529
MSNEA (Chen et al., 2022b) | 65.3 | 76.8 | 81.2 | 54.0 | 0.708 | 44.3 | 62.6 | 69.8 | 85.1 | 0.529
PathFusion${}_{\text{GCN-Align}}$ | 55.0 | 71.3 | 77.0 | 66.4 | 0.626 | 47.6 | 64.9 | 71.4 | 80.7 | 0.558
PathFusion${}_{\text{(w/o iteration)}}$ | 80.1 | 87.5 | 90.0 | 27.2 | 0.836 | 74.8 | 83.9 | 86.8 | 44.8 | 0.790
PathFusion | 87.7 | 93.0 | 94.5 | 22.7 | 0.902 | 80.5 | 88.0 | 90.1 | 36.4 | 0.839
Improvements over best baselines | 22.4 | 16.2 | 13.3 | 31.3 | 0.194 | 28.9 | 19.7 | 15.7 | 48.7 | 0.245
|  |  |  |  |  |  |  |  |  |

###  5.1. Experimental Setup

We describe the experimental setup, including datasets, baselines, and
evaluation metrics.

Datasets. For evaluating the multi-modal ability of PathFusion, we use the
following datasets.

  * •

To evaluate PathFusion on the relational, visual and attribute modalities, we
use two real-world multi-modal datasets, FB15K-DB15K and FB15K-YG15K, provided
by (Liu et al., 2019a; Chen et al., 2020). These datasets are most
representative in multi-modal EA task (Chen et al., 2022b, 2020, a; Li et al.,
2023b), and contains the _relational_ , _visual_ , and _attribute_ modalities.
For all experiments, we use 20% of the total alignment pairs as the training
set (seed alignment), and use the rest of the pairs as the test set. Table 2
shows the statistics of the multi-modal datasets.

  * •

The above-mentioned datasets do not contain temporal modality. Thus, for
evaluating PathFusion on the ability of modeling _temporal modality_ , we use
two real-world Temporal KG datasets provided by (Xu et al., 2021, 2022). They
are extracted from ICEWS05-15 (García-Durán et al., 2018), YAGO (Suchanek et
al., 2007), and Wikidata (Vrandecic and Krötzsch, 2014), denoted as DICEWS,
WY50K, respectively. DICEWS has two settings on seed alignment ratios,
DICEWS-200 and DICEWS-1K that use 200 and 1,000 seed alignment pairs,
respectively. Similarly, WY50K has WY50K-1K, WY50K-5K, that use 1,000, and
5,000 seed alignment pairs, respectively. Table 3 shows the statistics of the
two datasets.

Baselines. We compare PathFusion with the following baselines, which are
categorized into three groups: Traditional methods, multi-modal methods and
Time-aware methods.

_Traditional Methods_ are those methods that only use the relational modality
(some (Wang et al., 2018; He et al., 2019) also use the attribute modality).
They include:

  * •

MTransE (Chen et al., 2017): Embeds different knowledge graphs separately in
distinct embedding spaces and provides transitions for entities to map aligned
entities.

  * •

IPTransE (Zhu et al., 2017): Iteratively adds newly aligned entity pairs to a
set for soft alignment and employs a parameter sharing strategy for different
knowledge graphs.

  * •

GCN-Align (Wang et al., 2018): Combines structural and attribute information
using graph convolutional networks.

  * •

BootEA (Sun et al., 2018): Iteratively labels likely alignments as training
data and reduces error accumulation during iterations through an alignment
editing method.

  * •

SEA (Pei et al., 2019): Utilizes both labeled entities and abundant unlabeled
entity information for alignment and incorporates the difference in point
degrees with adversarial training.

  * •

IMUSE (He et al., 2019): Employs a bivariate regression model to learn weights
for relation and attribute similarities for combining alignment results.

  * •

HyperKA (Sun et al., 2020a): Introduces a hyperbolic relational graph neural
network to explore low-dimensional hyperbolic embeddings for entity alignment.

_Multi-modal Methods_ are those methods that use relational, attribute, and
visual modalities. They include:

  * •

PoE (Liu et al., 2019a): Judges SameAs links between aligned entities by
assigning a high probability to true triples and a low probability to false
triples. Defines the overall probability distribution as the product of all
uni-modal experts.

  * •

MMEA (Chen et al., 2020): Generates entity representations of relational,
visual, and numerical knowledge, and then migrates them.

  * •

MCLEA (Lin et al., 2022): Proposes a multi-modal contrastive learning
framework to learn the representations of entities from different modalities.

  * •

MSNEA (Chen et al., 2022b): Introduces a multi-modal siamese network approach
to learn the representations of entities from different modalities.

  * •

MEAformer (Chen et al., 2022a): Proposes a multi-modal transformer-based
approach to learn the representations of entities from different modalities.

  * •

ACK-MMEA (Li et al., 2023b): Proposes an attribute-consistent knowledge graph
entity alignment approach considering multiple modalities.

_Temporal modality Methods_ are those methods that specifically consider both
relational and temporal modalities. They are

  * •

TEA-GNN (Xu et al., 2021): As the first time-aware EA model, it models
temporal information as embeddings.

  * •

TREA (Xu et al., 2022): Uses a temporal relation attention mechanism to model
temporal information.

  * •

STEA (Cai et al., 2022): Constructs a time similarity matrix by a dictionary
counting the relation between timestamps and entities.

  * •

DualMatch (Liu et al., 2023): Proposes to separately align entities using
relational and temporal information, and then fuse the results.

Table 5. The results on DICEWS and WY50K. Method | DICEWS-200 | DICEWS-1K
---|---|---
H@1$\uparrow$ |  H@10$\uparrow$ |  MRR$\uparrow$ |  H@1$\uparrow$ |  H@10$\uparrow$ |  MRR$\uparrow$
TEA-GNN (Xu et al., 2021) | 87.6 | 94.1 | 0.902 | 88.7 | 94.7 | 0.911
TREA (Xu et al., 2022) | 91.0 | 96.0 | 0.927 | 91.4 | 96.6 | 0.933
STEA (Cai et al., 2022) | 94.3 | 96.8 | 0.954 | 94.5 | 96.7 | 0.954
DualMatch (Liu et al., 2023) | 95.3 | 97.4 | 0.961 | 95.3 | 97.3 | 0.961
PathFusion | 95.3 | 97.2 | 0.961 | 95.0 | 97.0 | 0.958
Method | WY50K-1K | WY50K-5K
H@1$\uparrow$ |  H@10$\uparrow$ |  MRR$\uparrow$ |  H@1$\uparrow$ |  H@10$\uparrow$ |  MRR$\uparrow$
TEA-GNN (Xu et al., 2021) | 72.3 | 87.1 | 0.775 | 87.9 | 96.1 | 0.909
TREA (Xu et al., 2022) | 84.0 | 93.7 | 0.885 | 94.0 | 98.9 | 0.958
STEA (Cai et al., 2022) | 94.3 | 98.9 | 0.962 | 96.1 | 99.2 | 0.974
DualMatch (Liu et al., 2023) | 94.7 | 98.4 | 0.961 | 98.1 | 99.6 | 0.986
PathFusion | 96.4 | 98.4 | 0.972 | 97.7 | 99.6 | 0.984
|  |  |  |  |  |

Evaluation Metrics. We use the widely-adopted Hits@N (H@N, in percentage),
Mean Reciprocal Rank (MRR) and Mean Rank (MR) metrics to evaluate the
performance of PathFusion and the baselines. Hits@$N$ (in percentage) denotes
the proportion of correctly aligned entities in the top-$N$ ranks. MRR is the
average of the reciprocal ranks of the correctly aligned entities, where
reciprocal rank reports the mean rank of the correct alignment. MR is the
average of the ranks of the correctly aligned entities. Higher H@$N$, MRR, and
lower MR indicates higher EA accuracy.

Implementation Details. We choose several state-of-the-art models as the
backbone models for each modality. We choose DualAMN (Mao et al., 2021a) to
learn the knowledge graph relations, SwinTransformer (Liu et al., 2021b) to
encode images into vectors, RoBERTa (Liu et al., 2019b) to encode attribute
names into vectors. We iteratively refine the relational model for 3
iterations, with the hyper-parameters the same as (Mao et al., 2021a). We
allow one entity to have a maximum of 6 images. We set $k=10$ for the Sinkhorn
algorithm. We implement all our codes in Python and run all experiments on a
server with 8 NVIDIA RTX 3090 GPUs.

###  5.2. Main Results

Modalities and Evaluation Results. We evaluate PathFusion on the FB15K-DB15K
and FB15K-YG15K datasets, considering relational, visual, and attribute
modalities. Results are presented in Table 4 with * denoting our reproduced
baselines, $\bullet$ denoting results from original papers, and other results
from (Chen et al., 2022b). The best results are bold, and the best among
baselines are underlined. Our observations are as follows:

First, PathFusion outperforms the baselines significantly. On FB15K-DB15K,
PathFusion surpasses the best baseline by 22.4% in Hits@1 and 0.194 in MRR. On
FB15K-YG15K, PathFusion outperforms the best baseline by 28.9% in Hits@1 and
0.245 in MRR, demonstrating its effectiveness for multi-modal EA. Second,
interestingly, some recent multi-modal approaches perform worse than the
single-modal baseline DualAMN (Mao et al., 2021a). These approaches fail to
effectively fuse information from different modalities. In contrast,
PathFusion successfully fuses information and outperforms the baselines by a
large margin.

Temporal and Relational Modalities. We evaluate PathFusion on the DICEWS and
WY50K datasets, focusing on temporal and relational modalities. Results in
Table 5 show that PathFusion outperforms two temporal-aware EA models (Xu et
al., 2021, 2022) by a significant margin, despite not explicitly modeling the
temporal modality. Additionally, PathFusion is comparable to the state-of-the-
art temporal-aware EA model DualMatch (Liu et al., 2023), while being a
general framework applicable to different modalities. DualMatch, on the other
hand, is specifically designed for temporal-aware EA and cannot be applied to
other modalities.

Figure 4. H@1 and MRR vs. Iteration Round

###  5.3. Ablation Study

We present various variants of PathFusion in Table 4.
PathFusion${}_{\;\text{GCN-Align}}$ refers to PathFusion with GCN-Align (Wang
et al., 2018), a classic EA model, as its relational model.
PathFusion${}_{\;\text{(w/o iteration)}}$ refers to PathFusion without
iterative refinement of the relational model. First, each variant exhibits
performance degradation, demonstrating the effectiveness of each component of
PathFusion. Second, even without iterative refinement, our proposed PathFusion
outperforms all previous methods. Third, while PathFusion${}_{\;\text{GCN-
Align}}$ outperforms GCN-Align (Wang et al., 2018), it is still inferior to
some baselines. This highlights the importance of the relational modality in
multi-modal EA.

We also offer a comprehensive analysis of the impact of removing individual
modalities, including Attribute, Visual, and all side modalities in each
iteration, from PathFusion, as depicted in Figure 4. It is clear that removing
any of these modalities results in a decline in performance, highlighting the
effectiveness of each modality. Additionally, the absence of the iterative
refinement process leads to a decrease in accuracy, further demonstrating the
effectiveness of IRF.

###  5.4. Study of MSP on Different Modalities

As mentioned in Section 4, MSP is a general framework that can be applied to
different modalities to obtain the alignment matrices. To evaluate the
effectiveness of MSP on different modalities, we assess the alignment matrices
generated by MSP for each modality. First, we apply MSP to each modality to
obtain the alignment matrices and then use these matrices to perform zero-shot
EA. We evaluate PathFusion for its zero-shot EA ability on different
modalities by directly evaluating the alignment matrices obtained by MSP on
each modality. In each setting, MSP is only allowed to use the corresponding
modality.

Attribute modality. We evaluate MSP on its EA ability for the attribute
modality by establishing a variant with only attribute modality information
(referred to as MSP (Attr.)). We compare MSP (Attr.) with ACK-MMEA (Li et al.,
2023b), a baseline introduced in 2023 that leverages all accessible
modalities. We also establish a baseline that encodes both attribute names and
values as a whole sentence using language models and then calculates the
similarity of these ”sentence” embeddings (denoted as AttrSim). The results
are presented in Table 6. It can be observed that MSP outperforms AttrSim by a
significant margin. Furthermore, MSP (Attr.) outperforms ACK-MMEA even using
only the attribute modality, with a significant 15% relatively improvement on
MRR of FB15K-YG15K dataset. This demonstrates the effectiveness of MSP on the
attribute modality. Lastly, AttrSim performs better on the FB15K-YG15K dataset
because YAGO has limited attribute types, making it easier to distinguish the
attribute values.

(a) FB15K-DB15K

(b) FB15K-YG15K

Figure 5. H@1 and MRR vs. Number of Images Table 6. The results of Attribute Modality. ACK-MMEA utilizes all the accessible modalities, while AttrSim and MSP (Attr.) only utilize the attribute modality. Method | FB15K-DB15K | FB15K-YG15K
---|---|---
H@1$\uparrow$ | H@10$\uparrow$ | MRR$\uparrow$ | H@1$\uparrow$ | H@10$\uparrow$ | MRR$\uparrow$
ACK-MMEA | 30.4 | 54.9 | 0.387 | 28.9 | 49.6 | 0.360
AttrSim | 3.7 | 11.4 | 0.064 | 10.7 | 23.9 | 0.153
MSP (Attr.) | 32.4 | 51.2 | 0.384 | 34.7 | 57.0 | 0.416
|  |  |  |  |  |

Visual modality. In our dataset, it’s possible for a single entity to be
associated with multiple images. For the FB15K dataset, the images are sourced
from the image links provided by MMKB (Liu et al., 2019a). In the cases of
YG15K and DB15K, we gather images from Wikimedia. In our experiments, we’ve
chosen to utilize the first six images for each entity. However, if there are
more than six images associated with a particular entity, we cap the selection
at six.

Here, we vary the number of images from 1 to 6 in MSP and evaluate the
performance of PathFusion on the FB15K-DB15K and FB15K-YG15K datasets using
only the visual modality. We also compare the results obtained using both the
max and sum operators of MSP. The results are presented in Fig. 5. As
expected, with more images, the performance of MSP gradually improves.
Additionally, using the max operator consistently outperforms the sum
operator. This is because the max operator better captures the most important
information from the images.

##  6\. Conclusions

This paper presents a novel method called PathFusion for multi-modal EA in
KGs, mitigating the challenges of inconsistent and inefficient modality
modeling. PathFusion addresses the limitations of incompleteness and
overlapping entities in KGs by incorporating additional modalities, enhancing
the effectiveness and robustness of PathFusion first proposes MSP, introducing
a unified modeling technique that simplifies the alignment process by
constructing paths connecting entities and modality nodes to model multiple
modalities. This enables direct transformation of information into alignment
results. Furthermore, PathFusion introduces IRF, an effective fusion method
that iteratively fuses information from different modalities using the path as
an information carrier. The experimental results demonstrate the superiority
of PathFusion over state-of-the-art methods in multi-modal EA. Specifically,
PathFusion achieves more accurate and comprehensive entity alignment, with
$22.4\%-28.9\%$ absolute improvement on Hits@1, and $0.194-0.245$ absolute
improvement on MRR. The use of MSP and IRF contributes to a more holistic
representation of entities across different KGs, leading to improved alignment
results.

## Limitations

Scalability. PathFusion has a quadratic complexity due to the nature of EA.
This is a common problem for EA methods, as we need to compare each entity
pair from different KGs. Also, there lack of a large-scale multi-modal EA
dataset for evaluation. Some recent works (Ge et al., 2022; Gao et al., 2022;
Mao et al., 2022b) have proposed to use of approximate methods to reduce the
number of entity pairs. However, these methods do not apply to PathFusion
because they are designed mostly for relational EA, and inevitably hurt the EA
accuracy as a tradeoff. Also, there are large-scale GNN training and inference
infrastructures that can be applied to EA (Wang et al., 2022; Yin et al.,
2022; Tan et al., 2023; Wang et al., 2023) We leave the scalability issue for
future work.

Modality Selection. PathFusion requires the modality nodes to be explicitly
provided. However, in real-world applications, the modality nodes are not
always available. We leave the modality selection problem for future work.
Also, PathFusion does not consider textual modality, which is an important
modality in KGs. This is due to the potential test data leakage problem. We
leave resolving the problem for future work.

Aligning KGs with different modality distributions. PathFusion requires the
two KGs to align have roughly similar modality distributions. This is because
PathFusion uses the modality nodes to guide the alignment process. However, in
real-world applications, the two KGs may have different modality
distributions. We leave the problem for future work.

## References

  * (1)
  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto García-Durán, Jason Weston, and Oksana Yakhnenko. 2013\.  Translating Embeddings for Modeling Multi-relational Data. In _NeurIPS_. 2787–2795.
  * Cai et al. (2022) Li Cai, Xin Mao, Meirong Ma, Hao Yuan, Jianchao Zhu, and Man Lan. 2022\.  A Simple Temporal Information Matching Mechanism for Entity Alignment Between Temporal Knowledge Graphs.  _arXiv preprint arXiv:2209.09677_ (2022).
  * Cao et al. (2023) Yong Cao, Xianzhi Li, Huiwen Liu, Wen Dai, Shuai Chen, Bin Wang, Min Chen, and Daniel Hershcovich. 2023\.  Pay More Attention to Relation Exploration for Knowledge Base Question Answering.  _arXiv preprint arXiv:2305.02118_ (2023).
  * Chen et al. (2020) Liyi Chen, Zhi Li, Yijun Wang, Tong Xu, Zhefeng Wang, and Enhong Chen. 2020\.  MMEA: Entity Alignment for Multi-modal Knowledge Graph. In _KSEM_. 134–147.
  * Chen et al. (2022b) Liyi Chen, Zhi Li, Tong Xu, Han Wu, Zhefeng Wang, Nicholas Jing Yuan, and Enhong Chen. 2022b.  Multi-modal siamese network for entity alignment. In _KDD_. 118–126.
  * Chen et al. (2021) Muhao Chen, Weijia Shi, Ben Zhou, and Dan Roth. 2021\.  Cross-lingual entity alignment with incidental supervision. In _EACL_. 645–658.
  * Chen et al. (2017) Muhao Chen, Yingtao Tian, Mohan Yang, and Carlo Zaniolo. 2017\.  Multilingual Knowledge Graph Embeddings for Cross-lingual Knowledge Alignment. In _IJCAI_. 1511–1517.
  * Chen et al. (2022a) Zhuo Chen, Jiaoyan Chen, Wen Zhang, Lingbing Guo, Yin Fang, Yufeng Huang, Yuxia Geng, Jeff Z. Pan, Wenting Song, and Huajun Chen. 2022a.  MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid.  _arXiv Pripint arXiv:2212.14454_ (2022).
  * Cuturi (2013) Marco Cuturi. 2013\.  Sinkhorn Distances: Lightspeed Computation of Optimal Transport. In _NeurIPS_. 2292–2300.
  * Gao et al. (2022) Yunjun Gao, Xiaoze Liu, Junyang Wu, Tianyi Li, Pengfei Wang, and Lu Chen. 2022\.  ClusterEA: Scalable Entity Alignment with Stochastic Training and Normalized Mini-batch Similarities. In _KDD_. 421–431.
  * García-Durán et al. (2018) Alberto García-Durán, Sebastijan Dumancic, and Mathias Niepert. 2018.  Learning Sequence Encoders for Temporal Knowledge Graph Completion. In _EMNLP_. 4816–4821.
  * Ge et al. (2021) Congcong Ge, Xiaoze Liu, Lu Chen, Baihua Zheng, and Yunjun Gao. 2021.  Make It Easy: An Effective End-to-End Entity Alignment Framework. In _SIGIR_. 777–786.
  * Ge et al. (2022) Congcong Ge, Xiaoze Liu, Lu Chen, Baihua Zheng, and Yunjun Gao. 2022.  LargeEA: Aligning Entities for Large-scale Knowledge Graphs.  _PVLDB_ 15, 2 (2022), 237–245.
  * He et al. (2019) Fuzhen He, Zhixu Li, Qiang Yang, An Liu, Guanfeng Liu, Pengpeng Zhao, Lei Zhao, Min Zhang, and Zhigang Chen. 2019.  Unsupervised Entity Alignment Using Attribute Triples and Relation Triples. In _DASFAA_ , Vol. 11446. 367–382.
  * Hu et al. (2021) Xuming Hu, Fukun Ma, Chenyao Liu, Chenwei Zhang, Lijie Wen, and Philip S Yu. 2021\.  Semi-supervised Relation Extraction via Incremental Meta Self-Training. In _Findings of EMNLP_. 487–496.
  * Jiang et al. (2022) Chengyue Jiang, Yong Jiang, Weiqi Wu, Pengjun Xie, and Kewei Tu. 2022.  Modeling Label Correlations for Ultra-Fine Entity Typing with Neural Pairwise Conditional Random Field.  _arXiv preprint arXiv:2212.01581_ (2022).
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-Supervised Classification with Graph Convolutional Networks. In _ICLR_.
  * Li et al. (2019) Chengjiang Li, Yixin Cao, Lei Hou, Jiaxin Shi, Juanzi Li, and Tat-Seng Chua. 2019\.  Semi-supervised Entity Alignment via Joint Knowledge Embedding Model and Cross-graph Model. In _EMNLP_. 2723–2732.
  * Li et al. (2023b) Qian Li, Shu Guo, Yangyifei Luo, Cheng Ji, Lihong Wang, Jiawei Sheng, and Jianxin Li. 2023b.  Attribute-Consistent Knowledge Graph Representation Learning for Multi-Modal Entity Alignment. In _WWW_. 2499–2508.
  * Li et al. (2023a) Yangning Li, Jiaoyan Chen, Yinghui Li, Yuejia Xiang, Xi Chen, and Hai-Tao Zheng. 2023a.  Vision, Deduction and Alignment: An Empirical Study on Multi-Modal Knowledge Graph Alignment. In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 1–5.
  * Lin et al. (2022) Zhenxi Lin, Ziheng Zhang, Meng Wang, Yinghui Shi, Xian Wu, and Yefeng Zheng. 2022\.  Multi-modal Contrastive Representation Learning for Entity Alignment.  _arXiv preprint arXiv:2209.00891_ (2022).
  * Liu et al. (2021a) Fangyu Liu, Muhao Chen, Dan Roth, and Nigel Collier. 2021a.  Visual Pivoting for (Unsupervised) Entity Alignment. In _AAAI_. 4257–4266.
  * Liu et al. (2023) Xiaoze Liu, Junyang Wu, Tianyi Li, Lu Chen, and Yunjun Gao. 2023.  Unsupervised Entity Alignment for Temporal Knowledge Graphs. In _WWW_. 2528–2538.
  * Liu et al. (2019a) Ye Liu, Hui Li, Alberto García-Durán, Mathias Niepert, Daniel Oñoro-Rubio, and David S. Rosenblum. 2019a.  MMKG: Multi-modal Knowledge Graphs. In _ISWC_. 459–474.
  * Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * Liu et al. (2020) Zhiyuan Liu, Yixin Cao, Liangming Pan, Juanzi Li, and Tat-Seng Chua. 2020.  Exploring and Evaluating Attributes, Values, and Structures for Entity Alignment. In _EMNLP_. 6355–6364.
  * Liu et al. (2021b) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021b.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In _ICCV_.
  * Mao et al. (2022a) Xin Mao, Meirong Ma, Hao Yuan, Jianchao Zhu, ZongYu Wang, Rui Xie, Wei Wu, and Man Lan. 2022a.  An Effective and Efficient Entity Alignment Decoding Algorithm via Third-Order Tensor Isomorphism. In _ACL_. 5888–5898.
  * Mao et al. (2021a) Xin Mao, Wenting Wang, Yuanbin Wu, and Man Lan. 2021a.  Boosting the Speed of Entity Alignment 10$\times$: Dual Attention Matching Network with Normalized Hard Sample Mining. In _WWW_. 821–832.
  * Mao et al. (2021b) Xin Mao, Wenting Wang, Yuanbin Wu, and Man Lan. 2021b.  From Alignment to Assignment: Frustratingly Simple Unsupervised Entity Alignment. In _EMNLP_. 2843–2853.
  * Mao et al. (2022b) Xin Mao, Wenting Wang, Yuanbin Wu, and Man Lan. 2022b.  LightEA: A Scalable, Robust, and Interpretable Entity Alignment Framework via Three-view Label Propagation. In _EMNLP_. 825–838.
  * Mao et al. (2020a) Xin Mao, Wenting Wang, Huimin Xu, Man Lan, and Yuanbin Wu. 2020a.  MRAEA: An Efficient and Robust Entity Alignment Approach for Cross-lingual Knowledge Graph. In _WSDM_. 420–428.
  * Mao et al. (2020b) Xin Mao, Wenting Wang, Huimin Xu, Yuanbin Wu, and Man Lan. 2020b.  Relational Reflection Entity Alignment. In _CIKM_. 1095–1104.
  * Pei et al. (2019) Shichao Pei, Lu Yu, Robert Hoehndorf, and Xiangliang Zhang. 2019.  Semi-Supervised Entity Alignment via Knowledge Graph Embedding with Awareness of Degree Difference. In _WWW_. 3130–3136.
  * Suchanek et al. (2007) Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007\.  Yago: a core of semantic knowledge. In _WWW_. 697–706.
  * Sun et al. (2021) Zequn Sun, Muhao Chen, and Wei Hu. 2021.  Knowing the No-match: Entity Alignment with Dangling Cases. In _ACL_. 3582–3593.
  * Sun et al. (2020a) Zequn Sun, Muhao Chen, Wei Hu, Chengming Wang, Jian Dai, and Wei Zhang. 2020a.  Knowledge Association with Hyperbolic Knowledge Graph Embeddings. In _EMNLP_. 5704–5716.
  * Sun et al. (2017) Zequn Sun, Wei Hu, and Chengkai Li. 2017.  Cross-Lingual Entity Alignment via Joint Attribute-Preserving Embedding. In _ISWC_. 628–644.
  * Sun et al. (2018) Zequn Sun, Wei Hu, Qingheng Zhang, and Yuzhong Qu. 2018\.  Bootstrapping Entity Alignment with Knowledge Graph Embedding. In _IJCAI_. 4396–4402.
  * Sun et al. (2019) Zequn Sun, JiaCheng Huang, Wei Hu, Muhao Chen, Lingbing Guo, and Yuzhong Qu. 2019\.  TransEdge: Translating Relation-Contextualized Embeddings for Knowledge Graphs. In _ISWC_. 612–629.
  * Sun et al. (2020b) Zequn Sun, Chengming Wang, Wei Hu, Muhao Chen, Jian Dai, Wei Zhang, and Yuzhong Qu. 2020b.  Knowledge Graph Alignment Network with Gated Multi-Hop Neighborhood Aggregation. In _AAAI_. 222–229.
  * Sun et al. (2020c) Zequn Sun, Qingheng Zhang, Wei Hu, Chengming Wang, Muhao Chen, Farahnaz Akrami, and Chengkai Li. 2020c.  A Benchmarking Study of Embedding-based Entity Alignment for Knowledge Graphs.  _PVLDB_ 13, 11 (2020), 2326–2340.
  * Tan et al. (2023) Zeyuan Tan, Xiulong Yuan, Congjie He, Man-Kit Sit, Guo Li, Xiaoze Liu, Baole Ai, Kai Zeng, Peter Pietzuch, and Luo Mai. 2023\.  Quiver: Supporting GPUs for Low-Latency, High-Throughput GNN Serving with Workload Awareness.  _arXiv preprint arXiv:2305.10863_ (2023).
  * Trisedya et al. (2019) Bayu Distiawan Trisedya, Jianzhong Qi, and Rui Zhang. 2019\.  Entity Alignment between Knowledge Graphs Using Attribute Embeddings. In _AAAI_. 297–304.
  * Vrandecic and Krötzsch (2014) Denny Vrandecic and Markus Krötzsch. 2014.  Wikidata: a free collaborative knowledgebase.  _Commun. ACM_ 57, 10 (2014), 78–85.
  * Wang et al. (2023) Qiange Wang, Xin Ai, Yanfeng Zhang, Jing Chen, and Ge Yu. 2023.  HyTGraph: GPU-Accelerated Graph Processing with Hybrid Transfer Management. In _2023 IEEE 39th International Conference on Data Engineering (ICDE)_. IEEE, 558–571.
  * Wang et al. (2022) Qiange Wang, Yanfeng Zhang, Hao Wang, Chaoyi Chen, Xiaodong Zhang, and Ge Yu. 2022\.  Neutronstar: distributed GNN training with hybrid dependency management. In _Proceedings of the 2022 International Conference on Management of Data_. 1301–1315.
  * Wang et al. (2018) Zhichun Wang, Qingsong Lv, Xiaohan Lan, and Yu Zhang. 2018\.  Cross-lingual Knowledge Graph Alignment via Graph Convolutional Networks. In _EMNLP_. 349–357.
  * Xu et al. (2021) Chengjin Xu, Fenglong Su, and Jens Lehmann. 2021.  Time-aware Graph Neural Network for Entity Alignment between Temporal Knowledge Graphs. In _EMNLP_. 8999–9010.
  * Xu et al. (2022) Chengjin Xu, Fenglong Su, Bo Xiong, and Jens Lehmann. 2022\.  Time-aware Entity Alignment using Temporal Relational Attention. In _WWW_. 788–797.
  * Yin et al. (2022) Peiqi Yin, Xiao Yan, Jinjing Zhou, Qiang Fu, Zhenkun Cai, James Cheng, Bo Tang, and Minjie Wang. 2022\.  DGI: Easy and Efficient Inference for GNNs.  _arXiv preprint arXiv:2211.15082_ (2022).
  * Zeng et al. (2020) Weixin Zeng, Xiang Zhao, Jiuyang Tang, and Xuemin Lin. 2020\.  Collective Entity Alignment via Adaptive Features. In _ICDE_. 1870–1873.
  * Zhang et al. (2021) Xin Zhang, Guangwei Xu, Yueheng Sun, Meishan Zhang, and Pengjun Xie. 2021.  Crowdsourcing Learning as Domain Adaptation: A Case Study on Named Entity Recognition. In _ACL_. 5558–5570.
  * Zhu et al. (2017) Hao Zhu, Ruobing Xie, Zhiyuan Liu, and Maosong Sun. 2017\.  Iterative Entity Alignment via Joint Knowledge Embeddings. In _IJCAI_. 4258–4264.