#  NativE: Multi-modal Knowledge Graph Completion in the Wild

Yichi Zhang  Zhejiang UniversityHangZhouZhejiangChina
<EMAIL_ADDRESS>,  Zhuo Chen  Zhejiang
UniversityHangZhouZhejiangChina<EMAIL_ADDRESS>,  Lingbing Guo  Zhejiang
UniversityHangZhouZhejiangChina<EMAIL_ADDRESS>,  Yajing Xu  Zhejiang
UniversityHangZhouZhejiangChina<EMAIL_ADDRESS>,  Binbin Hu  Ant
GroupHangZhouZhejiangChina<EMAIL_ADDRESS>,  Ziqi Liu  Ant
GroupHangZhouZhejiangChina<EMAIL_ADDRESS>,  Wen Zhang∗ 1Zhejiang
University
2Zhejiang University-Ant Group Joint Laboratory of Knowledge Graph
3Alibaba-Zhejiang University Joint Institute of Frontier
TechnologyHangZhouZhejiangChina<EMAIL_ADDRESS>and  Huajun Chen∗
1Zhejiang University
2Zhejiang University-Ant Group Joint Laboratory of Knowledge Graph
3Alibaba-Zhejiang University Joint Institute of Frontier
TechnologyHangZhouZhejiangChina<EMAIL_ADDRESS>
(2024)

###### Abstract.

Multi-modal knowledge graph completion (MMKGC) aims to automatically discover
the unobserved factual knowledge from a given multi-modal knowledge graph by
collaboratively modeling the triple structure and multi-modal information from
entities. However, real-world MMKGs present challenges due to their diverse
and imbalanced nature, which means that the modality information can span
various types (e.g., image, text, numeric, audio, video) but its distribution
among entities is uneven, leading to missing modalities for certain entities.
Existing works usually focus on common modalities like image and text while
neglecting the imbalanced distribution phenomenon of modal information. To
address these issues, we propose a comprehensive framework NativE to achieve
MMKGC in the wild. NativE proposes a relation-guided dual adaptive fusion
module that enables adaptive fusion for any modalities and employs a
collaborative modality adversarial training framework to augment the
imbalanced modality information. We construct a new benchmark called WildKGC
with five datasets to evaluate our method. The empirical results compared with
21 recent baselines confirm the superiority of our method, consistently
achieving state-of-the-art performance across different datasets and various
scenarios while keeping efficient and generalizable. Our code and data are
released at https://github.com/zjukg/NATIVE.

Multi-modal Knowledge Graphs, Knowledge Graph Completion, Multi-modal Fusion,
Adversarial Learning

$*$ Corresponding authors

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††conference:
The 47th International ACM SIGIR Conference on Research and Development in
Information Retrieval; July 14-18, 2024; Washington D.C., USA††isbn:
978-1-4503-XXXX-X/18/06††ccs: Computing methodologies Knowledge representation
and reasoning††ccs: Computing methodologies Semantic networks Figure 1. The
diversity and imbalance nature in MMKGs. We report the modalities included in
each MMKG in (a) and the statistical information about the modality
information distribution across dataset/entity in TIVA in (b).

##  1\. Introduction

Multi-modal knowledge graphs (MMKGs) (Liu et al., 2019; Chen et al., 2024b,
2023a) represent a rich extension of traditional knowledge graphs (KGs) (Wang
et al., 2017a; Liang et al., 2022), enriching the structure triples in the
form of (head entity, relation, tail entity) with comprehensive multi-modal
entity attributes. MMKGs are representative and have become the significant
infrastructure in many AI fields such as recommender systems (Wang et al.,
2019a), computer vision (Gu et al., 2023), and nature language processing
(Yasunaga et al., 2021; Zhang et al., 2023b; Chen et al., 2023c).

Investigating how to leverage structural knowledge in KGs and MMKGs is
beneficial for many areas of AI research. However, KGs and MMKGs usually face
critical incompleteness issues as they entail many unobserved factual
knowledge. This phenomenon makes knowledge graph completion (KGC) (Liang et
al., 2022) a significant task to automatically discover new knowledge in the
given KGs. Conventional KGC methods (Bordes et al., 2013; Yang et al., 2015;
Trouillon et al., 2016; Sun et al., 2019) generally emphasize learning
structural embeddings to model the triple structure and measure the triple
plausibility. Additionally, MMKGs accomplish the task more holistically by
incorporating information from different modalities such as images and text
into the KGC model, extending the task to multi-modal KGC (MMKGC) (Xu et al.,
2022; Cao et al., 2022; Li et al., 2023).

Existing MMKGC methods (Xie et al., 2017; Sergieh et al., 2018; Wang et al.,
2021; Li et al., 2023) typically treat the multi-modal information of entities
as auxiliary multi-modal embeddings and incorporate them to enhance the entity
representations. However, these methods neglect two vital problems for MMKGC
in real-world scenarios: the diversity problem and the imbalance problem. The
diversity problem arises from the diverse modalities in contemporary
information media. As the use of varied modalities such as numeric (Wang et
al., 2022), audio (Wang et al., 2023), and video (Pan et al., 2022) data
increases in MMKG construction, a summary in Figure 1(a) highlights the range
of modalities included in the different MMKGs. However, the existing MMKGC
methods are primarily designed for the prevalent image and text modalities,
limiting their generalization to diverse modalities. The imbalance problem
arises from the imbalanced distribution of modality information (Chen et al.,
2023b), implying that some important modality information would be missing in
the real-world KGs. Figure 1(b) shows the total amount and entity-wise
distribution of different modalities in the TIVA (Wang et al., 2023),
confirming the uneven distribution of modalities across the datasets. Existing
works do not focus on this problem or solve it by naive initialization (Xie et
al., 2017), leading to inadequate utilization of modality information.

Aiming to solve these key issues of MMKGC in the wild, we propose a novel
framework NativE, which can process and fuse Numeric, Audio, Text, Image,
Video, and any other modalities into the Embedding space with adaptive fusion
and adversarial augmentation. NativE comprises two key modules called
Relation-guided Dual Adaptive Fusion (ReDAF) module and Collaborative Modality
Adversarial Training (CoMAT) module respectively. The ReDAF facilitates
diverse multi-modal fusion with any input modalities with relation guidance to
moderate the weight of each modality. The CoMAT designs an adversarial
training strategy to augment the imbalanced modality information with
Wasserstein distance (Arjovsky et al., 2017) based objective. In addition, we
undertake a theoretical analysis to prove the rationale of our designs. We
construct a new benchmark called WildKGC with five MMKG datasets to evaluate
our method against 21 recent baselines with further exploration. Our
contributions can be summarized as three-fold:

  * •

Innovative framework. We propose a new framework called NativE to address the
diversity and imbalance problems of MMKG in the wild. NativE can achieve
adaptive fusion with any modality with relation guidance to address the
diversity problem of MMKG while augmenting the imbalanced modality information
by collaborative modality adversarial training.

  * •

Theoretical analysis. We performed a theoretical analysis to prove the
legitimacy and soundness of our design.

  * •

Comprihensive experiments. We construct a new benchmark to evaluate the MMKGC
task in the wild and conduct comprehensive experiments to demonstrate the
effectiveness, efficiency, and generalization of our NativE framework.

##  2\. Related Works

###  2.1. Knowledge Graph Completion

Knowledge graph completion (KGC) (Liang et al., 2022) is an essential task in
the community, aiming to discover the unobserved triples in the given KG.
Conventional KGC methods are usually embedding-based, which embed the entities
and relations of KGs into the continuous vector space and learn the embeddings
based on the existing triple structure. This technique is also called
knowledge graph embedding (KGE) (Wang et al., 2017a). Typically, conventional
KGE models are designed with different score functions to measure the
plausibility of triples with a general target to assign higher scores for
positive triples and lower scores for negative triples.

The existing KGE models can be divided into two main categories: translation-
based methods and semantic matching methods. Translation-based methods such as
TransE (Bordes et al., 2013), TransD (Ji et al., 2015), RotatE (Sun et al.,
2019), OTE (Tang et al., 2020), and PairRE (Chao et al., 2021) modeling the
triple structure as relational translation from the head entity to the tail
entity, which design distance-based score functions as the plausibility
measurement. Semantic matching methods such as DistMult (Yang et al., 2015),
ComplEx (Trouillon et al., 2016), TuckER (Balazevic et al., 2019) exploit
similarity-based scoring functions based on tensor decomposition. Some methods
(Dettmers et al., 2018; Zhu et al., 2021; Zhang et al., 2023c; Yao et al.,
2019; Liang et al., 2024, 2023) also attempt to extract structural semantics
with deep neural networks.

###  2.2. Multi-modal Knowledge Graph Completion

Multi-modal knowledge graph completion (MMKGC) further considers utilizing the
complex multi-modal information in the MMKGs to benefit the KGC. Current
mainstream methods usually extend the conventional KGE models with more
flexible multi-modal embeddings of entities such as visual embeddings and
textual embeddings. These embeddings are extracted with pre-trained models and
represent the entity feature from multi-views. In our taxonomy, the MMKGC
methods can be further divided into three categories. The first category is
the modal fusion methods. These methods (Xie et al., 2017; Sergieh et al.,
2018; Pezeshkpour et al., 2018; Wang et al., 2019b, 2021; Cao et al., 2022;
Lee et al., 2023; Zhang et al., 2024; Chen et al., 2024a) design elegant
approaches to achieve multi-modal fusion in the same representation space. The
second category is modal ensemble methods. These methods (Zhao et al., 2022;
Li et al., 2023) learn the respective models for each modality and make joint
predictions with ensemble learning. The third category is the negative
sampling (NS) enhanced methods. As mentioned before, NS is an important
technology for KGE model training. Therefore, some methods (Zhang et al.,
2023a; Xu et al., 2022; Zhang and Zhang, 2022) attempt to enhance the NS
process and generate high-quality negative samples by utilizing the multi-
modal information. The problem scenarios of existing MMKGC methods are
relatively simple and usually consider the text and image modalities, while
some of the work just considers the numerical modality. Also, they do not
consider the data imbalance problem in MMKGs. In our work, we plan to
accomplish the MMKGC task with a more unified perspective for more modalities
and more complex multi-modal data distribution in the wild.

###  2.3. Generative Adversarial Networks

Generative adversarial networks (GAN) (Goodfellow et al., 2014) is a milestone
progress in the field of deep learning. GAN proposes to train a pair of
discriminator and generator by playing a min-max game between them and achieve
better performance, which has been widely used in various fields such as
computer vision (Arjovsky et al., 2017; Karras et al., 2019), natural language
processing (Yu et al., 2017; Croce et al., 2020), information retrieval (Wang
et al., 2017b; Zhang et al., 2022), and recommender systems (Wei et al., 2023;
Yu et al., 2017; Wang et al., 2020). In the KGC field, there are some methods
(Cai and Wang, 2018; Wang et al., 2018; Tang et al., 2022; Lu et al., 2022)
employs a GAN-based framework to enhance the negative sampling (Bordes et al.,
2013) process. For example, KBGAN (Cai and Wang, 2018) utilizes reinforcement
learning (RL) with GAN to learn a better sampling policy. MMKRL (Lu et al.,
2022) designs an adversarial training strategy for MMKGC. However, these
methods tend to be oriented towards conventional KGC and often require RL,
which leads to a more limited effect. Our work is the first to involve multi-
modal entity information in MMKGCs and propose a unified framework to enhance
MMKGC models.

##  3\. Task Definition

Figure 2. The overview of our NativE framework. NativE consists of two main
modules called relation-guided dual adaptive fusion (ReDAF) module and
collaborative modality adversarial training (CoMAT) module respectively. ReDAF
is designed to fuse any input modality with modality adaptive weights and
relational guidance. CoMAT aims to augment the imbalanced modality information
in an adversarial manner by constructing synthetic triples to play a min-max
game.

A KG can be typically represented as
$\mathcal{K}=(\mathcal{E},\mathcal{R},\mathcal{T})$ where
$\mathcal{E},\mathcal{R}$ are the entity set, the relation set respectively.
$\mathcal{T}=\\{(h,r,t)\mid h,t\in\mathcal{E},r\in\mathcal{R}\\}$ is the
triple set. Furthermore, MMKGs have a modality set denoted as $\mathcal{M}$,
encapsulating different modalities (image $I$, text $T$, numeric $N$, audio
$A$, video $V$) in the MMKGs. For a given modality $m\in\mathcal{M}$, the set
of modal information is denoted as $\mathcal{X}_{m}$. For an entity
$e\in\mathcal{E}$, its modality information $m$ is denoted as
$\mathcal{X}_{m}(e)$, which is an empty set if the corresponding modal
information is missing. For different modalities, the elements in it have
different forms. For instance, $\mathcal{X}_{m}(e)$ can be a set of images
when $m=I$ and some video clips when $m=V$. Note that the graph structure
($S$) is also an intrinsic modality for each entity and the structural
information is already embodied in the triple set $\mathcal{T}$.

The general purpose of the MMKGC task is to learn a score function
$\mathcal{F}:\mathcal{E}\times\mathcal{R}\times\mathcal{E}\rightarrow\mathbb{R}$
to discriminate the plausibility of a given triple $(h,r,t)$. In this context,
a higher score implies a more plausible triple. To enable differentiable
computations with gradient-based optimization, entities and relations are
embedded into continuous vector spaces, which is called knowledge graph
embedding (KGE). The embeddings of entity $e$ and relation $r$ are denoted as
$\bm{e}\in\mathbb{R}^{d_{e}}$ and $\bm{r}\in\mathbb{R}^{d_{r}}$, where
$d_{e},d_{r}$ are the embedding dimensions. Convectional KGE models focus
solely on the structural information of triple, rendering these embeddings as
structural embeddings. The score function $\mathcal{F}(h,r,t)$ leverages the
structural embeddings to calculate the triple scores. For MMKGs, the multi-
modal entity information should also be considered in the score function,
implying additional embeddings $\bm{e}_{m}$ for varying modalities
$m\in\mathcal{M}$ to represent the multi-modal feature of an entity $e$. This
expansion consequently complicates the score function by considering multi-
modal embedding integration. During the training stage, negative sampling (NS)
(Bordes et al., 2013) is widely used to construct manual negative triples for
contrastive learning as KGs usually consist of only observed positive triples.
Given a positive triple $(h,r,t)$, the head $h$ or tail $t$ is randomly
replaced by another entity $e\in\mathcal{E}$ in the NS process. The negative
triple set can be denoted as
$\mathcal{T}^{\prime}=\\{(h^{\prime},r,t)\mid(h,r,t)\in\mathcal{T}\cap
h^{\prime}\in\mathcal{E}\setminus\\{h\\}\\}\cup\\{(h,r,t^{\prime})\mid(h,r,t)\in\mathcal{T}\cap
t^{\prime}\in\mathcal{E}\setminus\\{t\\}\\}$. During the inference stage, the
KGC model is usually evaluated with the link prediction task (Bordes et al.,
2013). The target of link prediction is to predict the missing head or tail
entity in the given query $(?,r,t)$ or $(h,r,?)$. For instance, in tail
prediction, the entire set of entities $\mathcal{E}$ will be the candidate set
during evaluation. For each $e\in\mathcal{E}$, the plausibility score of the
triple $(h,r,e)$ is calculated and then ranked across the entire candidate
set. A higher rank of the ground truth $(h,r,t)$ represents better model
performance.

##  4\. Methodology

In this section, we will introduce the proposed MMKGC framework in detail. We
refer to our model as NativE, designed to represent and combine multiple data
modalities (Numeric, Audio, Text, Image, Video, and more) from MMKGs into
multi-modal Embeddings with adversarial augmentation. This ability readies
NativE to deliver robust prediction capabilities in the wild while facing
diversity and imbalance problems. In NativE, we design two new modules called
relation-guided dual adaptive fusion and collaborative modality adversarial
training to address the problems mentioned before.

###  4.1. Modality Encoding

To leverage the modality information, we first perform modality encoding to
capture modality feature, which is a very common step in different MMKGC
methods. The raw feature $f_{m}$ of the entity $e$ in the modality $m$ is
extracted as:

(1)
$f_{m}=\frac{1}{|\mathcal{X}_{m}(e)|}\sum\nolimits_{x_{e,m}\in\mathcal{X}_{m}(e)}\mathbf{PE}_{m}(x_{e,m})$

where $x_{e,m}$ is one of the modality $m$ elements for entity $e$ and
$\mathbf{PE}_{m}$ is the corresponding pre-trained encoder. The encoders,
typically pre-trained on extensive datasets, extract deep semantic
characteristics distinctive to each modality. For example, we can employ BERT
(Devlin et al., 2019) as the textual encoder and VGG16 (Simonyan and
Zisserman, 2015) as the image encoder. The numeric information can be also
extended as a sequence and employ BERT to capture its modality information.
The pre-trained encoders are frozen to capture modality features and will not
be fine-tuned during the training process. Moreover, the modal embedding for
entity $e\in\mathcal{E}$ of modality $m\in\mathcal{M}$ is indicated as
$\bm{e}_{m}=\mathcal{P}_{m}\left(f_{m}\right)\in\mathbb{R}^{d_{e}}$, where
${d_{e}}$ is the embedding dimension and $\mathcal{P}_{m}$ is a projection
layer for the modality $m$, aiming to project the different modal embeddings
into the same vector space. Each $\mathcal{P}_{m}$ comprises a two-layer MLP
with ReLU (Glorot et al., 2011) as the activation function.

The modality encoding process yields distinct modal embeddings $\bm{e}_{m}$
for each entity $e$. In the MMKG context, embeddings from other modalities
serve as auxiliary information to enhance the structure of the entity. For
uniform representation of multi-modal information, we assign the structural
modality as $S$ and the structural embeddings $\bm{e}$ as $\bm{e}_{S}$.
However, structural embeddings for entities deviate from other modalities.
They are set to be learnable parameters to fit the triple structural
information rather than relying on the modality encoding offered by pre-
trained encoders.

###  4.2. Relation-guided Dual Adaptive Fusion

To extract the rich semantic information contained within entities, it is
customary to fuse these multi-modal embeddings. Existing methods often employ
fusion mechanisms such as concatenation (Sergieh et al., 2018), dot product
(Li et al., 2023), or gating (Wang et al., 2021) to accomplish modality
fusion. Nevertheless, these methods are primarily designed for specific
modalities such as text and images, which may not adequately serve scenarios
intertwined with a richer mix of modalities. Furthermore, given the inherent
imbalance in KGs, different modalities need to play different roles and offer
diverse evidence for robust prediction within varying contexts.

To address the mentioned problems, we propose a Relation-guided Dual Adaptive
Fusion (abbreviated as ReDAF) module within our framework. ReDAF includes an
adaptive fusion mechanism using a set of adaptive weights $\omega_{m}$ for
different modalities, which can be dynamically adjusted when the modal
information is imbalanced. For example, when a certain modal is missing and
the corresponding modal embedding is randomly initialized, the adaptive weight
will be decreased for this modality. Meanwhile, we design a relational-wise
temperature $\zeta_{r}$ to further modulate the weight distribution, thereby
providing relational context for entity representation. This leads to dynamic
weight adjustments populated both over different entities and under different
relational contexts, which is the reason we name this module dual adaptive
fusion. In more specific terms, the head entity $h$ of the triple $(h,r,t)$
has a relational context $r$ and the adaptive weight for each modal $m$ of $h$
is denoted as:

(2)
$\omega_{m}(h,r)=\frac{\exp(\mathcal{V}\odot\mathrm{Tanh}(\bm{h}_{m})/\sigma(\zeta_{r}))}{\sum_{n\in\mathcal{M}\cup\\{S\\}}\exp(\mathcal{V}\odot\mathrm{Tanh}(\bm{h}_{n})/\sigma(\zeta_{r}))}$

where $\mathcal{V}$ is a learnable vector and $\odot$ is the point-wise
operator. $\mathrm{Tanh}()$ is the tanh function. $\sigma$ represents the
sigmoid function to limit the relational-wise temperature in $(0,1)$, aiming
to amplify the differences between different modal weights. With the adaptive
weights, the joint embedding of the head entity $h$ is aggregated as:

(3) $\bm{h}_{joint}=\sum\nolimits_{m\in
M\cup\\{S\\}}\omega_{m}(h,r)\bm{h}_{m}$

The joint embedding $\bm{t}_{joint}$ of tail $t$ can be also obtained
similarly.

Classical MMKGC methods typically employ entity-centric modality fusion
methods, regardless of their triple context. The strength of our design lies
in its ability to dynamically modulate the modality weights for different
entities across different triples, permitting these weights to be dynamically
adjusted by their relational context. Therefore, ReDAF facilitates dual-
adaptive multi-modal fusion, considering both the modal information of the
entity and the relational context. Furthermore, ReDAF is a versatile modality
fusion module, capable of processing an unlimited number of input modalities
rather than considering specified modalities.

Once obtaining the joint embedding of entities, we employ a RotatE (Sun et
al., 2019) score function to discriminate the triple plausibility, the score
function is denoted as:

(4) $\mathcal{F}(h,r,t)=-||\bm{h}_{joint}\circ\bm{r}-\bm{t}_{joint}||$

where $\circ$ is the rotation operator in complex space. A higher score
represents higher triple plausibility. We chose RotatE as the score function
as RotatE can model the most common relational patterns. During training, we
employ a negative sampling-based loss function to optimize the parameters,
which can be represented as:

(5) $\displaystyle\mathcal{L}_{kgc}=$
$\displaystyle\sum\nolimits_{(h,r,t)\in\mathcal{T}}-\log\sigma(\gamma+\mathcal{F}(h,r,t))$
$\displaystyle-$
$\displaystyle\sum_{i=1}^{K}p(h_{i}^{\prime},r_{i}^{\prime},t_{i}^{\prime})\log\sigma(-\mathcal{F}(h_{i}^{\prime},r_{i}^{\prime},t_{i}^{\prime})-\gamma)$

where $\sigma$ is the sigmoid function; $\gamma$ is a fixed margin;
$(h_{i}^{\prime},r_{i}^{\prime},t_{i}^{\prime})\in\mathcal{T}^{\prime},(i=1,2,\dots,K)$
are $K$ negative samples for triple $(h,r,t)$. Besides,
$p(h_{i}^{\prime},r_{i}^{\prime},t_{i}^{\prime})$ is the self-adversarial
weight (Sun et al., 2019) proposed in RotatE, it can be denoted as:

(6)
$p(h_{i}^{\prime},r_{i}^{\prime},t_{i}^{\prime})=\frac{\exp\left(\beta\mathcal{F}(h_{i}^{\prime},r_{i}^{\prime},t_{i}^{\prime})\right)}{\sum_{j=1}^{k}\exp\left(\beta\mathcal{F}(h_{j}^{\prime},r_{j}^{\prime},t_{j}^{\prime})\right)}$

where $\beta$ is a temperature to control the negative triple weight. This is
a setting commonly used in KGC models.

###  4.3. Collaborative Modality Adversarial Training

The key function of the proposed ReDAF framework, despite achieving dual
adaptive multi-modal fusion amongst imbalanced and diverse multi-modal data,
is essentially feature selection for prediction, leaving the original
imbalanced modality information intact. Inspired by the idea of generative
adversarial networks (Goodfellow et al., 2014; Tygar, 2011), we propose a
Collaborative Modality Adversarial Training (abbreviated as CoMAT) module to
augment the modality embeddings. The CoMAT module enhances the multi-modal
embeddings through adversarial training, using entity-specific collaborative
modality data to balance the multi-modal information distribution.

Drawing from the design principles of classical Wasserstein GAN (WGAN)
(Arjovsky et al., 2017), our goal is to establish a min-max game between the
discriminator $\mathcal{D}$ and the generator $\mathcal{G}$ as:

(7) $\max_{\mathcal{D}}\min_{\mathcal{G}}\mathbb{E}_{x\sim
p_{real}}[{\mathcal{D}}(x)]-\mathbb{E}_{x^{\prime}\sim\mathcal{G}}[{\mathcal{D}}(x^{\prime})]$

where $x$ is the data sample. In this min-max game, the discriminator
$\mathcal{D}$ is assigned to discriminate the input data sample $x$ with a
score while the generator $\mathcal{G}$ aims to generate a synthetic data
sample $x^{\prime}$. With the adversarial training, the generator
$\mathcal{G}$ adapts to the actual distribution of the data $x$ and the
discriminator $\mathcal{D}$ can learn to judge the plausibility of the input
data.

In our scenario, the data $x$ corresponds to the entity embeddings in the KG.
As mentioned previously, each entity is represented by several modal
embeddings, which can denoted as
$\bm{e}_{real}=\\{\bm{e}_{m_{1}},\bm{e}_{m_{2}}\dots;\bm{e}_{m_{N}}\\}$ where
$m_{i}\in\mathcal{M}\cup\\{S\\}$ is each modality. We expect to design a pair
of $\mathcal{G}$ and $\mathcal{D}$ to learn the joint distribution of the
multi-modal embeddings and augment the imbalanced multi-modal information.
Therefore, $\mathcal{G}$ seeks to produce a collection of synthetic
embeddings, defined as:
$\bm{e}_{syn}=\\{\bm{e}^{\prime}_{m_{1}},\bm{e}^{\prime}_{m_{2}}\dots;\bm{e}^{\prime}_{m_{N}}\\}$,
and these embeddings can form a new synthetic entity denoted as $e^{*}$. By
facilitating adversarial training between real and synthetic entities, we can
enrich the multi-modal embeddings of the entity.

####  4.3.1. The Design of Generator

In the design of CoMAT, the generator $\mathcal{G}$ is implemented by a two-
layer MLP. $\mathcal{G}$ processes the input random noisy $z$ and the concated
real multi-modal embedding $\bm{e}_{real}$ to generate augmented synthetic
embeddings:

(8)
$\bm{e}_{syn}=\mathcal{G}(\bm{e}_{real},z)=\mathbf{MLP}_{2}\big{(}\delta\left(\mathbf{MLP}_{1}[\bm{e}_{real},z]\big{)}\right)$

where $\bm{e}_{real}$ is obtained by concatenating the different multi-modal
embeddings collaboratively (as illustrated in Figure 2) and the generated
synthetic embedding $\bm{e}_{syn}$ has the same shape of $\bm{e}_{real}$. We
can split $\bm{e}_{syn}$ by dimension to yield the generated embedding
$\bm{e}^{\prime}_{m}$ of each modality generated by $\mathcal{G}$. The
ultimate goal of this design is to emulate the probability distribution of the
multimodal embedding for each entity. Given a triple $(h,r,t)$, synthetic
embeddings for both head and tail can be generated for both head and tail,
denoted as $\bm{h}_{gen}$ and $\bm{t}_{gen}$ respectively. Besides, the
corresponding synthetic entities are denoted as $h^{*},t^{*}$.

####  4.3.2. The Design of Discriminator

The next step is to design the discriminator $\mathcal{D}$ to scoring the
synthetic embeddings. In existing works in other fields (Arjovsky et al.,
2017; Liu et al., 2022), $\mathcal{D}$ is usually implemented with another
two-layer MLP. However, in the KGC task, the score function $\mathcal{F}$
mentioned in Equation 4 serves as a natural alternative for the discriminator.
For a given triple $(h,r,t)$ and the synthetic entities $h^{*},t^{*}$
generated by $\mathcal{G}$, we can construct a series of synthetic triples
denoted as $\\{(h^{*},r,t),(h,r,t^{*}),(h^{*},r,t^{*})\\}$. These synthetic
triples’ scores can be computed with the ReDAF module and the score function
$\mathcal{F}$, indicating the plausibility of the synthetic embeddings in the
triple context. We can denote these synthetic triples as a synthetic set
$\mathcal{S}(h,r,t)$. Therefore, in the design of CoMAT, the adversarial
training loss can be denoted as:

(9)
$\mathcal{L}_{adv}=\sum_{(h,r,t)\in\mathcal{T}}\Big{(}-\mathcal{F}(h,r,t)+\frac{1}{|\mathcal{S}|}\sum_{(h^{*},r,t^{*})\atop\in\mathcal{S}(h,r,t)}\mathcal{F}(h^{*},r,t^{*})\Big{)}$

Note that the synthetic entities, analogous to real entities, also have
several multi-modal embeddings like the real entities. They can obtain their
joint embeddings with the ReDAF and calculate the final score subsequently.
Therefore, the final min-max game between $\mathcal{D}$ and $\mathcal{G}$ is
represented as:

(10) $\min_{\mathcal{D}}\max_{\mathcal{G}}\mathcal{L}_{adv}$

According to the previous setting, the parameters of $\mathcal{G}$ encompass
all the parameters in the two-layer MLP, while the parameters of $\mathcal{D}$
include the existing real embeddings, the extra parameters mentioned in ReDAF.
The two parts are iteratively optimized with the adversarial loss to achieve
convergence.

Table 1. Statistical information of the five MMKGs in our WildKGC benchmark.
We report the statistical information in the KGs. For each modality, we list
its feature dimension and the number of entities with the corresponding modal
information.

Dataset | #Entity | #Relation | #Train | #Valid | #Test | Image | Text | Numeric | Audio | Video
---|---|---|---|---|---|---|---|---|---|---
Num | Dim | Num | Dim | Num | Dim | Num | Dim | Num | Dim
MKG-W (Xu et al., 2022) | 15000 | 169 | 34196 | 4276 | 4274 | 14463 | 383 | 14123 | 384 | - | - | - | - | - | -
MKG-Y (Xu et al., 2022) | 15000 | 28 | 21310 | 2665 | 2663 | 14244 | 383 | 12305 | 384 | - | - | - | - | - | -
DB15K (Liu et al., 2019) | 12842 | 279 | 79222 | 9902 | 9904 | 12818 | 4096 | 9078 | 768 | 11022 | 768 | - | - | - | -
TIVA (Wang et al., 2023) | 11858 | 16 | 20071 | 2000 | 2000 | 11636 | 2048 | 11858 | 300 | - | - | 2441 | 128 | 10269 | 2048
KVC16K (Pan et al., 2022) | 16015 | 4 | 180190 | 22523 | 22525 | 14822 | 768 | 14822 | 768 | - | - | 14822 | 768 | 14822 | 768

###  4.4. Overall Training Objective

The final training objective of NativE combines the above-mentioned KGC
training loss $\mathcal{L}_{kgc}$ and the adversarial loss $\mathcal{L}_{adv}$
together. The discriminator $\mathcal{D}$ would minimize $\mathcal{L}_{kgc}$
and discriminate the synthetic entities generated by $\mathcal{G}$.
Conversely, the generator $\mathcal{G}$ aims to generate high-score synthetic
entities. Therefore, the training objective of $\mathcal{D}$ and $\mathcal{G}$
can be expressed separately as:

(11)
$\begin{cases}\mathcal{L}_{\mathcal{D}}=\mathcal{L}_{kgc}+\lambda_{1}\mathcal{L}_{adv}\\\
\mathcal{L}_{\mathcal{G}}=-\mathcal{L}_{adv}+\lambda_{2}\mathcal{L}_{gp}\end{cases}$

where $\lambda_{1},\lambda_{2}$ are the loss weights, $\mathcal{L}_{gp}$ is
the gradient penalty (Gulrajani et al., 2017), commonly used for stable WGAN
training, denoted as:

(12)
$\mathcal{L}_{gp}=-\sum\nolimits_{(h^{*},r,t^{*})\in\mathcal{S}(h^{*},r,t^{*})}\left(||\nabla\mathcal{F}(h^{*},r,t^{*})||_{2}-1\right)^{2}$

These two losses will be iteratively minimized during training.

###  4.5. Theoretical Analysis

In the previous sections, we give an intuitive motivation and design for the
CoMAT module. We further provide theoretical analysis to discuss the rationale
for our design. The success of the WGAN restriction discriminator’s scoring
function relies on its adherence to the K-Lipschitz condition, typically
accomplished through a two-layer MLP with gradient penalty (Arjovsky et al.,
2017; Liu et al., 2022). However, our design employs the RotatE score function
$\mathcal{F}$ as a discriminator. We can prove that function $\mathcal{F}$ is
also a K-Lipschitz function for $h$ and $t$.

Proposition. Let function $\mathcal{F}=-||\bm{h}\circ\bm{r}-\bm{t}||$. Then
for any $h_{1},h_{2}$ in the function domain, there exists a scalar $K$ which
satisfies:

(13) $||\mathcal{F}(h_{1},r,t)-\mathcal{F}(h_{2},r,t)||\leq
K||\bm{h}_{1}-\bm{h}_{2}||$

Proof. According to the properties of trigonometric inequalities, we have the
following derivations:

(14) $\displaystyle LHS$
$\displaystyle=\big{|}\big{|}||\bm{h}_{1}\circ\bm{r}-\bm{t}||-||\bm{h}_{2}\circ\bm{r}-\bm{t}||\big{|}\big{|}$
$\displaystyle\leq||(\bm{h}_{1}\circ\bm{r}-\bm{t})-(\bm{h}_{2}\circ\bm{r}-\bm{t})||$
$\displaystyle=||(\bm{h}_{1}-\bm{h}_{2})\circ
r||=||\bm{r}||\times||\bm{h}_{1}-\bm{h}_{2}||$

Besides, in the RotatE score function, the relation embeddings $\bm{r}$ are
limited with unit modulus (Sun et al., 2019) which means $||\bm{r}||\leq 1$
for any $r\in\mathcal{R}$. Let $K=1$, the proof is complete. This establishes
that $\mathcal{F}$ is K-Lipschitz for head $h$. A similar argument confirms
the same for the tail t. From the preceding analyses, it emerges that our
WGAN-based design, harnessing the score function from the MMKGC model, is
theoretically sound and rational for scoring for the synthetic entities. It
makes the design of the whole framework more collaborative and natural.
Indeed, CoMAT serves as a versatile framework that can be extendable to other
embedding-based MMKGC models as well. We will go through the empirical
evaluation to further demonstrate its effectiveness and generalization.

##  5\. Experiments and Evaluation

In this section, we first introduce our experimental procedure and settings,
followed by a comprehensive discussion of extensive experiments to highlight
the strengths of our method across a variety of scenarios. The following six
research questions (RQ) are the key questions that we explore in the
experiments.

  * RQ1.

Can our model NativE outperform the existing baseline and make substantial
progress in the MMKGC task?

  * RQ2.

Can NativE maintain robust performance in the MMKGC task when the modality
information is imbalanced?

  * RQ3.

How much do each module in the NativE contribute to the final results? Are
these modules reasonably designed?

  * RQ4.

Is the CoMAT strategy we designed universal and general enough to be applied
in other MMKGC models?

  * RQ5.

How does the training efficiency of our model compare to existing methods?

  * RQ6.

Are there intuitive cases to straightly demonstrate the effectiveness of
NativE?

###  5.1. Experiment Settings

Table 2. The main MMKGC results on WildKGC benchmark. We list the modalities
considered by each method where S/I/T denotes Structure/Image/Text. ”All”
represents that the method can process any number of input modalities. The
best results in baselines are underlined and we highlight the SOTA results in
bold. The flag ${\dagger}$ denotes the adversarial training baselines.

Method | Modality | MKG-W | MKG-Y | DB15K | KVC16K | TIVA
---|---|---|---|---|---|---
MRR | Hit@1 | MRR | Hit@1 | MRR | Hit@1 | Hit@10 | MRR | Hit@1 | Hit@10 | MRR | Hit@1 | Hit@10
Uni-modal KGC Methods
TransE | S | 29.19 | 21.06 | 30.73 | 23.45 | 24.86 | 12.78 | 47.07 | 8.54 | 0.64 | 23.42 | 83.85 | 83.20 | 84.15
DistMult | S | 20.99 | 15.93 | 25.04 | 19.33 | 23.03 | 14.78 | 39.59 | 6.37 | 3.03 | 12.61 | 82.27 | 81.15 | 84.22
ComplEx | S | 24.93 | 19.09 | 28.71 | 22.26 | 27.48 | 18.37 | 45.37 | 12.85 | 7.48 | 23.18 | 80.67 | 77.67 | 86.10
RotatE | S | 33.67 | 26.80 | 34.95 | 29.10 | 29.28 | 17.87 | 49.66 | 14.33 | 8.25 | 26.17 | 84.59 | 83.47 | 86.95
PairRE | S | 34.40 | 28.24 | 32.01 | 25.53 | 31.13 | 21.62 | 49.30 | - | - | - | - | - | -
Multi-modal KGC Methods
IKRL | S+I | 32.36 | 26.11 | 33.22 | 30.37 | 26.82 | 14.09 | 49.09 | 11.11 | 5.42 | 22.39 | 67.71 | 63.72 | 75.67
TBKGC | S+I+T | 31.48 | 25.31 | 33.99 | 30.47 | 28.40 | 15.61 | 49.86 | 5.39 | 0.35 | 15.52 | 81.57 | 78.75 | 86.05
TransAE | S+I | 30.00 | 21.23 | 28.10 | 25.31 | 28.09 | 21.25 | 41.17 | 10.81 | 5.31 | 21.89 | 79.57 | 74.95 | 88.07
MMKRL${\dagger}$ | S+I+T | 30.10 | 22.16 | 36.81 | 31.66 | 26.81 | 13.85 | 49.39 | 8.78 | 3.89 | 18.34 | 85.03 | 81.92 | 90.10
RSME | S+I | 29.23 | 23.36 | 34.44 | 31.78 | 29.76 | 24.15 | 40.29 | 12.31 | 7.14 | 22.05 | 40.01 | 30.55 | 51.35
VBKGC | S+I+T | 30.61 | 24.91 | 37.04 | 33.76 | 30.61 | 19.75 | 49.44 | 14.66 | 8.28 | 27.04 | 74.07 | 66.87 | 85.85
OTKGE | S+I+T | 34.36 | 28.85 | 35.51 | 31.97 | 23.86 | 18.45 | 34.23 | 8.77 | 5.01 | 15.55 | 35.28 | 30.45 | 41.98
IMF | S+I+T | 34.50 | 28.77 | 35.79 | 32.95 | 32.25 | 24.20 | 48.19 | 12.01 | 7.42 | 21.01 | 55.46 | 41.87 | 77.57
QEB | All | 32.38 | 25.47 | 34.37 | 29.49 | 28.18 | 14.82 | 51.55 | 12.06 | 5.57 | 25.01 | 74.25 | 66.10 | 88.35
VISTA | S+I+T | 32.91 | 26.12 | 30.45 | 24.87 | 30.42 | 22.49 | 45.94 | 11.89 | 6.97 | 21.27 | 76.07 | 70.67 | 86.60
Negative Sampling Methods
KBGAN ${\dagger}$ | S | 29.47 | 22.21 | 29.71 | 22.81 | 25.73 | 9.91 | 51.93 | 13.72 | 7.54 | 25.88 | 85.44 | 82.45 | 90.10
MANS | S+I | 30.88 | 24.89 | 29.03 | 25.25 | 28.82 | 16.87 | 49.26 | 10.42 | 5.21 | 20.45 | 85.70 | 82.70 | 90.62
MMRNS | S+I+T | 35.03 | 28.59 | 35.93 | 30.53 | 32.68 | 23.01 | 51.01 | 13.31 | 7.51 | 24.68 | 83.12 | 83.05 | 83.25
NativE | S+I+T | 36.58 | 29.56 | 39.04 | 34.79 | 36.74 | 26.87 | 54.65 | 15.26 | 8.56 | 28.29 | 91.27 | 90.57 | 91.27
NativE | All | 36.58 | 29.56 | 39.04 | 34.79 | 37.16 | 28.01 | 54.13 | 15.76 | 9.23 | 28.55 | 92.10 | 91.40 | 92.85
Improvement | +4.42% | +3.39% | +5.40% | +3.05% | +13.71% | +15.75% | +5.24% | +7.50% | +11.47% | +5.58% | +7.46% | +10.05% | +2.46%

####  5.1.1. Datasets

To better explore the MMKGC tasks in a more complex and diverse environment,
we construct a new MMKGC benchmark including five different datasets. While
four of the datasets are from prior studies, one is completely new. Our
benchmark includes the following datasets:

  * •

MKG-Y and MKG-W (Xu et al., 2022) are two MMKGs derived from YAGO (Suchanek et
al., 2007) and Wikidata (Vrandecic and Krötzsch, 2014) with images and texts
proposed by (Xu et al., 2022).

  * •

DB15K (Liu et al., 2019) is an MMKG with image, text, and numerical
information proposed by (Liu et al., 2019), which is a subset of DBpedia
(Lehmann et al., 2015).

  * •

TIVA (Wang et al., 2023) is an MMKG with image, text, audio, and video
information modality with 12K entities.

  * •

KVC16K (Pan et al., 2022) is modified from KuaiPedia (Pan et al., 2022), a
video concept encyclopedia. We reorganize it into an MMKG and leverage the
image/text/audio/video features provided by the original authors.

We refer to our benchmark as WildKGC with the 5 MMKGs. Detailed information
about the datasets is presented in Table 1. The raw modality features are
inherited from the original datasets.

####  5.1.2. Task and Evaluation Protocol

We conduct link prediction (Bordes et al., 2013) task on the five datasets,
which is a significant task of KGC. We have introduced the setting of link
prediction in Section 3. Following existing works, we use rank-based metrics
(Sun et al., 2019) like mean reciprocal rank (MRR) and Hit@K (K=1, 3, 10) to
evaluate the results.

Besides, the filter setting (Bordes et al., 2013) is applied to remove the
candidate triples already existing in the training set for fair comparisons.

####  5.1.3. Baseline Methods for Comparisons

In our experiments, we employ 21 different state-of-the-art KGC and MMKGC
models as our baselines for a comprehensive comparison and analysis. The
baselines can be divided into four categories: uni-modal (conventional) KGC
methods, multi-modal KGC methods, negative sampling methods, and numeric-aware
KGC methods.

i) Uni-modal KGC methods. We select 5 state-of-the-art uni-modal KGC methods
including TransE (Bordes et al., 2013), DistMult (Yang et al., 2015), ComplEx
(Trouillon et al., 2016), RotatE (Sun et al., 2019), PairRE (Chao et al.,
2021), which design elegant score functions and learn the structural
embeddings of the given KG without any multi-modal information.

ii) Multi-modal KGC methods. We employ 10 different MMKGC methods that
consider both multi-modal information and the triple structural information
including IKRL (Xie et al., 2017), TBKGC (Sergieh et al., 2018), TransAE (Wang
et al., 2019b), RSME (Wang et al., 2021), MMKRL (Lu et al., 2022), VBKGC
(Zhang and Zhang, 2022), OTKGE (Cao et al., 2022), IMF (Li et al., 2023), QEB
(Wang et al., 2023) and VISTA (Lee et al., 2023). Comparison with these
methods can verify the effectiveness of our model NativE. Meanwhile, among
these methods, MMKRL design an adversarial training framework. The design
concepts of both methods bear some resemblance to our design of CoMAT.

iii) Negative sampling methods. We employ 3 different negative sampling
methods to compare with our method, including KBGAN (Cai and Wang, 2018), MANS
(Zhang et al., 2023a), MMRNS (Xu et al., 2022). Among these methods, KBGAN
(Cai and Wang, 2018) is an adversarial negative sampling method designed for
conventional KGC, which applies reinforcement learning to optimize the models.
MANS (Zhang et al., 2023a) and MMRNS (Xu et al., 2022) are two negative
sampling strategies designed for MMKGC, which utilize the multi-modal
information to enhance the negative sampling process.

iv) Numeric-aware KGC methods. We employ 3 popular numeric-aware KGC methods
including KBLRN (García-Durán and Niepert, 2018), LiteralE (Kristiadi et al.,
2019), and KGA (Wang et al., 2022). These methods consider the numerical
information to enhance the KGC models. However, their design can not be
generalized to other modalities such as image and text as they are designed
only for numerical modality augmentation.

All of the selected baselines are embedding-based KGC and MMKGC methods. Other
methods such as text-based methods (Yao et al., 2019; Saxena et al., 2022) or
GNN-based methods (Zhu et al., 2021) are not considered as they are orthogonal
to our design.

####  5.1.4. Implemention Details

We implement our NativE framework based on OpenKE (Han et al., 2018), a famous
open-source KGC library. We conduct each experiment on a Linux server with
Ubuntu 20.04.1 operating system and a single NVIDIA A800 GPU.

In the NativE, we fix the batch size to 1024 and set the training epoch to
1000. The embedding dimensions $d_{e},d_{r}$ are tuned from
$\\{150,200,250\\}$ and the negative sampling number $K$ is tuned from
$\\{32,64,128\\}$. The margin $\gamma$ is tuned from $\\{3,6,9,12\\}$ and the
temprature $\beta$ is set to 2. In the CoMAT, the random noise dimension is
set to 64, and the coefficient $\lambda_{1},\lambda_{2}$ are tuned from
$\\{1e^{-2},1e^{-3},1e^{-4},1e^{-5}\\}$. We optimize the model with Adam
(Kingma and Ba, 2015) and the learning rate is tuned from
$\\{1e^{-3},1e^{-4},1e^{-5}\\}$ for both $\mathcal{G}$ and $\mathcal{D}$ in
the adversarial training setting. For baselines, we reproduce the results on
WildKGC following the methodology and parameter setting described in the
original papers and their open-source official code. Some of the baseline
results refer to MMRNS (Xu et al., 2022).

###  5.2. Main Results (RQ1)

The main results of MMKGC experiments are presented in Table 2. We first
compare our method NativE with 18 different KGC baselines. For equitable
comparisons, we account for all modalities leveraged by each method and
conduct two different sets of experiments on our model. The first set
considers only modalities common to all datasets (structure/image/text) and
the second set considers all the modalities inherent to each specific dataset.

Table 2 shows that NativE outperforms all the existing baselines and achieves
new SOTA results. Specifically, NativE significantly surpasses the baselines
by a large margin on Hit@1 on DB15K (15.75%), KVC16K (11.47%), and TIVA
(10.05%), indicating a marked improvement in its accurate reasoning ability.
When employing only the common modalities (structure/image/text) considered by
most mainstream methods, NativE can still perform better than the baseline
methods, signifying its ability to efficiently utilize information from
different modalities. The performance comparison with two adversarial-based
methods MMKRL (Lu et al., 2022) and KBGAN (Cai and Wang, 2018) demonstrates
the superior effectiveness of our adversarial module design which can fully
unleash the power of multi-modal information of the entities.

Table 3. Comparasions among numeric-aware methods on DB15K. S/N denote the structure/numeric modalitites. Model | Modality | MRR | Hit@1 | Hit@3 | Hit@10
---|---|---|---|---|---
KBLN | S+N | 24.68 | 19.26 | 27.27 | 34.89
LiteralE | S+N | 31.29 | 24.24 | 34.59 | 44.98
KGA | S+N | 33.62 | 25.82 | 37.73 | 47.85
NativE | S+N | 36.18 | 27.27 | 41.25 | 52.51
NativE | All | 37.16 | 28.01 | 42.25 | 54.13

Besides, none of the MMKGC baselines in Table 2 has considered the numerical
information presented in the DB15K. Accordingly, we conduct another experiment
to compare our model NativE with other numerical KGC baselines. These methods’
design principles significantly differ from typical MMKGCs and lack
generalizability to other modalities, so we compare them with numerical
methods only on the DB15K dataset. From Table 3 we can observe that NativE
still outperforms all baselines when considering only the structure and
numeric modalities. As the diversity of modal information increases, the
effectiveness of the model is further enhanced.

###  5.3. Imbalanced MMKGC Experiments (RQ2)

(a) Imbalance MMKGC Experiments (b) Group-wise Analysis for Modality-missing
Triples

Figure 3. The imbalance MMKGC results. We report the MRR and Hit@10 results on the DB15K datasets. Further, we divide the test triples into three groups according to whether there was complete modal information and tally their experimental results separately, where: Group1 (both h and t are modality-complete); Group2 (one of h, r is modality-missing); Group3 (both h and t are modality-missing). Table 4. The ablation study results on DB15K. We conduct three groups (G1/G2/G3) of experiments to validate the effectiveness of different modalities, the ReDAF module, and the CoMAT module respectively. Setting | MRR | Hit@1 | Hit@10
---|---|---|---
G1 | w/o image | 36.88 | 27.08 | 54.63
w/o text | 36.09 | 26.86 | 52.92
w/o numeric | 36.74 | 26.87 | 54.65
G2 | w/o $\mathcal{P}_{m}$ | 35.58 | 26.12 | 52.69
w/o $\zeta_{r}$ | 35.84 | 25.79 | 53.96
w/o $\omega_{m}$ | 35.65 | 25.23 | 54.10
w/ concat | 26.01 | 11.16 | 50.66
w/ product | 33.98 | 24.97 | 50.47
G3 | w/o CoMAT | 31.17 | 18.25 | 53.52
w/o $\mathcal{L}_{gp}$ | 37.14 | 27.93 | 54.21
w/ vanilla GAN | 36.91 | 27.02 | 54.74
w/ MLP as $\mathcal{D}$ | 36.34 | 27.12 | 53.31
Full NativE Model | 37.16 | 28.01 | 54.13

To better illustrate the performance of NativE in complex modality imbalance
scenarios, we perform a series of MMKGC experiments in imbalance scenarios. We
introduce a new parameter imbalanced proportion denoted as $\eta$ to quantify
the imbalance in datasets, representing the percentage of entities lacking
modalities information. For instance, $\eta=0.3$ means that there is 30%
missing modality information in the given MMKG.

Specifically, we perform a random division of the original dataset randomly
selecting the corresponding proportion $\eta$ of modal information. Besides,
we distinguish between two imbalance stratifications: entity-level imbalance
and modality-level imbalance. For the entity-level imbalance division, we
randomly drop all the modality information of a selected entity. For the
modality-level imbalance, we randomly drop the modality information for a
proportion $\eta$ across all entities, implying an entity might exhibit
varying modal information quantities. The missing modality information dropped
in these settings will be initiated randomly and the MMKGC experiments on the
imbalanced datasets are shown in Figure 3.

The results from the imbalanced experiments present that NativE maintains its
performance when the modality information is imbalanced. As imbalance
intensifies, the performance of the baseline models TBKGC (Sergieh et al.,
2018) and QBE (Wang et al., 2023) experiences a significant downturn, while
our methods show relative stability. Further, the impact of imbalanced
modality is more noticeable on coarse-grained metrics such as Hit@10,
suggesting that the completeness of modality information has a greater
influence on coarse-grained entity ranking. Additionally, a surplus of
irrelevant noise within the modality information hampers the accurate
inference of the model, leading to a marginal improvement in the performance
of NativE when the imbalance rate is increased. It is also noticeable that the
CoMAT design has a significant effect on model performance, retaining its
usefulness even in unbalanced situations.

We evaluate the imbalanced modality impact on different triples by counting
separately the results of triple at different levels of missing entity modal
information, which is shown in Figure LABEL:figure::distribution(b). In this
figure, the proportion $\eta=0.5$. Interestingly, the model performance is not
necessarily worse when modality information is fully missing (Group 3) in the
triples compared to a partial missing (Group 2). This outcome arises because
modality-level missing is much less likely to result in a complete lack of
entity modal information, preserving some modality information instrumental
for the final prediction, thereby leading to a relatively better performance
of the model in this case (MRR of Group 3). Conversely, entity-level missing
scenario deprives some entities of all modality information, causing a
complete loss of valid information in the modality, leading to poor
performance on coarse-grained metrics like Hit@10.

###  5.4. Ablation Study (RQ3)

We conduct extensive ablation studies to validate the effectiveness of each
module in our design. In the ablation study, we conduct three groups of
experiments to validate the contribution of each module in NativE by removing
the corresponding key component. Each of the three groups of experiments
targets the different modalities, the ReDAF module, and the CoMAT module. The
ablation study results are summarized in Table 4. From the results in G1, we
can conclude that each modality contributes to the overall performance,
despite the image modal seemingly having a lesser role in comparison to the
text. G2 shows the essential role of key components in the ReDAF module, as
their removal leads to a marked performance decline. Besides, compared with
other modality fusion strategies widely used by other MMKGC models like concat
and dot-product, our NativE module still outperforms. From the results in G3,
we can find that the CoMAT leads to a huge performance boost, especially on
the precision metrics like Hit@1. Besides, when removing the gradient penalty
loss or changing the adversarial loss to a vanilla GAN version with log-
likelihoods (Goodfellow et al., 2014) loss, the model performance still
decreases. We also try to replace the discriminator $\mathcal{D}$ with a two-
layer MLP, but the final result is worse than the original NativE. This
suggests that the WGAN (Arjovsky et al., 2017) framework employed by us works
better for the MMKGC scenario, affirming the correctness of our prior analyses
and proofs in the MMKGC setting.

###  5.5. Further Analysis (RQ4 & RQ5)

####  5.5.1. Generalization Analysis (RQ4)

Figure 4. The generalization experiments of the CoMAT module on three
different MMKGC models. We report the MRR and Hit@1 results on the DB15K
dataset.

As we mentioned before, CoMAT is a general framework for enhancing MMKGC
models based on embedding. To prove this point, we apply CoMAT on more MMKGC
models (IKRL (Xie et al., 2017), TBKGC (Sergieh et al., 2018), and VBKGC
(Zhang and Zhang, 2022)) and demonstrate the results in Figure 4.

We can make a conclusion that the MMKGC models trained w/ CoMAT can obtain
significant performance gain. This suggests that CoMAT can be used as a
general adversarial enhancement framework in different MMKGC models.

Figure 5. The results of the efficiency experiment. We report the MRR and
Hit@1 results on the KVC16K/DB15K datasets.

####  5.5.2. Efficency Analysis (RQ5)

Another important concern is the efficiency of the adversarial methods. This
is because the iterative training strategies would lead to more computation.
Therefore, we evaluate the efficiency of several MMKGC methods. The methods we
have chosen cover both non-adversarial (OTKGE (Cao et al., 2022), MMRNS (Xu et
al., 2022), IMF (Li et al., 2023)) and adversarial approaches (KBGAN (Cai and
Wang, 2018), MMKRL (Lu et al., 2022)). We evaluate the efficiency and
performance of different models with the same batch size of 1024 and dimension
of 250 on the same device. As shown in Figure 5, we can observe that NativE
makes a good trade-off between efficiency and performance, achieving the best
results while keeping relatively fast and efficient. Though the adversarial
training module CoMAT slows down the training to some extent, this latency is
within acceptable limits and there are significant gains in the model
performance.

###  5.6. Case Study (RQ6)

Figure 6. The relation-wise temperature distribution and high-frequency
relations that can raise each modality weight.

This section presents clear examples to further elucidate our design. We
previously emphasize that the relation guidance in ReDAF can zoom in on the
differences between the different modalities and sift through them to find the
useful parts, as the relation-wise temperatures are limited in $(0,1)$ with a
sigmoid function. We first demonstrate the distribution of the temperatures as
shown in Figure 6. We can observe that the distribution is diversified among
relations. To better illustrate this, we filter out and list the most frequent
relations that pull the weights of the corresponding modality higher for each
modality, which can be found in Figure 6 (b). We can find that each different
relationship will have a different dependence on the different modal
information of the entity when making predictions. For example, image
information can provide some intuitive visual information. When predicting
movie genre, such image information would greatly benefit the result.
Meanwhile, the affiliation, nationality, and death place of a person can be
usually found in the text descriptions which makes text modality valuable when
predicting the results of these relations.

##  6\. Conclusion

In this paper, we propose a novel MMKGC framework NativE to rectify the
diversity and imbalance issues of the MMKGs. NativE consists of two core
designs called ReDAF and CoMAT. ReDAF proposes a relation-guided dual adaptive
fusion method to incorporate adaptive features of any modalities to obtain
joint representations while CoMAT aims to enhance the information about the
imbalanced modes by a training strategy based on WGAN. We perform an in-depth
theoretical analysis to justify our design’s rationale. We construct a new
MMKGC benchmark and conduct comprehensive experiments on it against 21
baselines to show the effectiveness, generalization, and efficiency of our
framework. In the future, we think the MMKGC task and the downstream
application of MMKG in more complex real scenarios remain great challenges to
be solved.

###### Acknowledgements.

This work is founded by National Natural Science Foundation of China (
NSFC62306276 / NSFCU23B2055 / NSFCU19B2027 / NSFC91846204 ), Zhejiang
Provincial Natural Science Foundation of China (No. LQ23F020017), Ningbo
Natural Science Foundation (2023J291), Yongjiang Talent Introduction Programme
(2022A-238-G), Fundamental Research Funds for the Central Universities
(226-2023-00138).

## References

  * (1)
  * Arjovsky et al. (2017) Martín Arjovsky, Soumith Chintala, and Léon Bottou. 2017.  Wasserstein GAN.  _CoRR_ abs/1701.07875 (2017).
  * Balazevic et al. (2019) Ivana Balazevic, Carl Allen, and Timothy M. Hospedales. 2019.  TuckER: Tensor Factorization for Knowledge Graph Completion. In _EMNLP/IJCNLP (1)_. Association for Computational Linguistics, 5184–5193.
  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto García-Durán, Jason Weston, and Oksana Yakhnenko. 2013.  Translating Embeddings for Modeling Multi-relational Data. In _NIPS_. 2787–2795.
  * Cai and Wang (2018) Liwei Cai and William Yang Wang. 2018.  KBGAN: Adversarial Learning for Knowledge Graph Embeddings. In _Proc. of NAACL_.
  * Cao et al. (2022) Zongsheng Cao, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. 2022.  OTKGE: Multi-modal Knowledge Graph Embeddings via Optimal Transport. In _NeurIPS_.
  * Chao et al. (2021) Linlin Chao, Jianshan He, Taifeng Wang, and Wei Chu. 2021.  PairRE: Knowledge Graph Embeddings via Paired Relation Vectors. In _Proc. of ACL_.
  * Chen et al. (2023a) Zhuo Chen, Jiaoyan Chen, Wen Zhang, Lingbing Guo, Yin Fang, Yufeng Huang, Yichi Zhang, Yuxia Geng, Jeff Z. Pan, Wenting Song, and Huajun Chen. 2023a.  MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid. In _ACM Multimedia_. ACM, 3317–3327.
  * Chen et al. (2024a) Zhuo Chen, Yin Fang, Yichi Zhang, Lingbing Guo, Jiaoyan Chen, Huajun Chen, and Wen Zhang. 2024a.  The Power of Noise: Toward a Unified Multi-modal Knowledge Graph Representation Framework.  arXiv:2403.06832 [cs.CL]
  * Chen et al. (2023b) Zhuo Chen, Lingbing Guo, Yin Fang, Yichi Zhang, Jiaoyan Chen, Jeff Z. Pan, Yangning Li, Huajun Chen, and Wen Zhang. 2023b.  Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment. In _ISWC_ _(Lecture Notes in Computer Science, Vol. 14265)_. Springer, 121–139.
  * Chen et al. (2023c) Zhuo Chen, Wen Zhang, Yufeng Huang, Mingyang Chen, Yuxia Geng, Hongtao Yu, Zhen Bi, Yichi Zhang, Zhen Yao, Wenting Song, Xinliang Wu, Yi Yang, Mingyi Chen, Zhaoyang Lian, Yingying Li, Lei Cheng, and Huajun Chen. 2023c.  Tele-Knowledge Pre-training for Fault Analysis. In _ICDE_. IEEE, 3453–3466.
  * Chen et al. (2024b) Zhuo Chen, Yichi Zhang, Yin Fang, Yuxia Geng, Lingbing Guo, Xiang Chen, Qian Li, Wen Zhang, Jiaoyan Chen, Yushan Zhu, Jiaqi Li, Xiaoze Liu, Jeff Z. Pan, Ningyu Zhang, and Huajun Chen. 2024b.  Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey.  _CoRR_ abs/2402.05391 (2024).
  * Croce et al. (2020) Danilo Croce, Giuseppe Castellucci, and Roberto Basili. 2020.  GAN-BERT: Generative Adversarial Learning for Robust Text Classification with a Bunch of Labeled Examples. In _ACL_. Association for Computational Linguistics, 2114–2119.
  * Dettmers et al. (2018) Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018.  Convolutional 2D Knowledge Graph Embeddings. In _AAAI_. AAAI Press, 1811–1818.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _NAACL-HLT (1)_. Association for Computational Linguistics, 4171–4186.
  * García-Durán and Niepert (2018) Alberto García-Durán and Mathias Niepert. 2018.  KBlrn: End-to-End Learning of Knowledge Base Representations with Latent, Relational, and Numerical Features. In _UAI_. AUAI Press, 372–381.
  * Glorot et al. (2011) Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011.  Deep Sparse Rectifier Neural Networks. In _AISTATS_ _(JMLR Proceedings, Vol. 15)_. JMLR.org, 315–323.
  * Goodfellow et al. (2014) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014.  Generative Adversarial Nets. In _Proc. of NeurIPS_.
  * Gu et al. (2023) Xin Gu, Guang Chen, Yufei Wang, Libo Zhang, Tiejian Luo, and Longyin Wen. 2023.  Text with Knowledge Graph Augmented Transformer for Video Captioning. In _CVPR_. IEEE, 18941–18951.
  * Gulrajani et al. (2017) Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville. 2017.  Improved Training of Wasserstein GANs. In _NIPS_. 5767–5777.
  * Han et al. (2018) Xu Han, Shulin Cao, Xin Lv, Yankai Lin, Zhiyuan Liu, Maosong Sun, and Juanzi Li. 2018.  OpenKE: An Open Toolkit for Knowledge Embedding. In _Proc. of EMNLP_.
  * Ji et al. (2015) Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. 2015.  Knowledge Graph Embedding via Dynamic Mapping Matrix. In _ACL (1)_. The Association for Computer Linguistics, 687–696.
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila. 2019.  A Style-Based Generator Architecture for Generative Adversarial Networks. In _CVPR_. Computer Vision Foundation / IEEE, 4401–4410.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization. In _ICLR (Poster)_.
  * Kristiadi et al. (2019) Agustinus Kristiadi, Mohammad Asif Khan, Denis Lukovnikov, Jens Lehmann, and Asja Fischer. 2019.  Incorporating Literals into Knowledge Graph Embeddings. In _ISWC (1)_ _(Lecture Notes in Computer Science, Vol. 11778)_. Springer, 347–363.
  * Lee et al. (2023) Jaejun Lee, Chanyoung Chung, Hochang Lee, Sungho Jo, and Joyce Jiyoung Whang. 2023.  VISTA: Visual-Textual Knowledge Graph Representation Learning. In _EMNLP (Findings)_. Association for Computational Linguistics, 7314–7328.
  * Lehmann et al. (2015) Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Sören Auer, and Christian Bizer. 2015.  DBpedia - A large-scale, multilingual knowledge base extracted from Wikipedia.  _Semantic Web_ (2015).
  * Li et al. (2023) Xinhang Li, Xiangyu Zhao, Jiaxing Xu, Yong Zhang, and Chunxiao Xing. 2023.  IMF: Interactive Multimodal Fusion Model for Link Prediction. In _WWW_. ACM, 2572–2580.
  * Liang et al. (2024) Ke Liang, Yue Liu, Sihang Zhou, Wenxuan Tu, Yi Wen, Xihong Yang, Xiangjun Dong, and Xinwang Liu. 2024.  Knowledge Graph Contrastive Learning Based on Relation-Symmetrical Structure.  _IEEE Trans. Knowl. Data Eng._ 36, 1 (2024), 226–238.
  * Liang et al. (2023) Ke Liang, Lingyuan Meng, Meng Liu, Yue Liu, Wenxuan Tu, Siwei Wang, Sihang Zhou, and Xinwang Liu. 2023.  Learn from Relational Correlations and Periodic Events for Temporal Knowledge Graph Reasoning. In _SIGIR_. ACM, 1559–1568.
  * Liang et al. (2022) Ke Liang, Lingyuan Meng, Meng Liu, Yue Liu, Wenxuan Tu, Siwei Wang, Sihang Zhou, X Liu, and F Sun. 2022.  A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multimodal.  (2022).
  * Liu et al. (2022) Juncheng Liu, Zequn Sun, Bryan Hooi, Yiwei Wang, Dayiheng Liu, Baosong Yang, Xiaokui Xiao, and Muhao Chen. 2022.  Dangling-Aware Entity Alignment with Mixed High-Order Proximities. In _NAACL-HLT (Findings)_. Association for Computational Linguistics, 1172–1184.
  * Liu et al. (2019) Ye Liu, Hui Li, Alberto García-Durán, Mathias Niepert, Daniel Oñoro-Rubio, and David S. Rosenblum. 2019.  MMKG: Multi-modal Knowledge Graphs. In _ESWC_ _(Lecture Notes in Computer Science, Vol. 11503)_. Springer, 459–474.
  * Lu et al. (2022) Xinyu Lu, Lifang Wang, Zejun Jiang, Shichang He, and Shizhong Liu. 2022.  MMKRL: A robust embedding approach for multi-modal knowledge graph representation learning.  _Appl. Intell._ 52, 7 (2022), 7480–7497.
  * Pan et al. (2022) Haojie Pan, Yuzhou Zhang, Zepeng Zhai, Ruiji Fu, Ming Liu, Yangqiu Song, Zhongyuan Wang, and Bing Qin. 2022.  Kuaipedia: a Large-scale Multi-modal Short-video Encyclopedia.  _CoRR_ abs/2211.00732 (2022).
  * Pezeshkpour et al. (2018) Pouya Pezeshkpour, Liyan Chen, and Sameer Singh. 2018.  Embedding Multimodal Relational Data for Knowledge Base Completion. In _EMNLP_. Association for Computational Linguistics, 3208–3218.
  * Saxena et al. (2022) Apoorv Saxena, Adrian Kochsiek, and Rainer Gemulla. 2022.  Sequence-to-Sequence Knowledge Graph Completion and Question Answering. In _ACL (1)_. Association for Computational Linguistics, 2814–2828.
  * Sergieh et al. (2018) Hatem Mousselly Sergieh, Teresa Botschen, Iryna Gurevych, and Stefan Roth. 2018.  A Multimodal Translation-Based Approach for Knowledge Graph Representation Learning. In _*SEM@NAACL-HLT_. Association for Computational Linguistics, 225–234.
  * Simonyan and Zisserman (2015) Karen Simonyan and Andrew Zisserman. 2015.  Very Deep Convolutional Networks for Large-Scale Image Recognition. In _ICLR_.
  * Suchanek et al. (2007) Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007.  Yago: a core of semantic knowledge. In _WWW_. ACM, 697–706.
  * Sun et al. (2019) Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019.  RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space. In _ICLR (Poster)_. OpenReview.net.
  * Tang et al. (2020) Yun Tang, Jing Huang, Guangtao Wang, Xiaodong He, and Bowen Zhou. 2020.  Orthogonal Relation Transforms with Graph Context Modeling for Knowledge Graph Embedding. In _Proc. of ACL_.
  * Tang et al. (2022) Zhenwei Tang, Shichao Pei, Zhao Zhang, Yongchun Zhu, Fuzhen Zhuang, Robert Hoehndorf, and Xiangliang Zhang. 2022.  Positive-Unlabeled Learning with Adversarial Data Augmentation for Knowledge Graph Completion. In _Proc. of IJCAI_.
  * Trouillon et al. (2016) Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016.  Complex Embeddings for Simple Link Prediction. In _ICML_ _(JMLR Workshop and Conference Proceedings, Vol. 48)_. JMLR.org, 2071–2080.
  * Tygar (2011) J. D. Tygar. 2011.  Adversarial Machine Learning.  _IEEE Internet Comput._ 15, 5 (2011), 4–6.
  * Vrandecic and Krötzsch (2014) Denny Vrandecic and Markus Krötzsch. 2014.  Wikidata: a free collaborative knowledgebase.  _Commun. ACM_ 57, 10 (2014), 78–85.
  * Wang et al. (2022) Jiang Wang, Filip Ilievski, Pedro A. Szekely, and Ke-Thia Yao. 2022.  Augmenting Knowledge Graphs for Better Link Prediction. In _IJCAI_. ijcai.org, 2277–2283.
  * Wang et al. (2017b) Jun Wang, Lantao Yu, Weinan Zhang, Yu Gong, Yinghui Xu, Benyou Wang, Peng Zhang, and Dell Zhang. 2017b.  IRGAN: A Minimax Game for Unifying Generative and Discriminative Information Retrieval Models. In _SIGIR_. ACM, 515–524.
  * Wang et al. (2021) Meng Wang, Sen Wang, Han Yang, Zheng Zhang, Xi Chen, and Guilin Qi. 2021.  Is Visual Context Really Helpful for Knowledge Graph? A Representation Learning Perspective. In _ACM Multimedia_. ACM, 2735–2743.
  * Wang et al. (2018) Peifeng Wang, Shuangyin Li, and Rong Pan. 2018.  Incorporating GAN for Negative Sampling in Knowledge Representation Learning. In _Proc. of AAAI_.
  * Wang et al. (2017a) Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017a.  Knowledge Graph Embedding: A Survey of Approaches and Applications.  _IEEE Trans. Knowl. Data Eng._ 29, 12 (2017), 2724–2743.  https://doi.org/10.1109/TKDE.2017.2754499
  * Wang et al. (2019a) Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. 2019a.  KGAT: Knowledge Graph Attention Network for Recommendation. In _KDD_. ACM, 950–958.
  * Wang et al. (2023) Xin Wang, Benyuan Meng, Hong Chen, Yuan Meng, Ke Lv, and Wenwu Zhu. 2023.  TIVA-KG: A Multimodal Knowledge Graph with Text, Image, Video and Audio. In _ACM Multimedia_. ACM, 2391–2399.
  * Wang et al. (2020) Xiang Wang, Yaokun Xu, Xiangnan He, Yixin Cao, Meng Wang, and Tat-Seng Chua. 2020.  Reinforced Negative Sampling over Knowledge Graph for Recommendation. In _WWW_. ACM / IW3C2, 99–109.
  * Wang et al. (2019b) Zikang Wang, Linjing Li, Qiudan Li, and Daniel Zeng. 2019b.  Multimodal Data Enhanced Representation Learning for Knowledge Graphs. In _IJCNN_. IEEE, 1–8.
  * Wei et al. (2023) Wei Wei, Chao Huang, Lianghao Xia, and Chuxu Zhang. 2023.  Multi-Modal Self-Supervised Learning for Recommendation. In _WWW_. ACM, 790–800.
  * Xie et al. (2017) Ruobing Xie, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2017.  Image-embodied Knowledge Representation Learning. In _IJCAI_. ijcai.org, 3140–3146.
  * Xu et al. (2022) Derong Xu, Tong Xu, Shiwei Wu, Jingbo Zhou, and Enhong Chen. 2022.  Relation-enhanced Negative Sampling for Multimodal Knowledge Graph Completion. In _ACM Multimedia_. ACM, 3857–3866.
  * Yang et al. (2015) Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015.  Embedding Entities and Relations for Learning and Inference in Knowledge Bases. In _ICLR (Poster)_.
  * Yao et al. (2019) Liang Yao, Chengsheng Mao, and Yuan Luo. 2019.  KG-BERT: BERT for Knowledge Graph Completion.  _CoRR_ abs/1909.03193 (2019).
  * Yasunaga et al. (2021) Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021.  QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering. In _NAACL-HLT_. Association for Computational Linguistics, 535–546.
  * Yu et al. (2017) Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017.  SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient. In _AAAI_. AAAI Press, 2852–2858.
  * Zhang et al. (2022) Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, and Weizhu Chen. 2022.  Adversarial Retriever-Ranker for Dense Text Retrieval. In _ICLR_. OpenReview.net.
  * Zhang et al. (2023a) Yichi Zhang, Mingyang Chen, and Wen Zhang. 2023a.  Modality-Aware Negative Sampling for Multi-modal Knowledge Graph Embedding. In _IJCNN_. IEEE, 1–8.
  * Zhang et al. (2023b) Yichi Zhang, Zhuo Chen, Yin Fang, Lei Cheng, Yanxi Lu, Fangming Li, Wen Zhang, and Huajun Chen. 2023b.  Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering.  _CoRR_ abs/2311.06503 (2023).
  * Zhang et al. (2024) Yichi Zhang, Zhuo Chen, Lei Liang, Huajun Chen, and Wen Zhang. 2024.  Unleashing the Power of Imbalanced Modality Information for Multi-modal Knowledge Graph Completion.  _CoRR_ abs/2402.15444 (2024).
  * Zhang et al. (2023c) Yichi Zhang, Zhuo Chen, Wen Zhang, and Huajun Chen. 2023c.  Making Large Language Models Perform Better in Knowledge Graph Completion.  _CoRR_ abs/2310.06671 (2023).
  * Zhang and Zhang (2022) Yichi Zhang and Wen Zhang. 2022.  Knowledge Graph Completion with Pre-trained Multimodal Transformer and Twins Negative Sampling.  _CoRR_ abs/2209.07084 (2022).
  * Zhao et al. (2022) Yu Zhao, Xiangrui Cai, Yike Wu, Haiwei Zhang, Ying Zhang, Guoqing Zhao, and Ning Jiang. 2022.  MoSE: Modality Split and Ensemble for Multimodal Knowledge Graph Completion. In _EMNLP_. Association for Computational Linguistics, 10527–10536.
  * Zhu et al. (2021) Zhaocheng Zhu, Zuobai Zhang, Louis-Pascal A. C. Xhonneux, and Jian Tang. 2021.  Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction. In _NeurIPS_. 29476–29490.