# Node-based Knowledge Graph Contrastive Learning for Medical Relationship
Prediction

Zhiguang Fan  Sun Yat-sen UniversityGuangZhouChina<EMAIL_ADDRESS>,
Yuedong Yang  Sun Yat-sen UniversityGuangZhouChina<EMAIL_ADDRESS>,
Mingyuan Xu∗ Guangzhou National Laboratory, ChinaGuangZhouChina
<EMAIL_ADDRESS>and  Hongming Chen∗ Guangzhou National Laboratory,
ChinaGuangZhouChina<EMAIL_ADDRESS>
(2024)

###### Abstract.

The embedding of Biomedical Knowledge Graphs (BKGs) generates robust
representations, valuable for a variety of artificial intelligence
applications, including predicting drug combinations and reasoning disease-
drug relationships. Meanwhile, contrastive learning (CL) is widely employed to
enhance the distinctiveness of these representations. However, constructing
suitable contrastive pairs for CL, especially within Knowledge Graphs (KGs),
has been challenging. In this paper, we proposed a novel node-based
contrastive learning method for knowledge graph embedding, NC-KGE. NC-KGE
enhances knowledge extraction in embeddings and speeds up training convergence
by constructing appropriate contrastive node pairs on KGs. This scheme can be
easily integrated with other knowledge graph embedding (KGE) methods. For
downstream task such as biochemical relationship prediction, we have
incorporated a relation-aware attention mechanism into NC-KGE, focusing on the
semantic relationships and node interactions. Extensive experiments show that
NC-KGE performs competitively with state-of-the-art models on public datasets
like FB15k-237 and WN18RR. Particularly in biomedical relationship prediction
tasks, NC-KGE outperforms all baselines on datasets such as PharmKG8k-28,
DRKG17k-21, and BioKG72k-14, especially in predicting drug combination
relationships. We release our code at https://github.com/zhi520/NC-KGE.

Contrastive Learning, Graph Neural Network, Medical Relationship Prediction,
Biomedical Knowledge Graph

††copyright: acmcopyright††journalyear: 2024††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2024; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Data Mining Link Prediction

##  1\. Introduction

Knowledge graphs (KGs) are intricate networks, representing the human
knowledge, where nodes denote meaningful entities and edges signify their
relationships. Specifically, Biomedical Knowledge Graphs (BKGs), such as TTD,
PharmGKB, OMIM and DrugBank, start from the high-quality manually curated
biomedicine knowledge and capture the intricate interactions among biomedical
entities like genes, chemicals, and diseases. They play a pivotal role in many
artificial intelligence applications including drug repositioning, adverse
drug reaction analysis, and proteomics data analysis. Recently, Knowledge
graph embedding methods were developed to generate compact, distributed
representations of entities and relations for performance improvements. Once
learned, these embeddings can be analyzed using diverse scoring techniques to
provide probability scores for triplets, valuable for downstream tasks such as
prediction, clustering, and visualization. Especially, biomedical relationship
prediction, uncovers hidden associations between biological entities,
enhancing disease understanding and biomarker discovery, illustrating
biological pathways.

Recent KGE methods can be roughly categorized into structure-based embeddings
and enhanced knowledge graph embeddings. The structure-based embeddings focus
on the usage of knowledge graph facts and can be further categorized into 3
types. (1) translational distance models such as TransE (Bordes et al., 2013),
RotatE (Sun et al., 2019), QuatE (Zhang et al., 2019), DualE (Cao et al.,
2021) and HAKE (Zhang et al., 2020a). (2) semantic matching methods including
RESCAL (Nickel et al., 2011), TATEC (García-Durán et al., 2014), DistMult
(Yang et al., 2014), HolE (Nickel et al., 2016), ComplEx (Trouillon et al.,
2016), ANALOGY (Liu et al., 2017) and SimplE (Kazemi and Poole, 2018). (3)
neural-based methods, especially GNN-based methods including RGCN
(Schlichtkrull et al., 2018), SACN (Shang et al., 2019), KBGAT (Nathani et
al., 2019), A2N (Bansal et al., 2019), CompGCN (Vashishth et al., 2019) and
SE-GNN (Li et al., 2022a). The enhanced knowledge graph embeddings aim to use
additional information for a better representation, including (1) text-
enhanced knowledge embedding such as TEKE (Wang et al., 2016), DKRL (Xie et
al., 2016), KG-BERT (Yao et al., 2019), (2) logic-enhanced embedding like KALE
(Guo et al., 2016), (3) image-enhanced embedding and etc. Although these
methods have gain improvements in powerful representation of entities in
knowledge graphs, it is still challenging for biomedical relationship
predictions due to knowledge bases from the biomedical domain are usually
sparse, redundant and incomplete.

In the past, graph contrastive learning, a self-supervised method, has
achieved significant success in generating generalized, transferable, and
robust representations for graph structured data. This success has illuminated
the path for learning knowledge graph embeddings. Essentially, contrastive
learning aims to extract hidden information between samples by bringing
similar samples closer and pushing dissimilar ones apart in latent space. Its
core objective is to distinguish between a pair of representations from the
two augmentations of the same sample (positives) apart from the k pairs of
representation from the other (negative) samples. Constructing highly
confident contrastive pairs is crucial for the discriminative power of
contrastive learning models. However, due to the intricate structures within
knowledge graphs, defining these contrastive pairs is challenging.
Consequently, there are only limited attempts to integrate contrastive
learning strategies with Knowledge Graph Embedding (KGE) methods.

SimKGC (Wang et al., 2022), for instance, creates contrastive pairs using
semantic similarity through language models, deviating from the previous graph
contrastive learning models that fully mined information underlying graph
structures. However, the effectiveness of contrastive KGE methods based on
semantic similarity heavily relies on the specific language models used.
Another approach, KGE-SymCL (Liang et al., 2023b), utilizes the semantic
similarity of entities in relation-symmetrical positions to construct positive
contrastive samples. However, extracting the structure of symmetric relations
is a tedious process and there is no significant improvement compared to
SimKGC. In biomedical knowledge graphs, entities represent gene codes,
targets, or chemical compounds. The features generated by language models may
lead to inaccurate semantic estimation in this context. Therefore, there is a
pressing need for a more stable and universally applicable contrastive
learning criterion tailored for biomedical knowledge graphs.

In our study, we introduced a new and versatile node-based contrastive method
for knowledge graph embeddings, termed NC-KGE, specifically designed for
predicting biomedical relationships. For a given fact triplet in knowledge
graph, NC-KGE identifies triplets in the knowledge graph where one entity and
the relation type remain the same as positive samples, while all other
triplets are considered negative samples. By maximizing the similarity score
between positive samples and minimizing it between negative ones using a
modified classic contrastive learning loss, NC-KGE enhances the convergence
speed during training and improve the performance of non-contrastive methods,
such as CompGCN and SE-GNN. Additionally, we incorporated a relation-aware
multi-head attention (RAMHA) mechanism into NC-KGE to enhance the utilization
of relation semantics and interactions among relations and entities. We
evaluated NC-KGE’s performance in relation prediction on both general public
datasets (FB15k-237 and WN18RR) and biomedical-focused knowledge graphs
(PharmKG8k-28, DRKG17k-21, and BioKG72k-14). Our extensive experiments
revealed that NC-KGE competes effectively with state-of-the-art models on
general knowledge graphs, and surpasses all baselines on biomedical knowledge
graphs, particularly excelling in predicting drug combination relationships.

##  2\. Related Work

Knowledge Graph Embedding (KGE) aims to encode the entities and relations to
the low dimensional vector or matrix space while maximally preserving its
topological properties. Recent existing KGE models can be roughly categorized
into structure based embedding and enhanced knowledge embeddings, reviewed by
Singh et al. (Choudhary et al., 2021) and MINERVINI et al. (Palmonari and
Minervini, 2020). In this work, structure-based knowledge graph embedding
methods are more related than enhanced knowledge graph embeddings.

###  2.1. Structure-based Knowledge graph embedding methods

Translation distance models interpret relations as translations operation from
a head node to an tailor node in latent space, e.g., TransE (Bordes et al.,
2013), TransH (Wang et al., 2014), TransR (Lin et al., 2015) and etc. TransE
was among the initial attempts to represent relations as addition operations
between entities. TransH projects entities onto relation-specific hyperplanes,
allowing entities to play different roles in various relations. TransR maps
nodes and relations into distinct entities spaces and relation-specific
spaces. RotaE (Sun et al., 2019) treats the relation as a rotation operation.
PairRE (Chao et al., 2020) can encode complex relationships and multiple
relationship patterns at the same time. Moreover, HousE (Li et al., 2022b)
involves a novel parameterization based on the designed Householder
transformations for rotation and projection.

Semantic matching models, including RESCAL (Nickel et al., 2011), DistMult
(Yang et al., 2014), ComplEx (Trouillon et al., 2016), ConvE (Dettmers et al.,
2018), SimplE (Kazemi and Poole, 2018), CrossE (Huang et al., 2022), QuatE
(Zhang et al., 2019), DualE (Cao et al., 2021), are developed based on
similarity scoring functions. RESCAL utilizes a bilinear similarity function
to compute the scores of knowledge triples and assumes that positive triples
have higher scores than negative ones. DisMult simplifies the bilinear
similarity function through using diagonal matrix. ComplEx further generalized
DisMult by using complex embeddings and Hermitian dot products. Besides, the
advantage of quaternion representations is leveraged by QuatE to enrich the
correlation information between head and tail entities based on relational
rotation quaternions. Inspired by it, DualE is proposed to gain a better
expressive ability by projecting the embeddings in dual quaternion space.

Neural-based methods, including ConvE (Dettmers et al., 2018), RGCN
(Schlichtkrull et al., 2018), SACN (Shang et al., 2019), KBGAT (Nathani et
al., 2019), A2N (Bansal et al., 2019), CompGCN (Vashishth et al., 2019) and
SE-GNN (Li et al., 2022a). For example, ConvE introduces the use of
convolutional layers to extract information. RGCN introduces a relation-
specific transformation to integrate relation information with message
aggregation. RGHAT (Zhang et al., 2020b) incorporates a two-level attention
mechanism, addressing relations and entities separately. KE-GCN (Yu et al.,
2021) introduces a joint propagation method to update node and edge embeddings
simultaneously. CompGCN proposes various composition operations for neighbor
aggregation to model the structure pattern of multi-relational graph. RAGAT
(Liu et al., 2021) constructs separate message functions for different
relations, which aims at exploiting the heterogeneous characteristics of
knowledge graphs. SE-GNN (Li et al., 2022a), with its three levels of semantic
evidence, achieves in-depth knowledge representation by meticulously merging
these layers through multi-layer aggregation, leading to highly extrapolative
knowledge representations.

###  2.2. Contrastive Learning on Knowledge Graph

Graph Contrastive learning (GCL) operates by mining the hidden information in
intra-data in a self-supervised manner. These methods, such as GRACE (Zhu et
al., 2020), GraphCL (You et al., 2020), AutoGCL (Yin et al., 2022) and iGCL
(Liang et al., 2023a) etc, has been proven highly successful in both nodes
representing learning, relation prediction, classification, graph generation
and anomaly detections. Recently, only a few researchers attempt to extend
graph contrastive learning into knowledge graph embedding learning. SimKGC
(Wang et al., 2022) tend to combine samples with high semantic similarity as
positive pairs. This method heavily rely on semantic similarity predicted by
the specific language models. However, the language model may estimate sematic
similarity inaccurately for the biomedical entities including gene codes,
targets, or chemical compounds, making these methods fails for biomedical
knowledge graphs. KGE-SymCL (Liang et al., 2023b) utilizes the semantic
similarity of entities in relation-symmetrical positions to construct positive
contrastive samples. However, extracting the structure of symmetric relations
is a tedious process and there is no significant improvement compared to
SimKGC.

##  3\. Preliminary

Knowledge Graph (KG) is composed of the fact triplets, denoted as
$\mathcal{G}=\left\\{\left(e_{h},r,e_{t}\right)\mid
e_{h},e_{t}\in\mathcal{E},r\in\mathcal{R}\right\\}$, where $\mathcal{E}$ is
the set of entities (i.e., nodes), $\mathcal{R}$ is the set of relations
(i.e., edge types), $e_{h},e_{t}$ represents the head and tail entity,
respectively, and $r$ represents the relation between them. Relation
Prediction, also known as Link Prediction or Knowledge Completion, involves
predicting missing links or relationships in a knowledge graph. For a given
head entity $e_{h}\in\mathcal{E}$ and a relation $r\in\mathcal{R}$, the
objective is to identify the most suitable tail entity $e_{t}\in\mathcal{E}$,
forming a new plausible triple $\left(e_{h},r,e_{t}\right)$ within
$\mathcal{G}$. Here, we approach this task by scoring all the candidates
$\left\\{\left(e_{h},r,e_{t}^{\prime}\right)\mid
e_{t}^{\prime}\in\mathcal{E}\right\\}$, maximizing the scores for genuine
triples and minimizing the scores for all the other candidates.

##  4\. NC-KGE Methods

As shown in Figure 1, we show the overall process of NC-KGE. For a given fact
triple $\left(e_{h},r,e_{t}\right)$ in biomedical knowledge graph
$\mathcal{G}$, NC-KGE first constructs positive and negative augmented samples
for contrastive learning. Then, a learnable KGE model was used to generate the
embeddings for entities and relations in $\mathcal{G}$. Thirdly, a similarity
function $S$ is used to score the triple embeddings
$S\left(z_{h},x_{r},z_{t}^{*}\right)$ of both positive and negative samples,
and calculates the contrastive loss to optimize KGE model by maximizing the
scores of genuine triples and minimizing the scores for all the other
candidates. During inference, NC-KGE measures the scores of triple embeddings
$S\left(z_{h},x_{r},z_{t}^{*}\right)$ for all candidates $e_{t}^{*}$ when
provided $\left(e_{h},r,?\right)$, and the candidate $e_{t}$ with highest
scores will form a plausible triple with $e_{h}$ and $r$ in $\mathcal{G}$.
Additionally, finding the head entity $e_{h}$ for the provided
$\left(?,r,e_{t}\right)$ can be effortlessly transformed into a similar
process.

Figure 1. The overall framework of NC-KGE.

###  4.1. Construction of node-based contrastive pairs

In biomedical knowledge graph $\mathcal{G}$, the entity $e_{h}$ can form the
same relationship with multiple other entities $\left\\{e_{t}\mid
e_{t}\in\mathcal{E}\right\\}$. For example, a target entity can correspond to
multiple inhibitors, and one disease may be associated with multiple genes.
For a given triple $\left(e_{h},r,e_{t}\right)$, supporting another entity
$e_{t}^{+}$ forms the same relation $r$ with the head entity $e_{h}$,
$e_{t}^{+}$ is defined as a positive entity for $e_{h}$ and
$r_{i},\left(e_{h},r,e_{t}^{+}\right)$ is a positive triple pair for
$\left(e_{h},r,e_{t}\right)$. Additionally, the triple
$\left(e_{h},r,e_{t}\right)$ is also regarded as a positive pair for itself.
All the other entities $\left\\{e_{t}^{-}\right\\}$ which don’t form the
relation $r$ with $e_{h}$ are defined as negative entity of $e_{h}$, and
$\left(e_{h},r,e_{t}^{-}\right)$ is also regarded as a negative pair for
$\left(e_{h},r,e_{t}\right)$.

Figure 2. Relation-aware multi-head attention based KGE model.

###  4.2. Relation-aware multi-head attention based KGE model

Here, we proposed a relation-aware multi-head attention mechanism (RAMHA) by
integrating the relations between entities into attention computations to
enhance the utilization of relation semantics and interactions among relations
and entities in NC-KGE. As shown in Figure 2, RAMHA mainly includes three
phases: attention computing, message passing and information aggregating. By
stacking multiple layers of RAMHA, NC-KGE generates the knowledge graph
embeddings for both entities and relations.

Supporting the embedding of entity $e_{u}$ is denoted as $z_{u}^{l}$ and
$x_{r}^{l}$ represents the embedding of relation type $r$ between $e_{u}$ and
$e_{v}$ in $l$th layer of GAN, the next layer will aggregate information to
$e_{u}$ from its neighbor $e_{v}$ according to their relations $r$ in an
attention-based message passing process as shown in Equation 1.

(1)
$z_{u}^{l+1}=\Bigg{\|}_{c=1}^{C}\Bigg{(}\sum_{v\in\mathcal{N}(u)}\sum_{r\in\mathcal{R}(u,v)}\alpha_{u,r,v}^{l+1,c}*Message_{u,r,v}^{l+1,c}\Bigg{)}$

where $\alpha_{u,r,v}^{l+1,c}$ is the attention weight for the embedding
triple $\left(z_{u}^{l},x_{r}^{l},z_{v}^{l}\right)$ for $c$th head in multi-
head attentions, $Message_{u,r,v}$ denotes the information aggregated from
$e_{v}$ to $e_{u}$ with relation $r$, $\|$ represents the concatenation
operation between the total $C$ heads attentions. The embedding of relation r
is updated in $l+1$ layer with Equation 2 and 3.

(2) $x_{r}^{l+1}=\Big{\|}_{c=1}^{C}(x_{r}^{l+1,c})$

(3) $x_{r}^{l+1,c}=MLP_{r}^{l,c}\left(x_{r}^{l}\right)$

where $MLP_{r}^{l,c}$ is a relation-type specific MLP network in $l$th layer
for $c$th heads.

The attention weight $\alpha_{u,r,v}^{l,c}$ is as shown in Equation 4 $\sim$ 7

(4) $\alpha_{u,r,v}^{l+1,c}=\frac{\left\langle
q_{u}^{l,c},MLP_{r,1}^{l,c}\left(k_{v}^{l,c}\star
x_{r}^{l+1}\right)\right\rangle}{\sum_{w\in\mathcal{N}(u)}\sum_{r^{\prime}\in\mathcal{R}(u,w)}\left\langle
q_{u}^{l,c},MLP_{r^{\prime},1}^{l,c}\left(k_{w}^{l,c}\star
x_{r^{\prime}}^{l+1,c}\right)\right\rangle}$

(5) $q_{u}^{l,c}=MLP_{q}^{l,c}\left(z_{u}^{l}\right)$

(6) $k_{u}^{l,c}=MLP_{k}^{l,c}\left(z_{u}^{l}\right)$

(7) $v_{u}^{l,c}=MLP_{v}^{l,c}\left(z_{u}^{l}\right)$

where $q^{l,c}$ and $k^{l,c}$ is the query vector and key vector for entities
in multi-head attentions, $MLP_{r,1}^{l,c}$ is a relation-type specific MLP
without any bias terms, $\mathcal{N}(u)$ represents the set of neighbors of
entity $e_{u}$, $\mathcal{R}(u,w)$ denotes the set of relations between
$e_{u}$ and $e_{w}$, $\langle
q,k\rangle=\exp\left(\frac{q^{T_{*k}}}{\sqrt{d}}\right)$, and $\star$
represents the circular-correlation mentioned in HolE (Nickel et al., 2016).

The message passing information Message $e_{u,r,v}^{l,c}$ is as follows:

(8) ${Message}_{u,r,v}^{l+1,c}=MLP_{r,2}^{l,c}\left(v_{t}^{l,c}\star
x_{r}^{l+1,c}\right)$

where $MLP_{r,2}^{l,c}$ is also an unbiased relation-type specific MLP as
$MLP_{r,1}^{l,c}$.

###  4.3. Similarity scoring functions for relation predictions

Once embeddings of entities and relations are obtained, NC-KGE introduced a
learnable similarity function to score the embeddings of triplets in the
knowledge graph as same as ConvE. It is a convolution over 2D shaped
embeddings as formulated in Equation 9.

(9) $\psi\left(e_{u},r,e_{v}\right)=f\left(\text{ reshape
}\left(f\left(\left[\overline{e_{u}},\bar{r}\right]\otimes
w\right)\right)*W\right)\cdot e_{v}$

Where $\overline{e_{u}},\bar{r}$ denote a 2D reshaping of $e_{u}$ and relation
type $r$, [] denotes a concatenation operation, $\otimes w$ represents a
2D-convolutional layer with filters $w$, $f$ denotes a non-linear function,
$W$ is a linear transformation matrix, and $\cdot$ represents the inner
product operation. We need to note that NC-KGE can also integrate the classic
statistic similarity measure functions as TransE, DisMult, ComplEx and SimplE
as listed Table 1, but extensive experiments have shown that a learnable
similarity function in Equation 9 outperforms these traditional similarity
measures, as discussed in Section 5.4 Results.

Table 1. Examples of triple similarity measure functions proposed by TransE, DistMult, ComplEx, SimplE and ConvE. For triples $\left(e_{i},r_{j},e_{k}\right)$, we use $\mathrm{e}_{i},\mathrm{r}_{j},\mathrm{e}_{k}$ to represent the embedding of its components (in SimplE, they have two parts, which we use index to represent). $\|\cdot\|_{p}$ represents the $p$-norm; $\langle\cdot,;\rangle$ is the generalized three-way dot product; $\operatorname{Re}(\cdot)$ is the real part of the complex number; $\bar{e}_{k}$ is the complex conjugate of the complex-valued vector $\mathrm{e}_{k}$. Model | Function
---|---
TransE (Bordes et al., 2013) | $-\left\|\mathrm{e}_{i}+\mathrm{r}_{j}-\mathrm{e}_{k}\right\|_{p}$
DistMult (Yang et al., 2014) | $\left\langle\mathrm{e}_{i},\mathrm{r}_{j},\mathrm{e}_{k}\right\rangle$
ComplEx (Trouillon et al., 2016) | $\operatorname{Re}\left(\left\langle\mathrm{e}_{i},\mathrm{r}_{j},\overline{\mathrm{e}}_{k}\right\rangle\right)$
SimplE (Kazemi and Poole, 2018) | $\frac{1}{2}\left(\left\langle\mathrm{e}_{i1},\mathrm{r}_{j1},\mathrm{e}_{k1}\right\rangle+\left\langle\mathrm{e}_{i2},\mathrm{r}_{j2},\mathrm{e}_{k2}\right\rangle\right)$
ConvE (Dettmers et al., 2018) | $f\left(vec\left(f\left(concat\left(\overline{e_{i}},\overline{r_{j}}\right)*\omega\right)\right)W\right)e_{k}$

###  4.4. Contrastive learning objective

NC-KGE aims to mining the hidden information between entities and relations by
maximizing the similarity score between positive samples and minimizing it
between negative ones. Thus, a classic contrastive training objective function
are introduced into NC-KGE for a given triplet $\left(e_{h},r,e_{t}\right)$ as
shown in Equation 10.

(10)
$\small\mathcal{L}=-\log\frac{\sum_{k=1}^{K^{+}}\exp\left(\bar{S}\left(z_{h},x_{r},z_{t}^{+}\right)/\tau\right)}{\sum_{k=1}^{K^{+}}\exp\left(\bar{S}\left(z_{h},x_{r},z_{t}^{+}\right)/\tau\right)+Q\sum_{k=1}^{K^{-}}\exp\left(\bar{S}\left(z_{h},x_{r},z_{t}^{-}\right)/\tau\right)}$

where the construction $K^{+}$and $K^{-}$ are the number of positive triple
pairs and negative triple pairs, respectively, $Q$ is a scaling weight for
negative pairs, $\tau$ is a temperature factor controlling the discriminable
capability of KGE model to negative pairs. To avoid extreme temperature
coefficients affecting contrastive learning effects, we adopt a simulated
annealing strategy to adjust the temperature factor dynamically in the range
of [0.1,1.5] according to the MRR metrics. Additionally, we need to note that
the similarity scores $S$ were layer-normalized to compute the contrastive
loss, denoted as $\bar{S}$, to avoid the numerical overflow in exponential
operation and underflow in logarithmic operations.

##  5\. Experiments

Two commonly used benchmark datasets FB15k-237 and WN18RR (Daza et al., 2021)
for KGE methods are utilized to evaluate the performance of NC-KGE on
relation-predictions. We also perform benchmarks on three Bio-medical specific
datasets including PharmKG8k-28, DRKG17k-21 and BioKG72k-14, derived from
PharmKG, Drug Repositioning Knowledge Graph, and BioKG datasets. In these
benchmarks, we could further evaluate the performance of NC-KGE on relation-
predictions between different bio-meaningful entities. The detailed
description of datasets is shown in Table 8 of Appendix A.

###  5.1. Baselines

The baselines are composed of three types: Translation Model, Semantic
Matching Model and GNN-based Model. Translation Models include TransE (Bordes
et al., 2013), RotatE (Sun et al., 2019) and PaiRE (Chao et al., 2020).
Semantic Matching Models include DistMult (Yang et al., 2014), ComplEx
(Trouillon et al., 2016), TuckER (Balažević et al., 2019), ConvE (Dettmers et
al., 2018), InteractE (Vashishth et al., 2020) and PROCRUSTES (Peng et al.,
2021). GNN-based Models include HyConvE (Wang et al., 2023), MEKER (Chekalina
et al., 2022), RAGAT (Liu et al., 2021), HRGAT (Zheng et al., 2020), R-GCN
(Schlichtkrull et al., 2018), KBGAT (Nathani et al., 2019), A2N (Bansal et
al., 2019), SACN (Shang et al., 2019), CompGCN (Vashishth et al., 2019) and
SE-GNN (Li et al., 2022a)

###  5.2. Task and Evaluation

Relation Prediction, also termed Link Prediction, aims at inferring missing
facts based on the facts in Knowledge Graph. Similar to question answering, we
assess the quality of relation-predictions using the following ranking task:
for all triplets $\left(e_{h},r,e_{t}\right)$ in both training sets and test
sets, (1) we hidden the tail entity $e_{t}$. (2) we compute the similarity
scores $S\left(e_{h},r,e_{t}^{*}\right)$ for all $e_{t}^{*}\in\mathcal{E}$ as
discussed in Section 4.3. (3) we sort values by decreasing order and (4)
record the rank of the correct entity $e_{t}$. An identical process is
repeated for prediction $e_{h}$.

Two kinds of metrics are introduced for evaluation, including the proportion
of correct entities ranked in the top (such as top 1,3,10, denoted as Hits@1,
Hits@3, Hits@10), and the mean reciprocal rank (MRR). Let $r_{th}$ be the rank
of the correct triplet $t=\left(e_{h},r,e_{t}\right)$ among all possible
triples when hidden the head entity, while $r_{tt}$ represent its rank when
hidden the tail entity. MRR is the average of reciprocal rank of a set of
correct fact triplets $\mathcal{T}$ as shown in Equation 11.

(11)
$MRR=\frac{1}{2|\mathcal{T}|}\sum_{t\in\mathcal{T}}\frac{1}{r_{th}}+\frac{1}{r_{tt}}$

The Hits at 1 metric (H@1) is obtained by counting the number of times the
correct triple appears at position 1. The H@3 and H@10 are computed similarly,
considering the first 3 and 10 positions, respectively.

###  5.3. Experimental Setup

In our benchmark experiments, we employed a KGE encoder in NC-KGE comprising
two layers of RAMHA. The RAMHA model utilized 10 heads, and the hidden layer
dimension for all MLP was 200. For Equation 10, the positive pair number for
each fact triplet $K^{+}$ was set to 1 while all the negative pairs are used
in node-based contrastive training. To stabilize the training phase and
prevent overfitting, we introduced a dropout rate of 0.2 and applied batch
layer normalization between the RAMHA layers. The AdamW (Loshchilov and
Hutter, 2017) optimizer and cosine annealing learning rate scheduler
(Loshchilov and Hutter, 2016) is used in training. The patience of simulated
annealing strategy for temperature factor is set to 50 according to MRR
metric.

Table 2. Model reports on FB15k-237 and WN18RR test set. The best results are in bold. † denotes that results are from the published paper. Other results are from SE-GNN (Li et al., 2022a). Dataset | FB15k-237 | WN18RR
---|---|---
Task | Link Prediction | Link Prediction
Metric $\rightarrow$ | MRR | H@1 | H@3 | H@10 | MRR | H@1 | H@3 | H@10
Model $\downarrow$ |  |  |  |  |  |  |  |
TransE | 0.294 | - | - | 0.465 | 0.226 | - | - | 0.501
RotatE | 0.338 | 0.241 | 0.375 | 0.533 | 0.476 | 0.428 | 0.492 | 0.571
PaiRE | 0.351 | 0.256 | 0.387 | 0.544 | - | - | - | -
DistMult | 0.241 | 0.155 | 0.263 | 0.419 | 0.430 | 0.390 | 0.440 | 0.490
ComplEx | 0.247 | 0.158 | 0.275 | 0.428 | 0.440 | 0.410 | 0.460 | 0.510
TuckER | 0.358 | 0.266 | 0.394 | 0.544 | 0.470 | 0.443 | 0.482 | 0.526
ConvE | 0.325 | 0.237 | 0.356 | 0.501 | 0.430 | 0.400 | 0.440 | 0.520
InteractE | 0.354 | 0.263 | - | 0.535 | 0.463 | 0.430 | - | 0.528
PROCRUSTES | 0.345 | 0.249 | 0.379 | 0.541 | 0.474 | 0.421 | 0.502 | 0.569
HyConvE† | 0.339 | 0.212 | - | 0.458 | 0.461 | 0.432 | - | 0.534
MEKER† | 0.359 | 0.268 | 0.392 | 0.539 | 0.477 | 0.437 | 0.488 | 0.545
R-GCN | 0.248 | 0.151 | - | 0.417 | - | - | - | -
KBGAT | 0.157 | - | - | 0.331 | 0.412 | - | - | 0.554
A2N | 0.317 | 0.232 | 0.348 | 0.486 | 0.450 | 0.420 | 0.460 | 0.510
SACN | 0.350 | 0.260 | 0.390 | 0.540 | 0.470 | 0.430 | 0.480 | 0.540
CompGCN | 0.355 | 0.264 | 0.390 | 0.535 | 0.479 | 0.443 | 0.494 | 0.546
SE-GNN | 0.365 | 0.271 | 0.399 | 0.549 | 0.484 | 0.446 | 0.509 | 0.572
NC-KGE (our) | 0.366 | 0.273 | 0.392 | 0.542 | 0.486 | 0.447 | 0.499 | 0.556
Table 3. The results of models on DRKG17k-21, BioKG72k-14 and PharmKG8k-28 Datasets. † denotes that we reproduce the results using the codes 111https://github.com/DeepGraphLearning/. For other results, we implemented the official codes. Dataset | DRKG17k-21 | BioKG72k-14 | PharmKG8k-28
---|---|---|---
Task | Link Prediction | Link Prediction | Link Prediction
Metric $\rightarrow$ | MRR | H@1 | H@3 | H@10 | MRR | H@1 | H@3 | H@10 | MRR | H@1 | H@3 | H@10
Model $\downarrow$ |  |  |  |  |  |  |  |  |  |  |  |
TransE | 0.321 | 0.035 | 0.558 | 0.744 | 0.116 | 0.026 | 0.149 | 0.276 | 0.116 | 0.038 | 0.127 | 0.269
DistMult | 0.240 | 0.175 | 0.254 | 0.371 | 0.045 | 0.021 | 0.041 | 0.083 | 0.218 | 0.152 | 0.237 | 0.335
ComplEx | 0.099 | 0.036 | 0.087 | 0.227 | 0.111 | 0.073 | 0.118 | 0.174 | 0.124 | 0.064 | 0.128 | 0.244
TruckER | 0.460 | 0.411 | 0.535 | 0.557 | 0.226 | 0.174 | 0.237 | 0.327 | 0.182 | 0.103 | 0.202 | 0.336
HRGAT | 0.540 | 0.483 | 0.582 | 0.720 | 0.103 | 0.061 | 0.105 | 0.185 | 0.134 | 0.063 | 0.144 | 0.271
SACN | 0.487 | 0.393 | 0.534 | 0.665 | 0.179 | 0.118 | 0.192 | 0.299 | 0.156 | 0.085 | 0.170 | 0.296
CompGCN | 0.562 | 0.466 | 0.619 | 0.739 | 0.221 | 0.170 | 0.230 | 0.321 | 0.193 | 0.110 | 0.216 | 0.352
SE-GNN | 0.575 | 0.481 | 0.631 | 0.746 | 0.237 | 0.183 | 0.248 | 0.343 | 0.206 | 0.120 | 0.232 | 0.374
NC-KGE (our) | 0.590 | 0.505 | 0.637 | 0.747 | 0.240 | 0.185 | 0.256 | 0.344 | 0.228 | 0.145 | 0.252 | 0.390

###  5.4. Results

Here, we first evaluate the performance of NC-KGE on general knowledge graph
FB15k-237 and WN18RR, the benchmark results are as shown in Table2. Compared
with 17 baselines, NC-KGE outperforms all the translation distance-based and
semantic matching-based models, and is competitive with the SOTA method, SE-
GNN. NC-KGE obtains obvious improvement compared to CompGCN, a typical GNN-
based model, indicating node-based contrastive learning and RAMHA sufficient
for better knowledge graph embedding.

For bio-medical specific knowledge graphs, NC-KGE outperforms all the
baselines as shown in Table 3. In PharmKG8k-28, the knowledge entity can be
clearly categorized into four bio-medical meaningful types: gene, disease and
chemical compounds, Thus, the relation prediction can be divided into 6 types
according to the types of head and tail entity of triplet. Then, we further
evaluate the relation prediction performance of NC-KGE on these subdomains. As
shown in Table 4, NC-KGE give obviously better results for Chemical-Chemical
and Disease-Disease relation predictions, such as drug combination
relationship and disease complications, which MRR is 0.483 and 0.471,
respectively. Both of these subdomains are belonging to “Interactions”
category in PharmKG. Similarly, NC-KGE also achieve the best results, MRR is
0.640 on Chemical-Chemical relation predictions, since no Chemical-Disease and
Disease-Disease relations exists as shown in Table 5.

Additionally, NC-KGE proves to be more efficient for training, as illustrated
in Figure 3, Figure 5, and Table 7. The Mean Reciprocal Rank (MRR) on test
sets converges faster with NC-KGE than with any other methods. When we
substitute the node-based contrastive objective with other functions like
BCELoss (Binary CrossEntropy), MPLoss (Maximize Positive Sample Softmax Loss)
and MRLoss (Margin Ranking Loss), both accuracy and efficiency decrease. These
findings highlight the efficiency and potential applications of NC-KGE in
biomedical knowledge graphs.

Table 4. Experiments on the PharmKG8k-28 subdataset Dataset | PharmKG8k-28
---|---
Task | Link Prediction
Metric $\rightarrow$ | MRR | H@1 | H@3 | H@10
SubDataset $\downarrow$ |  |  |  |
Gene-Chemical | 0.160 | 0.093 | 0.169 | 0.296
Chemical-Disease | 0.153 | 0.082 | 0.160 | 0.291
Disease-Disease | 0.471 | 0.377 | 0.507 | 0.653
Gene-Gene | 0.244 | 0.156 | 0.275 | 0.413
Gene-Disease | 0.175 | 0.103 | 0.186 | 0.320
Chemical-Chemical | 0.483 | 0.367 | 0.537 | 0.713
ALL | 0.228 | 0.145 | 0.252 | 0.390
Table 5. Experiments on the DRKG17k-21 subdataset Dataset | DRKG17k-21
---|---
Task | Link Prediction
Metric $\rightarrow$ | MRR | H@1 | H@3 | H@10
SubDataset $\downarrow$ |  |  |  |
Gene-Chemical | 0.333 | 0.258 | 0.363 | 0.480
Gene-Gene | 0.243 | 0.160 | 0.272 | 0.400
Gene-Disease | 0.222 | 0.138 | 0.223 | 0.447
Chemical-Chemical | 0.640 | 0.549 | 0.693 | 0.811
ALL | 0.590 | 0.505 | 0.637 | 0.747
Table 6. Results of baselines + NCL. Dataset | PharmKG8k-28
---|---
Task | Link Prediction
Metric $\rightarrow$ | MRR | H@1 | H@3 | H@10
Model $\downarrow$ |  |  |  |
CompGCN | 0.193 | 0.110 | 0.216 | 0.352
SE-GNN | 0.206 | 0.120 | 0.232 | 0.374
CompGCN + NCL | 0.194 | 0.112 | 0.217 | 0.354
SE-GNN + NCL | 0.212 | 0.127 | 0.237 | 0.377
Table 7. KGE with diffirent loss functions. Dataset | PharmKG8k-28
---|---
Task | Link Prediction
Metric $\rightarrow$ | MRR | H@1 | H@3 | H@10
Model $\downarrow$ |  |  |  |
KGE + BCELoss | 0.222 | 0.141 | 0.244 | 0.379
KGE + MPLoss | 0.205 | 0.124 | 0.226 | 0.363
KGE + MRLoss | 0.163 | 0.082 | 0.176 | 0.326
KGE+ NCLoss | 0.228 | 0.145 | 0.252 | 0.390

(a) Multi-model Convergence.

(b) Multi-loss Convergence on KGE.

Figure 3. Performance of the models on PharmKG8k-28.

(a) Convergence of SE-GNN + NCL.

(b) Convergence of CompGCN + NCL.

Figure 4. Performance of the baseline models on PharmKG8k-28.

###  5.5. Discussion

The Universality of NC-KGE. NC-KGE is a straightforward yet effective method
that integrates contrastive learning with non-contrastive Knowledge Graph
Embedding (KGE) techniques to enhance performance on biomedical knowledge
graphs. In our study, we have integrated NC-KGE with existing models like
CompGCN and SE-GNN, using PharmKG8k-28 as a representative example. The
evaluation metrics for both CompGCN and SE-GNN demonstrated improvements, as
outlined in Table 6. Furthermore, the evolution of Mean Reciprocal Rank (MRR)
on the test set is depicted in Figure 4. NC-KGE not only enhances the accuracy
of the models but also accelerates the convergence of CompGCN and SE-GNN. This
suggests a general improvement resulting from node-based contrastive learning.

The performance of NC-KGE with different similarity measure functions. In this
paper, we consider the effect of different similarity measure functions on NC-
KGE. As shown in Table 9 of Appendix A, we consider 3 common classes of metric
functions, including TransE based on translation, DistMult, SimplE and ComplEx
based on tensor decomposition, ConvE with parameters based on convolutional
networks. Among the five similarity measure functions, NC-KGE combined with
ConvE has the best performance.

The performance of NC-KGE under various temperature coefficients. In this
section, we discuss the temperature hyperparameter for NC-KGE. From previous
experience, smaller temperature coefficient usually makes the model tend to
distinguish more difficult negative samples, and the learned embedding will be
smoother. However, this may also lead to the model paying too much attention
to difficult negative samples, resulting in insufficient attention to common
negative samples, which is not conducive to learning good embeddings. As shown
in Table 9 of Appendix A, we show the temperature coefficient is set to 0.1,
0.2, 0.3, 0.5, 0.7, 0.8, 1.0, 1.1, 1.2, 1.3, 1.5 and dynamic adjustment. Among
them, the temperature coefficient based on the dynamic adjustment strategy
achieves better performance. However, when the temperature coefficient is set
to a certain value, the performance of the model does not change
significantly.

The performance of NC-KGE with different number of negative samples. The
number of negative samples is always an important discussion topic for self-
supervised learning or unsupervised learning. In general, more negative
samples will allow the model to see more diverse data points, which will
improve performance. However, more negative samples also mean higher sampling
cost. So we need to make a trade-off between the two. Here, we also show the
performance of NC-KGE with different number of negative samples. As shown in
Table 9 of Appendix A, we find that NC-KGE achieves a decent performance when
the number of negative samples for contrastive learning is 1000, and it
performs best when the number is the whole.

##  6\. Conclusion

In this study, we introduced a node-based contrastive knowledge graph
embedding method called NC-KGE for relation prediction in biomedical knowledge
graphs. NC-KGE is a simple and versatile approach that can be seamlessly
integrated into existing KGE methods to enhance training convergence and
improve overall performance. This distinctive feature allows NC-KGE to fully
leverage advancements in state-of-the-art models. In the future work, we will
expand the application of NC-KGE to diverse downstream tasks, including but
not limited to knowledge-guided molecular property prediction and the
prediction of higher-order biomedical n-ary relationships. By leveraging
semantically rich pre-trained node and relation embeddings, we aim to maximize
the potential of NC-KGE in these domains.

###### Acknowledgements.

Thanks to the readers.

## References

  * (1)
  * Balažević et al. (2019) Ivana Balažević, Carl Allen, and Timothy M Hospedales. 2019\.  Tucker: Tensor factorization for knowledge graph completion.  _arXiv preprint arXiv:1901.09590_ (2019).
  * Bansal et al. (2019) Trapit Bansal, Da-Cheng Juan, Sujith Ravi, and Andrew McCallum. 2019.  A2N: Attending to neighbors for knowledge graph inference. In _Proceedings of the 57th annual meeting of the association for computational linguistics_. 4387–4392.
  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013\.  Translating embeddings for modeling multi-relational data.  _Advances in neural information processing systems_ 26 (2013).
  * Cao et al. (2021) Zongsheng Cao, Qianqian Xu, Zhiyong Yang, Xiaochun Cao, and Qingming Huang. 2021.  Dual quaternion knowledge graph embeddings. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 35. 6894–6902.
  * Chao et al. (2020) Linlin Chao, Jianshan He, Taifeng Wang, and Wei Chu. 2020\.  Pairre: Knowledge graph embeddings via paired relation vectors.  _arXiv preprint arXiv:2011.03798_ (2020).
  * Chekalina et al. (2022) Viktoriia Chekalina, Anton Razzhigaev, Albert Sayapin, and Alexander Panchenko. 2022.  MEKER: Memory Efficient Knowledge Embedding Representation for Link Prediction and Question Answering.  _arXiv preprint arXiv:2204.10629_ (2022).
  * Choudhary et al. (2021) Shivani Choudhary, Tarun Luthra, Ashima Mittal, and Rajat Singh. 2021.  A survey of knowledge graph embedding and their applications.  _arXiv preprint arXiv:2107.07842_ (2021).
  * Daza et al. (2021) Daniel Daza, Michael Cochez, and Paul Groth. 2021\.  Inductive Entity Representations from Text via Link Prediction. In _Proceedings of the Web Conference 2021_. 798–808.
  * Dettmers et al. (2018) Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018.  Convolutional 2d knowledge graph embeddings. In _Thirty-second AAAI conference on artificial intelligence_.
  * García-Durán et al. (2014) Alberto García-Durán, Antoine Bordes, and Nicolas Usunier. 2014\.  Effective blending of two and three-way interactions for modeling multi-relational data. In _Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2014, Nancy, France, September 15-19, 2014. Proceedings, Part I 14_. Springer, 434–449.
  * Guo et al. (2016) Shu Guo, Quan Wang, Lihong Wang, Bin Wang, and Li Guo. 2016.  Jointly embedding knowledge graphs and logical rules. In _Proceedings of the 2016 conference on empirical methods in natural language processing_. 192–202.
  * Huang et al. (2022) Hongren Huang, Chen Li, Xutan Peng, Lifang He, Shu Guo, Hao Peng, Lihong Wang, and Jianxin Li. 2022\.  Cross-knowledge-graph entity alignment via relation prediction.  _Knowledge-Based Systems_ 240 (2022), 107813.
  * Ioannidis et al. (2020) Vassilis N. Ioannidis, Xiang Song, Saurav Manchanda, Mufei Li, Xiaoqin Pan, Da Zheng, Xia Ning, Xiangxiang Zeng, and George Karypis. 2020\.  DRKG - Drug Repurposing Knowledge Graph for Covid-19.  https://github.com/gnn4dr/DRKG/.
  * Kazemi and Poole (2018) Seyed Mehran Kazemi and David Poole. 2018.  Simple embedding for link prediction in knowledge graphs.  _arXiv preprint arXiv:1802.04868_ (2018).
  * Li et al. (2022a) Ren Li, Yanan Cao, Qiannan Zhu, Guanqun Bi, Fang Fang, Yi Liu, and Qian Li. 2022a.  How does knowledge graph embedding extrapolate to unseen data: a semantic evidence view. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 36. 5781–5791.
  * Li et al. (2022b) Rui Li, Jianan Zhao, Chaozhuo Li, Di He, Yiqi Wang, Yuming Liu, Hao Sun, Senzhang Wang, Weiwei Deng, Yanming Shen, et al. 2022b.  House: Knowledge graph embedding with householder parameterization. In _International Conference on Machine Learning_. PMLR, 13209–13224.
  * Liang et al. (2023a) Huidong Liang, Xingjian Du, Bilei Zhu, Zejun Ma, Ke Chen, and Junbin Gao. 2023a.  Graph contrastive learning with implicit augmentations.  _Neural Networks_ 163 (2023), 156–164.
  * Liang et al. (2023b) Ke Liang, Yue Liu, Sihang Zhou, Wenxuan Tu, Yi Wen, Xihong Yang, Xiangjun Dong, and Xinwang Liu. 2023b.  Knowledge Graph Contrastive Learning Based on Relation-Symmetrical Structure.  _IEEE Transactions on Knowledge and Data Engineering_ (2023).
  * Lin et al. (2015) Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015.  Learning entity and relation embeddings for knowledge graph completion. In _Twenty-ninth AAAI conference on artificial intelligence_.
  * Liu et al. (2017) Hanxiao Liu, Yuexin Wu, and Yiming Yang. 2017.  Analogical inference for multi-relational embeddings. In _International conference on machine learning_. PMLR, 2168–2178.
  * Liu et al. (2021) Xiyang Liu, Huobin Tan, Qinghong Chen, and Guangyan Lin. 2021\.  RAGAT: Relation aware graph attention network for knowledge graph completion.  _IEEE Access_ 9 (2021), 20840–20849.
  * Loshchilov and Hutter (2016) Ilya Loshchilov and Frank Hutter. 2016.  Sgdr: Stochastic gradient descent with warm restarts.  _arXiv preprint arXiv:1608.03983_ (2016).
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ (2017).
  * Nathani et al. (2019) Deepak Nathani, Jatin Chauhan, Charu Sharma, and Manohar Kaul. 2019.  Learning attention-based embeddings for relation prediction in knowledge graphs.  _arXiv preprint arXiv:1906.01195_ (2019).
  * Nickel et al. (2016) Maximilian Nickel, Lorenzo Rosasco, and Tomaso Poggio. 2016\.  Holographic embeddings of knowledge graphs. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 30.
  * Nickel et al. (2011) Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. 2011\.  A three-way model for collective learning on multi-relational data. In _Icml_.
  * Palmonari and Minervini (2020) Matteo Palmonari and Pasquale Minervini. 2020.  Knowledge graph embeddings and explainable AI.  _Knowledge Graphs for Explainable Artificial Intelligence: Foundations, Applications and Challenges_ 47 (2020), 49.
  * Peng et al. (2021) Xutan Peng, Guanyi Chen, Chenghua Lin, and Mark Stevenson. 2021\.  Highly efficient knowledge graph embedding learning with orthogonal procrustes analysis.  _arXiv preprint arXiv:2104.04676_ (2021).
  * Schlichtkrull et al. (2018) Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018.  Modeling relational data with graph convolutional networks. In _European semantic web conference_. Springer, 593–607.
  * Shang et al. (2019) Chao Shang, Yun Tang, Jing Huang, Jinbo Bi, Xiaodong He, and Bowen Zhou. 2019\.  End-to-end structure-aware convolutional networks for knowledge base completion. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 3060–3067.
  * Sun et al. (2019) Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019\.  Rotate: Knowledge graph embedding by relational rotation in complex space.  _arXiv preprint arXiv:1902.10197_ (2019).
  * Toutanova and Chen (2015) Kristina Toutanova and Danqi Chen. 2015.  Observed versus latent features for knowledge base and text inference. In _Proceedings of the 3rd workshop on continuous vector space models and their compositionality_. 57–66.
  * Trouillon et al. (2016) Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016.  Complex embeddings for simple link prediction. In _International conference on machine learning_. PMLR, 2071–2080.
  * Vashishth et al. (2020) Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, Nilesh Agrawal, and Partha Talukdar. 2020.  Interacte: Improving convolution-based knowledge graph embeddings by increasing feature interactions. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 34. 3009–3016.
  * Vashishth et al. (2019) Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha Talukdar. 2019.  Composition-based multi-relational graph convolutional networks.  _arXiv preprint arXiv:1911.03082_ (2019).
  * Walsh et al. (2020) Brian Walsh, Sameh K. Mohamed, and Vít Novácek. 2020\.  BioKG: A Knowledge Graph for Relational Learning On Biological Data. In _CIKM ’20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020_ , Mathieu d’Aquin, Stefan Dietze, Claudia Hauff, Edward Curry, and Philippe Cudré-Mauroux (Eds.). ACM, 3173–3180.  https://doi.org/10.1145/3340531.3412776
  * Wang et al. (2023) Chenxu Wang, Xin Wang, Zhao Li, Zirui Chen, and Jianxin Li. 2023.  HyConvE: A Novel Embedding Model for Knowledge Hypergraph Link Prediction with Convolutional Neural Networks. In _Proceedings of the ACM Web Conference 2023_. 188–198.
  * Wang et al. (2022) Liang Wang, Wei Zhao, Zhuoyu Wei, and Jingming Liu. 2022\.  SimKGC: Simple contrastive knowledge graph completion with pre-trained language models.  _arXiv preprint arXiv:2203.02167_ (2022).
  * Wang et al. (2016) Zhigang Wang, Juanzi Li, Zhiyuan Liu, and Jie Tang. 2016\.  Text-enhanced representation learning for knowledge graph. In _Proceedings of International joint conference on artificial intelligent (IJCAI)_. 4–17.
  * Wang et al. (2014) Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014\.  Knowledge graph embedding by translating on hyperplanes. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 28.
  * Xie et al. (2016) Ruobing Xie, Zhiyuan Liu, Jia Jia, Huanbo Luan, and Maosong Sun. 2016.  Representation learning of knowledge graphs with entity descriptions. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 30.
  * Yang et al. (2014) Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2014.  Embedding entities and relations for learning and inference in knowledge bases.  _arXiv preprint arXiv:1412.6575_ (2014).
  * Yao et al. (2019) Liang Yao, Chengsheng Mao, and Yuan Luo. 2019.  KG-BERT: BERT for knowledge graph completion.  _arXiv preprint arXiv:1909.03193_ (2019).
  * Yin et al. (2022) Yihang Yin, Qingzhong Wang, Siyu Huang, Haoyi Xiong, and Xiang Zhang. 2022.  Autogcl: Automated graph contrastive learning via learnable view generators. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 36. 8892–8900.
  * You et al. (2020) Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. 2020\.  Graph contrastive learning with augmentations.  _Advances in neural information processing systems_ 33 (2020), 5812–5823.
  * Yu et al. (2021) Donghan Yu, Yiming Yang, Ruohong Zhang, and Yuexin Wu. 2021\.  Knowledge embedding based graph convolutional network. In _Proceedings of the Web Conference 2021_. 1619–1628.
  * Zhang et al. (2019) Shuai Zhang, Yi Tay, Lina Yao, and Qi Liu. 2019\.  Quaternion knowledge graph embeddings.  _Advances in neural information processing systems_ 32 (2019).
  * Zhang et al. (2020a) Zhanqiu Zhang, Jianyu Cai, Yongdong Zhang, and Jie Wang. 2020a.  Learning hierarchy-aware knowledge graph embeddings for link prediction. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 34. 3065–3072.
  * Zhang et al. (2020b) Zhao Zhang, Fuzhen Zhuang, Hengshu Zhu, Zhi-Ping Shi, Hui Xiong, and Qing He. 2020b.  Relational Graph Neural Network with Hierarchical Attention for Knowledge Graph Completion. In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_. AAAI Press, 9612–9619.  https://doi.org/10.1609/aaai.v34i05.6508
  * Zheng et al. (2020) Shuangjia Zheng, Jiahua Rao, Ying Song, Jixian Zhang, Xianglu Xiao, Evandro Fei Fang, Yuedong Yang, and Zhangming Niu. 2020\.  PharmKG: a dedicated knowledge graph benchmark for bomedical data mining.  _Briefings in Bioinformatics_ 22, 4 (12 2020), bbaa344.  https://doi.org/10.1093/bib/bbaa344 arXiv:https://academic.oup.com/bib/article-pdf/22/4/bbaa344/39139048/bbaa344.pdf
  * Zhu et al. (2020) Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. 2020.  Deep graph contrastive representation learning.  _arXiv preprint arXiv:2006.04131_ (2020).

##  Appendix A Appendix

The detailed description of datasets is as follows:

  * •

FB15k-237 (Toutanova and Chen, 2015) dataset contains knowledge base relation
triples and textual mentions of Freebase node pairs, as used in the work
published in (Toutanova and Chen, 2015). The knowledge base triples are a
subset of the FB15K (Bordes et al., 2013), originally derived from Freebase.
The inverse relations are removed in FB15k-237. To obtain node descriptions
and types, we employ the datasets made available by (Daza et al., 2021).

  * •

WN18RR (Dettmers et al., 2018) is created from WN18 (Bordes et al., 2013),
which is a subset of WordNet. WN18 consists of 18 relations and 40,943 nodes.
However, many text triples obtained by inverting triples from the training
set. Thus WN18RR dataset (Dettmers et al., 2018) is created to ensure that the
evaluation dataset does not have inverse relation test leakage. To obtain node
descriptions and types, we employ the datasets made available by (Daza et al.,
2021).

  * •

PharmKG8k-28 is built based on PharmKG dataset (Zheng et al., 2020). PharmKG
is a multi-relational, attributed biomedical KG, composed of more than 500 000
individual interconnections between genes, drugs and diseases, with 29
relation types over a vocabulary of $\sim$ 8000 disambiguated nodes. Each node
in PharmKG is attached with heterogeneous, domain-specific information
obtained from multi-omics data, i.e. gene expression, chemical structure and
disease word embedding, while preserving the semantic and biomedical features.
We obtained PharmKG8k-28 dataset after simple deduplication cleaning of
PharmKG.

  * •

DRKG17k-21 is built based on Drug Repositioning Knowledge Graph (DRKG)
(Ioannidis et al., 2020). DRKG is a comprehensive biological knowledge graph
relating genes, compounds, diseases, biological processes, side effects and
symptoms. DRKG includes information from six existing databases including
DrugBank, Hetionet, GNBR, String, IntAct and DGIdb, and data collected from
recent publications particularly related to Covid19.

  * •

BioKG72k-14 is built based on BioKG dataset (Walsh et al., 2020). BioKG is a
new more standardised and reproducible biological knowledge graph which
provides a compilation of curated relational data from open biological
databases in a unified format with common, interlinked identifiers. BioKG can
be used to train and assess the relational learning models in various tasks
related to pathway and drug discovery.

Table 8. Statistics of the datasets. | FB15k-237 | WN18RR | PharmKG8k-28 | DRKG17k-21 | BioKG72k-14
---|---|---|---|---|---
Relations | 237 | 11 | 28 | 21 | 14
Nodes | 14,541 | 40,943 | 7,599 | 17,508 | 72,569
Triples | 310,116 | 93,003 | 480,500 | 587,425 | 1,157,739
| Training
Nodes | 14,505 | 40,559 | 7,520 | 16,252 | 70,108
Triples | 272,115 | 86,835 | 384,880 | 470,526 | 927,348
| Validation
Nodes | 9,809 | 5,173 | 6,637 | 8,186 | 43,806
Triples | 17,535 | 3,034 | 47,570 | 58,156 | 114,617
| Test
Nodes | 10,348 | 5,323 | 6,628 | 8,242 | 43,989
Triples | 20,466 | 3,134 | 48,050 | 58,743 | 115,774
Table 9. Ablation experiments on the PharmKG8k-28 and FB15k-237 datasets.
’Neg100’ means that the number of negative samples is 100. ’Tem0.1’ means that
the temperature coefficient is 0.1. ’DynamicTem’ means to adopt the dynamic
temperature coefficient strategy.

Dataset | PharmKG8k-28
---|---
Task | Link Prediction
Metric $\rightarrow$ | MRR | H@1 | H@3 | H@10
Model $\downarrow$ |  |  |  |
NC-KGE + TransE | 0.140 | 0.077 | 0.147 | 0.263
NC-KGE + DistMult | 0.169 | 0.097 | 0.177 | 0.317
NC-KGE + SimplE | 0.200 | 0.126 | 0.217 | 0.344
NC-KGE + ComplEx | 0.205 | 0.134 | 0.220 | 0.345
NC-KGE +ConvE | 0.228 | 0.145 | 0.252 | 0.390
NC-KGE + Neg100 | 0.214 | 0.126 | 0.241 | 0.385
NC-KGE + Neg500 | 0.214 | 0.126 | 0.241 | 0.384
NC-KGE + Neg1000 | 0.220 | 0.133 | 0.246 | 0.386
NC-KGE +NegAll | 0.228 | 0.145 | 0.252 | 0.390
NC-KGE + Tem0.1 | 0.218 | 0.137 | 0.239 | 0.376
NC-KGE + Tem0.2 | 0.220 | 0.138 | 0.242 | 0.379
NC-KGE + Tem0.3 | 0.217 | 0.135 | 0.240 | 0.377
NC-KGE + Tem0.5 | 0.220 | 0.138 | 0.242 | 0.379
NC-KGE + Tem0.7 | 0.225 | 0.142 | 0.247 | 0.386
NC-KGE + Tem0.8 | 0.223 | 0.140 | 0.247 | 0.383
NC-KGE + Tem1.0 | 0.225 | 0.142 | 0.248 | 0.386
NC-KGE + Tem1.1 | 0.225 | 0.143 | 0.248 | 0.386
NC-KGE + Tem1.2 | 0.227 | 0.145 | 0.250 | 0.384
NC-KGE + Tem1.3 | 0.225 | 0.142 | 0.250 | 0.386
NC-KGE + Tem1.5 | 0.225 | 0.142 | 0.249 | 0.386
NC-KGE +DynamicTem | 0.228 | 0.145 | 0.252 | 0.390

Dataset | FB15k-237
---|---
Task | Link Prediction
Metric $\rightarrow$ | MRR | H@1 | H@3 | H@10
Model $\downarrow$ |  |  |  |
NC-KGE + TransE | 0.321 | 0.235 | 0.352 | 0.494
NC-KGE + DistMult | 0.322 | 0.235 | 0.353 | 0.499
NC-KGE + SimplE | 0.318 | 0.232 | 0.347 | 0.494
NC-KGE + ComplEx | 0.323 | 0.236 | 0.354 | 0.500
NC-KGE +ConvE | 0.362 | 0.270 | 0.392 | 0.542
NC-KGE + Neg100 | 0.342 | 0.248 | 0.377 | 0.531
NC-KGE + Neg500 | 0.352 | 0.259 | 0.387 | 0.538
NC-KGE + Neg1000 | 0.354 | 0.258 | 0.388 | 0.539
NC-KGE +NegAll | 0.362 | 0.270 | 0.392 | 0.542
NC-KGE + Tem0.1 | 0.347 | 0.255 | 0.382 | 0.531
NC-KGE + Tem0.2 | 0.348 | 0.255 | 0.384 | 0.530
NC-KGE + Tem0.3 | 0.348 | 0.258 | 0.381 | 0.528
NC-KGE + Tem0.5 | 0.348 | 0.257 | 0.381 | 0.529
NC-KGE + Tem0.7 | 0.348 | 0.254 | 0.383 | 0.534
NC-KGE + Tem0.8 | 0.347 | 0.256 | 0.381 | 0.531
NC-KGE + Tem1.0 | 0.349 | 0.257 | 0.382 | 0.535
NC-KGE + Tem1.1 | 0.349 | 0.256 | 0.383 | 0.536
NC-KGE + Tem1.2 | 0.348 | 0.258 | 0.384 | 0.537
NC-KGE + Tem1.3 | 0.348 | 0.259 | 0.385 | 0.536
NC-KGE + Tem1.5 | 0.349 | 0.258 | 0.386 | 0.536
NC-KGE +DynamicTem | 0.362 | 0.270 | 0.392 | 0.542

(a) Convergence on FB15k-237.

(b) Convergence on WN18RR.

Figure 5. Performance of the models on FB15k-237 and WN18RR.


contents=  Submitted to the Proceedings of the US Community Study on the
Future of Particle Physics (Snowmass 2021) ,
scale=1,placement=top,opacity=1,color=black,position=3.25in,1.2in

# Snowmass2021 - White Paper
Collider Physics Opportunities of Extended Warped Extra-Dimensional Models

Kaustubh Agashe  Maryland Center for Fundamental Physics, Department of
Physics, University of Maryland, College Park, MD 20742, USA  Jack H. Collins
SLAC National Accelerator Laboratory, 2575 Sand Hill Road, Menlo Park, CA
94025, USA  Peizhi Du  C.N. Yang Institute for Theoretical Physics, Stony
Brook University, Stony Brook, NY, 11794, USA  Majid Ekhterachian  Theoretical
Particle Physics Laboratory (LPTP), Institute of Physics, EPFL, Lausanne,
Switzerland  Sungwoo Hong  Department of Physics, The University of Chicago,
5720 S Ellis Ave, Chicago, IL 60637 , USA  Argonne National Laboratory, 9700
S. Cass Avenue, Lemont, IL 60439, USA  Doojin Kim  Mitchell Institute for
Fundamental Physics and Astronomy, Department of Physics and Astronomy, Texas
A&M University, College Station, TX 77845, USA  Rashmish K. Mishra  Harvard
University, 17 Oxford Street, Cambridge, MA, 02138, USA  Deepak Sathyan
Maryland Center for Fundamental Physics, Department of Physics, University of
Maryland, College Park, MD 20742, USA

###### Abstract

While the warped extra dimensional models provide an attractive solution to
both the gauge and the flavor hierarchy problems, the mass scale of new
particles predicted by the minimal models would be beyond the reach of the
LHC. Models of extended warped extra dimensions have been proposed to evade
these issues and their collider implications have been investigated for the
last decade. This white paper summarizes the recent developments in the
context of collider phenomenology. The strategies and lessons are broad, and
provide a template to extend the experimental program, to cover a wider class
of signals in other new physics scenarios as well.

###### Contents

  1. 1 Executive Summary
  2. 2 Extended Warped Model
  3. 3 Collider Signatures
    1. 3.1 Cascade Decays of Warped Vector Resonances
      1. 3.1.1 Tri-EW Boson Signals
      2. 3.1.2 Boosted di-EW Boson Signals
    2. 3.2 Four-jet Signals of KK Graviton
  4. 4 Future Prospects
  5. 5 Conclusions

##  1 Executive Summary

Although the monumental discovery of the Higgs particle puts the final piece
of the Standard Model (SM), the stability of its mass against high-scale
physics still remains as an open question that the SM cannot answer. The
Randall-Sundrum model with two branes [1, 2] is a promising framework to
address this issue, and the warped extra-dimensional models [3, 4, 5, 6],
where even the SM gauge and matter fields propagate in the bulk, additionally
provide an attractive solution to the flavor hierarchy problem. These models
predict rich phenomenology at the Large Hadron Collider (LHC), but no
conclusive signals have been observed yet; in fact, flavor/CP violation
constraints (without extra symmetries) already suggest that the mass scale of
relevant new particles be of $\mathcal{O}(10)$ TeV [7, 8, 9, 10] even before
the LHC was turned on.

In light of this situation, models of extended warped extra dimensions [11]
have been proposed to avoid these potential issues while not only keeping the
virtue of solving gauge/flavor hierarchy problem but letting new particles
[e.g., Kaluza-Klein (KK) excitations] accessible at the LHC. In this white
paper, we discuss the idea of models of extended warped extra dimension and
their phenomenological implications on energy-frontier collider phenomenology.
The signal topologies and the kinematic regimes in which they are produced,
are novel and challenging for detection. The developed collider stragies are
applicable to other BSM scenarios where such signals are relevant, and in
general they allow the experimental programs to have sensitivity to broad
class of signals.

##  2 Extended Warped Model

The basic idea is to postulate an additional brane beyond the “Higgs” brane,
where the SM Higgs field is localized, and to assume that the SM gauge fields
(together with gravity) can propagate further in the extended bulk down to the
additional brane (henceforth called IR brane). See also Figure 1. By contrast,
the SM matter fields are confined in-between the usual UV brane and the Higgs
brane, so if the Higgs brane is set to be $\mathcal{O}(10)$ TeV (at the
expense of a little hierarchy issue), flavor/CP violation constraints can be
avoided [11]. Now the IR brane is allowed to be of $\mathcal{O}(2-3)$ TeV
without any severe tension with the existing limits, thus the LHC is capable
of probing KK modes of the SM gauge fields and gravity. In addition, the usual
leading decay channels of the lightest KK modes into top quark and Higgs boson
are suppressed. This effect permits erstwhile subdominant channels to emerge
significant. Therefore, collider phenomenology that has received less
attention becomes an important aspect in exploring extra-dimensional models at
the energy-frontier facilities, throughout the upcoming decades.

Figure 1: Extended warped model with all SM gauge fields (and gravity) in the
extended bulk. The colored lines show the schematic profiles of the
corresponding fields.

##  3 Collider Signatures

In this section, we summarize various collider signatures of KK vector
resonances followed by KK graviton.

###  3.1 Cascade Decays of Warped Vector Resonances

A class of well-motivated scenarios is the process where a singly produced
gauge KK particle (say, $A_{\rm KK}$) decays to a corresponding SM gauge
particle (say, $A$) and a radion (denoted by $\varphi$) which subsequently
decays into a pair of the SM gauge particles which are possibly different from
the upstream gauge particle $A$ [12] (see Figure 2):

$pp\to A_{\rm KK},~{}A_{\rm KK}\to
A\;\varphi,~{}\varphi\to\gamma\gamma/WW/ZZ/gg\,.$ (1)

In other words, the expected experimental signature involves a two-step
cascade decay of a warped vector resonance, resulting in various combinations
of three SM gauge bosons in the final state with sizable production cross-
sections: for example, $\gamma gg$, $g\gamma\gamma$, $ggg$, $gV_{h}V_{h}$, and
$W_{l}gg$, with $V_{h}$ and $W_{l}$ denoting hadronic massive SM gauge bosons
and leptonic $W$, respectively. This cascade nature of the decay can make the
signal identification challenging (e.g. due to combinatorial ambiguities, or
due to merging of some of the final states, if sufficiently boosted). One can
imagine such signals easily appearing in other contexts, making this line of
investigation much more general. A sensitivity study for those channels has
been performed, reporting that they would allow for an excess of $\sim
3\sigma$ to more than $\sim 10\sigma$ at the high-luminosity (HL) LHC [12].
Note that the radion cannot be lighter than $\sim 1$ TeV in the scenario where
all SM gauge field propagate in the extended bulk, since there are strong
bounds from its di-photon decay with production from gluon fusion (both
couplings are at respective full/vanilla strength). So, in this case, the two
SM gauge bosons from radion decay are well-separated.

Figure 2: Cascade decays of KK gauge bosons, $A_{\rm KK}$ into a radion and
the coresponding SM gauge boson, $A=g/W/Z/\gamma$, followed by the radion
decaying into a pair of any SM gauge bosons.

####  3.1.1 Tri-EW Boson Signals

An intriguing model variation under the aforementioned extended framework is
to allow only electroweak (EW) gauge fields to propagate in the extended bulk
while keeping the gluon field confined in-between the UV and Higgs branes [13]
(see the left panel of Figure 3). Under this model setup, tri-boson signals,
e.g., tri-photon, tri-$W$, $W\gamma\gamma$ etc are well motivated (see Figure
4). Depending on the mass choices, in the case of three massive gauge bosons,
they can be significantly boosted so that a novel signature of three “fat”
$W/Z$ jets can arise. We have performed a sensitivity study for the above tri-
boson signatures, using the jet substructure techniques, and found that all
channels would allow for more than $\sim 4-5\sigma$ significance even with an
integrated luminosity of 300 fb-1 [13].

Figure 3: (Left) Extended warped model with SM EW gauge fields in the extended
bulk. (Right) Extended warped model with the hypercharge gauge field in the
extended bulk. Figure 4: Cascade decays of KK EW gauge bosons ($V_{\rm KK}$)
into a radion and the corresponding SM EW gauge boson, $V=W/Z/\gamma$,
followed by the radion decaying into a pair of SM EW gauge bosons.

A more specific version involves only hypercharge gauge boson having access to
the extended bulk (see the right panel of Figure 3), in which case KK $W$ and
KK gluon are very heavy, but the KK hypercharge gauge boson is potentially
within LHC reach [13].

####  3.1.2 Boosted di-EW Boson Signals

An interesting region of parameter space for the above tri-boson signal is the
region where the mass gap between the KK EW gauge and the radion is so large
that the radion is significantly boosted. The point is that a light radion is
now allowed, i.e., satisfies di-photon search bounds, since radion couplings
to SM gluons, hence its production is highly suppressed, even if decays to di-
photons are still present. As a result, its decay products, for example, two
$W/Z$ are not just significantly boosted, but merged, forming a “fat” jet with
multi-layered substructures (see Figure 5). This signature is not well
captured by existing searches in which conventional boosted techniques are
adopted, so a dedicated search strategy with a targeted jet reconstruction
algorithm is needed to improve the signal sensitivity.

Figure 5: Cascade decay of KK $W/Z$ into boosted/merged $WW/ZZ$, plus an
isolated $W/Z$. Note that this corresponds to the same Feynman diagram as in
Figure 4, but with the radion being light, thus boosted.

We have developed respective jet substructure techniques to be sensitive to
fully hadronic boosted di-boson jets and semi-leptonic boosted di-boson jets
[14]. The application of the techniques to the fully hadronic boosted radion
(i.e., $\varphi\to W_{h}W_{h}$) and the semi-leptonic boosted radion (i.e.,
$\varphi\to W_{h}W_{l}$) suggests that the associated searches at the LHC with
an integrated luminosity of 300 fb-1 allow us to probe regions of parameter
space in the $m_{W_{\rm KK}}-m_{\varphi}$ plane that have been unexplored by
the existing searches. The techniques developed are generic enough to be
straightforwardly applicable to similar boosted di-boson resonances in other
models. Indeed, the fully hadronic $3\;W$ channel along above lines has
already been searched for by CMS [15] and similarly the semi-leptonic $3\;W$
signal [16]: in both studies, both light and heavy radion were considered. One
can also study hypercharge-only in extended bulk model with a light radion,
giving boosted/merged di-photon (or $Z+\gamma$) from light radion decay (see
Figure 6). Presence of a light scalar is however a generic possibility, and
the boosted regime of its production is quite relevant to study for the
experimental programs to have a wide coverage, therefore making this line of
investigation again quite broad.

Figure 6: Cascade decay of KK hypercharge gauge boson into boosted/merged di-
photon (Left) or boosted/merged $Z+\gamma$ (Right), plus an isolated photon.
Note that this corresponds to the same Feynman diagram as in Figure 4, but
with the radion being light, thus boosted.

###  3.2 Four-jet Signals of KK Graviton

As gravity always propagates in the extended bulk, its lightest KK mode can be
accessible at the LHC, too. Like the KK gauge modes, its dominant decay
channels into top quark and Higgs pairs in the standard framework become
subdominant in the extended framework, whereas the decays to a radion pair and
to a KK gauge particle and its corresponding SM gauge particle begin to “stand
out”. An interesting signature is the four-jet final state stemming from the
processes, $G_{\rm KK}\to\varphi\varphi,~{}\varphi\to gg$ (called “radion
channel”) and $G_{\rm KK}\to gg_{\rm KK},~{}g_{\rm KK}\to
g\varphi,~{}\varphi\to gg$ (called “KK gluon channel”), with $G_{\rm KK}$ and
$g_{\rm KK}$ denoting the lightest KK graviton mode and the lightest KK gluon
mode, respectively (see Figure 7). These channels become more significant
especially in the model variation where the SM gluon field is propagating in
the extended bulk while the propagation of EW fields is restricted to the
Higgs brane [17] (see Figure 8), in part because radion decays to di-photon
(golden channel) are then extremely suppressed.

Depending on the underlying mass spectrum, one channel is leading to the
other, and different search strategies are motivated due to the difference of
the associated event topologies. We have performed a sensitivity study with a
few well-motivated benchmark sets of parameter values giving well-separated
four jets, and found that the HL-LHC (3,000 fb-1) would be sensitive to the KK
graviton signals by $\sim 2.5-5\sigma$ significance [17]. It was further found
that the radion channel can be the first discovery channel of KK graviton due
to the effectiveness of the cuts designed for the event topology of the radion
channel signal, while the KK gluon channel can serve as a cross-check to
understand the structure of the underlying model.

Figure 7: Cascade decay of KK graviton into two radions (Left) or a radion,
plus a KK gluon (Right). Figure 8: Extended warped model with only SM gluon
(and gravity) in extended bulk.

##  4 Future Prospects

We can consider a couple of follow-up studies. While a series of our works
have shown that the (HL-)LHC would be able to explore parameter space of the
extended warped models, the high-energy LHC ($\sqrt{s}=27$ TeV), the 100 TeV
future collider and a muon collider will be excellent venues to explore deep
in the parameter space (toward larger masses) or to carry out the precision
studies for the models (upon discovery). So, it would be interesting (and
rather straightforward) to investigate prospects of KK gauge particles and KK
graviton at these future colliders, in terms of their experimental reaches and
post-discovery analyses. Moreover, since such signals are already on the radar
of CMS, we hope that the LHC collaborations will pursue this for Run III of
the LHC and HL-LHC as well.

Second, the above-described KK graviton signals with light radions ($\lesssim
500$ GeV), which are relatively less constrained by existing bounds, are
interesting to investigate. Once again, such a light radion is allowed for
only gluon living in the extended bulk, since radion couplings to SM photons,
hence that decay channel is now highly suppressed, even if production via SM
gluon fusion is still at the vanilla/full strength. Owing to substantial mass
gaps between such radion and KK states, the gluonic radion (i.e., $\varphi\to
gg$) is significantly boosted, hence manifests itself as a merged radion jet
(akin to boosted/merged $WW$ mentioned earlier). So, the KK graviton signal in
the radion channel will appear as a dijet event, while that in the KK gluon
channel will appear as a trijet event. Dedicated jet substructure techniques
can reveal these signals, allowing us to explore different regions of
parameter space. In fact, CMS has already studied signals for such a light
radion arising from decay of direct production of KK gluon (vs. from KK
graviton mentioned above), giving an isolated gluon, plus a boosted/merged di-
gluon (see Figure 9) [18]. Obviously, this experimental jet substructure
algorithm developed for the boosted/merged di-gluon can also be used for the
KK graviton case in Figure 7.

The challenging and universal nature of the class of signals discussed in this
white paper is useful for providing a well-motivated testing ground for novel
data driven methods. For example, Ref. [19] used machine learning (ML)
techniques to identify such signals, i.e., with BSM resonances decaying into
“fat” object made of SM particles, thus with a multi-prong structure. Further,
this topology was an important and one of the most challenging ones to detect
in anomaly detection challenges [20], which inform the community of the
limitations of existing methods, and allow development of new ones.

Figure 9: Cascade decay of KK gluon into boosted/merged di-gluon, plus an
isolated gluon. Note that this corresponds to the same Feynman diagram as in
Figure 2, but with the radion being light, thus boosted.

##  5 Conclusions

Models of extended warped extra dimensions are an attractive framework as they
retain the virtue of addressing the gauge/flavor hierarchy problems like the
standard ones and contain new particles of a few TeV (or even lighter),
allowing the models to be tested even in the upcoming LHC runs. A summary of
the generalized tri-boson-type signals from gauge KK covering all
possibilities is shown in Table 1, which includes cases where the two SM gauge
bosons from radion decay are merged. A similar compilation for “quadri”-boson
signals from KK graviton can be easily done, generalizing Figure 7 to the case
with all SM gauge fields residing in the extended bulk. Furthermore, collider
techniques for searching for such new particles can inspire the search effort
for other new physics models giving similar experimental signatures. In
particular, the studies above serves as motivation to analyse cascade decays
of BSM resonances in general, i.e., within other frameworks/models as well:
again, instead of directly decaying into SM particles, BSM heavy particle can
readily decay into other/lighter BSM particles which subsequently decay into
SM particles. Given rich phenomenology and potential impacts on collider
physics, the research on the extended warped model will be an important aspect
of not only the energy-frontier program but the theory-frontier program in the
next decade.

| Singal features
---|---
Radion mass (KK scale $\gtrsim 3$ TeV) |  General topology |  (I). All SM gauge in the extended bulk [12] |  (II). Only EW in the extended bulk [13, 14] |  (III). Only QCD in the extended bulk [17]
Heavy radion ($\gtrsim 1$ TeV) |  isolated three bosons |  mixture of gluons and $W/Z/\gamma$ (Fig. 2) |  mixture of $W/Z/\gamma$ (Fig. 4) |  3 gluons/jets
Light radion ($O(100)$ GeV) |  boosted di-boson $+$ an isolated boson |  (ruled out by di-photon searches) |  $W/Z/\gamma+$ boosted $WW/ZZ/Z\gamma/\gamma\gamma$ (Figs. 5 and 6) |  gluon $+$ boosted di-gluon (Fig. 9)
Table 1: Summary of various triboson signals from KK gauge particle production
and decay into radion.

## Acknowledgments

The work of KA and DS was supported in part by the NSF grant PHY-1914731 and
by the Maryland Center for Fundamental Physics. The work of JHC is supported
by the U.S. Department of Energy, Office of Science under contract DE-
AC02-76SF00515. PD is supported in part by Simons Investigator in Physics
Award 623940 and NSF award PHY-1915093. The work of ME was supported by the
Swiss National Science Foundation under contract 200020-188671 and through the
National Center of Competence in Research SwissMAP. The work of SH is
supported by a DOE grant DE-SC-0013642 and a DOE grant DE-AC02-06CH11357. The
work of DK is supported by the DOE Grant No. DE-SC0010813. The work of RKM is
supported by the National Science Foundation under Grant No. NSF PHY-1748958
and NSF PHY-1915071.

## References

  * [1] L. Randall and R. Sundrum, _A Large mass hierarchy from a small extra dimension_ , _Phys. Rev. Lett._ 83 (1999) 3370–3373, [hep-ph/9905221].
  * [2] L. Randall and R. Sundrum, _An Alternative to compactification_ , _Phys. Rev. Lett._ 83 (1999) 4690–4693, [hep-th/9906064].
  * [3] R. Sundrum, _Tasi 2004 lectures: To the fifth dimension and back_ , in _Theoretical Advanced Study Institute in Elementary Particle Physics: Physics in D $\geqq$ 4_, pp. 585–630, 8, 2005.  hep-th/0508134.
  * [4] C. Csaki, J. Hubisz and P. Meade, _TASI lectures on electroweak symmetry breaking from extra dimensions_ , in _Theoretical Advanced Study Institute in Elementary Particle Physics: Physics in D $\geqq$ 4_, pp. 703–776, 10, 2005.  hep-ph/0510275.
  * [5] H. Davoudiasl, S. Gopalakrishna, E. Ponton and J. Santiago, _Warped 5-Dimensional Models: Phenomenological Status and Experimental Prospects_ , _New J. Phys._ 12 (2010) 075011, [0908.1968].
  * [6] E. Ponton, _TASI 2011: Four Lectures on TeV Scale Extra Dimensions_ , in _Theoretical Advanced Study Institute in Elementary Particle Physics: The Dark Secrets of the Terascale_ , pp. 283–374, 2013.  1207.3827.  DOI.
  * [7] C. Csaki, A. Falkowski and A. Weiler, _The Flavor of the Composite Pseudo-Goldstone Higgs_ , _JHEP_ 09 (2008) 008, [0804.1954].
  * [8] M. Blanke, A. J. Buras, B. Duling, S. Gori and A. Weiler, _$\Delta$ F=2 Observables and Fine-Tuning in a Warped Extra Dimension with Custodial Protection_, _JHEP_ 03 (2009) 001, [0809.1073].
  * [9] M. Bauer, S. Casagrande, U. Haisch and M. Neubert, _Flavor Physics in the Randall-Sundrum Model: II. Tree-Level Weak-Interaction Processes_ , _JHEP_ 09 (2010) 017, [0912.1625].
  * [10] B. Keren-Zur, P. Lodone, M. Nardecchia, D. Pappadopulo, R. Rattazzi and L. Vecchi, _On Partial Compositeness and the CP asymmetry in charm decays_ , _Nucl. Phys. B_ 867 (2013) 394–428, [1205.5803].
  * [11] K. Agashe, P. Du, S. Hong and R. Sundrum, _Flavor Universal Resonances and Warped Gravity_ , _JHEP_ 01 (2017) 016, [1608.00526].
  * [12] K. S. Agashe, J. Collins, P. Du, S. Hong, D. Kim and R. K. Mishra, _LHC Signals from Cascade Decays of Warped Vector Resonances_ , _JHEP_ 05 (2017) 078, [1612.00047].
  * [13] K. Agashe, J. H. Collins, P. Du, S. Hong, D. Kim and R. K. Mishra, _Dedicated Strategies for Triboson Signals from Cascade Decays of Vector Resonances_ , _Phys. Rev. D_ 99 (2019) 075016, [1711.09920].
  * [14] K. Agashe, J. H. Collins, P. Du, S. Hong, D. Kim and R. K. Mishra, _Detecting a Boosted Diboson Resonance_ , _JHEP_ 11 (2018) 027, [1809.07334].
  * [15] CMS collaboration, A. Tumasyan et al., _Search for resonances decaying to three W bosons in the hadronic final state in proton-proton collisions at $\sqrt{s}$ = 13 TeV_, 2112.13090.
  * [16] CMS collaboration, A. Tumasyan et al., _Search for resonances decaying to three W bosons in proton-proton collisions at $\sqrt{s}$ = 13 TeV_, 2201.08476.
  * [17] K. Agashe, M. Ekhterachian, D. Kim and D. Sathyan, _LHC Signals for KK Graviton from an Extended Warped Extra Dimension_ , _JHEP_ 11 (2020) 109, [2008.06480].
  * [18] CMS collaboration, A. Tumasyan et al., _Search for high-mass resonances decaying to a jet and a Lorentz-boosted resonance in proton-proton collisions at $\sqrt{s}$ = 13 TeV_, 2201.02140.
  * [19] J. Filipek, S.-C. Hsu, J. Kruper, K. Mohan and B. Nachman, _Identifying the Quantum Properties of Hadronic Resonances using Machine Learning_ , 2105.04582.
  * [20] G. Kasieczka et al., _The LHC Olympics 2020 a community challenge for anomaly detection in high energy physics_ , _Rept. Prog. Phys._ 84 (2021) 124201, [2101.08320].