# Using Graph Algorithms to Pretrain Graph Completion Transformers

Jonathan Pilault1 , Michael Galkin1, Bahare Fatemi2∗,
Perouz Taslakian3∗, David Vasquez4, Christopher Pal1,4,5
1Polytechnique Montreal & Mila, 2 Google Research,
3Samsung AI, 4ServiceNow Research, 5Canada CIFAR AI Chair
<EMAIL_ADDRESS>Work performed
at ServiceNow Research.

###### Abstract

Recent work on Graph Neural Networks has demonstrated that self-supervised
pretraining can further enhance performance on downstream graph, link, and
node classification tasks. However, the efficacy of pretraining tasks has not
been fully investigated for downstream large knowledge graph completion tasks.
Using a contextualized knowledge graph embedding approach, we investigate five
different pretraining signals, constructed using several graph algorithms and
_no external data_ , as well as their combination. We leverage the versatility
of our Transformer-based model to explore _graph structure generation_
pretraining tasks, typically inapplicable to most graph embedding methods. We
further propose a new path-finding algorithm guided by information gain and
find that it is the best-performing pretraining task across three downstream
knowledge graph completion datasets. In a multitask setting that combines all
pretraining tasks, our method surpasses some of the latest and strong
performing knowledge graph embedding methods on all metrics for fb15k-237, on
Hit@1 for wn18rr and on MRR and hit@10 for jf17k (a knowledge hypergraph
dataset).

##  1 Introduction

Transfer learning has recently emerged as a powerful technique in several
domains Ramachandran et al. (2017); Pennington et al. (2014); Tomar et al.
(2017); Subramanian et al. (2018); Oquab et al. (2014); Yosinski et al.
(2014); Trinh et al. (2019), in which a model is first pretrained on relevant
tasks before being fine-tuned on a downstream task. In most modern vision and
NLP applications, such pretraining is often based on the versatile Transformer
model Vaswani et al. (2017) using self-supervised learning on unlabeled data
Devlin et al. (2019); Raffel et al. (2020); Bao et al. (2022). Transformer-
based and self-supervised pretraining have also been applied in the graph
representation learning scenarios. These studies, however, focus mostly on
non-relational graphs Thakoor et al. (2022); Fatemi et al. (2021) or small
molecular graphs Ying et al. (2021), leaving pretraining approaches for
Knowledge Graphs (KG) relatively unexplored.

Many Graph Neural Network (GNN) techniques use positional embeddings of the
entities in the relation along with entity/relation embeddings to represent
tuples You et al. (2019). Transformer Language Models Vaswani et al. (2017)
use positional embeddings in a similar way. Transformers can perform
_contextualized_ link prediction by masking out one of the entities in an
input tuple. In an arity-2 graph, a KG can be viewed as a set of triplets
$\mathcal{G}=\\{(h_{i},r_{i},t_{i})\\}$, $i\in\\{1,\dotsc,T\\}$, where $h_{i}$
is the head entity, $t_{i}$ is the tail entity, and $r$ is the relation
between $h_{i}$ and $t_{i}$. We can feed the sequence $h_{i}$, $r_{i}$, [mask]
or [mask], $r_{i}$, $t_{i}$, where [mask] represents the entity to predict.
Since Transformers can process sequences of arbitrary length, we can go beyond
triple-based KGs, and consider the more general case of _Knowledge
Hypergraphs_ (KHGs, Fatemi et al. (2020)), where edges are composed of tuples
with an arbitrary number of entities. With the versatility of our base model,
we can also perform other tasks, such as autoregressive path generation or
query conditioned neighborhood prediction, that GNNs or KG embeddings cannot.

In this paper, we study the effectiveness of various graph-based pretraining
schemes that _do not use external data_ applied to a Transformer model,
trained either separately or jointly, and evaluated on the downstream task of
KG completion (link prediction). Not using external data is instrumental in
situations where data is scarce, confidential, or classified, for example,
medical records, crime networks, or terrorist activity. Our techniques can be
applied to a generalization of KGs, i.e. KHGs. Our contributions are as
follows: (i) formalizing and introducing five pretraining objectives suitable
for sequence-based KG Transformers; (ii) introducing a new path-finding
algorithm that is guided by information gain; (iii) evaluating the effect of
pretraining strategies on triple-based KGs as well as on KHGs.

##  2 Related Works

Our method is a _Contextualized KG Embedding_ method Wang et al. (2019) that
uses _Self-Supervised Graph Representation_ pretraining.

#### Knowledge Graph Embedding

methods encode distributed representations of entities and relations using a
continuous vector. The representation is learned via a scoring function that
measures the plausibility that a given triple $\\{(h_{i},r_{i},t_{i})\\}$
exists. Such embeddings maximize the plausibility of observed triples, which
may not be predictive enough for downstream tasks Wang et al. (2015); Wei et
al. (2015). To make the embeddings more transferable, researchers have
incorporated other types of information. For example, _external_ data sources
can complement a KG embedding by also encoding entity types Guo et al. (2015);
Xie et al. (2016b) or textual descriptions Xie et al. (2016a); Wang and Li
(2016). As explored here, we can also enhance KG embedding by _only_ using
structural features from the graph. For example, relational paths and multi-
hop relationships between entities (see section A for more details), are a
type of structural information that has proven useful Lin et al. (2015);
Toutanova et al. (2016); Das et al. (2017) for KG completion.

#### Self-Supervised Graph Representation

learning techniques broadly fall into three categories and include methods
that: (1) use random walk procedures to encode diverse neighborhoods Perozzi
et al. (2014); Grover and Leskovec (2016); Hamilton et al. (2017a); (2)
reconstruct a graph’s adjacency matrix Kipf and Welling (2016); Hasanzadeh et
al. (2019); Fatemi et al. (2021); (3) maximize mutual information between
local node representations and global graph representations Veličković et al.
(2019); You et al. (2020); Liang et al. (2021). For applications in chemistry
and biology, most pre-training techniques also use additional datasets and
signals like MAE loss Hu et al. (2020a); Ying et al. (2021), however, our
technique does not use _external_ data. With the exception of Deep Relational
Graph Infomax (drgi) Liang et al. (2021), most self-supervised pretraining
schemes based on graph structure have never been applied to solve large KG
completion tasks. Other techniques such as kg-bert Yao et al. (2019) use a
bert-based Devlin et al. (2019) pretrained language models with entity
descriptions for KG link prediction. Again, such work relies on external data
and additional information (descriptions) while our pretraining signals are
self-contained.

##  3 Methodology

In this section, we provide an overview of the five different graph algorithms
that we use to create pretraining tasks for our experiments: (1) Relational
Shortest Path sequence generation (sp); (2) Information gain Path sequence
generation (ip); (3) K-Hop Neighbor prediction (khn); (4) Invariant Adjacency
matrix classification (iva); (5) Local Clustering Coefficient estimation
(lcc). Please see Appendix A for more details on definitions and notation used
in this section. When jointly trained (all), we use all pretraining tasks and
prepend a task token to delineate each task. Note that for all, the total is
the unweighted sum of each pretraining task’s loss. An overview of the task
objectives is outlined in Table 1.

Task | Input | Target | Objective | Type
---|---|---|---|---
Path-based
sp | $\\{e_{i},r,e_{j}\\}$ | $\\{P_{rel}\\}_{r}$ | $\prod_{k}^{n}f(p_{k}|p_{<k},e_{i},r,e_{j})$ | SG
ip
Neighborhood Based
khn | $\\{e_{i},E_{\mathcal{N}_{k-1}}\\}$ | $\mathbf{O}_{\mathcal{N}_{k}}$ | $\text{KL}(\mathbf{O}_{\mathcal{N}_{k}}||f(e_{i},E_{\mathcal{N}_{k-1}}))$ | MP
iva | $\\{\mathbf{A},\widetilde{\mathbf{A}}|\mathbf{A}^{\prime}\\}$ | $\\{1|0\\}$ | $y_{i}log(f(\mathbf{A},\widetilde{\mathbf{A}}|\mathbf{A}^{\prime}))$ | BC
lcc | $\\{e_{i},E_{\mathcal{N}_{k-1}}\\}$ | $c_{e_{i}}$ | $(f(e_{i})-c_{e_{i}})_{k}^{2}$ | R
Table 1:  Overview of Graph Algorithm Pretraining Tasks. Rel=Uses Relations;
SG=Sequence Generation; BC=Binary Classification; MP=Multi-label Prediction;
R=Regression.

###  3.1 Relational Shortest Paths (sp)

The set of relational paths $\\{P_{rel}\\}_{r}$ as defined in Section A may
yield an exponential number of choices. In practice, we would like to limit
the number of such paths and hence, we need a way to select a subset of
$\\{P_{rel}\\}_{r}$. There are a few possible heuristics for selecting a
subset of such paths. For example, we can find shortest paths based on
Dijkstra’s algorithm and then just keep the sequences of relations joining two
entities $\\{e_{i},r,e_{j}\\}$. As specified in Table 1, we condition our path
sequence generation with $\\{e_{i},r,e_{j}\\}$. The next token generated is
the first relation on a sampled path $\\{P_{rel}\\}_{r}$. Our path-based
pretraining algorithms allow the model to explore ${\mathcal{G}}$ beyond the
tuples included in $\mathcal{G^{\prime}}$ since ${\mathcal{G}}$ includes
relational paths between entities that may not appear in any of the train,
validation, or test sets. A masked query “[Spain] [form of government] [mask]”
can have many relational paths linking the [Spain] entity to a target entity
that we want to predict. For example, a relational path may be “[nominated
for], [film/country]” that will link [Spain] to the target entity to be
predicted. When no path exist between $\\{e_{i},e_{j}\\}$, the model target is
the [no_path] token.

###  3.2 Information gain Paths (ip)

Input: Training set $\mathcal{G^{\prime}}$; query relation $r$; $k$ number of
top paths; max relational paths length (max hops) $l$

Output: at most $n=k^{l}$ relational paths $\\{P_{rel}\\}_{r}$;

1

$R^{\prime}\leftarrow\text{FindIncidentRelations}(r,\emptyset)$ // Algorithm 2

$R^{\prime}\leftarrow\text{TopEntropy}(k,R^{\prime})$ // Algorithm 3

2 $\\{P_{rel}\\}_{r}\leftarrow R^{\prime}$

3 for  _$i\in\\{1,\dotsc,l\\}$_ do

4 for  _$r^{\prime}\in R^{\prime}$ ; $p\in\\{P_{rel}\\}_{r}$_ do

5 $R^{\prime\prime}\leftarrow\text{FindIncidentRelations}(r^{\prime})$

$R^{\prime\prime}\leftarrow\text{BottomCondEntropy}(k,R^{\prime\prime},r^{\prime})$
// Algorithm 3

6

7 for  _$r^{\prime\prime}\in R^{\prime\prime}$_ do

8 if  _$r^{\prime}$ is the last element in $p$_ then

9 $\\{P_{rel}\\}_{r}\leftarrow\\{p\\}\ \cup\\{r^{\prime\prime}\\}$

10 end if

11

12 end for

13

14 end for

15 $R^{\prime}\leftarrow R^{\prime\prime}$

16 end for

Return: $\\{P_{rel}\\}_{r}$

Algo 1 Top_k Path Information Gain

While sp provides a measure of subgraph distance and connectivity, i.e. the
number of elements between entities $\\{e_{i},e_{j}\\}$ in the graph, it may
often yield reasoning chains having little semantic overlap with the original
query. For example, a masked query “[Spain] [form of government] [mask]”
produces many sp such as “[nominated for], [film/country]”, which is a
sequence of relations unrelated to “[form of government]”. Further, sp
algorithms select paths that minimize the distance (number of hops) between
two entities. It is therefore formed from shortest paths of the same length,
which may limit subgraph exploration. We propose _Information Gain Paths_ in
Algorithm 1. ip has the same initial condition $\\{e_{i},r,e_{j}\\}$ to start
generating the paths and has the same learning objective as sp. ip also helps
us reduce the number of paths by selecting the ones that constitute a
beneficial context for answering the query based on a measure of information
gain. Given a query tuple $Q_{r}$ (having relation $r$), the algorithm
progressively builds relational paths $\\{P_{rel}\\}_{r}$ starting from a
relation $r$ such that at each step, it selects the (at most) $k$ _incident_
relations that would yield the $k$ highest information gain for the paths
constructed so far.

With infinite $k$, the algorithm is identical to the breadth-first search as
no relations are ignored. Algorithm 1 bears some similarities with beam search
Goldberg and Reddy (1977). There are however a few differences: (1) for $l$
hops and $k$ beam size, we obtain a maximum of $k^{l}$ paths; (2) a relation
already in $P_{rel}$ cannot be used to form the next hop; (3) the paths are
formed via a back-chaining process starting from the last relation on path
$P_{rel}$ that connects $e_{i}$ and $e_{j}$; (4) an extra step is performed to
select paths in $\\{P_{rel}\\}_{r}$ linking entities in $Q_{r}$.

We let $\text{IG}({P_{rel}})$ denote the _Information Gain_ of relational path
$P_{rel}:r_{1},\dotsc,r_{l}\in{\mathcal{G}}^{\prime}$ having length $l$ as
$\text{IG}({P_{rel}})=H(r_{l})-\sum\limits_{i=1}^{l-1}H(r_{l-i}|r_{l-i+1})$.
We define the _Information Entropy_ $H(r_{l})$ for a relation $r_{l}$, and
entities $E(r_{l})$ that appear in a tuple in ${\mathcal{G}}^{\prime}$ having
relation $r_{l}$ as:

$\scriptstyle
H(r_{l})=-\left(\frac{|E(r_{l})|}{|{\mathcal{E}}|}\log(\frac{|E(r_{l})|}{|{\mathcal{E}}|})+\frac{|{\mathcal{E}}\setminus
E(r_{l})|}{|{\mathcal{E}}|}\log(\frac{|{\mathcal{E}}\setminus
E(r_{l})|}{|{\mathcal{E}}|})\right),$ (1)

where ${\mathcal{E}}$ is all entities in ${\mathcal{G}}^{\prime}$.

The _Conditional Information Entropy_ $H(r_{i-1}|r_{i})$ of two consecutive
relations is defined as:

$\displaystyle H(r_{i-1}|r_{i})=$
$\displaystyle-\frac{|E(r_{i-1})|}{|{\mathcal{E}}|}\biggl{(}\mathrm{U}\log(\mathrm{U})+\mathrm{V}\log(\mathrm{V})\biggr{)}$
(2)

where $\mathrm{U}=\frac{|E(r_{i-1})\cup E(r_{i})\setminus E(r_{i-1})\cap
E(r_{i})|}{|E(r_{i-1})|}$ and $\mathrm{V}=\frac{|E(r_{i-1})\cap
E(r_{i})|}{|E(r_{i-1})|}$.

| Pre | fb15k-237 | wn18rr | jf17k
---|---|---|---|---
Trained | MRR | H@1 | H@3 | H@10 | MRR | H@1 | H@3 | H@10 | MRR | H@1 | H@3 | H@10
Contextualized KG Embedding Results
kg-trsf 1 | $\times$ | 0.364 | 0.272 | 0.400 | 0.549 | 0.484 | 0.450 | 0.496 | 0.553 | 0.532 | 0.445 | 0.561 | 0.687
kg-trsf sp | ✓ | 0.363 | 0.272 | 0.403 | 0.559 | 0.473 | 0.438 | 0.488 | 0.549 | 0.537 | 0.450 | 0.569 | 0.703
kg-trsf ip | ✓ | 0.370 | 0.273 | 0.408 | 0.571 | 0.489 | 0.454 | 0.505 | 0.573 | 0.549 | 0.457 | 0.582 | 0.723
kg-trsf khn | ✓ | 0.361 | 0.266 | 0.398 | 0.558 | 0.479 | 0.444 | 0.491 | 0.556 | 0.536 | 0.450 | 0.565 | 0.701
kg-trsf lcc | ✓ | 0.350 | 0.258 | 0.386 | 0.535 | 0.477 | 0.442 | 0.488 | 0.544 | 0.532 | 0.445 | 0.561 | 0.679
kg-trsf iva | ✓ | 0.358 | 0.264 | 0.397 | 0.552 | 0.479 | 0.445 | 0.494 | 0.556 | 0.538 | 0.453 | 0.568 | 0.705
kg-trsf all | ✓ | 0.380 | 0.277 | 0.422 | 0.591 | 0.499 | 0.456 | 0.511 | 0.588 | 0.554 | 0.465 | 0.594 | 0.715
SOTA KG Embedding Results
boxe | ✓ | 0.337 | 0.238 | 0.374 | 0.538 | 0.451 | 0.400 | 0.472 | 0.541 | 0.553 | 0.467 | 0.596 | 0.711
drgi | ✓ | 0.362 | 0.270 | 0.399 | 0.549 | 0.479 | 0.445 | 0.496 | 0.543 | — | — | — | —
star* | ✓ | 0.358 | 0.205 | 0.322 | 0.482 | 0.401 | 0.243 | 0.491 | 0.675 | — | — | — | —
lp-bert* | ✓ | 0.310 | 0.223 | 0.336 | 0.490 | 0.482 | 0.343 | 0.563 | 0.752 | — | — | — | —
Table 2:  Link prediction test results . H<EMAIL_ADDRESS>Results from: 1Wang et al.
(2019). In the table, bold is best and underline is second best. _*Uses
external data and is a contextualized KG embedding method_.

###  3.3 k-Hop Neighborhood prediction (khn)

Our next graph algorithm to create our pretraining data is based on the
observation that the representation of nodes in a neighborhood
$\mathcal{N}_{k}(e_{i})$ encodes subgraph structure and provides a more
powerful representation of $e_{i}$ Hamilton et al. (2017b). To coerce a
Transformer model to understand local structure up to $k$ hops, we ask the
model to generate $\mathbf{O}_{\mathcal{N}_{k}}$ of the next hop from Eq. 5,
given $E_{\mathcal{N}_{k-1}}$ and $e_{i}$. We input an arbitrarily ordered set
of entities $E_{\mathcal{N}_{k-1}}$ to condition our prediction. Note that
typically entities are ordered according to their appearance in the original
datasets. The loss to learn output entity occurrence probability of entities
for hops up to $k=3$ is
$L_{\textsl{{khn}}}=\sum^{k}_{i=1}\big{(}\text{KL}(\mathbf{O}_{\mathcal{N}_{k}}||f(e_{i},E_{\mathcal{N}_{k-1}}))\big{)}$,
where $E_{\mathcal{N}_{0}}=\\{\\}$ and KL is the Kullback–Leibler divergence
Kullback and Leibler (1951). Please note that we directly chose $k=3$ since it
typically performs better than $k=2$ in a variety of settings Nikolentzos et
al. (2020).

###  3.4 Invariant Adjacency matrix (iva)

One of the key issues with previous heuristics in Section 3.3 is that a
certain order is assumed. However, graph representations are more powerful
when they are order invariant Khasanova and Frossard (2017). To allow order
invariance, we propose a binary classification task based on adjacency
matrices $E_{\mathcal{N}_{k}}$ of the local neighborhood subgraph. The label
is $1$ if the Transformer is given inputs
$\\{\mathbf{A},\widetilde{\mathbf{A}}\\}$, where $\widetilde{\mathbf{A}}$ is
the equivalent adjacency matrix to $\mathbf{A}$ where the entity order is
randomly permuted. The label is $0$ if the Transformer is given inputs
$\\{\mathbf{A},\mathbf{A}^{\prime}\\}$, where $\mathbf{A}^{\prime}$ is a
corrupted adjacency matrix. We corrupt the adjacency matrix by randomly
swapping the columns or by randomly assigning different adjacency matrix
values. Since $\mathbf{A}$ is a symmetric matrix, the input to the Transformer
is a flattened sequence of the upper triangular matrix that contains (1)
column entities and (2)
$\text{row}_{1}(\mathbf{A}),\dots,\text{row}_{|E_{\mathcal{N}_{k}}|}(\mathbf{A})$.
For example, $\mathbf{A}=\begin{bmatrix}1&2&1\\\ 2&0&3\\\ 1&3&1\\\
\end{bmatrix}$ with columns $[e_{1},e_{2},e_{3}]$, will produce the flattened
sequence $[e_{1},e_{2},e_{3},1,2,1,0,3,1]$.

###  3.5 Local Clustering Coefficient (lcc)

For lcc, we use the same inputs as with khn in section 3.3. However, we are
trying to estimate the local clustering coefficient of a subgraph $k$-hops
away from $e_{i}$. The loss for $k=3$ is therefore
$L_{\textsl{{lcc}}}=\sum_{i=1}^{k}(f(e_{i})-c_{e_{i}})_{k}^{2}$.

##  4 Experiments

Experimental set-up and training details are left to Appendix C. We compare
our results to several strong baselines described in Appendix D. Our results
are presented in Table 2. Our baseline model, kg-trsf or “CoKe” Wang et al.
(2019), is a Transformer model that is trained from scratch on link
prediction.

###  4.1 Results and Discussion

Our new algorithm ip provides benefit over kg-trsf across _all_ datasets.
Further, we see that H@10 is typically better for all pretraining signals
compared to kg-trsf. Moreover, all pretraining signals (except lcc) provide
gains on the hypergraph jf17k compared to kg-trsf. We then see that using all
pretraining tasks jointly provides $5$% relative MRR score increase over the
unpretrained baseline, and often surpassing the competitive performances of
state-of-the-art models. The all results are in-line with recent graph
pretraining methods Hu et al. (2020a, b) that show that multitask pretraining
performs better than any individual task. Interestingly, our path-based
algorithm ip provides the largest single task increase in performance over the
unpretrained baseline.

Except for our wn18rr results, path-based pretraining surpasses neighborhood-
based pretraining. Compared to sp, we hypothesize that ip provides a higher
quality pretraining signal for two reasons: (1) paths are more diverse and (2)
paths are semantically closer to a query $Q_{r}$. sp produces paths that have
often redundant first 2-hop relations on a k-hop path $\\{P_{rel}\\}_{r}$.
Further, all the paths have the same minimum length and often transit through
high degree entities. As seen in Section 3.1, high degree entities do not
necessarily provide the most meaningful reasoning chains; though the ip-based
reasoning chains seem more semantically relevant. For example, for a masked
query “Spain”, “form of government” and [mask]”, ip paths are [“military
conflict/combatants”, “international organization/member states”], [“adjoining
relationship/adjoins”, “continents/countries within”] or [“organization
member/member of”, “international organization/member states”].

###  4.2 Analysis and Ablation

In this section, we discuss the choice and combination of tasks. An ablation
study is presented in table 3 that allows us to compare various multitask
combinations. We first notice that given sp, ip or sp \+ ip relational path-
based pretraining schemes, adding any other pretraining signal (khn, lcc or
iva) typically results in an improvement. khn provides the largest MRR
increase when combined with relational path-based signals, with sp \+ ip \+
khn providing a 0.06 improvement in MRR. iva comes in as the second best
signal to combine and lcc only improving MRR slightly.

Pretraining Signal Combination (MRR)
---
sp | 0.363 | ip | 0.370 |  sp \+ ip | 0.372
\+ khn | 0.367 | \+ khn | 0.373 | \+ khn | 0.378
\+ lcc | 0.364 | \+ lcc | 0.370 | \+ lcc | 0.374
\+ iva | 0.366 | \+ iva | 0.373 | \+ iva | 0.375
Table 3:  Task combination ablation study on fb15k-237 using MRR. Adding
individual signals to sp, ip or sp \+ ip pretraining schemes.

##  5 Conclusion

We have investigated the effectiveness of five graph algorithmic pretraining
schemes that do not rely on external data sources. On three downstream KG
completion tasks, we found that kg-trsf: (1) multitask pretraining results in
performance increases, (2) generally, pretrained models exhibit better
improvements in recall (H@10) rather than precision (H@1), and (3) the ip
pretraining tasks work best. Our study has important implications since it
shows that using various graph structural signals that do not rely on external
data can outperform strong baselines pretrained with external data.

## Limitations.

Our technique has a few limitations. First, the pretraining signals that we
used in this paper require a highly adaptive model. Out of the five
pretraining schemes, two include the task of relational path sequence
generation (sp and ip) and another is a local clustering coefficient
regression task (lcc). Such tasks typically cannot be performed by most Graph
Neural Networks or Graph Embedding methods. However, the versatility of our
Transformer-based technique also means that our model can be pretrained on
multiple modalities Mustafa et al. (2022); Lee et al. (2022). For example,
pretraining with text and KGs has already proven very powerful in language
generation tasks Agarwal et al. (2021). The point of our study was to show
that we can surpass other methods such as drgi and lp-bert (see Appendix D for
an overview of baselines) that use pretrained bert models and entity
descriptions. We suspect that enhancing our entity and relation
representations with text will only make the model even stronger at KG
completion. Finally, we notice that jointly training on all pretraining tasks
yields the best results. However, since it is possible that negative task
interference occurs (a negative side effect of multitask learning Zhang et al.
(2020); Pilault et al. (2021), a more throughout study of task combinations
can help unlock even better performances for each specific dataset.

## References

  * Abboud et al. (2020) Ralph Abboud, Ismail Ceylan, Thomas Lukasiewicz, and Tommaso Salvatori. 2020.  Boxe: A box embedding model for knowledge base completion.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 9649–9661. Curran Associates, Inc.
  * Agarwal et al. (2021) Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. 2021.  Knowledge graph based synthetic corpus generation for knowledge-enhanced language model pre-training.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3554–3565, Online. Association for Computational Linguistics.
  * Bao et al. (2022) Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2022.  BEit: BERT pre-training of image transformers.  In _International Conference on Learning Representations_.
  * Bollacker et al. (2008) Kurt D. Bollacker, Colin Evans, Praveen K. Paritosh, Tim Sturge, and Jamie Taylor. 2008.  Freebase: a collaboratively created graph database for structuring human knowledge.  In _SIGMOD Conference_.
  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Durán, Jason Weston, and Oksana Yakhnenko. 2013.  Translating embeddings for modeling multi-relational data.  In _Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2_ , NIPS’13, page 2787–2795, Red Hook, NY, USA. Curran Associates Inc.
  * Das et al. (2017) Rajarshi Das, Arvind Neelakantan, David Belanger, and Andrew McCallum. 2017.  Chains of reasoning over entities, relations, and text using recurrent neural networks.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_ , pages 132–141, Valencia, Spain. Association for Computational Linguistics.
  * Dettmers et al. (2018) Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018.  Convolutional 2d knowledge graph embeddings.  In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_ , pages 1811–1818. AAAI Press.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019.  Unified language model pre-training for natural language understanding and generation.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 13042–13054.
  * Fatemi et al. (2021) Bahare Fatemi, Layla El Asri, and Seyed Mehran Kazemi. 2021.  Slaps: Self-supervision improves structure learning for graph neural networks.  _Advances in Neural Information Processing Systems_ , 34:22667–22681.
  * Fatemi et al. (2020) Bahare Fatemi, Perouz Taslakian, David Vazquez, and David Poole. 2020.  Knowledge hypergraphs: Prediction beyond binary relations.  In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_ , pages 2191–2197. International Joint Conferences on Artificial Intelligence Organization.  Main track.
  * Goldberg and Reddy (1977) H. G. Goldberg and R. Reddy. 1977.  Speech understanding systems. Summary of results of the five-year research effort at Carnegie-Mellon University.  Interim Report Carnegie-Mellon Univ.
  * Grover and Leskovec (2016) Aditya Grover and Jure Leskovec. 2016.  node2vec: Scalable feature learning for networks.  In _Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 855–864.
  * Guo et al. (2015) Shu Guo, Quan Wang, Bin Wang, Lihong Wang, and Li Guo. 2015.  Semantically smooth knowledge graph embedding.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 84–94, Beijing, China. Association for Computational Linguistics.
  * Hamilton et al. (2017a) Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017a.  Inductive representation learning on large graphs.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Hamilton et al. (2017b) William L. Hamilton, Rex Ying, and Jure Leskovec. 2017b.  Representation learning on graphs: Methods and applications.  _IEEE Data Eng. Bull._ , 40(3):52–74.
  * Hasanzadeh et al. (2019) Arman Hasanzadeh, Ehsan Hajiramezanali, Krishna Narayanan, Nick Duffield, Mingyuan Zhou, and Xiaoning Qian. 2019.  Semi-implicit graph variational auto-encoders.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Hu et al. (2020a) Weihua Hu, Bowen Liu*, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. 2020a.  Strategies for pre-training graph neural networks.  In _International Conference on Learning Representations_.
  * Hu et al. (2020b) Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. 2020b.  GPT-GNN: generative pre-training of graph neural networks.  In _KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020_ , pages 1857–1867. ACM.
  * Khasanova and Frossard (2017) Renata Khasanova and Pascal Frossard. 2017.  Graph-based isometry invariant representation learning.  In _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 1847–1856. PMLR.
  * Kipf and Welling (2016) Thomas N. Kipf and Max Welling. 2016.  Variational graph auto-encoders.  _CoRR_ , abs/1611.07308.
  * Kullback and Leibler (1951) S. Kullback and R. A. Leibler. 1951.  On Information and Sufficiency.  _The Annals of Mathematical Statistics_ , 22(1):79 – 86.
  * Lee et al. (2022) Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang, Henryk Michalewski, and Igor Mordatch. 2022.  Multi-game decision transformers.
  * Li et al. (2022) Da Li, Ming Yi, and Yukai He. 2022.  LP-BERT: multi-task pre-training knowledge graph BERT for link prediction.  _CoRR_ , abs/2201.04843.
  * Liang et al. (2021) Shuang Liang, Jie Shao, Dongyang Zhang, Jiasheng Zhang, and Bin Cui. 2021.  Drgi: Deep relational graph infomax for knowledge graph completion.  _IEEE Transactions on Knowledge and Data Engineering_ , pages 1–1.
  * Lin et al. (2015) Yankai Lin, Zhiyuan Liu, Huanbo Luan, Maosong Sun, Siwei Rao, and Song Liu. 2015\.  Modeling relation paths for representation learning of knowledge bases.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 705–714, Lisbon, Portugal. Association for Computational Linguistics.
  * Mohamed et al. (2019) Sameh K. Mohamed, Vít Novácek, Pierre-Yves Vandenbussche, and Emir Muñoz. 2019.  Loss functions in knowledge graph embedding models.  In _DL4KG@ESWC_.
  * Mustafa et al. (2022) Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. 2022.  Multimodal contrastive learning with limoe: the language-image mixture of experts.
  * Nikolentzos et al. (2020) Giannis Nikolentzos, George Dasoulas, and Michalis Vazirgiannis. 2020.  k-hop graph neural networks.  _Neural Networks_ , 130:195–205.
  * Oquab et al. (2014) Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. 2014.  Learning and transferring mid-level image representations using convolutional neural networks.  In _2014 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1717–1724.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Perozzi et al. (2014) Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014.  Deepwalk: Online learning of social representations.  In _Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’14, page 701–710, New York, NY, USA. Association for Computing Machinery.
  * Pilault et al. (2021) Jonathan Pilault, Amine El hattami, and Christopher Pal. 2021.  Conditionally adaptive multi-task learning: Improving transfer learning in NLP using fewer parameters & less data.  In _International Conference on Learning Representations_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Ramachandran et al. (2017) Prajit Ramachandran, Peter Liu, and Quoc Le. 2017.  Unsupervised pretraining for sequence to sequence learning.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 383–391, Copenhagen, Denmark. Association for Computational Linguistics.
  * Subramanian et al. (2018) Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. 2018\.  Learning general purpose distributed sentence representations via large scale multi-task learning.  In _International Conference on Learning Representations_.
  * Thakoor et al. (2022) Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Mehdi Azabou, Eva L Dyer, Remi Munos, Petar Veličković, and Michal Valko. 2022.  Large-scale representation learning on graphs via bootstrapping.  In _International Conference on Learning Representations_.
  * Tomar et al. (2017) Gaurav Singh Tomar, Thyago Duque, Oscar Täckström, Jakob Uszkoreit, and Dipanjan Das. 2017.  Neural paraphrase identification of questions with noisy pretraining.  In _Proceedings of the First Workshop on Subword and Character Level Models in NLP_ , pages 142–147, Copenhagen, Denmark. Association for Computational Linguistics.
  * Toutanova et al. (2016) Kristina Toutanova, Victoria Lin, Wen-tau Yih, Hoifung Poon, and Chris Quirk. 2016\.  Compositional learning of embeddings for relation paths in knowledge base and text.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1434–1444, Berlin, Germany. Association for Computational Linguistics.
  * Trinh et al. (2019) Trieu H. Trinh, Minh-Thang Luong, and Quoc V. Le. 2019.  Selfie: Self-supervised pretraining for image embedding.  _CoRR_ , abs/1906.02940.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 5998–6008.
  * Veličković et al. (2019) Petar Veličković, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. 2019.  Deep graph infomax.  In _International Conference on Learning Representations_.
  * Wang et al. (2021) Bo Wang, Tao Shen, Guodong Long, Tianyi Zhou, Ying Wang, and Yi Chang. 2021.  Structure-augmented text representation learning for efficient knowledge graph completion.  In _Proceedings of the Web Conference 2021_ , WWW ’21, page 1737–1748, New York, NY, USA. Association for Computing Machinery.
  * Wang et al. (2019) Quan Wang, Pingping Huang, Haifeng Wang, Songtai Dai, Wenbin Jiang, Jing Liu, Yajuan Lyu, Yong Zhu, and Hua Wu. 2019.  Coke: Contextualized knowledge graph embedding.  _CoRR_ , abs/1911.02168.
  * Wang et al. (2015) Quan Wang, Bin Wang, and Li Guo. 2015.  Knowledge base completion using embeddings and rules.  In _Proceedings of the 24th International Conference on Artificial Intelligence_ , IJCAI’15, page 1859–1865. AAAI Press.
  * Wang and Li (2016) Zhigang Wang and Juanzi Li. 2016.  Text-enhanced representation learning for knowledge graph.  In _Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence_ , IJCAI’16, page 1293–1299. AAAI Press.
  * Watts and Strogatz (1998) D.J. Watts and S.H. Strogatz. 1998.  Collective dynamics of ’small-world’ networks.  _Nature_ , (393):440–442.
  * Wei et al. (2015) Zhuoyu Wei, Jun Zhao, Kang Liu, Zhenyu Qi, Zhengya Sun, and Guanhua Tian. 2015.  Large-scale knowledge base completion: Inferring via grounding network sampling over selected instances.  In _Proceedings of the 24th ACM International on Conference on Information and Knowledge Management_ , CIKM ’15, page 1331–1340, New York, NY, USA. Association for Computing Machinery.
  * Wen et al. (2016) Jianfeng Wen, Jianxin Li, Yongyi Mao, Shini Chen, and Richong Zhang. 2016.  On the representation and embedding of knowledge bases beyond binary relations.  In _Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016_ , pages 1300–1307. IJCAI/AAAI Press.
  * Xie et al. (2016a) Ruobing Xie, Zhiyuan Liu, Jia Jia, Huanbo Luan, and Maosong Sun. 2016a.  Representation learning of knowledge graphs with entity descriptions.  In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_ , AAAI’16, page 2659–2665. AAAI Press.
  * Xie et al. (2016b) Ruobing Xie, Zhiyuan Liu, and Maosong Sun. 2016b.  Representation learning of knowledge graphs with hierarchical types.  In _Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence_ , IJCAI’16, page 2965–2971. AAAI Press.
  * Yao et al. (2019) Liang Yao, Chengsheng Mao, and Yuan Luo. 2019.  Kg-bert: Bert for knowledge graph completion.  _ArXiv_ , abs/1909.03193.
  * Ying et al. (2021) Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. 2021.  Do transformers really perform badly for graph representation? In _Advances in Neural Information Processing Systems_.
  * Yosinski et al. (2014) Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014.  How transferable are features in deep neural networks?  In _Advances in neural information processing systems_ , pages 3320–3328.
  * You et al. (2019) Jiaxuan You, Rex Ying, and Jure Leskovec. 2019.  Position-aware graph neural networks.  In _International conference on machine learning_ , pages 7134–7143. PMLR.
  * You et al. (2020) Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. 2020.  Graph contrastive learning with augmentations.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 5812–5823. Curran Associates, Inc.
  * Zhang et al. (2020) Wen Zhang, Lingfei Deng, and Dongrui Wu. 2020.  Overcoming negative transfer: A survey.  _CoRR_ , abs/2009.00909.

##  Appendix A Definition and Notation

Without loss of generality, in this section, we formulate our definitions and
notation based on hypergraphs. Given a finite set of entities ${\mathcal{E}}$
and a finite set of relations ${\mathcal{R}}$, a _tuple_ is an ordered set of
the form $r(e_{1},\dots,e_{n})$, where $r\in{\mathcal{R}}$,
$e_{i}\in{\mathcal{E}}$ for all $i=1,\dots,n$ and $|r|=n$ is its _arity_. Let
${\mathcal{G}}$ be the set of ground truth tuples; that is, it specifies all
of the tuples that are true so that if a tuple is not in $\mathcal{G}$, it is
false. A _knowledge hypergraph_ (KHG) consists of a subset of the tuples
$\mathcal{G^{\prime}}\subseteq{\mathcal{G}}$. A knowledge graph is a special
case of a knowledge hypergraph where all relations have arity $2$. We let
$E(r)\subseteq{\mathcal{E}}$ denote the set of all entities that appear in a
tuple in ${\mathcal{G}}^{\prime}$ having relation $r$. That is,
$E(r)=\\{e_{i}|r(\dots,e_{i},\dots)\in{\mathcal{G}}^{\prime}\\}$. Similarly,
we let $R(e_{i})\subseteq{\mathcal{R}}$ denote the set of all relations that
appear in a tuple in ${\mathcal{G}}^{\prime}$ linked to entity $e_{i}$. That
is, $R(e_{i})=\\{r|r(\dots,e_{i},\dots)\in{\mathcal{G}}^{\prime}\\}$. We say
that two tuples are _incident_ in ${\mathcal{G}}^{\prime}$ if they share at
least one entity. Two entities are _connected_ in $\mathcal{G}$’ if they
appear together in a tuple. For example, $r_{1}(e_{1},e_{2},e_{3})$ is
incident to $r_{2}(e_{4},e_{3},e_{5})$ because they share $e_{3}$. Entities
$e_{2}$ and $e_{3}$ are connected because they both appear in the first tuple.
A query for the knowledge completion task is a tuple with one missing entity
that needs to be predicted. We let
$Q_{r}=[r,e_{1},\dots,\mbox{{[mask]}},\dots,e_{n}]_{q}$ denote a query tuple
with relation $r$, where [mask] is the placeholder token (the masked-out
entity) for the entity we want to predict.

A _path_ $P$ in a KHG is a sequence of tuples where two consecutive tuples
share at least one entity. We say that $P$ _connects_ entities $e_{i}$ and
$e_{j}$ if the first tuple in $P$ contains $e_{i}$ and the last tuple contains
$e_{j}$. A _relational path_ $P_{rel}$ between $e_{i}$ and $e_{j}$ is the
sequence of relations along the edges of a path connecting $e_{i}$ and $e_{j}$
such that no relation is repeated along the path (without cycles). For
example, $e_{1}$ and $e_{6}$ are connected through path
$P:r_{1}(e_{1},e_{2},e_{3}),r_{2}(e_{3},e_{4},e_{5}),r_{3}(e_{4},e_{5},e_{6})$,
and have a relational path $P_{rel}:r_{1},r_{2},r_{3}$. Given the set of
entities in queries $\\{Q_{r}\\}_{r}$, we define the set of all possible paths
between pairwise entities as $\\{P_{rel}\\}_{r}$.

The _k-hop neighborhood_ of entity $e_{i}$, denoted $\mathcal{N}_{k}(e_{i})$,
is the unordered set of entities
$E_{\mathcal{N}_{k}}=\\{e_{k},\dots,e_{j}\\}\subseteq{\mathcal{E}}$ enclosed
in a $k$-hop radius around $e_{i}$. We denote the relation-less adjacency
matrix of entities in $\mathcal{N}_{k}(e_{i})$ as
$\mathbf{A}_{E_{\mathcal{N}_{k}}}\in\mathbb{R}^{|E_{\mathcal{N}_{k}}|\times|E_{\mathcal{N}_{k}}|}$.
For a relational graph $\mathcal{G^{\prime}}$, we define a relation-less
adjacency matrix $\mathbf{A}_{E}$ as:

$\displaystyle\mathbf{A}_{E}=\sum_{r\in{\mathcal{R}}}\mathbf{A}_{E(r)},\mathbf{A}_{E}\in\mathbb{R}^{|E_{{\mathcal{E}}}|\times|E_{{\mathcal{E}}}|}.$
(3)

Adjacency $\mathbf{A}$ has a certain order of appearance of entities in
columns and rows which is arbitrarily set by the order in which dataset tuples
are sampled. We let $\widetilde{\mathbf{A}}$ be the equivalent adjacency
matrix with a permuted order of entities in columns and rows (e.g.:
$r_{1},r_{2},r_{3},r_{4}$ can become $r_{2},r_{3},r_{1},r_{4}$). The local
clustering coefficient $c_{e_{i}}$ Watts and Strogatz (1998) of
$\mathcal{N}_{k}(e_{i})$ measures the proportion of closed triangles in the
local k-hop neighborhood of an entity such that:

$\displaystyle
c_{e_{i}}=\frac{|(e_{1},e_{2}\in{\mathcal{E}}:e_{1},e_{2}\in\mathcal{N}_{k}(e_{i})|}{\binom{d_{e_{i}}}{2}},$
(4)

where $d_{e_{i}}=\sum_{e_{j}\in\mathcal{N}_{k}(e_{i})}\mathbf{A}[e_{i},e_{j}]$
is the node degree of $e_{i}$. For the pretraining task in section 3.3, we
define the probability that an entity $e_{i}$ is found in a in $k$-hop
neighborhood (probability of occurrence) $O(e_{i})$ as:

$\displaystyle O(e_{i})=\frac{d_{e_{i}}}{\sum_{e_{j}\in
E_{\mathcal{N}_{k}}}d_{e_{j}}},\text{ and
}\mathbf{O}_{\mathcal{N}_{k}}=\begin{bmatrix}O(e_{1})\\\ \vdots\\\
O(e_{|{\mathcal{E}}|})\end{bmatrix}$ (5)

##  Appendix B More details on ip Algorithms

1 Function  _FindIncidentRelations(_$r$ , $P_{rel}$_)_:

2 $R^{\prime}\leftarrow\emptyset$

3 for  _$r^{\prime}\in{\mathcal{R}}\backslash\\{r\\}$_ do

4 if  _$E(r^{\prime})\cap E(r)\neq\emptyset\textrm{{ and }}r^{\prime}\notin
P_{rel}$_ then

5 $R^{\prime}\leftarrow r^{\prime}$

6

7

8 return $R^{\prime}$

9End Function

Algo 2 Finding all relations incident to $r$ in ${\mathcal{G}}^{\prime}$.

1 Function  _TopEntropy(_$k$ , $R$_)_:

2 Compute $H(r)$ for all the relations $r\in R$ (Eq. (1)) return set ${r\in
R}$ with top-k highest $H(r)$

3End Function

4 Function  _BottomCondEntropy(_$k$ , $R^{\prime}$, $r$_)_:

5 Compute $H(r|r^{\prime})$ for all $r^{\prime}\in R$ (Eq. (2)) return set
${r^{\prime}}\in R^{\prime}$ with k lowest $H(r|r^{\prime})$

6End Function

Algo 3 Finding $r\in R$ with top $k$ entropy $H(r)$ or with with bottom $k$
conditional entropy $H(r|r^{\prime})$.

##  Appendix C More details on Experimental Set-Up

We use fb15k-237 Bollacker et al. (2008) (extracted from Freebase) and wn18rr
Dettmers et al. (2018). We also test our method on a hypergraph link
prediction task based off the jf17k Wen et al. (2016) dataset. Table 4
summarizes dataset properties.

Dataset | $|\mathcal{E}|$ | $|\mathcal{R}|$ | #train | #valid | #test | density
---|---|---|---|---|---|---
wn18rr | 40,943 | 11 | 86,835 | 3,034 | 3,134 | 2.1
fb15k-237 | 14,541 | 237 | 272,115 | 17,535 | 20,466 | 18.7
jf17k | 29,177 | 327 | 61,911 | 15,822 | 24,915 | 35.9
Table 4: Dataset Statistics.

The individual pretraining signals seem to lower precision (H@1) and increase
recall (H@10) compared to kg-trsf from scratch. Further, individual
pretraining signals have a larger positive effect as graph density increases.
Individual pretraining signals typically show MRR improvements for jf17k, the
densest of the KGs.

#### Training and Evaluation

MRR is the Mean Reciprocal Rank and H@(1,3,10) are HIT@ measures all commonly
used in link prediction Mohamed et al. (2019). For all tasks, we use the same
autoregressive Transformer model that applies a transformation $f$ on the
input and that is optimized with different loss functions. Our Transformer
model is a single monolithic architecture that uses a masking scheme similar
to UniLM Dong et al. (2019), allowing the model to play a variety of roles
(encoder-only, decoder-only, or encoder-decoder) and tackle a variety of
objectives (classification, regression or sequence generation). In our
experiments, we have used $L=12$ Transformer layers, with $D=256$ hidden
dimension, $A=12$ self-attention heads, an intermediate layer of size $2D$.
For path algorithms, we truncate our path sequences (max hops) to $l=4$, and,
for neighborhood-based algorithms, we limit $k=3$. The pretraining data
generated from each dataset is applied to separate model instances. For each
dataset, we save the weights of the pretrained model that performs best on the
evaluation set of the link prediction task. During link prediction finetuning,
we use a dropout rate $\rho\in\\{0.2,0.3,0.5\\}$, a label smoothing rate
$\zeta\in\\{0.6,0.7,0.8,0.9\\}$ and a learning rate of $\eta=5^{-4}$. In our
multitask setting (all), each task is sampled from the uniform distribution
$\frac{1}{|D_{t}|}$, for dataset $D_{t}$ of a pretraining task $t$. A batch
may therefore contain multiple tasks. We weight the losses of tasks from
$\tau=\\{\textsl{{sp}},\textsl{{ip}},\textsl{{khn}},\textsl{{lcc}},\textsl{{iva}}\\}$
according the available pretraining dataset size such that
$L_{\textsl{{all}}}=\sum_{t\in\tau}\alpha_{t}L_{t}$, where the
$\alpha_{t}=|D_{t}|/\sum_{t^{\prime}\in\tau}|D_{t}^{\prime}|$. For khn, note
that if $|E_{\mathcal{N}_{k}-1}|>1024$, we clip the token sequence for a
maximum length of 1024. At inference, each entity in $Q_{r}$ is masked and
evaluated once. We use two evaluation metrics: HIT@$n$ and Mean Reciprocal
Rank (MRR).

##  Appendix D More details on Baselines

In table 2, we compare our method against strong and recent KG Embedding
methods. boxe Abboud et al. (2020) is a spatio-translational graph embedding
model that uses logical rules (similar to paths) and that can support
hierarchical inference patterns and higher-arity relations (knowledge
hypergraphs or KHG). This is one of the rare methods that can be applied to
both graphs and hypergraphs. boxe achieves state-of-the-art results on jf17k
while remaining competitive on fb15k-237 and wn18rr. It is unclear however if
boxe is pretrainable111In table 2, we wrote “N/A” since we are not certain if
pretraining is applicable.. To our knowledge, drgi Liang et al. (2021) is the
only other KG completion method that was pretrained using only signals from
the graph structure. Similarly to Deep Graph Infomax Veličković et al. (2019),
drgi is pretrained on artifacts of the graph structure by maximizing the
mutual information between local and global graph representations. We were not
able to ascertain at the moment if drgi is applicable to KHGs. Both lp-bert Li
et al. (2022) and star Wang et al. (2021) are Transformer-based Contextualized
KG Embedding methods. Both techniques are improvements over kg-bert Yao et al.
(2019) that uses textual descriptions of entities and relations on top of a
bert model, which is pretrained on a large text corpus222The text pretraining
data is external data and not self-contained to the KG data. lp-bert also uses
a multitask pretraining step by jointly training on three denoising tasks:
random word masking in the descriptions, entity masking, and relation masking.
star combines kg-bert and TransE Bordes et al. (2013). star contextualizes the
translation function by embedding the head and relation descriptions with
bert. The authors claim that the technique is structure-aware since
translation-based graph embedding approaches conduct structure learning by
measuring spatial distance. Note that KHG tuples in jf17k can have up to 6
entities. In most cases, the description of all entities in a tuple exceeds
the standard bert maximum sequence length of 1024. For this reason, we were
not able to apply lp-bert and star to jf17k.


# Binding energies and thermal width $\Gamma$ of the QGP at finite quark
chemical potential

Siddhartha Solanki, Manohar Lal, and Vineet Kumar Agotiya  Department of
Physics, Central University of Jharkhand, Ranchi, India, 835 222.
Corresponding author id-<EMAIL_ADDRESS>
###### Abstract

The present work have investigated the properties of the heavy quarkonia in
the presence of finite quark chemical potential for different number of
flavors by using the quasi particle approach. The effect of the finite quark
chemical potential has been incorporated through the quasi-particle Debye mass
to examine the binding energies of the quarkonium states. From the imaginary
part of the potential we have calculated the thermal width of the ground state
of the quarkonia and found that the thermal width increases with finite quark
chemical potential. The dissociation temperature $(T_{D})$ of the $J/\psi$ and
$\Upsilon$ have been calculated in the presence of finite quark chemical
potential for different flavors. The effect of the finite quark chemical
potential on the mass spectra of the quarkonium states has been also studied.

qusi-particle Debye mass, Heavy quark complex potential, Number of flavors,
quark chemical potential, thermal width, dissociation temperature.

##  I Introduction

The ongoing heavy ion collision experiment at the relativistic heavy ion
collider (RHIC) accelerator in BNL, USA and the LHC collider at CERN,
Switzerland has their great importance for exploring the phase diagram of the
Quantum Chromo-dynamics (QCD) after the discovery of the Quark Gluon Plasma
(QGP), the fourth state of matter. The study of the fundamental forces between
quark and gluon is essential for the understanding of QCD and it has been
expected that at varying temperature, finite quark chemical potential scales
(low or high) and different phases contribute in the T-$\mu$ plane. For
instance, at small or vanishing temperature quarks and gluons are limited by
the strong force while at high temperature asymptotic freedom suggests a
somewhat different QCD medium which consists of rather weakly coupled
deconfined quarks and gluons, the so-called QGP. The anomalous suppression of
the $J/\psi$ production in heavy-ion collisions which has been observed
experimentally M.C.Abreu  in the depletion of the dilepton multiplicity in the
region of invariant mass which corresponds to the $J/\psi$ meson was proposed
long back as a possibly unmistakable sign of the beginning of deconfinement.
Matsui and Satz T.Matsui  argued that charmonium states generated before the
formation of a thermalized QGP would tend to melt in their way through the
deconfined medium, because the binding coulomb potential is screened by the
large number of color charges, thereby producing an anomalous drop in the
$J/\psi$ yields. The pair develops into the physical resonance during
formation time and passes through the plasma and the hadronic matter before
they leave the interacting system to decay into a dilepton to be detected.

Figure 1: The variation of potential with the distance ’r’ for the quasi-
particle, leading order and lattice parameterized Debye mass at $N_{f}$= $0$,
$2$, $3$ has been shown.

Figure 2: Shows the variation of quasi-particle Debye mass with finite quark
chemical potential at different values of temperature 2(a) and with
temperature at different values of finite quark chemical potential 2(b)
respectively.

Figure 3: Shows the variation of binding energy of $J/\psi$ and $\Upsilon$
with the temperature in figures 3(a) and 3(b) for different flavors at $N_{f}$
= $2$ and $3$.

This long ’trek’ within the interacting system is fairly dangerous for the
$J/\psi$. Even before the resonance occurs, it may be absorbed by the nucleons
streaming past it C.Gerschel . By the time the quarkonium resonance is formed,
the screening of the color forces in the plasma may be sufficient to inhibit a
binding of the charmonium T.Matsui  or an energetic gluon X.M.Xu  or a
comoving hadron could dissociate the resonance. Quarkonia at finite
temperature is an important tool for the studying QGP formation in heavy-ion
collisions. Many effort have been devoted to determine the $T_{D}$ of
quarkonium state in the deconfined medium, using either lattice calculations
of $Q\bar{Q}$ spectral function or non-relativistic calculations based upon
some effective screened potential. Lattice studies are directly based on QCD
and these studies answer most of the questions that arises while studying the
QCD phase diagram. However, in lattice studies the spectral function must be
extracted using rather limited sets of data from the Euclidian (imaginary
time) correlators, which are directly based on the lattice. This along with
the intrinsic technical difficulties of lattice calculations, limit the
reliability of the results obtained so far, and also their scope is
essentially limited to the mass of the ground states in each quarkonium
channel. Potential models, on the other hand, provide a simple and intuitive
framework to study the properties of quarkonium at finite temperature, from
which quantities can be calculated that are beyond the current possibilities
for lattice studies. Umeda and Alberico T.Umeda ; W.M.Alberico(2008)  have
shown that the lattice computations of mesons correlators at finite
temperature contain a constant contribution, due to the presence of zero modes
in the spectral functions. The problem of dissociation of bound states in a
hot QCD medium is of great importance in heavy-ion collisions as it provides
evidence for the creation of the QGP Leitch . The physical understanding of
the quarkonium dissociation within a deconfined medium has undergone several
refinements in the last couple of years Laine:2008cf ; BKP:2000 ; BKP:2001 ;
BKP:2002 ; BKP:2004 . As the heavy quark and anti-quark in a quarkonia state
are bound together by almost static (off-shell) gluons, therefore, the issue
of their dissociation boils down to how the gluon self-energy behaves at high
temperature. It has been noticed that the gluon self-energy has both real and
imaginary parts laine . Note that the real part lead to the Debye screening,
while the imaginary part leads to Landau damping and give rise the thermal
width to the quarkonia.
It indeed provides a useful way to examine quarkonium binding energies,
quarkonium wave functions, reaction rates, transition rates, and decay widths.
It further allows the extrapolation to the region of high temperatures by
expressing screening effects reflecting on the temperature dependence of the
potential and finite quark chemical potential. The effects of dynamics of
quarks on the stability of quarkonia can be studied by using finite quark
chemical potential extracted from thermodynamic quantities that are computed
in full QCD. At high temperatures, the deconfined phase of QCD exhibits
screening of static color-electric fields E.V.Shuryak ; GPY ; it is,
therefore, expected that the screening will lead to the dissociation of
quarkonium states. After the success at zero temperature while predicting
hadronic mass spectra, potential model descriptions have been also applied to
understand quarkonium properties at finite temperature and finite quark
chemical potential. It is well known that the production of $J/\psi$ and
$\Upsilon$ mesons in hadronic reactions occur in part via the production of
higher excited $c\bar{c}$ (or $b\bar{b}$) states and they decay into the
respective ground state. Since the lifetime of different quarkonium state is
much larger than the typical lifetime of the medium produced in nucleus-
nucleus collisions; their decay occurs almost completely outside the produced
medium lain ; he1 . Since the produced medium can be probed not only by the
ground state quarkonium but also by different excited quarkonium states. So,
the potential model in this context could be helpful in predicting for the
binding energies of various quarkonia state by setting up and solving
appropriate Schrodinger equation in the hot QCD medium. The first step towards
this is to model an appropriate medium dependent interquark interaction
potential at finite temperature and finite quark chemical potential.
Thereafter the dissociation of excited states of quarkonia has been studied.
The present work is different from the work U.Kakade  in which the authors
have studied the effect of baryonic chemical potential on the Quarkonium
states using leading order Debye mass. In the current work we have started
with the modified form of the potential V.Agotiya:2009  in which the effect of
the finite quark chemical potential has been incorporated through the quasi-
particle Debye mass. The energy eigen values of the ground states of the
charmonium and bottomonium for different flavors has been obtained by solving
the Schrodinger equation. Further we have studied the effect of the finite
quark chemical potential on the thermal width and the dissociation temperature
of the quarkonium states.
The present manuscript is organized in the following manner: a brief
description about the real and the imaginary part of the heavy quark potential
is given in section-II. Whereas in section-III the quark finite quark chemical
potential is introduced into the the quasi-particle Debye mass. In Section-IV,
we examine the binding energy and $T_{D}$ of various quarkonium states at
different values of flavor and the finite quark chemical potential. The effect
of the finite quark chemical potential on the mass spectra of the quarkonium
states is evaluated in the section-V. In section-VI we discuss the result of
present work. Finally, in section-VII, we conclude with the future prospects
of the present work.

Figure 4: Shows the variation of binding energy of $J/\psi$ and$\Upsilon$ with
the temperature in the figures 4(a) and 4(b) for different values of the
finite quark chemical potential at $N_{f}=3$.

Figure 5: Shows the variation of Binding energy of $\psi^{\prime}$ and
$\Upsilon^{\prime}$ with the temperature in the figures 5(a) and 5(b) for
different values of the finite quark chemical potential at $N_{f}$= $3$.

Figure 6: Mass spectra of $J/\psi$ and $\Upsilon$ with the temperature in the
figures 6(a) and 6(b) for different masses has been shown.

##  II Heavy quark complex potential

###  II.1 Real Part of the potential

Quantitative understanding of the bound state properties needs the exact
potential at finite-temperature which should be derived directly from QCD,
like the Cornell potential at zero temperature has been derived from pNRQCD
from the zeroth-order matching coefficient. Such derivations at finite
temperature for weakly-coupled plasma have been recently come up in the
literature Brambilla05 ; Brambilla08  but they are plagued by the existence of
temperature-driven hard as well as soft scales, $T$, $gT$, $g^{2}T$,
respectively. Due to these difficulties in finite temperature extension in
effective field theories, the lattice-based potentials become popular.
However, neither the free energy nor the internal energy can be directly used
as the potential. What kind of screened potentials should be used in the
Schrödinger equation which well describes is the bound states at finite
temperature is still an open question. The potential model based phenomenology
as well as the lattice QCD approaches infers that the quark-antiquark
interaction potential in presence of medium plays a crucial role in
understanding the nature of quark-antiquark bound state in the hot QCD/QGP
medium. The potential employed is commonly screening columns (Yukawa form)
Brambilla05 ; L.Kluberg . In case of finite temperature QCD we employ the
ansatz that the medium modification enter in the Fourier transform of heavy
quark potential V(k) as V.Agotiya:2009 .

$\tilde{V(k)}=\frac{V(k)}{\varepsilon(k)}$ (1)

Where, $\varepsilon(k)$ is dielectric permitivity which is obtain from the
static limit of the longitudnal part of the gluon self energy R.A.Schneider ;
H.A.Weldon .

$\varepsilon(k)=\left(1+\frac{\pi_{L}(0,k,T)}{k^{2}}\right)\equiv\left(1+\frac{m^{2}_{D}(T,\mu)}{k^{2}}\right)$
(2)

V(k) is the Fourier transform of the Cornell potential given as,

$\mbox{\boldmath$V$}(k)=-\sqrt{\frac{2}{\pi}}\frac{\alpha}{k^{2}}\\\
-\frac{4\sigma}{\sqrt{2\pi}k^{4}}$ (3)

Substituting the value of Eq.(2) and Eq.(3) in equation Eq.(1), and solving
using inverse Fourier transform, we get the medium modified potential
depending upon ’r’ V.Chandra:2007 ; R.A.Schneider ; A.Ranjan .

$\mbox{\boldmath$V$}(r,T,\mu)=\left(\frac{2\sigma}{m^{2}_{D}(T,\mu)}-\alpha\right)\frac{exp(-m_{D}(T,\mu)r)}{r}\\\
-\frac{2\sigma}{m^{2}_{D}(T,\mu)r}+\frac{2\sigma}{m_{D}(T,\mu)}-\alpha
m_{D}(T,\mu)$ (4)

In addition to standard Yukawa potential, this potential
$\mbox{\boldmath$V$}(r,T,\mu)$ given by Eq.(4) has also long-range coulomb
tail. The constant added to yield could appear naturally while performing the
basic computation of the real time static potential in the hot QCD, these
constant are required to yield the correct limit of
$\mbox{\boldmath$V$}(r,T,\mu)$ M.Laine  as T$\longrightarrow$0 and can also
found from the real and imaginary time correlators in thermal QCD medium
A.Beraudo . The potential in the Eq.(4) will look like, in the limiting case
$r\gg\frac{1}{m_{D}}$ as,

$V(r,T,\mu)\approx-\frac{2\sigma}{m^{2}_{D}(T,\mu)r}-\alpha m_{D}(T,\mu)$ (5)

Figure 7: Shows variation of the $J/\psi$ and $\Upsilon$ thermal width
$\Gamma$ with the temperature in the figures 7(a) and 7(b) for different quark
finite quark chemical potential.

Figure 8: Shows the real and imaginary binding energies of the $J/\psi$ and
$\Upsilon$ for different finite quark chemical potential at $N_{f}$= $0$ in
the figures 8(a) and 8(b).

###  II.2 Imaginary Part of the potential

Heavy quark complex potential contains both the real and the imaginary part
which were obtained by using the gluon self energy which in turn responsible
for both the Debye screening and the Landau damping respectively. However,
Debye screening is obtained by using both the retard and self-energy
propagator whereas the static limit of the symmetric self-energy has been used
for calculating the imaginary part of the potential. The imaginary part of the
potential has its importance while studying the threshold enhancement of the
bound state or the thermal width of the charmonium and bottomonium resonances.
This thermal width in spectral function is further used to determine the
$T_{D}$ of the quarkonium resonances. From the previous studies AgnesMocsy ;
M.LaineJHEP2007  is it clear that, the dissociation of the quarkonium states
takes place when thermal width becomes equal to the twice of the binding
energy.

Figure 9: Shows the real and imaginary binding energies of the $J/\psi$ and
$\Upsilon$ for different finite quark chemical potential at $N_{f}$= $2$ in
the figures 9(a) and 9(b).

Figure 10: Shows the real and imaginary binding energies of the $J/\psi$ and
$\Upsilon$ for different finite quark chemical potential at $N_{f}$= $3$ in
the figures 10(a) and 10(b)

The symmetric propagator for the imaginary part is given below:

$ImD_{F}^{00}(0,p)=\frac{-2\pi{T}{m_{D}^{2}}}{p\left(p^{2}+m_{D}^{2}\right)^{2}}$
(6)

Eq.6 gives the imaginary part of the dielectric function as;

$\epsilon^{-1}(p)=-\pi{T}{m_{D}^{2}}\
\frac{p^{2}}{p\left(p^{2}+m_{D}^{2}\right)^{2}}$ (7)

Thus, we obtained the imaginary part of the potential using the following
equation:

$V(r,T,\mu)=\int{\frac{d^{3}k}{\left(2\pi\right)^{\frac{3}{2}}}\left(e^{i.pr}-1\right)\frac{V(p)}{\epsilon(p)}}$
(8)

This implies

$ImV\left(r,T,\mu\right)=\int{\frac{d^{3}k}{\left(2\pi\right)^{\frac{3}{2}}}\left(e^{i.pr}-1\right)\left(-\sqrt{\frac{2}{\pi}}\frac{\alpha}{p^{2}}-\frac{4\sigma}{\sqrt{\pi}{p^{4}}}\right)}\\\
\times
p^{2}\left({\frac{-\pi^{2}{T}{m_{D}^{2}}}{p\left(p^{2}+m_{D}^{2}\right)^{2}}}\right)$
(9)

After performing the integration of the above Eq.9, the contribution due to
coulomb and the string term becomes:

$ImV\left(r,T,\mu\right)=-2{\alpha}{T}\int_{0}^{\infty}\\\
{\frac{dz}{{(z^{2}+1)}^{2}}\left(1-\frac{\sin{z}}{z\hat{r}}\right)+\frac{4\sigma{T}}{m_{D}^{2}}\int_{0}^{\infty}{\frac{dz}{{(z^{2}+1)}^{2}}\left(1-\frac{\sin{z\hat{r}}}{z\hat{r}}\right)}}$
(10)

Where z = $\frac{P}{m_{D}}$. This can be further simplified as:

$ImV\left(r,T,\mu\right)=\\-{\alpha}{T}\phi_{0}\left(\hat{r}\right)+\frac{2\sigma{T}}{m_{D}^{2}}\psi_{0}\left(\hat{r}\right)$
(11)

Where
$\phi_{0}\left(\hat{r}\right)=-{\alpha}{T}\left(\frac{{\hat{r}}^{2}}{9}\left(-4+3\gamma_{E}+3log(\hat{r})\right)\right)$
and
$\psi_{0}\left(\hat{r}\right)=-\frac{{\hat{r}}^{2}}{6}\
+\left(\frac{-107+60\gamma_{E}+60log(\hat{r})}{3600}\right){\hat{r}}^{4}+O({\hat{r}}^{5})$
For the limit $\hat{r}<<1$, we have

$ImV\left(r,T,\mu\right)=-T\left(\frac{\alpha{\hat{r}}^{2}}{3}+\frac{\sigma{\hat{r}}^{4}}{30m_{D}^{2}}\right)log\left(\frac{1}{\hat{r}}\right)$
(12)

Now, by using Eq.12 we calculate the thermal width $\Gamma$ of the resonance
state(1S). This can be done by folding with unperturbed (1S) Coulomb wave
function and is given by prd2016_vin ,

$\Gamma=\left(\frac{4T}{\alpha{m_{Q}^{2}}}+\frac{12\sigma{T}}{\alpha^{2}m_{Q}^{4}}\right)m_{D}^{2}log\frac{\alpha{m_{Q}}}{2m_{D}}$
(13)

##  III Quasi-particle Debye mass in the presence of finite quark chemical
potential

The perturbative nature of the leading order Debye mass in QCD coupling at
high temperature is known for a long time A.Rebhan . In QCD the Debye mass is
non-perturbative and gauge independent defined in E.Braaten . Debye mass is
also calculated for the two polyakov loops by Braaten and Nicto at high
temperature Y.Burnier . The basic definition of the Debye mass itself creates
a difficulty because of the gauge variant nature of electric correlators in
K.Kajantie . To overcome this problem many approaches have been purposed so
far K.Kajantie ; Anbazavov ; S.Nadkarni . Keeping in view of all the
interaction between the quasiparticle because of the quasi-parton, a number of
attempts have been made so far such as, effective mass model V.Goloviznin ;
A.Peshier , the effective mass with Polyakov loop M.D.Elia , model based on
PNJL and NJL A.Dumitru , effective fugacity model V.Chandra:2007 ;
V.Chandra:2009 . Quasi-particle model is important to describe the non-ideal
behavior of the QGP and the masses arise because of the surrounding matter in
the medium around the parton and this quasi-parton acquired the same quantum
number as the real particle i.e quarks and gluon P.K.Srivastava . Here we use
quasi-particle EoS M.Cheng . The quasi-particle Debye mass ${m_{D}}$ in terms
of the temperature and finite quark chemical potential for full QCD case is
given as,

Table 1: The dissociation temperature of the ${J/\psi}$ and ${\Upsilon}$ at $N_{f}=0$ has been calculated for the different values of finite quark chemical potential when the thermal width $\Gamma=2B.E$. $State$ | $\mu=300MeV$ | $\mu=325MeV$ | $\mu=350MeV$
---|---|---|---
$J/\psi$ | 0.8628 | 0.8571 | 0.8234
$\Upsilon$ | 0.9307 | 0.9209 | 0.9124
Table 2: The dissociation temperature of the ${J/\psi}$ and ${\Upsilon}$ at $N_{f}=2$ has been calculated for the different values of finite quark chemical potential when the thermal width $\Gamma=2B.E$. $State$ | $\mu=300MeV$ | $\mu=325MeV$ | $\mu=350MeV$
---|---|---|---
$J/\psi$ | 0.8652 | 0.8596 | 0.8546
$\Upsilon$ | 0.9354 | 0.9299 | 0.9248

$\displaystyle m^{2}_{D}\left(T,\mu\right)$ $\displaystyle=$ $\displaystyle
g^{2}(T)T^{2}\bigg{[}\bigg{(}\frac{N_{c}}{3}\times\frac{6PolyLog[2,z_{g}]}{\pi^{2}}\bigg{)}$
(14)
$\displaystyle+{\bigg{(}\frac{\hat{N_{f}}}{6}\times\frac{-12PolyLog[2,-z_{q}]}{\pi^{2}}\bigg{]}}$

and the value of $\hat{N_{f}}$ is,

$\displaystyle\hat{N_{f}}$ $\displaystyle=$
$\displaystyle\bigg{(}N_{f}+\frac{3}{\pi^{2}}\bigg{(}\frac{\mu^{2}}{T^{2}}\bigg{)}\bigg{)}$
(15)

and

$\displaystyle\mu$ $\displaystyle=$ $\displaystyle\frac{\mu_{b}}{3}$ (16)

U.Kakade , where $(\mu)$ defined the finite quark chemical potential and
$(\mu_{b})$ is finite baryonic chemical potential. After introducing the value
of $\hat{N_{f}}$ in the Eq.(14), we get the full expression of quasi-particle
debye mass in terms of temperature and finite quark chemical potential as,

$m^{2}_{D}\left(T,\mu\right)=T^{2}\left\\{\bigg{(}\frac{N_{c}}{3}Q^{2}_{g}\bigg{)}+\bigg{[}\frac{N_{f}}{6}+\frac{1}{2\pi^{2}}\bigg{(}\frac{\mu^{2}}{T^{2}}\bigg{)}\bigg{]}Q^{2}_{q}\right\\}$
(17)

Where, $Q_{g}$ and $Q_{q}$ are the effective charges given by the equations:

$z_{g,q}=a_{q,g}\exp\bigg{(}-\frac{b_{g,q}}{x^{2}}-\frac{c_{g,q}}{x^{4}}-\frac{d_{g,q}}{x^{6}}\bigg{)}.$
(18)

Here $x=T/T_{c}$ and $a$, $b$ and $c$ and $d$ are fitting parameters, for
equation of state in the quasi-particle description V.Chandra:2007 ;
V.Chandra:2009  respectively.

$\displaystyle Q^{2}_{g}$ $\displaystyle=$ $\displaystyle
g^{2}(T)\frac{6PolyLog[2,z_{g}]}{\pi^{2}}$ $\displaystyle Q^{2}_{q}$
$\displaystyle=$ $\displaystyle g^{2}(T)\frac{-12PolyLog[2,-z_{q}]}{\pi^{2}}$
(19)

Here, $g(T)$ is the QCD running coupling constant, $N_{c}=3$ ($SU(3)$) and
$N_{f}$ is the number of flavor, the function $PolyLog[2,z]$ having form,
$PolyLog[2,z]=\sum_{k=1}^{\infty}\frac{z^{k}}{k^{2}}$ and $z_{g}$ is the
quasi-gluon effective fugacity and $z_{q}$ is quasi-quark effective fugacity.
These distribution functions are isotropic in nature. Both $z_{g}$ and $z_{q}$
have a very complicated temperature dependence and asymptotically reach to the
ideal value unity. In our present analysis we have used the quasi-particle
Debye mass, $m_{D}^{QP}$ depending upon the temperature and finite quark
chemical potential for different number of flavors.

##  IV Binding energy and dissociation temperature

In this section, we have studied the effect of the finite quark chemical
potential on the binding energies and the dissociation temperature of
c$\bar{c}$ and b$\bar{b}$ resonances with different flavors number. To reach
this end we solve the Schrodinger equation for the complete understanding of
the quarkonia in the hot QGP medium. The binding energy of charmonium and
bottomonium state at $T$=$0$ is defined by the difference of energy between
the $m_{Q}$ (mass of quarkonia) and the bottom/open charm threshold. But
distance between the continuum threshold and the peak position is defined the
binding energy at finite value of temperature AgnesMocsy . Hence, the solution
of Schrodinger equation for the potential given by Eq.(5) gives the energy
eigen values for the ground and excited states of charmonium and bottomonium
i.e, $J/\psi$, $\psi^{\prime}$, $\Upsilon$ and $\Upsilon^{\prime}$ as:

$E_{n}=-\frac{1}{n^{2}}\frac{m_{Q}\sigma^{2}}{m^{4}_{D}}$ (20)

It has been observed that the binding energy decreases with the temperature
and also with finite quark chemical potential which is shown in the figures,
3, 4, 5 and 6. The binding energy of charmonium and bottomonium state at
particular values of temperature becomes smaller or equal to the value of mean
thermal energy; i.e, the state of quarkonia is said to be dissociated at that
given value of temperature. The dissociation temperature for the states of
charmonium and bottomonium is also discussed in P.Sandin ; prd2016_vin ;
prd2018_vin . The dissociation temperature of $J/\psi$ and $\Upsilon$ with
different number of flavors at different values of the finite quark chemical
potential has been obtained by using the condition: $\Gamma$ = 2B.E. The
intersecting point between the binding energy curve and the thermal width of a
quarkonium states ($J/\psi$ and $\Upsilon$) is taken as dissociation point of
such states.

Table 3: The dissociation temperature of the ${J/\psi}$ and ${\Upsilon}$ at $N_{f}=3$ has been calculated for the different values of finite quark chemical potential when the thermal width $\Gamma=2B.E$. $State$ | $\mu=300MeV$ | $\mu=325MeV$ | $\mu=350MeV$
---|---|---|---
$J/\psi$ | 0.8653 | 0.8597 | 0.8547
$\Upsilon$ | 0.9350 | 0.9247 | 0.9258
Table 4: Comparison of the mass spectra for $J/\psi$ and $\Upsilon$ obtained in the present work with the theoretical and experimental data. $State$ | present work | Exp.massTanabashi  | Error [in GeV]
---|---|---|---
$J/\psi$ | 3.1 | 3.096 | 0.00129
$\Upsilon$ | 9.5 | 9.460 | 0.00421

##  V Mass Spectra of Quarkonium state

For calculating the mass spectra of heavy quarkonia the relation is:

$M=2m_{Q}+B.E$ (21)

Here, mass spectra is equal to the sum of the exact formula of energy and
twice the quark-mass. Now, we substitute the values of $E_{nl}^{n}$ (B.E) in
the above equation,

$M=2m_{Q}+\frac{1}{n^{2}}\frac{m_{Q}\sigma^{2}}{m^{4}_{D}}$ (22)

Where, $m_{Q}$ is used for the mass of quarkonium state, such as charmonium
and bottomonium mass and $n$ is used for the states of quarkonium (n = $1$ for
ground states and n = $2$ for excited states).

##  VI Results and Discussion

In the present work we have investigated the propeties of the charmonium and
the bottomonium at finite temperature and quark chemical potential using quasi
particle approach. Figure 1 shows the variation of the potential with distance
(r) for different form of the Debye mass (e.g. Quasi-Particle, Leading order
and Lattice parametrized) at finite quark chemical potential $\mu$ = $300$ MeV
and $T$ = $150$ MeV with different number of flavours. It has been found that
the variation of potential increases with the distance (r). This potential is
not similar to the lattice free energy of heavy-quark in the deconfined phase,
which is a well known coulomb potential H.Satz , the Cornell potential is
solvable by one-dimensional Fourier-transform (FT) in the hot QCD medium and
has a similar form that has been used for the study of quarkonium properties
(which is consider like the color flux tube structure).
The variation of quasi-particle Debye mass with finite quark chemical
potential at different values of temperature (i.e, $150$, $160$ and $170$ MeV)
is shown in figure 2 (a), and the variation of quasi-particle Debye mass with
temperature at different values of finite quark chemical potential (i.e,
$150$, $200$ and $250$ MeV) is shown in the figure 2 (b) respectively. The
Debye screening mass at baryon density and temperature has been studied by the
lattice Taylor expansion method M.Doring . If we increase the value of finite
quark chemical potential, Debye mass increases for different temperatures is
shown in the figure 2 (a). But in the figure 2 (b), Debye mass increases with
the temperature for different quark chemical potential. The variation of the
binding energy for the $J/\psi$ and $\Upsilon$ with the temperature for
different number of the flavors $N_{f}$ = $2$, $3$ has been shown in figure 3
(a) and in figure 3 (b) respectively. When we increase the value of number of
flavors in the figure 3, the variation of binding energy decreases at $\mu$ =
$100$ MeV.
Figures 4 and 5 shows the variation of binding energy of $J/\psi$ and
$\psi^{\prime}$ in the figure 4 (a) and 5 (a), $\Upsilon$ and
$\Upsilon^{\prime}$ in the figure 4 (b) and 5 (b) respectively, with the
temperature at $N_{f}$ = $3$ for different values of the finite quark chemical
potential ($\mu$ = $100$, $110$, $120$ MeV). From these figures, we notice
that the binding energy $J/\psi$, $\psi^{\prime}$, $\Upsilon$ and
$\Upsilon^{\prime}$ decreases with the temperature as we increase the value of
finite quark chemical potential. So, it become an interesting fact about the
rate of exponential decay of binding energy. This behavior of binding energy
could be understood by the strongerness of Debye screening with increasing
values of the finite quark chemical potential $(\mu)$ and also the strength of
inter-quark potential, which is weaker as compared to the zero finite quark
chemical potential $\mu$ = $0$. The binding energy at finite value of
temperature and quark chemical potential gives information about the
dissociation of the quarkonium states (charmonium and bottomonium states).
Since, it is known that the binding energy is directly proportional to the
mass of quarkonia so, if the value of quarkonia mass increases, in other words
we turns towards higher masses i.e. from mass of $J/\psi$ ($1.5$ GeV) to mass
of $\Upsilon$ ($4.5$ GeV), we notice that the binding energy increases. The
mass spectra of the quarkonia states in the presence of finite quark chemical
potential has been also studied and its variation with the temperature has
been shown in the figures 6. It has been observed that the mass spectra of the
$J/\psi$ in the figure 6 (a) and $\Upsilon$ in the figure 6 (b) increases as
the mass of the quarkonia increaes. We have also made a comparison of the
masses of the $J/\psi$ and $\Upsilon$ with the experimental data Tanabashi
and this has been given in the Table-IV. The variation of the thermal width of
$J/\psi$ in the figure 7 (a) and $\Upsilon$ in the figure 7 (b) with the
temperature for different values of the finite quark chemical potential (i.e,
$200$, $250$ and $300$ MeV) has been shown. It has been seen that the thermal
width of the quarkonium states with the finite quark chemical potential
increase. The real and the imaginary part of the binding energies of the
$J/\psi$ in the figures 8 (a), 9 (a) and 10 (a) and $\Upsilon$ in the figures
8 (b), 9 (b) and 10 (b) with the $T/T_{c}$ for $N_{f}$ = $0$, $2$, $3$ at
$\mu$ = $300$, $325$, $350$ MeV has been shown respectively. From these
figures we have examined the dissociation temperature (By the intersection
point of the thermal width and two times of the binding energy of the
quarkonia) of the $J/\psi$ and $\Upsilon$ for different flavors $N_{f}$= $0$,
$2$, $3$ and the obtained values of the dissociation temperatures, given in
the Table-I, II and III respectively. The dissociation temperature of the
$J/\psi$ and $\Upsilon$ at $N_{f}$ = $0$ for the finite quark chemical
potential $\mu$ = $300$, $325$, $350$ MeV are found $0.8628$, $0.8571$,
$0.8234$ and $0.9307$, $0.9209$, $0.9124$ respectively (in terms of $T_{c}$).
Similarly at $N_{f}$ = $2$ the dissociation temperature of the $J/\psi$ and
$\Upsilon$ for different finite quark chemical potential $\mu$ = $300$, $325$,
$350$ MeV are $0.8652$, $0.8596$, $0.8546$ and $0.9354$, $0.9299$, $0.9248$
respectively (in unit of $T_{c}$). Finally, the dissociation point of the
$J/\psi$ and $\Upsilon$ for different values of the finite quark chemical
potential $\mu$ = $300$, $325$, $350$ MeV was found as $0.8653$, $0.85970$,
$8547$ and $0.9350$, $0.9247$, $0.9258$ respectively (in unit of $T_{c}$) when
the number of flavor was taken as $N_{f}$ = $3$.

##  VII Conclusion and future outcomes

The current work have determined the quarkonium dissociation behavior in the
hot and dense QGP medium, in the presence of the finite quark chemical
potential. It has been observed that the Cornell potential with distance ’r’,
increases for the different Debye screening. The behavior of binding energy
with temperature has also seen by the earlier studies prd2016_vin  and with
anisotropic parameter at zero finite quark chemical potential in prd2018_vin .
Here we consider the finite values of the quark chemical potential. It is
noticed that the binding energy for the different quarkonium states decreases
with the temperature as we increase the value of finite quark chemical
potential. The dissociation temperature of the quarkonium states, on the other
hand, increases with the increasing values of the finite quark chemical
potential. This can be seen from the Table-I, II and III for number of flavor
$N_{f}$= $0$, $2$, $3$ respectively. The maximum error in the mass spectra of
the $J/\psi$ and $\Upsilon$ deduced by using Chi square function is $0.0019$
and $0.00421$ (in GeV) respectively.
The present work might be helpful in exploring the studies of the compact
objects like neutron stars. Since the Compressed Baryonic Matter (CBM)
experiment at FAIR is exploring the quark gluon plasma at higher baryon
densities, so such type of theoretical studies may contribute to the physics
of compact bodies with high baryon densities.

##  VIII Acknowledgments

One of the author, V.K.Agotiya acknowledge the Science and Engineering
Research Board (SERB) Project No. EEQ/2018/000181 New Delhi for the research
support in basic sciences.

##  IX Declarations

###  IX.1 Funding

No any other funding.

###  IX.2 Conflict of interest

The author declare no competing interests.

## References

  * (1) M. C. Abreu et al, Physics Letters B 477 no.1-3 pp.28-36 (2000).
  * (2) T. Matsui and H. Satz, Physics Letters B 178 no.4 pp.416-422 (1986).
  * (3) C. Gerschel and J. Hufner, Physics Letters B 207 pp.253 (1988).
  * (4) X. M. Xu, D. Kharzeev, H. Satz and X. N. Wang, Physical Review C 53 no.6 pp.3051 (1996).
  * (5) T. Umeda, Physical Review D 75 pp.094502 (2007).
  * (6) W. M. Alberico, A. Beraudo, A. De. Pace and A. Molinari, Physical Review D 77 pp.017502 (2008).
  * (7) M. J. Leitch [PHENIX Collaboration] arXiv: 0806.1244 [nucl-ex].
  * (8) M. Laine, Nuclear Physics A 820 pp.25C (2009).
  * (9) D. Pal, B. K. Patra and D. K. Srivastava, European Physics Journal C 17, pp.179-186, (2000).
  * (10) Binoy Krishna Patra and D. K. Srivastava, Physics Letters B 505 no.1-4 pp.113-118 (2001).
  * (11) Binoy Krishna Patra, V. J. Menon, Nuclear Physics A 708 pp.353-364 (2002).
  * (12) Binoy Krishna Patra and V. J. Menon, The European Physical Journal C 37 pp.115-121 (2004).
  * (13) Y. Burnier, M. Laine, M. Vepsáláinen, Physics Letters B 678 no.1 pp.86-89 (2009).
  * (14) E. V. Shuryak, Physics Reports 61 no.2 pp.71-158 (1980).
  * (15) D. J. Gross, R. D. Pisarki and L. G. Yaffe, Reviews of Modern Physics 53 pp.43 (1981).
  * (16) M. Laine, Journal of High Energy Physics 04 pp.124 (2011).
  * (17) M. He, R. J. Fries and R. Rapp, Physics Letters B 701 pp.445 (2011).
  * (18) N. Brambilla, A. Pineda, J. Soto, and A. Vairo, Reviews of Modern Physics 77 no.4 pp.1423-1496 (2005).
  * (19) N. Brambilla, J. Ghiglieri, A. Vairo and P. Petreczky, Physical Review D 78 no.01 pp.4017 (2008).
  * (20) L. Kluberg and H. Satz, https://arXiv:hep-ph/0901.3831.
  * (21) V. Agotiya et al, Physical Review C 80 no.2 id-025210 (2009).
  * (22) R. A. Schneider, Physical Review D 66 no.3 id-036003 (2002).
  * (23) H. A. Weldon, Physical Review D 26 no.6 pp.1394-1407 (1982).
  * (24) V. Chandra, R. Kumar and V. Ravishankar, Physical Review C 76 no.6 id-054909 (2007).
  * (25) A. Ranjan and V. Ravishankar, https://arXiv:0707.3697.
  * (26) M. Laine, O. Philipsen, M. Tassler, and P. Romatschke, Journal of High Energy Physics 03 pp.054 (2007).
  * (27) A. Beraudo, J. P. Blaizot and C. Ratti, Nuclear Physics A 806 no.1-4 pp.312-338 (2008).
  * (28) M. Laine, O. Philipsen and M. Tassler, Journal of High Energy Physics 09,pp.066 (2007).
  * (29) A. Rebhan, Physical Review D 48 no.9 pp.R3967 (1993).
  * (30) E. Braaten and A. Nieto, Physical Review Letters 73 no.18 pp.2402 (1994).
  * (31) Y. Burnier and A. Rothkopf, Physics Letters B 753 pp.232-236 (2016).
  * (32) K. Kajantie, M. Laine, J. Peisa, A. Rajantie, K. Rummukainen and M. E. Shaposhnikov, Physical Review Letters 79 no.17 pp.3130 (1997).
  * (33) Anbazavov, F. Kirsch, P. Petrezky and S. Mukherjee, Physical Review D 91 no.5 id-054503 (2015).
  * (34) S. Nadkarni, Physical Review D 33 no.12 pp.3738 (1986).
  * (35) V. Goloviznin and H. Satz, Zeitschrift fur Physik C particlse and fields 57 pp.671-675 (1993).
  * (36) A. Peshier, B. Kampfer, O. P. Pavlenko, and G. Soff, Physical Review D 54 no.3 pp.2399-2402 (1996).
  * (37) M. D’Elia, A. Di. Giacomo, and E. Meggiolaro, Physical Review D 67 no.11 id-114504 (2003).
  * (38) A. Dumitru and R. D. Pisarski, Physics Letters B 525 no.1-2 pp.95-100 (2002).
  * (39) V. Chandra, V. K. Agotiya and B. K. Patra, https://arXiv:0901.2084v1 (2009).
  * (40) P. K. Srivastava, S. K. Tiwari and C. P. Singh, Physical Review D 82 no.01 id-014023 (2010).
  * (41) M. Cheng et al, Physical Review D 77 no.01 id-014511 (2008).
  * (42) U. Kakade and B. K. Patra, Physical Review C 92 no.2 id-024901 (2015).
  * (43) A. Mocsy and P. Petreczky, Physics Review Letters 99 no.21 id-211602 (2007).
  * (44) M. Margotta, K. McCarty, C. McGahan, M. Strick-land and D. Y. Elorriaga, Physical Review D 83 no.10 id-105019 (2011).
  * (45) L. Thakur, U. Kakade and B. K. Patra, Physical Review D 89 id-094020 (2014).
  * (46) P. Sandin, M. Ogren and M. Gulliksson, Physical Review E 93 no.3 id-033301 (2016).
  * (47) V. K. Agotiya, V. Chandra, M. Y. JamaL and I. Nilima, Physical Review D 94 no.9 id-094006 (2016).
  * (48) M Y Jamal, I Nilima, V. Chandra and V. K. Agotiya, https://arXiv:1805.04763 (2018).
  * (49) H. Satz, Reports on Progress in Physics 63 pp-1511 https://arXiv:hep-ph/0007069 (2000).
  * (50) M. Doring, S. Ejiri, O. Kaczmarek, F. Karsch and E. Laermann, Proceedings of Science (XXIIIrd international symposium on lattice field theory) LAT2005 pp.193 (2005).
  * (51) M. Tanabashi, C. Carone, T. G. Trippe and C. G. Wohl, Phys. Rev. D 98 546-548 (2018).