]Institute of Science and Technology Austria, 3400 Klosterneuburg, Austria

# Opponents and proponents of the war in Ukraine in Russian social media: who
are they?

Alesya Sokolova<EMAIL_ADDRESS>[

###### Abstract

Understanding the personality of Russians who support the war in Ukraine is
one of the key steps to understanding how this war became possible. However,
during the war, traditional sociological methods are not always applicable.
Social media provides an alternative source of what is inside people’s heads.
In this paper, I compare the political identities, values, and interests of
social media users in Russia who hold a strong position for or against the war
in Ukraine. I collect data from VK, the most popular Russian social media
platform, and analyze self-filled profile information as well as the groups
that the users subscribed to. I found that proponents of the war tend to have
a weaker political identity (self-identified as "moderate") compared to
opponents, who specify it more precisely (often, but not limited to,
"liberal"). Additionally, the values of the proponents more frequently align
with those promoted by the Russian government, such as orthodoxy and family.
Despite these differences, pro-war and anti-war users share many common
interests, as evidenced by their subscriptions to the same groups focused on
music, history, and sport. When asked to state the most important trait in
people (a field users can fill in VK), the most frequent answer for both
groups is "kindness and honesty". The analysis results, in addition to
contributing to the understanding of public opinion in Russia, can be utilized
for predicting one’s position on the war based on their social media profile.

###### keywords:

War in Ukraine, Russia, Social media, Political position, Identity, Prediction

##  1 Introduction

On February 24, 2022, Russia invaded Ukraine, leading to thousands of deaths
and millions of displaced in Ukraine, tightening repression and thousands of
arrested dissenters in Russia, and economic and social upheaval in the rest of
the world. The question of how many Russians support the invasion and why they
do so became a significant part of public discussion.

The traditional methods to measure public opinion, such as polls, during times
of war and under repressive pressure are skewed by respondents’ fears.
Experiments show that a significant percentage of Russian citizens are likely
to be hiding their true views about the conflict Chapkovski and Schaub (2022).
Additionally, research shows that opponents of the war and the regime were
more likely to be concerned about responding to pollsters and making political
statements than supporters Re: Russia (2023); Chronicles (2022). However,
survey data from three independent projects (Levada Center Levada-center
(2023), the Chronicles project Chronicles (2023), and Russian Field Russian
Field (2023)) indicates similar distributions of responses on crucial issues,
which indicates some relevance of this survey data Re: Russia (2023). Such
data shows that consistent war supporters and opponents make up approximately
only a quarter of the population each, while the position of the remaining
part tends to lean towards the support of the war, but is mixed Chronicles
(2023); Russian Field (2023).

Another way to study public opinion and perception of the war in Ukraine is
through in-depth interviews. In contrast to polls, which can capture a general
trend of the attitude to the war, in-depth interviews allow tracking of
precise narratives and complex perceptions, which often cannot fit into
standardized sets of intelligible positions. Such a method confirms that most
of the supporters of the war do not have a consistent and strong political
view and, moreover, are often depoliticized Ishchenko and Zhuravlev (2022);
Public Sociology Lab (2022, 2023).

Both of these approaches do not usually focus on the values and personalities
of respondents, which may be one of the key steps to understanding how their
position developed. Meanwhile, social media provides us with an alternative
source of data, from which the information on public opinion can be extracted
Murphy et al. (2014); Dong and Lian (2021), and that was extensively used to
study political phenomena Tucker et al. (2018); Bright (2018).

In this paper, to extract the information about personalities of proponents
and opponents of the war, I use data from VK, the largest Russian social media
platform, which is frequently used in studies on political and social
movements in Russia Enikolopov et al. (2020); Bykov and Martyanov (2021);
Philippov (2022); Petrov (2022).

This social media, however, has some peculiarities that affect the contingent
of its users. For the last decade, it is under the specific attention of the
Centre for Combating Extremism (Centre E), a unit within the Russian Ministry
of Internal Affairs heavily criticized for repressing opposition activists
Open Democracy (2012); Wired (2022); Earl et al. (2022). For publications,
reposts, comments, and likes posted on their VK pages, many Russian citizens
were sentenced to fines, suspended sentences, and imprisonment. For this
reason, many opposition users stopped using it years before the full-scale
invasion of Ukraine. However, it still remained a platform for political
communication and protest coordination for some fraction of users Philippov
(2022); Bykov and Martyanov (2021). Then, several days after the start of the
full-scale invasion, Russia accepted new laws designed to criminalize public
anti-war statements McCarthy et al. (2023), pushing many of the remaining
opposition users to leave VK or at least to erase there any signs of the
opposition views. Additionally, Russia blocked other popular social media
platforms – Instagram and Facebook – and announced their parent company Meta
extremist organization Stoner (2022), and many pro-war users moved from there
to VK.

For these reasons, VK is far from being a representative source of the
statistics on the number of supporters and opponents of the war. However, in
my analysis, I do not focus on the comparison of amounts of pro-war and anti-
war activity. Instead, I focus on the views, values, and interests of people.
While it is still possible to find enough active opponents of the war on VK,
the mentioned peculiarities should not affect the reliability of the results
of this study.

For the current research, I collect a dataset of the profiles of the users who
clearly express a position for or against the war in the posts they write on
VK. I analyze their self-filled profile fields such as "political views",
"religion", "personal priority" and "important in others". Additionally, I
analyze the groups they subscribed to, to identify and compare their
interests.

Since I consider only those who are confident in their position enough to
write a post on it, my analysis is more focused on the consistent supporters
and opponents of the war (who, according to the Chronicles project Chronicles
(2023), make up 22% and 20% of the total population correspondingly), than on
those whose position is not so strongly defined.

##  2 Methods

###  2.1 Dataset

For collecting a dataset of the users who support (pro-war class) and do not
support (anti-war class) the war, I used VK API. Firstly, I found pro-war and
anti-war posts. Then, I added the suitable profiles of the authors of the
posts to the dataset.

To find pro-war (anti-war) posts, I used three types of search requests:

  * -

Pro-war (anti-war) hashtags

  * -

Offensive language that supporters (opponents) of the war use with respect to
opponents (supporters)

  * -

Terms to describe the war in Ukraine.

The detailed description of the search requests can be found in Appendix A.
The date of publishing of posts was from Mai 1, 2022, to January 6, 2023.

Then, I collected the profiles of the authors of the pro-war (anti-war) posts.
From this set of profiles, I removed the following:

  * -

Profiles that went to both pro-war and anti-war classes (e.g. if the user uses
both pro-war and anti-war hashtags in their posts)

  * -

Users who have less than one subscription to the groups

  * -

Users whose country of residence (according to the profile) is not Russia, or
who did not set it

  * -

Users who went to anti-war class, but used pro-war hashtags that were not used
for search (for the detailed description, see Appendix A).

The final dataset contains 10551 profiles, 7284 of which used pro-war
vocabulary, and 3267 – anti-war. For each profile in the dataset, the
following information was extracted (the standard VK fields with a choice of
answers):

  * -

Sex (female or male)

  * -

Date of birth

  * -

City

  * -

Political views (apathetic, communist, socialist, moderate, liberal,
conservative, monarchist, ultraconservative, libertarian)

  * -

Religion (Judaism, Orthodoxy, Catholicism, Protestantism, Islam, Buddhism,
Confucianism, Secular Humanism, Pastafarianism, other)

  * -

Personal priority (family and children, career and money, entertainment and
leisure, science and research, improving the world, personal development,
beauty and art, fame and influence)

  * -

Important in others (intellect and creativity, kindness and honesty, health
and beauty, wealth and power, courage and persistence, humor and love for
life).

Also, subscriptions to the groups ("publics", as they are called in VK) were
extracted for each user.

The potential problem with this dataset could be the fake accounts ("trolls"),
created by companies associated with the government to promote its interests
Prier (2017). To check if the accounts I found actually belong to real people,
I compared the distribution of activity counters of the users who were
allocated to the pro-war class with those allocated to the anti-war class and
found that they look natural in both cases (see Appendix B). It makes it
unlikely that a significant fraction of the accouts was created artificially
to promote some position. This result is not surprising: the goal of trolls is
to imitate public activity ("astroturfing"), and for this, writing comments
and putting "likes" is more efficient than writing posts on a personal page.
Additionally, trolls tend to not use public accounts (so their posts and other
personal information are not accessible from search or VK API) Radio Free
Europe – Radio Liberty (2021).

###  2.2 Considering bias in the distributions

To check the accuracy of the classification of posts as pro-war or anti-war
based on keywords, I read 311 posts manually. From those classified as anti-
war (157 posts), 95 (60%) posts actually contained anti-war positions, 43
(27%) contained pro-war positions, and from 19 (12%) I could not determine the
position. From those classified as pro-war (154 posts), 144 (93.5%) posts
contained pro-war positions, 2 (1.3%) – anti-war positions, and from 8 (5.2%)
I could not determine the position.

I assumed that if the position is undefined from the posts, then the user
supports or opposes the war with equal probability. Then, the fraction of
actual opponents of the war in the class of anti-war users is 67%, and the
fraction of proponents of the war in the class of pro-war users is 96%.

To calculate the error of these values, I quadratically summed up half of the
fraction of undefined posts with the statistical error of the binomial
distribution. As a result, the fraction of the anti-war users in the anti-war
class is

$f_{\textrm{anti-war}}=0.67\pm 0.07,$ (1)

and the fraction of pro-war users in the corresponding class is

$f_{\textrm{pro-war}}=0.96\pm 0.03.$ (2)

Knowing these values, the initial distributions can be corrected to remove the
bias caused by inaccurate classification. If $\tilde{p}_{\textrm{anti-
war}}(x)$ and $\tilde{p}_{\textrm{pro-war}}(x)$ are the distributions of some
parameter in the biased dataset (e.g. x = {"political views" = "liberal"}, and
$\tilde{p}_{\textrm{anti-war}}(x)$ is a fraction of the users in anti-war
class who specified their political views as liberal), and $p_{\textrm{anti-
war}}(x)$ and $p_{\textrm{pro-war}}(x)$ are the actual unbiased distributions,
then:

$\tilde{p}_{\textrm{anti-war}}(x)=f_{\textrm{anti-war}}p_{\textrm{anti-
war}}(x)+(1-f_{\textrm{anti-war}})p_{\textrm{pro-war}}(x),$ (3)
$\tilde{p}_{\textrm{pro-war}}(x)=f_{\textrm{pro-war}}p_{\textrm{pro-
war}}(x)+(1-f_{\textrm{pro-war}})p_{\textrm{anti-war}}(x).$ (4)

From these equations, the unbiased distributions can be easily expressed in
terms of dataset distributions:

$p_{\textrm{anti-war}}(x)=\frac{f_{\textrm{pro-war}}\tilde{p}_{\textrm{anti-
war}}(x)-(1-f_{\textrm{anti-war}})\tilde{p}_{\textrm{pro-
war}}(x)}{f_{\textrm{anti-war}}+f_{\textrm{pro-war}}-1},$ (5)

$p_{\textrm{pro-war}}(x)=\frac{f_{\textrm{anti-war}}\tilde{p}_{\textrm{pro-
war}}(x)-(1-f_{\textrm{pro-war}})\tilde{p}_{\textrm{anti-
war}}(x)}{f_{\textrm{anti-war}}+f_{\textrm{pro-war}}-1}.$ (6)

All the distributions in the section 3 are corrected with Equation 5 and
Equation 6.

###  2.3 Machine learning for predicting support of the war

The information about the user subscriptions can be used to train a model for
predicting positions about the war in Ukraine. To do this, I used the method
described in Kosinski et al. (2013), where the personality traits and
political positions were predicted based on the likes on Facebook. The only
difference is that I use subscriptions instead of likes.

I constructed a matrix $A$, the rows of which are users, and the columns are
accounts that are followed by more than 20 people from the dataset. Each
element of the matrix $a_{ij}$ is equal to 0 or 1, depending on whether user
$i$ is subscribed to account $j$. Users subscribed to less than two accounts
were excluded from the sample. After that, some random pro-war users were
excluded from the sample, so that the number of elements in pro-war and anti-
war classes was the same.

For the resulting matrix, dimensionality reduction using Singular Value
Decomposition (SVD) was applied, with a final dimension of 100. This reduced
the matrix size from (5514, 11110) to (5514, 100). Each row of this matrix
still corresponds to one user, while the columns now represent some
combination of subscriptions.

Finally, the users, whose class was checked by reading the posts manually,
were excluded from the training dataset. These users are to be used for the
test of the accuracy of the model. The training was done with logistic
regression.

##  3 Results

###  3.1 Demography

I start the comparison of profiles of pro-war and anti-war users with
demography.

Gender is a mandatory field in VK, so it is filled in by all the users, and
its distribution in our sample is shown in Figure 1(b). Among the supporters
of the war, there are more women, which is surprising since most surveys
Russian Field (2023); Russia Watcher (2022) indicate that women support the
war less than men, and this has been a consistent result since the beginning
of the Russian invasion. Therefore, it is worth talking about who writes posts
about their position more often instead of who supports the war. Apparently,
women write pro-war posts more actively than men.

Figure 1: (a) Distribution of users by gender, (b) city and (c) age. All the
distributions are corrected as described in subsection 2.2.

The city is indicated by 2863 (88%) users from the anti-war class, and 6569
(90%) – from the pro-war class. As shown in Figure 1(b), a half of the
proponents of the war in VK (51%) lives in Moscow and Saint-Petersburg, while
among pro-war users this fraction is only 18%. In contrast, 27% of suppporters
of the war live in small localities (with population less than 100k), while
only 6% of anti-war users live in such places. It is consistent with the
common beliefs that large cities are more opposition. One should notice that
this result can be biased, because some of the people could leave Russia or
move to the different region, but not change their location in VK.

Let’s now move on to the age distribution. The date of birth is filled in by
1564 (48%) users from the anti-war class and 3958 (54%) – from the pro-war.
The picture Figure 1(c) shows a 3 year floating average of user’s age. Note
that this data reflects the demographics of VK users rather than of the
population as a whole, therefore their absolute values do not allow making any
conclusions, and one can only compare the age difference between supporters
and opponents of the war. The average age of people supporting the war is 45
years old, the median is also 45. For opponents of the war, these values are
42 and 40 years accordingly. As expected Chronicles (2022), the supporters of
the war turned out to be older than the opponents.

###  3.2 Views and values

The next characteristics I analyzed were the similarities and differences of
self-filled user views and values between supporters and opponents of the war.

I start with political views (see Figure 2(a)) – they are indicated by 995
(14%) people from the pro-war class and 662 (20%) from the anti-war. Among
anti-war-minded people, the percentage of people identified as liberals is
significantly larger than among pro-war, for whom it is one of the least
popular political views. This result is totally consistent with the popular
worldview in Russia, according to which liberal views are incompatible with
pro-government ideology РИА Новости (2022); ТАСС (2022).

Figure 2: Distribution of user views: (a) political views, (b) religion, (c)
personal priority, (d) important in others. All the distributions are
corrected as described in subsection 2.2.

Among pro-war users, the most popular views are "moderate". It is possible
that this variant is a default option for people who do not have any strong
political orientation. Therefore, it may indicate lower interest in politics
or even political apathy of the supporters of the war. It confirms the result
that Russian propaganda derives its effectiveness from political apathy rather
than its ability to persuade Maxim Alyukov (2022).

Interestingly, in terms of the ratio of socialists and communists, as well as
monarchists, there is almost no difference between anti-war and pro-war users.
The result that the supporters of the war are fans of the USSR not more than
opponents is also confirmed in subsection 3.3.

Religious views are indicated in their profile by 482 (15%) users with an
anti-war position and 728 (10%) with a pro-war. Only 20% of the anti-war users
define their religion as Orthodoxy, while among the supporters of the war,
this number reaches almost 65%.

The most popular answer among the war opponents in our sample is the free-form
option, which is shown as “Other” in the diagram. Here are some examples of
the answers in this field (translated from Russian): “God exists”, “priest of
the Orthodox Church”, “God is not exactly what the church says”, “believe in
love”, “not religious”, “believe in myself ". Thus, it is difficult to
determine the religiosity of this category of users.

According to surveys Levada Center (2022), 71% of Russians consider themselves
Orthodox. However, only 22% of the respondents attended religious services
more than once a year, and 49% believe in life after death. Therefore,
Orthodox identity does not necessarily mean following the Orthodox religion.
Considering the result that the number of "Orthodox" people among the
supporters of the war in VK is more than 3 times higher than among opponents,
one can assume that Orthodox identity may be based on supporting the values
and political course of the Russian Orthodox Church, which leans in line with
the government course.

The next profile fields that I analyzed are "personal priority" (indicated in
589 (18%) anti-war and 999 (14%) pro-war users) and "important in others"
(indicated in 597 (18%) and 1066 (15%) users, respectively). Interestingly,
anti-war and pro-war users do not differ a lot in these parameters. The most
significant difference is that “family and children” are more important for
war supporters (about half of the pro-war users chose this option), while war
opponents are slightly more likely to choose “self-development” (38%). This
difference is especially interesting in the context of the rhetoric of the
Russian government and Putin, who actively advocates for "traditional values",
the main component of which is a strong family Президент России (2022).

###  3.3 Subscriptions and interests

The next thing I explored is how the VK-publics a user follows are related to
support or non-support of the war.

For each public, I calculated what percentage of users from the sample are
subscribed to it (i.e. popularity), which is shown on the vertical axis at
Figure 3(a), and the fraction of the public subscribers who support and do not
support the war (which is shown on the horizontal axis).

Figure 3: Distribution of subscriptions of the users. (a) VK-publics, ranged
by their popularity and a degree of pro-war/anti-war orientation. Each point
is a public. The size of a point is the total number of subscribers of this
public. On the vertical axis is the fraction of the users from the selection,
subscribed to this public (in other words, popularity among the selection). On
the horizontal axis is the fraction of anti-war users in this public (e.g. x =
0.75 means that among all the subscribers of this public from the selection,
there are 75% anti-war users and 25% pro-war). X and Y coordinates are
calculated with Equation 7 and Equation 8. Some publics are highlighted by
magenta, to give examples of their names (translated from Russian). (b) VK-
publics classified into topics (see Appendix C), where each point corresponds
to a public. The horizontal axis is the same as in (a). The size of the point
here is the popularity among the selection (this parameter is on the vertical
axis in (a)). The weighted average positions of the publics on the horizontal
axis for each topic are shown in magenta, as well as the standard error of the
distribution.

I denote a fraction of users from the anti-war class subscribed to the public
$i$ as $\tilde{p}^{i}_{\textrm{anti-
war}}=\frac{N^{\textrm{subscribed}}_{\textrm{anti-
war}}}{N^{\textrm{total}}_{\textrm{anti-war}}}$, and correspondingly
$\tilde{p}^{i}_{\textrm{pro-war}}=\frac{N^{\textrm{subscribed}}_{\textrm{pro-
war}}}{N^{\textrm{total}}_{\textrm{pro-war}}}$. These values should be
corrected with Equation 5 and Equation 6 to get unbiased distributions
$p^{i}_{\textrm{anti-war}}$ and $p^{i}_{\textrm{pro-war}}$. Then, these
distributions are used to get $x_{i}$ and $y_{i}$ for each public $i$ in
Figure 3(a):

$x_{i}=\frac{p^{i}_{\textrm{anti-war}}}{p^{i}_{\textrm{anti-
war}}+p^{i}_{\textrm{pro-war}}},$ (7)

$y_{i}=\frac{p^{i}_{\textrm{anti-war}}+p^{i}_{\textrm{pro-war}}}{2}.$ (8)

The more to the right the public in the Figure 3(a), the more opponents of the
war are subscribed to it, and the fewer supporters. And the higher it is, the
more popular it is with users of both categories. Publics that are in the
middle are equally popular among both supporters and opponents of the war.

All publics can be approximately divided into 3 groups. On the right are
opposition accounts: "TV channel Rain", "Navalny", "Meduza", etc. On the left
are pro-war political accounts: "United Russia", "Russian Ministry of
Defense", "Putin V.V.", etc. And in the middle - publics on neutral topics. It
is worth noting here that many (but not all) opposition publics in VK are
blocked for people located in Russia, which may affect the results of the
analysis.

An interesting observation here is an absence of clustering: most publics are
followed by both supporters and opponents of the war. Therefore, the pro-war
and anti-war users (even those who have a strong position enough to write a
post on it) often coexist in the same information fields, and consume the same
content, when it is not connected to the war.

Figure 3(b) shows interests, highlighted by keywords (Appendix C) in the names
of publics, and their average position on the axis of support/non-support of
the war (the same horizontal axis as in Figure 3(a)). For the detail of
calculating the average position, see Appendix D. The wider the magenta line,
the greater the spread – for example, among the publics about a country house,
there exist the publics with almost only subscribers who support the war, but
also the publics where the fraction of supporters and opponents is almost
equal.

This analysis shows that many interests are shared by both anti-war and pro-
war users. For example, history, sports, and music are similarly interesting
for both user classes. Interestingly, almost equal number of supporters and
opponents of the war are interested in USSR, which complements the result from
subsection 3.2, showing that there is almost equal number of communists and
socialists among pro-war and anti-war users.

However, apart from the obvious political interests, there are some topics
that users of one of the groups are more interested in than the other. For
example, war supporters are more interested in family and cooking, which is in
line with previous results showing that family and children are more important
to war supporters. At the same time, opponents of the war are more often
interested in learning English and traveling.

###  3.4 Predicting position based on subscriptions

Machine learning to determine whether a person supports war or not, based on
information about subscriptions. Details on how it works can be found in
subsection 2.3. The accuracy of the model was determined on users (with more
than 2 subscribtions) whose posts were read and classified manually (72 anti-
war posts and 160 pro-war ones).

It turned out that it is possible to predict that the user adheres to the
anti-war position in 90% of cases based on their subscriptions. For pro-war
users, the fraction of correctly guessed positions is 70%.

##  4 Conclusion

Understanding the personality of the pro-war Russians is essential for
understanding the roots of the war, as well as for finding ways to prevent
similar situations in the future. Social media provide an extensive source of
information on people’s personalities and can be successfully utilized for
this purpose.

My analysis shows that war supporters on VK are more interested in what is
similar to the “traditional values” in terms of the Russian government: family
and children are more important to them, and they usually consider themselves
Orthodox. Meanwhile, opponents of the war are more interested in learning
English, travelling, books and science.

However, despite different political views and different interests on some
issues, supporters and opponents of the war have some commonalities. They
constantly coexist in the same information space: they read the same content
about history, sports, and music. Representatives of both groups – both
supporters and opponents of the war – most often value kindness and honesty in
people, and this value does not depend on whether the person supports the war.

About the political believes, supporters of the war most often define their
political views on VK as "moderate", which may indicate their weak political
identity or apoliticality, while opponents of the war often can determine
their political orientation more precisely, with majority of them consisting
of liberals, but also with significant fractions of socialists and
conservatives. Additionally, opponents and supporters of the war are equally
interested in the topic of the USSR, and the number of communists and
socialists among these categories is the same.

The extra practical result of the analysis is that based on the user
subscriptions, the machine learning model can predict the position regarding
the war. For anti-war users, the accuracy of such prediction is 90%, while for
pro-war users it is 70%.

Acknowledgments

I am grateful to Novaya gazeta Europe and Teplitsa for organizing the
hackathon Projector 2023, as well as my team at this hackathon, where I had a
chance to present the results of the current research for the first time,
improve it by gaining insights during discussions with experts and team
members, and the inspiration that it gave to me. I am also thankful to Leonid
Yuldashev and Equalitie for useful discussions and support.

## Statements and declarations

Data availability

The anonymized data collected during the current study, as well as the code
for analysis, are available in the Zenodo repository,
https://doi.org/10.5281/zenodo.8125674.

Funding

There was no funding for this research.

Competing interests

There is no conflict of interest.

##  Appendix A Keywords for data collection

Appendix A contains search words used for dataset collection.

Table 1: Search words for dataset collection

Class | Search word – original | Search word – translated | Notes
---|---|---|---
Pro-war |  |  |

СвоихНеБросаем & #WeDontLeaveOurs  Meaning: "we don’t leave our people in
trouble" Pro-war #ZaНаших #ForOurs  Meaning: "for our people". "Z" as a
military symbol 111The letter Z is widely used as a symbol of the Russian army
during the war in Ukraine. O and V are also sometimes used in this context The
New York Times (2022) Pro-war ZOV –  "Z", "O", "V" as military symbols 111The
letter Z is widely used as a symbol of the Russian army during the war in
Ukraine. O and V are also sometimes used in this context The New York Times
(2022) Pro-war укрофашисты Ukrofascists Meaning: "Ukrainian fascists" Pro-war
бандеровцы Bandera  Meaning: "people supporting nationalist Stepan Bandera"
Pro-war неонацисты neo-nazis – Pro-war хохлы –  Meaning: "Ukrainians"
(offensive) Pro-war реабилитация нацизма rehabilitation of Nazism – Pro-war
специальная военная операция special military operation  The term used by
Russian authorities with respect to the war in Ukraine Anti-war #НетВойне
#NoWar – Anti-war #Нет_Войне #No_War – Anti-war #No_War – – Anti-war
#StopRussianAggression – – Anti-war #CloseTheSky –  A call to Europe to close
the sky over Ukraine Anti-war #RussiansAgainstWar – – Anti-war Путин хуйло
Putin is a dick – Anti-war zвастика zwastika Swastika with "Z" 111The letter Z
is widely used as a symbol of the Russian army during the war in Ukraine. O
and V are also sometimes used in this context The New York Times (2022) Anti-
war zомби zombie Zombie with "Z" 111The letter Z is widely used as a symbol of
the Russian army during the war in Ukraine. O and V are also sometimes used in
this context The New York Times (2022) Anti-war zомбирование zombie (process)
Zombie with "Z" 111The letter Z is widely used as a symbol of the Russian army
during the war in Ukraine. O and V are also sometimes used in this context The
New York Times (2022) Anti-war путлер Putler  Combination of "Putin" and
"Hitler" Anti-war рашисты ruscists Meaning: "Russian fascists" Anti-war
российское вторжение Russian invasion –

The profiles, initially classified as anti-war, but whose posts contained the
hashtags from Table 2, were removed from the dataset, since they are likely
pro-war or their polarity is undefined.

Table 2: Keywords identifying that the author of the post initially classified as anti-war should be removed from the dataset Keyword – original | Keyword – translated | Notes
---|---|---
#ZaМир | #ForPeace | "Z" as a military symbol 111The letter Z is widely used as a symbol of the Russian army during the war in Ukraine. O and V are also sometimes used in this context The New York Times (2022)
#ЗаМир | #ForPeace | –
#ЗаНаших | #ForOurs | Meaning: "for our people"
#МыВместе | #WeAreTogether | –
#ZаПрезидента | #ForPresident | "Z" as a military symbol 111The letter Z is widely used as a symbol of the Russian army during the war in Ukraine. O and V are also sometimes used in this context The New York Times (2022)
#ЗаПрезидента | #ForPresident | –
#НетФашизму | #NoFascism | –
#ZаРоссию | #ForRussia | "Z" as a military symbol 111The letter Z is widely used as a symbol of the Russian army during the war in Ukraine. O and V are also sometimes used in this context The New York Times (2022)
#ЗаРоссию | #ForRussia | –
#ЕР | #UR | "United Russia" (political party)
#ЕдинаяРоссия | #UnitedRussia | "United Russia" (political party)
#МариупольРусскийГород | #MariupolIsRussianCity | –
#ДаПобеде | #YesVictory | Opposite to "no war"
#ZаПобеду | #ForVictory | "Z" as a military symbol 111The letter Z is widely used as a symbol of the Russian army during the war in Ukraine. O and V are also sometimes used in this context The New York Times (2022)
#ЗаПобеду | #ForVictory | –
#ВремяПомогать | #TimeToHelp | –
#ГероиZ | #HeroesZ | "Z" as a military symbol 111The letter Z is widely used as a symbol of the Russian army during the war in Ukraine. O and V are also sometimes used in this context The New York Times (2022)
#СВО | #SMO | Special military operation
#ДНР | #DPR | Donetsk People’s Republic
#ЛНР | #LPR | Lugansk People’s Republic
#ZаНамиПравда | #TruthIsWithUs | "Z" as a military symbol 111The letter Z is widely used as a symbol of the Russian army during the war in Ukraine. O and V are also sometimes used in this context The New York Times (2022)
#ЗаНамиПравда | #TruthIsWithUs | –
#ZаПравду | #ForTruth | "Z" as a military symbol 111The letter Z is widely used as a symbol of the Russian army during the war in Ukraine. O and V are also sometimes used in this context The New York Times (2022)
#ZаПравду | #ForTruth | –
#ДжекиЧан | #JackieChan | See 222Hashtag #НетВойне (#NoWar) was used in a YouTube short about Jakie Chan that people reposted to VK. Its content is not connected to the war in Ukraine
|  |

##  Appendix B Comparison of activity counters between classes

All the user activity counters, such as the number of videos, audio tracks,
photos, friends, gifts, groups, followers, pages (meaning subscriptions to
group pages, "publics"), subscriptions (to other user pages), follow the usual
Kwak et al. (2010); Ma et al. (2022); Silva et al. (2013) for such parameters
power law, that looks linear in double-logarithmic scale for large values
(Figure 4). The maximum number of some of these counters in VK is restricted
to 10000 (friends, audios, and groups).

Figure 4: Comparison of activity counters between classes in double-
logarithmic scale

Despite some differences in user counters between users from pro-war and anti-
war classes, both distributions look natural, making it unlikely that a
significant fraction of the accounts were created artificially specifically
for promoting the pro-war position.

##  Appendix C Topics of publics

To define topics of the publics, I used keywords. The publics was considered
to belong to some topics if its name contained at least one of the keywords in
any form of case. E.g. the publics was considered to be about history if there
was "history" or "historical" in its name. The full list of the keywords for
topics is listed in Table 3.

Table 3: Keywords to define topics of the publics Topic | Keyword – original | Keyword – translated
---|---|---
Navalny |  навальный, фбк, navalny |  | Navalny,
---
ACF (Anti-Corruption Foundation)
Prigozhin | пригожин, вагнер, чвк |  | Prigozhin, Wagner,
---
PMC (private military company)
English |  английский, english | English
SMO |  сво, z111The letter Z is widely used as a symbol of the Russian army during the war in Ukraine. O and V are also sometimes used in this context The New York Times (2022), zov111The letter Z is widely used as a symbol of the Russian army during the war in Ukraine. O and V are also sometimes used in this context The New York Times (2022) | SMO (Special Military Operation)
Country house (dacha) | дача, огород, дачный | cottage (country house), garden
Family |  |  семья, ребенок, родитель,
---
родительство
family, child, parent, parenting
Cooking |  |  рецепт, кулинария, кухня,
---
кулинарный, вкусно, вкусный
| recipe, cooking, kitchen, culinary,
---
delicious
Traveling |  |  путешествие, туризм,
---
турист, путешествовать,
туристический
| journey, tourism, tourist, travel,
---
touristic
Books |  |  книга, чтение, литература,
---
поэзия, поэт, писатель
| book, reading, literature, poetry,
---
poet, writer
Science | наука, научный | science, scientific
Orthodoxy |  |  православие, христианство,
---
православный, христианский
| Orthodoxy, Christianity, Orthodox,
---
Christian
Psychology | психология, психологический | psychology, psychological
Movies | кино, фильм | cinema, film, movie
USSR | ссср, советский | USSR, Soviet
History | история, исторический | history, historical
Sports |  |  спорт, зож, тренировка,
---
спортивный, тренироваться,
фитнес
| sports, healthy, workout, sportive,
---
exercise, fitness
Music | музыка, музыкальный | music, musical
|  |

##  Appendix D Weighted average

To calculate the average pro-war/anti-war ratio for publics on a particular
topic, I use the equation for weighted average:

$\bar{x}=\frac{\Sigma x_{i}y_{i}}{\Sigma y_{i}},$ (9)

where $x_{i}$ and $y_{i}$ are calculated with Equation 7 and Equation 8
correspondingly.

A square of the standard deviation of the weighted average is

$SE^{2}=\frac{n}{(n-1)(\Sigma
y_{i})^{2}}(\Sigma(y_{i}x_{i}-\bar{y}\bar{x})^{2}-2\bar{x}\Sigma(y_{i}-\bar{y})(y_{i}x_{i}-\bar{y}\bar{x})+\bar{x}^{2}\Sigma(y_{i}-\bar{y})^{2}),$
(10)

where $n$ is a number of elements (publics in the topic).

## References

  *   * Chapkovski and Schaub (2022) Chapkovski, P., Schaub, M.: Solid support or secret dissent? a list experiment on preference falsification during the russian war against ukraine. Research & Politics 9(2), 20531680221108328 (2022) https://doi.org/10.1177/20531680221108328
  * Re: Russia (2023) Re: Russia: Russian Field: Support for Non-support of Peace and War. https://re-russia.net/en/analytics/059/ Accessed 2023-05-21
  * Chronicles (2022) Chronicles: Wartime Censorship. How to Measure Fear? https://www.chronicles.report/en/chapter2 Accessed 2023-05-21
  * Levada-center (2023) Levada-center: Conflict with Ukraine: Assessments for March 2023. https://www.levada.ru/en/2023/04/07/conflict-with-ukraine-assessments-for-march-2023/ Accessed 2023-05-21
  * Chronicles (2023) Chronicles: A Year of War as Seen by Russians. https://www.chronicles.report/en/chapter9 Accessed 2023-05-21
  * Russian Field (2023) Russian Field: Год «специальной военной операции» в Украине: отношение россиян. https://russianfield.com/godsvo Accessed 2023-05-21
  * Ishchenko and Zhuravlev (2022) Ishchenko, V., Zhuravlev, O.: Imperialist ideology or depoliticization? why russian citizens support the invasion of ukraine. HAU: Journal of Ethnographic Theory 12(3), 668–676 (2022) https://doi.org/10.1086/723802
  * Public Sociology Lab (2022) Public Sociology Lab: Далёкая близкая война: Как россияне воспринимают военные действия в Украине. http://publicsociology.tilda.ws/war_report Accessed 2023-05-21
  * Public Sociology Lab (2023) Public Sociology Lab: Смириться с неизбежностью: Как россияне оправдывают военное вторжение в Украину? http://publicsociology.tilda.ws/report2 Accessed 2023-05-21
  * Murphy et al. (2014) Murphy, J., Link, M.W., Childs, J.H., Tesfaye, C.L., Dean, E., Stern, M., Pasek, J., Cohen, J., Callegaro, M., Harwood, P.: Social Media in Public Opinion Research: Executive Summary of the Aapor Task Force on Emerging Technologies in Public Opinion Research. Public Opinion Quarterly 78(4), 788–794 (2014) https://doi.org/10.1093/poq/nfu053
  * Dong and Lian (2021) Dong, X., Lian, Y.: A review of social media-based public opinion analyses: Challenges and recommendations. Technology in Society 67, 101724 (2021) https://doi.org/10.1016/j.techsoc.2021.101724
  * Tucker et al. (2018) Tucker, J.A., Guess, A., Barbera, P., Vaccari, C., Siegel, A., Siegel, A., Sanovich, S., Stukal, D., Nyhan, B.: Social Media, Political Polarization, and Political Disinformation: A Review of the Scientific Literature. Preprint at http://dx.doi.org/10.2139/ssrn.3144139 (2018)
  * Bright (2018) Bright, J.: Explaining the Emergence of Political Fragmentation on Social Media: The Role of Ideology and Extremism. Journal of Computer-Mediated Communication 23(1), 17–33 (2018) https://doi.org/10.1093/jcmc/zmx002
  * Enikolopov et al. (2020) Enikolopov, R., Makarin, A., Petrova, M.: Social media and protest participation: Evidence from russia. Econometrica 88(4), 1479–1514 (2020) https://doi.org/10.3982/ECTA14281
  * Bykov and Martyanov (2021) Bykov, I., Martyanov, D.: Studying political communities in vk.com with network analysis. Galactica Media: Journal of Media Studies 3(1), 64–78 (2021) https://doi.org/%****␣sn-article.bbl␣Line␣250␣****10.46539/gmd.v3i1.144
  * Philippov (2022) Philippov, I.B.: The “moscow case” as a factor of protest communication activity in the social network “vk”. RUDN Journal of Political Science 24(3), 530–544 (2022) https://doi.org/10.22363/2313-1438-2022-24-3-530-544
  * Petrov (2022) Petrov, I.: Anti-vaccination movement on vk: Information exchange and public concern. In: Alexandrov, D.A., Boukhanovsky, A.V., Chugunov, A.V., Kabanov, Y., Koltsova, O., Musabirov, I., Pashakhin, S. (eds.) Digital Transformation and Global Society, pp. 108–121. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-93715-7_8
  * Open Democracy (2012) Open Democracy: The Dark Doings of Russia’s Centre E. https://www.opendemocracy.net/en/odr/dark-doings-of-russias-centre-e/ Accessed 2023-05-21
  * Wired (2022) Wired: How the Kremlin Infiltrated Russia’s Facebook. https://www.wired.co.uk/article/vk-russia-democracy Accessed 2023-05-21
  * Earl et al. (2022) Earl, J., Maher, T.V., Pan, J.: The digital repression of social movements, protest, and activism: A synthetic review. Science Advances 8(10), 8198 (2022) https://doi.org/10.1126/sciadv.abl8198
  * McCarthy et al. (2023) McCarthy, L.A., Rice, D., Lokhmutov, A.: Four months of "discrediting the military": Repressive law in wartime russia. Demokratizatsiya: The Journal of Post-Soviet Democratization 31(2), 125–160 (2023)
  * Stoner (2022) Stoner, K.: The war in ukraine: How putin’s war in ukraine has ruined russia. Journal of Democracy 33(3), 38–44 (2022) https://doi.org/10.1353/jod.2022.0038
  * Prier (2017) Prier, J.: Commanding the trend: Social media as information warfare. Strategic Studies Quarterly 11(4), 50–85 (2017) https://doi.org/10.2307/26271634
  * Radio Free Europe – Radio Liberty (2021) Radio Free Europe – Radio Liberty: Inside The ’Propaganda Kitchen’ – A Former Russian ’Troll Factory’ Employee Speaks Out. https://www.rferl.org/a/russian-troll-factory-hacking/31076160.html Accessed 2023-06-08
  * Kosinski et al. (2013) Kosinski, M., Stillwell, D., Graepel, T.: Private traits and attributes are predictable from digital records of human behavior. PNAS 110(15), 5802–5805 (2013) https://doi.org/10.1073/pnas.1218772110
  * Russia Watcher (2022) Russia Watcher: Research. https://russiawatcher.com/research Accessed 2023-06-10
  * Chronicles (2022) Chronicles: Mobilization: an Express Survey. https://www.chronicles.report/en/chapter7 Accessed 2023-06-10
  * РИА Новости (2022) РИА Новости: "Я не такой, как все": либеральная интеллигенция давно служит нацизму. https://ria.ru/20220531/liberaly-1791830210.html Accessed 2023-06-10
  * ТАСС (2022) ТАСС: Путин: либерализм дошел до абсурда, объявляя альтернативные взгляды угрозами демократии. https://tass.ru/politika/16174585 Accessed 2023-06-10
  * Maxim Alyukov (2022) Maxim Alyukov: Propaganda, Political Apathy, and Authoritarianism in Russia. https://jordanrussiacenter.org/news/propaganda-political-apathy-and-authoritarianism-in-russia/ Accessed 2023-06-10
  * Levada Center (2022) Levada Center: Religiosity. https://www.levada.ru/en/2022/05/18/religiosity/ Accessed 2023-07-04
  * Президент России (2022) Президент России: Указ Президента Российской Федерации от 09.11.2022 г. № 809. http://www.kremlin.ru/acts/bank/48502 Accessed 2023-07-05
  * The New York Times (2022) The New York Times: The Letter ‘Z’ Has Become a Symbol for Russians Who Support the Invasion of Ukraine. https://www.nytimes.com/2022/03/07/world/europe/russia-letter-z-ivan-kuliak.html Accessed 2023-05-23
  * Kwak et al. (2010) Kwak, H., Lee, C., Park, H., Moon, S.: What is twitter, a social network or a news media? In: Proceedings of the 19th International Conference on World Wide Web. WWW ’10, pp. 591–600. Association for Computing Machinery, New York, NY, USA (2010). https://doi.org/10.1145/1772690.1772751 . https://doi.org/10.1145/1772690.1772751
  * Ma et al. (2022) Ma, X., Li, C., Bailey, J., Wijewickrema, S.: Finding Influentials in Twitter: A Temporal Influence Ranking Model (2022)
  * Silva et al. (2013) Silva, T.H., Melo, P.O.S.V.d., Almeida, J.M., Salles, J., Loureiro, A.A.F.: A picture of instagram is worth more than a thousand words: Workload characterization and application. In: 2013 IEEE International Conference on Distributed Computing in Sensor Systems, pp. 123–132 (2013). https://doi.org/10.1109/DCOSS.2013.59


# Pairwise Learning for Neural Link Prediction

Zhitao Wang
WeChat Pay, Tencent
<EMAIL_ADDRESS>
Yong Zhou
WeChat Search, Tencent
<EMAIL_ADDRESS>
Litao Hong
WeChat Pay, Tencent
<EMAIL_ADDRESS>
Yuanhang Zou
WeChat Search, Tencent
<EMAIL_ADDRESS>
Hanjing Su
WeChat Pay, Tencent
<EMAIL_ADDRESS>
Shouzhi Chen
WeChat Pay, Tencent
<EMAIL_ADDRESS>

###### Abstract

In this paper, we aim at providing an effective Pairwise Learning Neural Link
Prediction (PLNLP) framework. The framework treats link prediction as a
pairwise learning to rank problem and consists of four main components, i.e.,
neighborhood encoder, link predictor, negative sampler and objective function.
The framework is flexible that any generic graph neural convolutions or link
prediction specific neural architectures could be employed as neighborhood
encoder. For link predictor, we design different scoring functions, which
could be selected based on different types of graphs. In negative sampler, we
provide several sampling strategies, which are problem specific. As for
objective function, we propose to use an effective ranking loss, which
approximately maximizes the standard ranking metric AUC. We evaluate the
proposed PLNLP framework on 4 link property prediction datasets of Open Graph
Benchmark (OGB), including ogbl-ddi, ogbl-collab, ogbl-ppa and ogbl-ciation2.
PLNLP achieves top 1 performance on ogbl-ddi and ogbl-collab, and top 2
performance on ogbl-ciation2 only with basic neural architecture. The
experimental results demonstrate the effectiveness of PLNLP.

##  1 Introduction

With a variety of real-world applications, link prediction has been recognized
of great importance and attracted increasing attention from the research
community in past decade (Lü and Zhou, 2011; Martínez et al., 2017). For
instance, link prediction methods could help infer potential protein-protein
interactions to efficiently save human effort on blind checking (Airoldi et
al., 2008). Also, link prediction techniques could be used to predict new
friendships between uses on social media, or to discover potential user-to-
item relationships on E-commerce sites, such that user experience could be
improved (Adamic and Adar, 2003; Koren et al., 2009).

In the literature, heuristic-based methods are probably the most
representative link prediction algorithms. The key idea of most existing
heuristic methods is to measure the similarity of two target nodes based on
their neighborhood information. The success of these heuristics has
demonstrated the importance of neighborhood information of target node-pair.
However, heuristic methods often have weak applicability and expressiveness in
dealing with different types of networks for its simple-form and hand-crafted
information of neighborhood. A previous survey found that all of heuristics
methods failed to perform consistently across multiple networks (Lü and Zhou,
2011). The needs of prior knowledge or expensive trial and error are
inevitable in choosing appropriate heuristics for different networks. Thanks
to effective feature learning ability of neural networks, a series of neural
link prediction models (Zhang and Chen, 2017; Kipf and Welling, 2016b; Wang et
al., 2020, 2019, 2021) were proposed, of which the generalization ability was
successfully improved.

Existing neural link prediction methods pay much attention on designing more
expressive neural architectures, while some basic properties of the problem
are often neglected. For example, most neural models treat link prediction as
a binary classification problem and naturally adopt a cross entropy loss
function. However, this learning schema seems not to be suitable for the link
prediction problem. First, link classification is extremely imbalanced due to
the natural sparsity of most graphs. Although under-sampling could be adopted,
there would be information loss during sampling process and what ratio of
sampling is hard to decide. Second, most link prediction evaluation protocols
do not aim at labeling positive pairs as 1 while negative pairs as 0, but ask
for ranking positive pairs higher than negative pairs. Therefore, employing
cross-entropy function seems not to be so direct to the objective of the link
prediction task.

Based on above understanding and our previous research (Wang et al., 2020,
2019, 2021), we provide an effective and generic pairwise learning neural link
prediction framework in this paper, named PLNLP. The framework adopts a
pairwise learning to rank schema and consists of four main components, i.e.,
neighborhood encoder, link predictor, negative sampler and objective function.
The neighborhood encoder aims at extracting expressive neighborhood
information of input node-pair. Any generic graph neural convolution, such as
GCN (Kipf and Welling, 2016a) and SAGE (Hamilton et al., 2017), or link
prediction specific neural architecture, such as SEAL (Zhang and Chen, 2018),
NANs (Wang et al., 2020) and HalpNet (Wang et al., 2021), could be employed as
neighborhood encoder. For link predictor, we design different scoring
functions, which could be selected based on different types of graphs. In
negative sampler, we provide several negative sampling strategies, which are
problem specific. As for objective function, we propose to use an effective
ranking loss, which approximately maximizes the standard ranking metric AUC.
We evaluate the proposed PLNLP framework on 4 link property prediction
datasets of Open Graph Benchmark (OGB) (Hu et al., 2020), including ogbl-ddi,
ogbl-collab, ogbl-ppa and ogbl-ciation2. PLNLP with basic neural architecture
achieves top 1 performance on ogbl-ddi and ogbl-collab, and top 2 performance
on ogbl-ciation2. The performance demonstrates the effectiveness of PLNLP.

##  2 Related Work

Existing link prediction approaches can be categorized into three families:
heuristic feature based, latent embedding based and neural network based.

Heuristic Methods: Most heuristics measure node similarity with neighborhood
information. Popular heuristics include first-order methods common neighbors,
Jaccard index (Salton and McGill, 1986) and preferential attachment (Liben-
Nowell and Kleinberg, 2007); second-order methods, i.e., Adamic-Adar (Adamic
and Adar, 2003), resource allocation (Zhou et al., 2009); and high-order
heuristic SimRank (Jeh and Widom, 2002). These heuristics often fail to
capture complex latent formation features.

Embedding-based Methods: Embedding based methods aim at learning latent node
features. The most classical one is matrix factorization (MF) method (Menon
and Elkan, 2011), which aims at reconstructing adjacency matrix. Besides, a
series of unsupervised network representation learning models (Perozzi et al.,
2014; Tang et al., 2015; Grover and Leskovec, 2016; Hamilton et al., 2017),
are also applicable for link prediction. These methods learn generic latent
embeddings by preserving structure proximities from a probabilistic view and
predict links by composing node embeddings as edge features. PNRL (Wang et
al., 2017) is a state-of-the-art link prediction specific embedding method,
which simultaneously preserves proximities of observed structure and infers
hidden links.

NN-based Methods: Recently, some neural network-based link prediction models
were developed, which explore non-linear deep structural features with neural
layers. Variational graph auto-encoders (Kipf and Welling, 2016b) predict
links by encoding graph with graph convolutional layer (Kipf and Welling,
2016a). Another two state-of-the-art neural models WLNM (Zhang and Chen, 2017)
and SEAL (Zhang and Chen, 2018) use graph labeling algorithm to transfer union
neighborhood of two nodes (enclosing subgraph) as meaningful matrix and employ
convolutional neural layer or a novel graph neural layer DGCNN (Zhang et al.,
2018) for encoding.

Besides, in our previous work, we proposed a series of neighborhood attention
neural networks (Wang et al., 2020, 2019, 2021), in which different attention
mechanisms were designed to encode neighborhood information specific for link
prediction problem. For instance, in (Wang et al., 2020, 2019), we proposed
cross neighborhood attention and interactive attention mechanisms to capture
structural interactions between neighborhoods of the target node-pair.

##  3 Preliminaries

###  3.1 Graphs

Generally, a graph (network) is represented as $G=(V,E)$, where
$V=\\{v_{1},...,v_{N}\\}$ is the set of nodes, $E\subseteq V\times V$ is the
set of links, and the total number of distinct nodes is $N$. Also, a graph is
often denoted as an adjacency matrix $\mathbf{A}$, where $A_{i,j}=1$ if there
is a link from node $v_{i}$ to $v_{j}$, otherwise $A_{i,j}=0$. $\mathbf{A}$
will be symmetric, if the graph is undirected.

###  3.2 Neighborhood of Node

We use $\mathcal{N}^{h}(v_{i})$ to represent the $h$-hop neighborhood of node
$v_{i}\in V$, which is the set of nodes whose distance to $v_{i}$ (represented
as $d(v_{i},v_{j})$) is not greater than $h$. In this paper, we focus on
unweighted graph, thus the distance function $d(v_{i},v_{j})$ is directly
computed as the length of the shortest path between $v_{i}$ and $v_{j}$. We
call $v_{i}$ the center node and $v_{j}\in\mathcal{N}^{h}(v_{i})$ the
neighboring node within $h$-hop. To make the neighborhood also include the
unique information of the center node, we define that the center node $v_{i}$
is a neighboring node of itself, such that $v_{i}\in\mathcal{N}^{h}(v_{i})$.

###  3.3 Neighborhood Subgraph of Node-Pair

We use $\mathcal{G}^{h}(v_{i},v_{j})$ to represent the $h$-hop neighborhood
subgraph of the node pair $(v_{i},v_{j})$, which is extracted from the whole
graph $\mathcal{G}$. Formally, for any node $v_{k}$ in the neighborhood
subgraph $\mathcal{G}^{h}(v_{i},v_{j})$, it should satisfy $d(v_{k},v_{i})\leq
h$ and $d(v_{k},v_{j})\leq h$, i.e.,
$v_{k}\in\mathcal{N}^{h}(v_{i})\cup\mathcal{N}^{h}(v_{j})$.

###  3.4 Link Prediction

Link prediction problems are categorized as temporal link prediction which
predicts potential new links on an evolving network, and structural link
prediction which infers missing links on a static network. In this paper, we
focus on structural link prediction. Given the partially observed structure of
a network, the goal of it is to predict the unobserved links. Formally, given
a partially observed network $G=(V,E)$, we represent the set of node-pairs
with unknown link status as $E^{?}=V\times V-E$, then the goal of structural
link prediction is to infer link status of node-pairs in $E^{?}$.

##  4 PLNLP Framework

The proposed framework is illustrated as Figure 1. Given an input graph,
negative sampler aims to draw negative samples and form training pairs. A
training pair consists of a positive sample, which is a node-pair with an
observed edge in input graph, and a negative sample, which is a node-pair
drawn by negative sampler. Neighborhood encoder is used to extract
neighborhood information of both positive and negative samples as the hidden
representations. Given the hidden representations, link predictor will
calculate link scores of both samples. With link scores of training pairs, the
model parameters will be optimized based on the pairwise ranking objective
function.

Figure 1: PLNLP Framework

###  4.1 Neighborhood Encoder

Neighborhood information has proved crucial for link prediction. Therefore, we
propose to use neighborhood neural encoder to extract structural information
of input samples. We consider two kinds of neighborhood encoder in this paper.
One is Node Neighborhood Encoder (NNE), which encodes the two nodes of a input
sample with their own neighborhood as two hidden representations, separately.
Any generic graph neural networks (GNN), e.g, GCN, GraphSAGE and GAT, could be
employed as NNE. Assume that input sample is $(v_{i},v_{j})$, NNEs aim to
extract hidden representations of the input sample as follows:

$\mathbf{h}_{i}=\textup{NNE}(\mathbf{x}_{i},\\{\textbf{x}_{k}\
|{v_{k}\in\mathcal{N}^{h}(v_{i})}\\}),\ \ \ \
\mathbf{h}_{j}=\textup{NNE}(\mathbf{x}_{j},\\{\textbf{x}_{l}\
|{v_{l}\in\mathcal{N}^{h}(v_{j})}\\})$ (1)

where $\mathbf{x}_{i}$ generally represents the input feature of node $v_{i}$.
If there is no input features, $\mathbf{x}_{i}$ represents a embedding vector
of node $v_{i}$, which is trainable parameter. $\mathbf{x}_{i}$ could also
represent the concatenation of input feature and node embedding. In this
framework, we only consider homogeneous graph, which means that all nodes
share a same NNE.

The other kind of neighborhood encoder is Edge level Neighborhood Encoder
(ENE), or called node-pair neighborhood encoder. Recently, a series of ENEs,
such as SEAL, NIAN and HalpNet, were proposed specifically for link prediction
problem. The main advantage of ENEs is capturing structural interactions
between the neighborhoods, which are ignored in NENs. ENEs often consider the
neighborhood subgraph of a sample (node-pair) as input, and encode it as one
hidden representation. Assume that input sample is $(v_{i},v_{j})$, ENEs
derive a hidden representation of the input sample as follows:

$\mathbf{h}_{ij}=\textup{ENE}(\mathbf{x}_{i},\mathbf{x}_{j},\\{\textbf{x}_{k}\
|{v_{k}\in\mathcal{G}^{h}(v_{i},v_{j})}\\})$ (2)

Similarly, $\mathbf{x}_{i}$ represents the input feature, or trainable
embedding, or the concatenation of input feature and embedding.

###  4.2 Link Score Predictor

After deriving the hidden representations either in node level or node-pair
(edge) level, the framework will calculate a linking score of the input
sample. We provide several selections of the scoring function.

Dot Predictor. If we use NNE to derive $\mathbf{h}_{i}$ and $\mathbf{h}_{j}$
of the input sample $(v_{i},v_{j})$, we can simply use a dot operator to
derive the score:

$s_{ij}=\mathbf{h}_{i}\cdot\mathbf{h}_{j}$ (3)

Bilinear Dot Predictor. Dot operator can be only used for undirected graph due
to its commutative property. For directed graph, we can adopt bilinear dot
operator to make the scoring function not commutative:

$s_{ij}=\mathbf{h}_{i}\mathbf{W}\mathbf{h}_{j}$ (4)

where $\mathbf{W}$ is a learnable matrix.

MLP Predictor. We can also employ a multi-layer perceptron (MLP) as the link
predictor. If we use ENEs to obtain hidden representation of the input sample,
the predictor is as follow:

$s_{ij}=\textup{MLP}(\mathbf{h}_{ij})$ (5)

If we use NNEs to obtain hidden representations, there are several possible
forms of MLP’s input. If the graph is undirected, we can adopt a widely used
commutative operator, i.e, hadamard product $\odot$:

$s_{ij}=\textup{MLP}(\mathbf{h}_{i}\odot\mathbf{h}_{j})$ (6)

If the graph is directed, we would prefer a non-commutative operator, such as
concatenation $||$:

$s_{ij}=\textup{MLP}(\mathbf{h}_{i}||\mathbf{h}_{j})$ (7)

###  4.3 Pairwise Learning with Ranking Objective

Due to the sparsity of networks, there often exists extreme imbalance between
linked pairs and non-linked pairs. Meanwhile, most link prediction tasks do
not aim at labeling positive pairs as 1 while negative pairs as 0, but ask for
ranking positive pairs higher than negative pairs. To be consistent with the
general objective of link prediction, we adopt the ranking idea for model
learning, which can be formalized as:

$s_{ij}>s_{kl},\forall{(v_{i},v_{j})\in E}\ \textup{and}\
\forall{(v_{k},v_{l})\in E^{-}}$ (8)

where $s_{ij}$ and $s_{kl}$ are the output scores of link predictor, $E^{-}$
is the set of true non-linked pairs. In fact, the above learning objective is
equivalent to maximize the Area Under the Curve (AUC), which is interpreted as
the probability of a positive sample ranking higher than a negative sample.
The empirical AUC value is defined as follow:

$\textup{AUC}=\sum_{(v_{i},v_{j})\in E}\sum_{(v_{k},v_{l})\in
E^{-}}\frac{\mathbbm{1}[f_{\theta}(v_{i},v_{j})>f_{\theta}(v_{k},v_{l})]}{|E|\times|E^{-}|}$
(9)

where $\mathbbm{1}[\cdot]$ is an indicator function that equals to 1 if
$f_{\theta}(v_{i},v_{j})>f_{\theta}(v_{k},v_{l})$, otherwise equals to 0.
$f_{\theta}(v_{i},v_{j})=s_{ij}$ represents the output of the neural link
prediction model, where $\theta$ denotes all parameters of the model.
Optimizing AUC is not straightforward since the gradient of this function is
either zero or not defined. Various techniques have been proposed to
approximate the AUC with a surrogate function. There are several possible
selections of surrogate functions, such as pairwise hinge loss, logistic loss
or exponential loss. In this paper, we simply select the square least
surrogate loss, which is proved consistent with AUC theoretically (Gao and
Zhou, 2015). Our framework is flexible to adopt any other surrogate function
that approximates AUC. The base AUC-optimization objective function is defined
as follow:

$O_{\textup{\scriptsize{AUC}}}=\min_{\theta}\sum_{{(v_{i},v_{j})\in
E,(v_{i},v_{k})\in
E^{-}}}\left(1-f_{\theta}\left(v_{i},v_{j}\right)+f_{\theta}\left(v_{i},v_{k}\right)\right)^{2}+\frac{\lambda}{2}||\theta||^{2}$
(10)

The above function forces the margin between positive samples and negative
samples to be 1. In some situations, this constraint is too strict for the
optimization. It can be relaxed by combining the hinge loss with above
function:

$O_{\textup{\scriptsize{Hinge-AUC}}}=\min_{\theta}\sum_{{(v_{i},v_{j})\in
E,(v_{i},v_{k})\in
E^{-}}}\left(\max\left(0,1-f_{\theta}(v_{i},v_{j})+f_{\theta}(v_{i},v_{k})\right)\right)^{2}+\frac{\lambda}{2}||\theta||^{2}$
(11)

The above function only forces the margin between positive samples and
negative samples to be larger than 1.

Furthermore, the margin may not be fixed as 1 if weights on training edges
(positive sample) are expected to be modeled. A straightforward way of
introducing sample weights is as follows:

$O_{\textup{\scriptsize{Weighted-Hinge-
AUC}}}=\min_{\theta}\sum_{{(v_{i},v_{j})\in E,(v_{i},v_{k})\in
E^{-}}}\gamma_{ij}\left(\max\left(0,\gamma_{ij}-f_{\theta}(v_{i},v_{j})+f_{\theta}(v_{i},v_{k})\right)\right)^{2}+\frac{\lambda}{2}||\theta||^{2}$
(12)

where $\gamma$ is an adaptive margin, which may correspond to normalized
weights of training edges.

In above objective functions, to prevent over-fitting problem, we use the L2
regularization on parameters with a weight $\lambda$. Given a positive pair
$(v_{i},v_{j})$ and a sampled negative pair $(v_{k},v_{l})$, the parameters
$\theta$ of the model are optimized by the stochastic gradient descent (SGD)
method. For most cases, we use the basic objective function in Eq. 10. When
sample weights are considered, the objective function of Eq. 12 is used.

###  4.4 Negative Sampling

In practice, true non-linked set $E^{-}$ is not available in the training
data. A conventional strategy is randomly sampling a negative node-pair
$(v_{k},v_{j})$, which has unknown link status and is assumed as negative
samples. For different problems or types of graphs, we may have different
sampling strategies.

Global Sampling. Global sampling represents that, for each positive sample, we
uniformly sample a negative node-pair from the set $E^{?}=V\times V-E$. This
strategy is suitable for the problem seeking for global ranking performance.
For example, in the protein-protein interaction, we are interested in
potential node-pairs, which are worth performing further analysis on, among
all possible node-pairs.

Local Sampling. Local sampling represents that, for a positive sample
$(v_{i},v_{j})$, we firstly select an anchor node saying $v_{i}$, then
uniformly sample a node $v_{k}$ and regard $(v_{i},v_{k})$ as the negative
sample. Instead of uniform distribution, other distribution, e.g, the power of
node degrees, can be applied to sample the negative node $v_{k}$. This
strategy is appropriate to the situation that aims to obtain good ranking for
individual nodes. For example, in a recommendation system, we would like to
recommend a good ranking list of items to each individual user.

Adversarial Sampling. The performance of random sampling strategy is not
always stable due to complete randomness. Similar problems of random negative
sampling have also been found in other tasks, e.g., knowledge graph embedding
(Wang et al., 2018; Cai and Wang, 2017) and image retrieval (Wu et al., 2017).
In our previous work (Wang et al., 2020), we proposed to use adversarial
learning technique to generate negative samples instead of random sampling. We
designed a generative model to generate high quality negative samples, which
aims at making difficulties to link prediction model. In this way, link
prediction model and negative sample generator play an adversarial game. By
continuously providing high quality negative samples, adversarial sampling
more robust than random sampling. We leave the evaluation of adversarial
sampling on ogb datasets as future work.

Negative Sample Sharing. Since the framework adopts pairwise schema, each
negative sample can only be used for one positive sample once, which is not
efficient. To make better use of negative samples, we propose a negative
sample sharing mechanism. As shown in Figure 2, assume that the total number
of positive sample is $m$, we firstly draw $m$ negative samples and construct
$m$ training pairs with same indexes. Given the negative samples, the sharing
mechanism will random permute the indexes of negative samples, and form $m$
new training pairs. The hyper-parameter num_neg indicates the mechanism will
random permute (num_neg-1) times of negative samples. By using this sharing
mechanism, we could create $m\times\texttt{num\\_neg}$ training pairs by only
sampling $m$ negative samples at each training epoch.

###  4.5 Data Augmentation with Random Walk

In some graphs, high-order structure information play a important role.
Although increasing the number of GNN layers could model the high-order
information, it also may leads to over-smoothing problem and low efficiency.
To this end, we propose to use data augmentation to introduce high-order
information at the input. A general technique to sample high order information
is the Random Walk. Given all nodes in the graph, we use the basic random walk
method to sample the high-order pairs. Assume the start point node is $v_{i}$
and its random walk is $\textup{RW}(v_{i})=\\{v_{k+1},...,v_{k+l}\\}$, where
$l$ represents the walk length , then the set of positive samples is augmented
as :

$E_{\textup{aug}}=E\cup\\{(v_{i},v_{j})|v_{j}\in\textup{RW}(v_{i}),\forall
v_{i}\in V\\}$ (13)

Meanwhile, the augmented pairs are associated with weights based on the steps
of walks. For example, in the walk,
$\textup{RW}(v_{i})=\\{v_{k+1},...,v_{k+l}\\}$, the augmented pair
$(v_{i},v_{k+l})$ is associated with the weight $1/l$. With different weights
of augmented pairs in $E_{\textup{aug}}$, we find that using the weight-
adaptive objective function in Eq. 12 is more effective. Therefore, it is
suggested using this objective function when random walk augmentation is
adopted.

Figure 2: Negative Sample Sharing Mechanism

##  5 Evaluation on OGB

Our code for evaluation is available at https://github.com/zhitao-wang/PLNLP.

###  5.1 Datasets and Evaluation Metrics

We evaluate the link prediction ability of PLNLP on Open Graph Benchmark (OGB)
data (Hu et al., 2020). Four data sets with different graph types are
evaluated, including ogbl-ddi, ogbl-collab, ogbl-citation2 and ogbl-ppa.

ogbl-ddi: The dataset is a homogeneous, unweighted, undirected graph,
representing the drug-drug interaction network. Each node represents a drug.
Edges represent interactions between drugs.

The task is to predict drug-drug interactions given information on already
known drug-drug interactions. The performance is evaluated by Hits@20: each
true drug interaction is ranked among a set of approximately 100,000 randomly-
sampled negative drug interactions, and count the ratio of positive edges that
are ranked at 20-place or above.

ogbl-collab: The dataset is an undirected graph, representing a subset of the
collaboration network between authors indexed by MAG. Each node represents an
author and edges indicate the collaboration between authors. All nodes come
with 128-dimensional features.

The task is to predict the future author collaboration relationships given the
past collaborations. Evaluation metric is Hits@50, where each true
collaboration is ranked among a set of 100,000 randomly-sampled negative
collaborations.

ogbl-ppa: The dataset is an undirected, unweighted graph. Nodes represent
proteins from 58 different species, and edges indicate biologically meaningful
associations between proteins.

The task is to predict new association edges given the training edges.
Evaluation metric is Hits@100, where each positive edge is ranked among
3,000,000 randomly-sampled negative edges.

ogbl-citation2: The dataset is a directed graph, representing the citation
network between a subset of papers extracted from MAG. Each node is a paper
with 128-dimensional word2vec features.

The task is to predict missing citations given existing citations. The
evaluation metric is Mean Reciprocal Rank (MRR), where the reciprocal rank of
the true reference among 1,000 negative candidates is calculated for each
source paper, and then the average is taken over all source papers.

###  5.2 Evaluation Settings and Results

The detailed settings of PLNLP in this paper are shown in Table 1. We only
employ basic node level neighborhood encoder, e.g., GCN or SAGE, to
demonstrate the effectiveness of the proposed framework. Some well-design edge
level neighborhood encoder specific for link prediction, such as SEAL, NANs
and HalpNet, may further improve the performance. But due to low efficiency of
edge level neighborhood encoders, we leave this part in the future work. We
treat all training datasets as unweighted and undirected graphs. As for MLP
predictor, we use hadamard product to get the input of MLP. It is worth noting
that we use validation set for training on ogbl-collab, which is allowed by
OGB. Meanwhile, we employ the trick from HOP-REC that we only use training
edges after year 2010 in ogbl-collab. Furthermore, we use random walk
augmentation with a walk length 10 for ogbl-collab.

Table 1: Settings of PLNLP on OGB Datasets | Loss Func. | Encoder | Predictor | Neg. Sampler |  Other Parameters
---|---|---|---|---|---
ddi | $O_{\textup{\scriptsize{AUC}}}$ |  SAGE layer = 2 dim = 512 dropout = 0.3 |  MLP layer = 2 dim = 512 dropout = 0.3 |  GLOBAL num_neg = 3 |  node emb. = 512 node feat. = no lr = 0.001 epochs = 500
collab | $O_{\textup{\scriptsize{Weighted-Hinge-AUC}}}$ |  SAGE layer = 1 dim = 256 dropout = 0.3 | DOT |  GLOBAL num_neg = 1 |  node emb. = 256 node feat. = no lr = 0.001 epochs = 800 random walk aug. = yes walk length = 10
ppa | $O_{\textup{\scriptsize{AUC}}}$ |  SAGE layer = 2 dim = 256 dropout = 0.3 | DOT |  GLOBAL num_neg = 3 |  node emb. = 256 node feat. = yes lr = 0.001 epochs = 200
citation2 | $O_{\textup{\scriptsize{AUC}}}$ |  GCN layer = 2 dim = 200 dropout = 0.0 |  MLP layer = 2 dim = 200 dropout = 0.0 |  LOCAL num_neg = 3 |  node emb. = 50 node feat. = yes lr = 0.001 epochs = 100

Following OGB rules, we evaluate PLNLP with 10 runs, without fixing random
seed. As for other state-of-the-art methods, we just copy the results from OGB
official leader board.

Table 2: Link Prediction Performance on OGB (Test Performance) |  ogbl-ddi Hits@20(%) |  ogbl-collab Hits@50(%) |  ogbl-ppa Hits@100(%) |  ogbl-citation2 MRR(%)
---|---|---|---|---
CN | $17.73\pm 0.00$ | $61.37\pm 0.00$ | $27.65\pm 0.00$ | $51.47\pm 0.00$
AA | $18.61\pm 0.00$ | $64.17\pm 0.00$ | $32.45\pm 0.00$ | $51.89\pm 0.00$
RA | $-$ | $-$ | $49.33\pm 0.00$ | $-$
AA+Proposal Set | $-$ | $65.48\pm 0.00$ | $-$ | $-$
RA+Proposal Set | $-$ | $-$ | $\textbf{53.24}\pm\textbf{0.00}$ | $-$
MF | $13.68\pm 4.75$ | $38.86\pm 0.29$ | $32.29\pm 0.94$ | $51.86\pm 4.43$
DeepWalk | $22.46\pm 2.90$ | $50.37\pm 0.34$ | $23.02\pm 1.63$ | $-$
Node2vec | $23.26\pm 2.09$ | $48.88\pm 0.54$ | $22.26\pm 0.83$ | $-$
HOP-REC | $-$ | $70.12\pm 0.16$ | $-$ | $-$
SAGE | $53.90\pm 4.74$ | $54.63\pm 1.12$ | $16.55\pm 2.40$ | $82.60\pm 0.36$
GCN | $37.07\pm 5.07$ | $47.14\pm 1.45$ | $18.67\pm 1.32$ | $84.74\pm 0.31$
SEAL | $30.56\pm 3.86$ | $64.74\pm 0.33$ | $48.80\pm 3.16$ | $\textbf{87.67}\pm\textbf{0.32}$
SAGE+Proposal Set | $74.95\pm 3.17$ | $-$ | $-$ | $-$
CFLP (w/ JKNet) | $86.08\pm 1.98$ | $-$ | $-$ | $-$
SAGE+Edge Attr | $87.81\pm 4.47$ | $-$ | $-$ | $-$
PLNLP | $\textbf{90.88}\pm\textbf{3.13}$ | $\textbf{70.59}\pm\textbf{0.29}$ | $32.38\pm 2.58$ | $\textbf{84.92}\pm\textbf{0.29}$

The averaged results with standard deviation are reported in the Table 2. Only
with basic graph neural architectures, PLNLP achieves top 1 performance on
ogbl-ddi and ogbl-collab, and top 2 performance on ogbl-ciation2. This
significantly demonstrates the effectiveness of PLNLP.

###  5.3 Ablation Study

Furthermore, we compare PLNLP against the generic classification learning
framework where the loss function is cross-entropy. In this ablation study, we
keep same neural architecture (same encoder, predictor with same parameters as
reported in Table 1) in the two frameworks. We use basic AUC objective
function Eq.10 for all datasets and do not use random walk augmentation for
ogbl-collab in this study. To guarantee fairness, we use same negative
sampling strategies and use the same number of negative samples at each epoch.
The results are shown in Table 3. It is found that PLNLP remarkably
outperforms the generic classification learning schema, which indicates the
proposed pairwise learning could maximize the performance of graph neural
models on link prediction problem.

Table 3: PLNLP vs. Classification Schema |  ogbl-ddi Hits@20(%) |  ogbl-collab Hits@50(%) |  ogbl-ppa Hits@100(%) |  ogbl-citation2 MRR(%)
---|---|---|---|---
| Test | Valid | Test | Valid | Test | Valid | Test | Valid
Classification | 70.70 | 68.02 | 64.97 | 99.17 | 16.55 | 17.24 | 84.64 | 84.73
PLNLP | 90.88 | 82.42 | 68.72 | 100.00 | 32.38 | 31.62 | 84.92 | 84.90

#### Acknowledgments

The authors greatly thank the great support for advanced research from
departments of WeChat Pay and WeChat Search.

## References

  * Adamic and Adar [2003] Lada A Adamic and Eytan Adar.  Friends and neighbors on the web.  _Social networks_ , 25(3):211–230, 2003.
  * Airoldi et al. [2008] Edoardo M Airoldi, David M Blei, Stephen E Fienberg, and Eric P Xing.  Mixed membership stochastic blockmodels.  _Journal of Machine Learning Research_ , 9(Sep):1981–2014, 2008.
  * Cai and Wang [2017] Liwei Cai and William Yang Wang.  Kbgan: Adversarial learning for knowledge graph embeddings.  _arXiv preprint arXiv:1711.04071_ , 2017.
  * Gao and Zhou [2015] Wei Gao and Zhi-Hua Zhou.  On the consistency of auc pairwise optimization.  In _Twenty-Fourth International Joint Conference on Artificial Intelligence_ , 2015.
  * Grover and Leskovec [2016] Aditya Grover and Jure Leskovec.  node2vec: Scalable feature learning for networks.  In _Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 855–864. ACM, 2016.
  * Hamilton et al. [2017] Will Hamilton, Zhitao Ying, and Jure Leskovec.  Inductive representation learning on large graphs.  In _Advances in neural information processing systems_ , pages 1024–1034, 2017.
  * Hu et al. [2020] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec.  Open graph benchmark: Datasets for machine learning on graphs.  _arXiv preprint arXiv:2005.00687_ , 2020.
  * Jeh and Widom [2002] Glen Jeh and Jennifer Widom.  Simrank: a measure of structural-context similarity.  In _Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 538–543. ACM, 2002\.
  * Kipf and Welling [2016a] Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  _arXiv preprint arXiv:1609.02907_ , 2016a.
  * Kipf and Welling [2016b] Thomas N Kipf and Max Welling.  Variational graph auto-encoders.  _NIPS Workshop on Bayesian Deep Learning_ , 2016b.
  * Koren et al. [2009] Yehuda Koren, Robert Bell, and Chris Volinsky.  Matrix factorization techniques for recommender systems.  _Computer_ , (8):30–37, 2009.
  * Liben-Nowell and Kleinberg [2007] David Liben-Nowell and Jon Kleinberg.  The link-prediction problem for social networks.  _Journal of the American society for information science and technology_ , 58(7):1019–1031, 2007.
  * Lü and Zhou [2011] Linyuan Lü and Tao Zhou.  Link prediction in complex networks: A survey.  _Physica A: statistical mechanics and its applications_ , 390(6):1150–1170, 2011.
  * Martínez et al. [2017] Víctor Martínez, Fernando Berzal, and Juan-Carlos Cubero.  A survey of link prediction in complex networks.  _ACM Computing Surveys (CSUR)_ , 49(4):69, 2017\.
  * Menon and Elkan [2011] Aditya Krishna Menon and Charles Elkan.  Link prediction via matrix factorization.  In _Joint european conference on machine learning and knowledge discovery in databases_ , pages 437–452. Springer, 2011.
  * Perozzi et al. [2014] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena.  Deepwalk: Online learning of social representations.  In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 701–710. ACM, 2014.
  * Salton and McGill [1986] Gerard Salton and Michael J McGill.  Introduction to modern information retrieval.  1986\.
  * Tang et al. [2015] Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei.  Line: Large-scale information network embedding.  In _Proceedings of the 24th International Conference on World Wide Web_ , pages 1067–1077. International World Wide Web Conferences Steering Committee, 2015.
  * Wang et al. [2018] PeiFeng Wang, Shuangyin Li, and Rong Pan.  Incorporating gan for negative sampling in knowledge representation learning.  In _AAAI_ , 2018.
  * Wang et al. [2017] Zhitao Wang, Chengyao Chen, and Wenjie Li.  Predictive network representation learning for link prediction.  In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 969–972. ACM, 2017\.
  * Wang et al. [2019] Zhitao Wang, Yu Lei, and Wenjie Li.  Neighborhood interaction attention network for link prediction.  In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_ , pages 2153–2156, 2019.
  * Wang et al. [2020] Zhitao Wang, Yu Lei, and Wenjie Li.  Neighborhood attention networks with adversarial learning for link prediction.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2020\.
  * Wang et al. [2021] Zhitao Wang, Wenjie Li, and Hanjing Su.  Hierarchical attention link prediction neural network.  _Knowledge-Based Systems_ , 232:107431, 2021.  ISSN 0950-7051.  doi: https://doi.org/10.1016/j.knosys.2021.107431.  URL https://www.sciencedirect.com/science/article/pii/S0950705121006936.
  * Wu et al. [2017] Chao-Yuan Wu, R Manmatha, Alexander J Smola, and Philipp Krahenbuhl.  Sampling matters in deep embedding learning.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 2840–2848, 2017.
  * Zhang and Chen [2017] Muhan Zhang and Yixin Chen.  Weisfeiler-lehman neural machine for link prediction.  In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , pages 575–583. ACM, 2017.
  * Zhang and Chen [2018] Muhan Zhang and Yixin Chen.  Link prediction based on graph neural networks.  In _Advances in Neural Information Processing Systems_ , pages 5165–5175, 2018.
  * Zhang et al. [2018] Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen.  An end-to-end deep learning architecture for graph classification.  In _Thirty-Second AAAI Conference on Artificial Intelligence_ , 2018\.
  * Zhou et al. [2009] Tao Zhou, Linyuan Lü, and Yi-Cheng Zhang.  Predicting missing links via local information.  _The European Physical Journal B_ , 71(4):623–630, 2009.