# Known by the company we keep: ‘Triadic influence’ as a proxy for
compatibility in social relationships

Miguel Ruiz-García  Departamento de Matemáticas, Universidad Carlos III de
Madrid, 28911 Leganés, Madrid, Spain  Juan Ozaita  Departamento de
Matemáticas, Universidad Carlos III de Madrid, 28911 Leganés, Madrid, Spain
Grupo Interdisciplinar de Sistemas Complejos (GISC), Universidad Carlos III de
Madrid, 28911 Leganés, Madrid, Spain  María Pereda  Grupo Interdisciplinar de
Sistemas Complejos (GISC), Universidad Carlos III de Madrid, 28911 Leganés,
Madrid, Spain  Grupo de Investigación Ingeniería de Organización y Logística
(IOL), Departamento Ingeniería de Organización, Administración de empresas y
Estadística, Escuela Técnica Superior de Ingenieros Industriales, Universidad
Politécnica de Madrid, 28006 Madrid, Spain  Antonio Alfonso  LoyolaBehLAB &
Department of Economics, Universidad Loyola Andalucía, 14004 Córdoba, Spain
Pablo Brañas-Garza LoyolaBehLAB & Department of Economics, Universidad Loyola
Andalucía, 14004 Córdoba, Spain  José A. Cuesta  Departamento de Matemáticas,
Universidad Carlos III de Madrid, 28911 Leganés, Madrid, Spain  Grupo
Interdisciplinar de Sistemas Complejos (GISC), Universidad Carlos III de
Madrid, 28911 Leganés, Madrid, Spain  Instituto de Biocomputación y Física de
Sistemas Complejos (BIFI), Universidad de Zaragoza, 50018 Zaragoza, Spain
Angel Sánchez  Departamento de Matemáticas, Universidad Carlos III de Madrid,
28911 Leganés, Madrid, Spain  Grupo Interdisciplinar de Sistemas Complejos
(GISC), Universidad Carlos III de Madrid, 28911 Leganés, Madrid, Spain
Instituto de Biocomputación y Física de Sistemas Complejos (BIFI), Universidad
de Zaragoza, 50018 Zaragoza, Spain

###### Abstract

Networks of social interactions are the substrate upon which civilizations are
built. Often, we create new bonds with people that we like or feel that our
relationships are damaged through the intervention of third parties. Despite
their importance and the huge impact that these processes have in our lives,
quantitative scientific understanding of them is still in its infancy, mainly
due to the difficulty of collecting large datasets of social networks
including individual attributes. In this work, we present a thorough study of
real social networks of 13 schools, with more than 3,000 students and 60,000
declared positive and negative relations, including tests for personal traits
of all the students. We introduce a metric—the ‘triadic influence’—that
measures the influence of nearest-neighbors in the relationships of their
contacts. We use neural networks to predict the relationships and to extract
the probability that two students are friends or enemies depending on their
personal attributes or the triadic influence. We alternatively use a high-
dimensional embedding of the network structure to also predict the
relationships. Remarkably, the triadic influence (a simple one-dimensional
metric) achieves the highest accuracy at predicting the relationship between
two students. We postulate that the probabilities extracted from the neural
networks—functions of the triadic influence and the personalities of the
students—control the evolution of real social networks, opening a new avenue
for the quantitative study of these systems.

Suggested keywords

Positive relationships help individuals thrive in society, whereas negative
ones can jeopardize our chances of success and happiness. Social relationships
arise from interactions between individuals and have been studied on different
time scales and contexts [1, 2]. As a result, social networks are formed, with
individuals as nodes and interactions as links [3], and they can be studied
and characterized using a complex network approach [4] in order to assess the
many implications of social structure in our lives [5]. A great deal of
research has been carried out on social networks by aggregating the
interactions that occur over a certain period of time to define links,
starting from the pioneering work of Moreno [6]. However, such an approach
does not capture the dynamics of relationships, which is necessary to advance
our understanding of the field [7]. Large efforts have been devoted to this
question in recent years, mainly using empirical data with different degrees
of time resolution, such as, e.g., letter exchanges [8], mobile phone
communications [9, 10], spatial mobility [11], or face-to-face interactions
[12, 13, 14] (see also Ref. [15] for a review). All these analyses have led to
many interesting insights on the evolution of relationships, but the issue of
the mechanisms that explain how/why these relationships are created and evolve
remains elusive.

Several models have been proposed to explain different aspects of the
empirical observations. The first attempts were devoted to reproduce some of
the structural properties observed in social networks, such as the small world
phenomena [16] or the rich-get-richer effect [17, 18]. Starnini et al. [19]
proposed a simple model based on random walks and individual attractiveness to
describe face-to-face interactions. For social networks, Jin et al. [20]
studied networks with exponential decay of tie strengths to represent
friendships. Other approaches have resorted to exponential random graph models
[21] or stochastic actor-oriented models [22]. Finally, regression models that
incorporate a selection of individual traits have also been considered for
online social networks [23]. Still, none of these approaches sheds light on
friendship formation in real life, taking into account the characteristics of
the individuals and how some relationships can influence others.

In this paper, we contribute towards the understanding of friendship formation
by adopting a different point of view, namely that of link prediction in
networks [24]. The problem of link prediction, as originally formulated, is
about temporal networks: given the graph of connections between certain
entities or nodes during some interval, the task is to predict the set of
links in a later interval. Notwithstanding this definition, the same idea
applies to many different situations, such as recommendation systems [25],
bioinformatics [26], scientific collaboration networks [27], criminal networks
[28], or even estimating the reliability of network data [29], to name a few.
In the case of online social networks, link prediction has been considered,
for example, by Song et al. [30] or Hao [31] (see Ref. [32] for a review).
Much less has been explored regarding real-world social networks, in
particular friendship networks [33], due to the difficulty of collecting data
on reasonably complete social networks that include personal attributes in
real settings. For this reason, the discussion has been devoted in many cases
to ego-networks (i.e., data on disconnected individuals who mentioned their
friends) and to the meaning of friendship [34].

In this work we study social networks collected in $13$ complete high schools
in Spain, containing more than $3,000$ individuals and $60,000$ declared
relationships between them. All students completed tests including information
about their self-declared gender, cognitive results, and other variables that
measured their selfishness/prosociality. Performing link prediction on this
data, we are able to extract the probability that two students will be
friends/enemies depending on their personalities. We also studied how this
probability is affected by other relationships, defining a metric that we have
termed _triadic influence_. Although we analyze static networks, our results
suggest that the probabilities that we extract determine the mechanisms that
control the initial formation of relationships and the evolution of the whole
social network.

## Results

Data collection was carried out in $13$ schools in different areas of Spain,
with a total of $3,395$ students. They were asked to choose with whom they
were related within their school by picking names from a school list. Then
they had to rate the relationship as very bad, bad, good, or very good, which
we codified as $-2$, $-1$, $+1$, and $+2$, respectively. We recovered $60,566$
declared relationship, see Supporting Information (SI) for more details. In
addition, we also collected data on the students’ gender (self-reported),
cognitive skills (measured by the cognitive reflection test, CRT), and their
prosociality (see Methods for details on these individual features). With this
information, we build a directed weighted network, with each link representing
a relationship that goes from the nominator to the nominee—two nodes can be
connected by links in both directions—weighted by the reported rating.
Additionally, each node represents one student and has his/her individual
attributes (gender, CRT and prosociality). Figure 1 presents a sketch of the
kind of social network that we will study. We have included several figures
studying the structure of the social networks in the Supporting Information,
see figures S1 to S3.

In this work, we study the correlations between the personal features of both
students and the type of relationship between them, as well as the influence
of other students on that relationship. We have used artificial neural
networks to perform link prediction within our dataset from two complementary
viewpoints: the first one focuses on the local structure, using the
personality traits of both students and the influence of the nearest-neighbors
as described in the next section; the second one uses only the structural
information of the network – the undirected and unweighted graph – to predict
relationships. In what follows, we discuss these two approaches separately.

Figure 1: Diagram of a social network that includes personality traits and
computation of the triadic influence. To predict the relationship from node
$0$ to node $1$ we can use the individual features of both students
(represented by the sliders within their body) and / or the triadic influence
$I_{01}$. The directions of these relationships are marked by arrows going
from the nominator to the nominee, whereas the weight/intensity is represented
with colors and edge labels (dark green: close friend, green: friend, yellow:
dislike, orange: enemy). The thick arrows highlight the relationships that
enter the calculation of $I_{01}$. To compute $I_{01}$ we select all directed
paths of length $2$ from node $0$ to node $1$
($0\rightarrow\text{node}\rightarrow 1$). In this example, they are 0-5-1 and
0-6-1. The path 0-3-1 is not a directed path (the direction of the edges is
$0\rightarrow 3\leftarrow 1$) and therefore is not included in the calculation
of $I_{01}$. Thus, $I_{01}=w_{05}w_{51}+w_{06}w_{61}=2\cdot 2+(-1)\cdot 2=2$.

### Predicting with the personality traits and the influence of the nearest-
neighbors

Figure 1 shows a sketch of the social network with all the information
available to perform link prediction. It shows the students (nodes) with their
traits (sliders) and relationships of different types between them. In this
section, we use only local properties of the network to predict the
relationship between two students, namely the individual features of both
students (e.g. nodes $0$ and $1$ in Fig. 1) and the directed weighted paths of
length $2$ between them. Specifically, we define a variable that we term
triadic influence as
$I_{ij}\equiv\left(W^{2}\right)_{ij}=\sum_{k}w_{ik}w_{kj}$, where $w_{ik}$ is
the weight of the link that goes from node $i$ to node $k$ (see Fig. 1 for an
example). The triadic influence condenses into one scalar the influence of
third parties; e.g., if node $i$ declares node $k$ as a friend and $k$ does
the same with $j$, it adds a positive number to $I_{ij}$ (your friend’s
friends are likely to be your friends), whereas a path containing links of
opposite sign will lead to a negative contribution (your enemy’s friends or
your friend’s enemies are likely to be your enemies). $I_{ij}$ adds up the
contribution from all directed paths of length $2$ between $i$ and $j$.

For simplicity, we will train a neural network to correctly classify all
relationships in the network into two classes: friends and enemies (see
Methods for more details). We used different combinations of the triadic
influence and the individual characteristics of the students as input for the
deep neural network (NN) and trained it to output the correct value for each
relationship in the training dataset (see Methods for a full description of
the neural network and the training process). With our procedure, we obtain
the probability that two students relate through a relationship belonging to
one of the two classes (friends or enemies) as a function of the corresponding
inputs. To avoid using a misleading metric of performance, since our classes
are unbalanced—there are many more declared friends than enemies—we assess the
performance of our method using the balanced accuracy on the test dataset. To
compute it, after training the NN we feed it with all relations in the test
dataset and assign the label “friend” or “enemy” to the class with the highest
probability. The balanced accuracy is then computed as

$\textrm{bAcc}=\frac{1}{2}\left(\frac{N_{+}^{C}}{N_{+}^{T}}+\frac{N_{-}^{C}}{N_{-}^{T}}\right),$

where $N_{\alpha}^{C}$ is the number of samples belonging to class $\alpha$
($+$ friend, or $-$ enemy) that were correctly classified from the total
number of samples belonging to that class ($N_{\alpha}^{T}$). This is more
informative than other performance metrics because if either the NN classified
everything in the same class or guessed at random, we would obtain
$\textrm{bAcc}=1/2$ regardless of the number of samples in each class, whereas
if all relations were correctly predicted, then $\textrm{bAcc}=1$ (see
Methods).

Figure 2 collects the accuracies achieved using the NN to predict the
relationships between students with different combinations of predictors. We
first study relationships $i\to j$ with non-zero triadic influence (i.e., with
at least one directed path of length $2$ from $i$ to $j$; see Fig. 1 and
Methods for more details). The results are shown in the four upper bars of
Fig. 2 (see the SI for the distribution of relationships per number of
directed paths of length $2$, Fig. S2). We train the classifier using four
sets of predictors: (1) triadic influence and personal information (gender,
CRT, and prosociality) of the pair of nodes, (2) triadic influence, (3)
personal information, and (4) only students’ prosociality. Just as a
clarification, in case (1) we use as input for the NN the triadic influence (a
scalar) and the individual traits of both students (a $6$-dimensional array)
to predict the correct label of that relation (friend or enemy). See Methods
for a detailed explanation about how the value of the considered features:
gender, CRT, and prosociality, are gathered and computed.

Figure 2: Balanced test accuracy for different choices of information used to
train the NN. Purple bars correspond to relationships where there is at least
one directed path of length $2$ from $i$ to $j$ ($(A^{2})_{ij}>0$, $A_{ij}$
being the adjacency matrix of the network). We train the classifier using four
sets of predictors: (1) triadic influence and personal information (gender,
CRT, and prosociality), (2) triadic influence alone, (3) personal information
alone, (4) just students’ prosociality. In all four cases we trained $10$
different NN with random initializations and show here the mean bAcc. Yellow
bars correspond to the bAcc for relationships that have no directed paths of
length $2$. In this case, we use just two sets of predictors: (5) personal
information and (6) students’ prosociality. These cases use $10$-fold cross-
validation to estimate the performance of the prediction. Error bars represent
the standard error of the mean in all cases.

The highest balanced accuracy, $86$%, is achieved using the triadic influence
as input, either in combination with personal information of both students (1)
or alone (2). It is remarkable that such a high accuracy for the prediction of
the nature of a relationship (friend/enemy) can be obtained with just a scalar
(the triadic influence), and that a 6-dimensional array containing information
about both students’ characteristics does not improve on that. This suggests
that the information on the individual features is already encoded in the
network of relationships and captured by the triadic influence. The rationale
for this is the following: if individuals $i$ and $j$ are both very prosocial,
then $i$ will be friends with $j$, as well as with many other intermediate
individuals, with high probability; and these intermediate individuals will
very likely be friends with $j$, since $j$ is also very prosocial. Hence,
$I_{ij}$ already encodes information about the prosociality of $i$ and $j$.
This suggests that $I_{ij}$ may act as a proxy for personal compatibility when
individual traits are not available.

On the other hand, using only the personal traits of both students (3) yields
$\textrm{bAcc}=60$%. We studied the three attributes (gender, CRT, and
prosociality) separately, and prosociality turned out to be the most
predictive. In fact, using only students’ prosociality to predict their
relationship (4) already yields $\textrm{bAcc}=57$%. Both figures are above
the accuracy that a random guess would reach ($50$%). Note that prosociality
is calculated with students’ answers to three simple questions (see Methods).
It is really remarkable that such a simple metric is already predictive for
the nature of the social relationship between two individuals.

Finally, we study separately the relationships that do not have directed paths
of length $2$ connecting $i$ to $j$ (i.e. $\left(A^{2}\right)_{ij}=0$, with
$A_{ij}$ the adjacency matrix of the network); therefore, there is no triadic
influence between $i$ and $j$. These results are shown by the two bottom bars
of Fig. 2. Since this dataset is much smaller ($2$% of all relationships, i.e.
$1,211$ out of a total of $60,566$; see the SI Fig. S2 for more details), we
assess the performance of the classifier using 10-fold cross-validation to
ensure that our results are robust. We study two sets of predictors: (5) the
complete personal information of the students (gender, CRT and prosociality)
and (6) just the prosociality. The mean bAcc for the 10 realizations within
10-fold cross-validation is $57$% for (5) and $55$% for (6). Note that the
mean bAcc seems to decrease compared to the case when
$\left(A^{2}\right)_{ij}>0$ (purple bars), although the significance of this
difference is low given that error bars corresponding to cases (3) and (5),
and (4) and (6) either overlap or are very close.

### Interpreting the probabilities learned by the neural network

It is important to note that until now we have chosen to assess the
performance of our prediction using bAcc for the sake of simplicity. However,
the NN learns more than this; in particular, it learns to predict the
probability that a relationship belongs to each of the classes in the dataset
(see Methods for a detailed explanation on how this is achieved through the
minimization of the cross-entropy loss function). The great advantage of using
low-dimensional inputs is that we can interpret what the NN is learning. We
can plot the probability that a sample belongs to a class (friend/enemy) as a
function of the different predictors. In Fig. 3 (a) we plot this probability
as a function of the triadic influence. We use the $10$ different NNs trained
for Fig. 2 (2) and plot the average probability of being friends and enemies
for a pair of students with a given triadic influence. The colored area around
both curves represents the standard deviation of the probabilities. The
probability of being friends saturates to $1$ when the triadic influence
$I_{ij}\gg 1$, and drops to $0$ if the triadic influence $I_{ij}\lesssim 0$
(the probability of being enemies is the complementary because both add up to
$1$). The probability curves of being friends and being enemies cross around
$I_{ij}\approx 5$. Note that this is the only information used when computing
the accuracy bAcc, because we identify each relationship with the most
probable one, as predicted by the NN. However, the probabilities learned by
the neural network (which minimize the cross-entropy loss, see Methods)
contain much more information and could be used to generate ensembles of
social networks or to simulate their evolution using stochastic Markov chains.
It is worth mentioning that, although the probability curves change abruptly
around $I_{ij}\approx 0$, this change slows down as the triadic influence
increases, thus displaying an asymmetric behavior on both sides of the
crossing point $I_{ij}\approx 5$.

Figures 3 (b) and (c) display the probability of being enemies and friends,
respectively, as a function of the prosociality of both students
(nominator/nominee), averaged over the $10$ simulations used for case (4) of
Fig. 2. Similarly to the case of the triadic influence, even though bAcc is
fully determined by the curve where the probability is $0.5$, the profiles
shown in these figures convey much more information. In particular, we can see
that the probability that two students with $0$ prosociality are enemies is
$70\%$, which is in line with what one would expect: selfish people declare to
have more enemies and are declared enemies more often than altruists (see the
Supporting Information, where this can also be directly observed in the raw
data, Fig. S4). Alternatively, two highly prosocial students are friends with
a probability higher than $60\%$. Note also that both colormaps are
approximately symmetric with respect to the diagonal. This implies
reciprocity: the probability that $i$ declares $j$ as a friend is
approximately the same as the probability that $j$ does the same with $i$.

Figure 3: Probabilities of being friends/enemies as a function of the triadic
influence and prosociality. Panel (a) shows the probability learned by the NN
as a function of the triadic influence. We performed 10 simulations that led
to the accuracy shown in the (2) bar in Fig. 2. Continuous lines in panel (a)
correspond to the mean, whereas the shaded area correspond to one standard
error of the mean. Panel (b) and (c) display the mean probabilities learnt by
the 10 NN used in Fig. 2 (4), they show the probability of having a
friendly/enmity relationship as a function of the prosociality of both
students, the nominator (from) and nominee (to). Both probabilities are
normalized to $1$.

### Predicting with the structural information of the social network alone

Figure 4: Distribution of balanced accuracy for the 13 high schools. Each
histogram is composed of a sample of $N=390$ points, which are different
simulations for the same treatment. The histograms are normalized so that the
area under the curve is $1$. Purple (dark) histogram represents _treatment I_
(see Methods), where we use a random pick of nodes as test set. Orange (light)
histogram represents _treatment II_ , where we pick a specific course from the
high school as the test set. The darker orange in the figure represents the
overlapping in the distribution. The same figure for the Random Forest method
can be checked in Fig.11.

In the previous sections we use local information—individual features and
triadic influence—to predict relationships. Complementary to this, in this
section we will attempt to make the same predictions using only the structure
of the network—excluding weights, link directions, and individual
features—hoping to shed light on the role played by the structure of the
network for the creation of different relationships. We will merge labels
$\\{+1,+2\\}$ into a unique “friends” label, and labels $\\{-1,-2\\}$ into a
unique “enemies” label, so that predictions can be binary. In order to do that
we will create _node embeddings_ by assigning to each node a $d$-dimensional
array of features—which will replace the array of individual features used in
the previous section. $128$-dimensional embeddings are created with Node2Vec
[35], an algorithm that explores the neighborhood of each node using biased
random walks (see Methods for more details and figures S5 and S6 of the SI).
The embeddings of all nodes are then used as inputs to train different models,
in order to predict the relationships in the network. We show here the case
where we train a neural network, although we have also used Random Forests
(see the SI, Fig. S7) obtaining similar results.

We create the embeddings for all nodes once and keep them throughout. We then
train a neural network to predict the relationship between pairs of students
(friends/enemies) using both their embeddings as input. This is akin to using
the individual features of both students in the previous section, only this
time embeddings encode information about the environment surrounding each
node. We have trained and tested the neural network using two alternative
treatments: in _treatment I_ we have chosen at random $20\%$ of the
relationships from all high schools as the test dataset, and trained the
neural network using the rest of the relationships; in _treatment II_ we have
created a test dataset with all the relationships inside one specific age
level from one high school, and trained the model using all the other
relationships. For treatment I, we trained the neural network 390 times, every
time changing the train and test datasets as well as the initialization of the
neural network (the embeddings do not change). For treatment II, there are 39
different age levels within the 13 high schools that we study, and we trained
10 different neural networks for each course—390 simulations in total.

The results corresponding to treatments I and II are summarized in Fig. 4. In
this figure we show the accuracy as a histogram after carrying out treatments
I and II for the 390 simulations—blue and orange bars, respectively. For
treatment I, where we train and test on random relationships, the average
accuracy is $\sim 75\%$, and the accuracy is always above $60\%$ (purple
bars). However, when we test on a complete age level that was excluded from
the training dataset the performance degrades (treatment II), the mean
accuracy is now $\sim 60\%$ (orange bars), and there are many instances where
the model is not doing better than a dummy model (bAcc $\sim 50\%$).

The fact that the model has predictive power using only structural information
shows that there is a structural difference between the environments of
friendly or adversarial relationships. Besides, since the predictive power of
the model decreases when testing on an isolated age level, this suggests that
the structure of most age levels contain specific information that is not
present in the rest of the data. In other words, some of the information that
is contained within an age level, and stored in the embedding of the students,
seems to be exclusive to that level.

## Discussion

In this paper, we have applied techniques for link prediction to gain insight
into the mechanisms behind the formation and evolution of social networks.
This has been possible due to the large amount of data that we have collected,
comprising individual features of more than $3,000$ students as well as their
corresponding network of personal relationships—over $60,000$ connections. The
picture of the network dynamics that emerges from our work is as follows. Some
initial relationships appear between pairs of students, promoted by their
prosocial stance. As a matter of fact, we have shown that the prosociality of
both students by themselves are capable of predicting isolated relationships
significantly better than a pure random guess. This is actually a very strong
claim, because many of those initial relationships are now hidden among many
other relationships that emerged afterwards, and the isolated ones that we can
find now are probably very sensitive to noise or trolling (e.g. students that
label randomly other peers as friends/enemies). We hypothesize that isolated
relationships continue to emerge until directed paths of length $2$ dominate
the dynamics of network formation. As discussed in previous sections, paths of
length $2$ are equivalent to intermediate students who can get two of their
contacts in touch with each other. This mediation, quantified by the triadic
influence, is an extremely good predictor of relationships, with accuracies as
high as 86%. Interestingly, when we focus on relationships that are not
isolated (there are directed paths of length 2 connecting both students),
prosociality is still a good predictor of them. This suggests that some of
these relationships might have originated as isolated relationships, and that
prosociality is still important even when the relationship is not isolated.
Complementary to this, we have observed that the accuracy achieved by the
triadic influence does not improve if we also provide personal information
about the students. This implies that the triadic influence somehow subsumes
the information on the students’ characteristics, rendering it irrelevant to
predict relationships. It is still an open question whether information
obtained from more elaborated personality tests could improve on the
predictions achieved by the triadic influence alone.

On the other hand, we have used state-of-the-art algorithms to create an
embedding for each student that contains information about their surrounding,
considering only the undirected and unweighted network. We have shown that
this structural information can be used to predict the type of relationship
between two students. The embedding of each node is created using a random
walk exploration of its surrounding, the depth of which is a parameter that we
can vary (see Methods). Depending on the typical length of the exploring
random walks this method can gather different structural information. The
maximum length of the random walks used in this study is $L=4$ (see the SI
Fig. S6). Therefore, the Node2Vec algorithm is exploring the local structure
of each student. This aligns with the results achieved using the triadic
influence, suggesting that the closest contacts in the network—the local
environment—are the ones that influence the creation/transformation of
relationships the most. Still, it is remarkable that the triadic influence—a
one-dimensional metric—is able to achieve a higher accuracy than the
prediction based on the embeddings. This suggests that the triadic influence
gathers more information than the metrics included in the embedding of the
nodes—although note also that we are using different information in both cases
because the embeddings are created irrespective of the weights and directions
of the relationships.

Interestingly, Ref. [36] suggests that individuals with similar genotypes may
not actively select into friendships. Instead, they may be placed into these
contexts by institutional mechanisms outside of their control. Our conclusions
could be interpreted in a similar way; the triadic influence may act as a
social force that encourage students that are compatible (incompatible) to
have positive (negative) relationships, akin to the popular knowledge “to be
judged by the company you keep”. In this case, prosociality would be still a
good predictor of the relationship even though it was the social context—the
triadic influence in our case—which promoted the relationship. This raises an
important point that we want to stress: predictability does not imply
causality. Another situation that highlights the difficulty of disentangling
cause and effect is that at the time we collected the data many relationships
that nucleated in isolation due to prosociality alone were now surrounded by
multiple directed paths of length $2$, and we have shown that the triadic
influence is a very good predictor of the label of these relationships, even
if their existence predated the paths entering the computation of the triadic
influence. Therefore, while our results suggest a nucleation mechanism based
on individual traits followed by a growth and evolution of the network
dominated by the triadic influence, they do not prove that this is indeed the
case. In order to assess to what extent this idea describes what is actually
happening in real networks, a possibility would be to use the probabilities
that we have learned through our link prediction techniques to simulate
growing/evolving networks, and then compare these simulations with real data.
In particular, it will be extremely interesting to collect data for the same
network at different times to test the plausibility of different mechanisms of
network evolution based on the probabilities learned here. If our proposal
remains a good candidate to explain how networks form and evolve, then
specific questions of interest arise, such as when the paths of length $2$
begin to dominate over the primitive relationships existing in a network or
how a local change in the sign of a relationship can lead to a cascade of
changes with global effects on the social network.

Finally, it is worth mentioning that our results come from data from a large
number of surveys but from a very specific population, namely, teenagers in
secondary schools in Spain. Thus, the generality of our results should be
validated by gathering similar data from other collectives and performing
similar analyses.

## Methods

### Data collection

Surveys were conducted in 13 Spanish high schools (mandatory education, 11 to
15 years of age). The surveys were delivered through a computer interface and
included direct questions about their relationships, as well as some others
aimed at identifying personal attributes. To elicit relationships, students
could choose from a list containing all the other students in their same year
within their own schools. The number of classes participating in the study in
each school depended on the availability of time and the decisions of the
school direction. The data corresponding to one of the schools, also included
in this work, was presented in full detail in Ref. [37]. For each student, we
collected:

  * •

General data: School ID, course, class, and a student ID assigned by the
software for the purpose of this study.

  * •

List of relationships: All the relationships declared by the student (very
good, good, bad and very bad) were collected with the student IDs of the
nomenees and the corresponding labels ($+2$, $+1$, $-1$, $-2$).

  * •

Individual traits:

    * –

Gender, which included 1789 males, 1720 females, and 4 non-binary people.

    * –

Cognitive reflection test (CRT), computed using the answer to 3 questions
about logic [38, 39], and yielding values $0$, $1$, $2$ and $3$.

    * –

Prosociality, evaluated through the answer to the three following questions
about sharing ($q_{i}$ ranks the level of selfishness of each answer):

      * *

What do you prefer? A) 10€ for you and 10€ for your partner ($q_{1}=0$) B) 10€
for you and 0€ for your partner ($q_{1}=1$).

      * *

What do you prefer? A) 10€ for you and 10€ for your partner. ($q_{2}=1$) B)
10€ for you and 20€ for your partner ($q_{2}=0$).

      * *

What do you prefer? A) 10€ for you and 10€ for your partner ($q_{3}=0$) B) 20€
for you and 0€ for your partner ($q_{3}=1$).

The selfishness score is $s=q_{1}+q_{2}+q_{3}$, and the prosociality index is
obtained as $p=1-(s/3)$.

### Predicting relationships using local information

Our social networks are directed graphs representing the relationships between
all the students within each of the high schools of our study. We kept only
the students that answered all the tests about their individual features
(described above), a total of 3395 students and 60566 relationships.
Relationships are gathered in the weighted adjacency matrix $W$, with elements
$w_{ij}\in\\{-2,-1,0,1,2\\}$ corresponding to the value of the relationship
that student $i$ declares to have with student $j$ ($w_{ij}=0$ if there is no
declared relationship). Note that $w_{ii}=0$ and that $W$ is not symmetric
(relations are not necesarily reciprocal). Additionally, the individual traits
described above (self-declared gender, CRT, and prosociality) are stored in
the nodes $n_{i}$ of the graph. A key quantity used in this work is the
triadic influence $I_{ij}\equiv(W^{2})_{ij}=\sum_{k}w_{ik}w_{kj}$. It
quantifies the aggregated contribution of the directed paths of length $2$
that go from $i$ to $j$. Note that triadic influence considers only directed
paths from $i$ to $j$, and that $I_{ij}\neq I_{ji}$ in general.

In order to use a neural network to predict the declared relationships between
students, we would like to avoid having highly unbalanced classes, and
therefore we define a task with only two classes: friends (we consider here
only $+2$ relationships) or enemies (we merge here relationships $-2$ and
$-1$). We have also considered a more unbalanced case, with the friend class
corresponding to relationships with labels +1 and +2 and the results were
qualitatively analogous. In any case, when we compute the triadic influence
$I_{ij}$, we keep all the labels in the network $\\{-2,-1,1,2\\}$ (see Fig. 1
for an example). In this section, we use a deep neural network with one hidden
layer, ReLu activation (see e.g. Ref. [40]), and $100$ hidden units. The input
dimension depends on the data we want to use to predict the relationship. Our
neural network is a nonlinear function of the inputs and the internal
parameters (numbers that change their value during training), which outputs a
vector of dimension two. Let us call these outputs
$f(\mathcal{I},\mathcal{W})_{i}$, where $\mathcal{I}$ stands for the inputs
corresponding to one specific relation (triadic influence, gender of both
students …), $\mathcal{W}$ are the internal parameters of the network and
$i=0,1$ indicates one of the two classes in our dataset (friends/enemies).
Then these outputs are put into a SoftMax function (see e.g. Ref. [40]) such
that

$q(\mathcal{I},\mathcal{W})_{i}\equiv\frac{e^{f(\mathcal{I},\mathcal{W})_{i}}}{e^{f(\mathcal{I},\mathcal{W})_{0}}+e^{f(\mathcal{I},\mathcal{W})_{1}}},$

where $q(\mathcal{I},\mathcal{W})_{i}$ can be interpreted as the probability
that a specific sample, characterized by inputs $\mathcal{I}$, belongs to
class $i=0,1$. Training the neural network amounts to minimizing a loss
function such that $q(\mathcal{I},\mathcal{W})_{i}$ resembles the _actual_
probability distribution $p(\mathcal{I})_{i}=\delta_{i,\ell(\mathcal{I})}$ for
each sample—$\ell(\mathcal{I})$ being the label of that input data and
$\delta_{i,j}=1$ if $i=j$ and $0$ otherwise. We use the cross-entropy loss
function

$\begin{split}\mathcal{L}&=-\sum_{k,i}p(\mathcal{I}_{k})_{i}\log(q(\mathcal{I}_{k},\mathcal{W})_{i})=-\sum_{k}\log(q(\mathcal{I}_{k},\mathcal{W})_{\ell(\mathcal{I}_{k})}),\end{split}$

where the index $k$ runs over all samples in the dataset. Note that if
$q(\mathcal{I}_{k},\mathcal{W})_{\ell(I_{k})}=1$ for all $k$, the network
would predict with $100$% certainty the correct label for all samples. In this
situation $\mathcal{L}=0$, indicating that for the set of parameters
$\mathcal{W}$ the function $\mathcal{L}$ reaches an absolute minimum. Hence,
training the neural network amounts to minimizing $\mathcal{L}$ with respect
to the parameters $\mathcal{W}$. We have used stochastic gradient descent with
an initial learning rate of $0.1$ and a decaying factor of $0.99$. We use a
minibatch of size $20$ and oversample the class with the smallest number of
samples so that each minibatch has the same number of samples from each class.
Unless otherwise stated, we minimize for $200$ steps and compute the accuracy
in the final step. In the case of the prediction of isolated relationships
(two bottom bars of Fig. 2), the dataset is greatly reduced. To ensure that
our results are robust we use a 10-fold cross-validation approach and report
the mean value and an error bar representing the standard deviation from the
mean. In this case, we train for $1000$ minimization steps using a dynamical
loss function with oscillations of amplitude $10$ and a period of $5$
minimization steps. A dynamical loss function weights the contribution of each
class to the loss function with proportionality factors that oscillate during
minimization. This process changes the topography of the loss function
landscape [41], and helps the model find deeper and wider minima of the loss
function (see Ref. [42] for further details).

### Predicting relationships using global information

The steps followed in the process of creating the embeddings and predicting
the class of a relationship are:

  * •

Passing the graph as an object to Node2Vec [35] yields a $128$-dimensional
vector for each node (an embedding). Node2Vec is defined by the two
hyperparameters $(p,q)$, which describe the space explored by the random
walks. We use $(p=1,q=4)$ after doing a hiperparameter optimization. The
characterization of the typical random walk in this process can be found in
the Supporting Information.

  * •

We merge the embeddings for each pair of nodes to create the embedding of each
edge (relationship), $\mathbf{e}$.

  * •

The structural representation for each edge, $\mathbf{e}$, is the input that
we use to predict the label, friends/enemies of the relationships in the
training dataset. We oversample the training data (test data are left
untouched) using the SMOTE technique [43]. This method produces new examples
by interpolating close existing points in the 128-dimensional space.

  * •

We apply two different machine learning procedures: a random forest, and an
artificial neural network.

The artificial neural network was implemented in the standard library
Tensorflow [44] with four hidden layers (sizes $128$, $64$, $32$, and $8$) and
the ReLu activation function. The final output included a sigmoid function.

###### Acknowledgements.

This work has been partly supported by grant PGC2018-098186-B-I00 (BASIC),
funded by MCIN/AEI/10.13039/501100011033 and by “ERDF A way of making Europe”.
M.R.-G. acknowledges support from the CONEX-Plus program funded by Universidad
Carlos III de Madrid and the European Union’s Horizon 2020 research and
innovation program under the Marie Skłodowska-Curie grant agreement No.
801538. P.B.-G. acknowledges support from Junta de Andalucía (PY18-FR-007) and
Agencia Andaluza de Cooperación Internacional para el Desarrollo
(AACID-0I008/2020) and MCIN (PID2021-126892NB-I00).

##  Appendix A Statistical analysis of the data

Our dataset compromises $13$ schools with $3395$ students and $60566$ declared
relationships, where we have already removed the students that did not answer
all the questions about personality (around 3% of the students) and the
relationships that included them. From the $13$ schools considered, $3$ of
them are in the Region of Madrid and the rest are in Andalucía. In this
Supplementary Materials we will name the schools in Andalucía as t11_1, …,
t11_10, whereas the schools in Madrid will be t1, t2 and t6.

Students can declare to have a very good ($+2$), good ($+1$), bad ($-1$) or
very bad ($-2$) relationship with any other student in their school. The
distribution of the four types of relationships changes among schools although
some features are common, see Fig. 5. For all schools, the most numerous
relationships are good ($+1$) relationships, accounting between $40$ and $50$%
in most schools. On the contrary, the least abundant relationships are the
very bad ones ($-2$). In addition to this, we see some differences between the
schools in Madrid and Andalucía but we leave a detailed study of these
features for a future work.

According to the main text, the most important information to predict a
relation between students $i$ and $j$ is the triadic influence. This quantity
accounts for the influence that third people have on other relationships, and
uses the directed paths of length $2$ that connect $i$ to $j$. For this
reason, it is interesting to know how many relationships have a particular
number of paths of length $2$ connecting the starting node ($i$) and the final
node ($j$). Figure 6 encodes this information. We have plotted each school
separately (color lines) and all the schools together (black line). From the
plot, we can see that $2$% of all relationships do not have any path of length
$2$ connecting the starting and final node. The violin plot helps to visualize
each school separately. Although there is variation between schools, the peak
of the joint distribution (black line) is around $5$ or $6$ paths of length
$2$.

Prosociality is computed for each student. Students answer 3 questions that
lead to a scalar that takes values $0,0.33,0.66\text{ or }1$ (see Methods).
The proportion of students per prosociality value is similar between schools,
see Fig. 7, and resembles the distribution of relationships (Fig. 5). The
largest group of students in most schools is students with prosociality
$0.66$, while the smallest group corresponds to antisocial students ($0$).
Again, there is an apparent difference between the schools in Andalucía and
Madrid. The latter seem to have a larger proportion of very social students
$1$ compared to the schools in Andalucía. This seems to be in good agreement
with the distribution of relationships where schools in Madrid showed more
very good relationships ($+2$). We leave a detailed study of this phenomenon
for future work.

Figure 8 shows the average number of friends/enemies that are nominated by (or
that nominate to) students of different prosociality values. Clear general
trends indicate that students of high prosociality nominate and are nominated
as friends more often, whereas students with low prosociality belong to
negative relationships in a larger proportion than highly prosocial students.
These results are in good agreement with the probabilities learnt by the
neural network in the main text, where students with low prosociality had a
larger probability of being enemies whereas highly prosocial students have a
larger probability of being friends.

##  Appendix B Predicting using only the structure of the network: additional
information on the creation of the embedding and alternative results using a
Random Forest

We include here some additional information to clarify how the Node2vec
algorithm works. Node2Vec builds the embedding for each node using random
walks. The configuration of hyperparameters that we have chosen ($p=1$ and
$q=4$) leads to local exploration. To create the embedding of one node we use
420 random walks, each of them composed of 30 movement attempts. In order to
get some intuition, we show in Fig.9 the path followed by three random walks
starting from different nodes, we show RWs that does not stay fixed in the
initial node, although that is the most probable situation. The local
exploration of the surrounding of the node by the random walks is confirmed by
Fig.10, which shows the distribution of lengths for the random walks with the
hyperparameters used in our work (see Methods).

Finally, to complement the results shown in the main text, we also include
here the results for a second method using the embeddings as input. We have
used a Random Forest (scikit-learn [45]). A Random Forest is an algorithm that
divides the data into random ensembles of predictors and data. From these
randomly chosen pieces of data, the algorithm builds and trains decision
trees. The final decision is then made on the most popular answer within this
population of decision trees. We used a maximum depth of 7 levels for our
decision trees. Fig. 11 depicts qualitatively analogous results to the ones
achieved with the neural network and shown in the main text.

Figure 5: Proportion of the different types of relationship at each school.
Students declare relationships with values $-2,-1,1,2$, from nemesis to best
friends. These relationships are directed, from student $i\rightarrow j$ with
value $W_{ij}$. We have studied $13$ schools with $3395$ students in total,
and $60566$ declared relationships. Most common relationships are $+1$
followed by $+2$, $-1$ and $-2$. We observe similar percentages across
schools.  Figure 6: Percentage of relationships according to the number of
paths of length 2 connecting the same nodes. For each relationship from $i$ to
$j$ we compute the number of paths of length 2 that go from $i$ to $j$, after
this we compute the number of relations with a specific number of length-2
paths. Upper panel displays a color line for each school (see legend) and a
thicker black line for the distribution corresponding to the complete dataset
(clustering all the schools together). Bottom panel show a violin plot for
each school to ease visualization. Figure 7: Distribution of prosocial
behavior across schools. Prosocial behavior takes values $0,0.33,0.67,1$ and
bars correspond to the percentage of students in each school that display a
certain level of prosociality. Distributions are similar across schools
although t1, t2 and t6 (schools in Madrid) seem to have a larger proportion of
very prosocial students ($1$) when compared to the other schools from
Andalucía.  Figure 8: Statistics for the number of friends and enemies
depending on the prosociality of the students. We merge here $-2$ and $-1$
relationships into an enemy category and $+2$ and $+1$ relationships as
friends. Panel (a) shows the average number of people nominated as friends by
a student of a specific prosociality, whereas panel (b) shows the average
number of people that nominate someone of a specific prosociality as a friend.
Panels (c) and (d) are analogous to (a) and (b) but for enemies. Students with
high (low) prosociality tend to nominate and be nominated as friends (enemies)
more frequentely. Error bars correspond to the error of the mean. Figure 9:
Network representation of the students in one of the classes in the dataset,
together with three examples of random walks used by Node2Vec to create the
embedding of one node. The random walks are controlled by the hyperparameters
(in our case $p=1$ and $q=4$), they measure the probability of exploring or
staying in a certain node along the path. From the figure we can see that
Node2Vec is exploring the local environment of the node.  Figure 10:
Distribution of path lengths in the random walks used by Node2Vec. This length
is the shortest path between the origin of the random walk and the farthest
node in that walk. A walk of length $0$ means staying at the same node. We can
therefore understand the locality of the algorithm. The shortest path length
has an average and standard deviation of $0.80\pm 1.07$, which shows again
that embeddings are mainly built with local structure. Figure 11:  Predictions
achieved with a Random Forest. Distribution of balanced accuracy for the 13
high schools. Each histogram is composed of a sample of $N=390$ points, that
are different simulations for the same treatment, and then normalized such
that the area of the histogram sums 1. The purple/dark histogram represents
_treatment I_ whereas the orange histogram represents _treatment II_ , that
is, using an specific age level from a high school as the test set.

## References

  * Jackson [2010] M. Jackson,  _Social and Economic Networks_ (Princeton University Press, Princeton, 2010).
  * Easley and Kleinberg [2010] D. Easley and J. Kleinberg,  _Networks, Crowds, and Markets: Reasoning About a Highly Connected World_ (Cambridge University Press, Cambridge, 2010).
  * Wasserman and Faust [1994] S. Wasserman and K. Faust,  _Social Network Analysis: Methods and applications_ (Cambridge University Press, Cambridge, 1994).
  * Newman [2010] M. E. J. Newman,  _Networks: An introduction_ (Oxford University Press, Oxford, 2010).
  * Dunbar [2020] R. Dunbar, Structure and function in human and primate social networks: Implications for diffusion, network stability and health, Proc. R. Soc. A 476, 20200446 (2020).
  * Moreno [1934] J. L. Moreno,  _Who Shall Survive? Foundations of Sociometry, Group Psychotherapy, and Sociodram_ (Beacon House, 1934).
  * Granovetter [1973] M. S. Granovetter, The strength of weak ties, Am. J. Sociology 78, 1360 (1973).
  * Oliveira and Barabasi [2005] J. G. Oliveira and A.-L. Barabasi, Darwin and einstein correspondence patterns, Nature 437, 1251 (2005).
  * Onnela  _et al._ [2007] J.-P. Onnela, J. Saramäki, J. Hyvönen, G. Szabó, D. Lazer, K. Kaski, J. Kertész, and A.-L. Barabási, Structure and tie strengths in mobile communication networks, Proc. Natl. Acad. Sci. USA 104, 7332 (2007).
  * Ureña Carrion  _et al._ [2020] J. Ureña Carrion, J. Saramäki, and M. Kivelä, Estimating tie strength in social networks using temporal communication data, EPJ Data Sci. 9, 37 (2020).
  * Brockmann  _et al._ [2006] D. Brockmann, L. Hufnagel, and T. Geisel, The scaling laws of human travel, Nature 439, 462 (2006).
  * Cattuto  _et al._ [2010] C. Cattuto, W. Van den Broeck, A. Barrat, V. Colizza, J.-F. Pinton, and A. Vespignani, Dynamics of person-to-person interactions from distributed rfid sensor networks, PLoS ONE 5, e11596 (2010).
  * Leecaster  _et al._ [2016] M. Leecaster, D. J. A. Toth, W. B. P. Pettey, J. J. Rainey, H. Gao, A. Uzicanin, and M. Samore, Estimates of social contact in a middle school based on self-report and wireless sensor data, PLoS ONE 11, e0153690 (2016).
  * Gelardi  _et al._ [2020] V. Gelardi, J. Godard, D. Paleressompoulle, N. Claidière, and A. Barrat, Measuring social networks in primates: wearable sensors versus direct observations, Proc. R. Soc. A 476, 20190737 (2020).
  * Holme and Saramäki [2012] P. Holme and J. Saramäki, Temporal networks, Phys. Rep. 519, 97 (2012).
  * Watts and Strogatz [1998] D. Watts and S. Strogatz, Collective dynamics of ‘small-world’ networks, Nature 393, 440 (1998).
  * Barabási and Albert [1999] A.-L. Barabási and R. Albert, Emergence of scaling in random networks, Science 286, 509 (1999).
  * Bianconi  _et al._ [2001] G. Bianconi, A.-L. Barabási, and R. Albert, Competition and multiscaling in evolving networks, Europhys. Lett. 54, 436 (2001).
  * Starnini  _et al._ [2013] M. Starnini, A. Baronchelli, and R. Pastor-Satorras, Modeling human dynamics of face-to-face interaction networks, Phys. Rev. Lett. 110, 168701 (2013).
  * Jin  _et al._ [2001] E. M. Jin, M. Girvan, and M. E. J. Newman, Structure of growing social networks, Phys. Rev. E 64, 046132 (2001).
  * Hanneke  _et al._ [2010] S. Hanneke, W. Fu, and E. P. Xing, Discrete temporal models of social networks, Electron. J. Stat. 4, 585 (2010).
  * Snijders  _et al._ [2010] T. A. B. Snijders, G. G. Van de Bunt, and C. E. G. Steglich, Introduction to stochastic actor-based models for network dynamics, Soc. Networks 32, 44 (2010).
  * Peter  _et al._ [2005] J. Peter, P. M. Valkenburg, and A. P. Schouten, Developing a model of adolescent friendship formation on the internet, Cyberpsychol. Behav. 8, 423 (2005).
  * Liben-Nowell and Kleinberg [2007] D. Liben-Nowell and J. Kleinberg, The link-prediction problem for social networks, J. Am. Soc. Inf. Sci. Tec. 58, 1019 (2007).
  * Lü  _et al._ [2012] L. Lü, M. Medo, C. H. Yeung, Y.-C. Zhang, Z.-K. Zhang, and T. Zhou, Recommender systems, Phys. Rep. 519, 1 (2012).
  * Airoldi  _et al._ [2008] E. M. Airoldi, D. M. Blei, S. E. Fienberg, E. P. Xing, and T. Jaakkola, Mixed membership stochastic block models for relational data with application to protein-protein interactions, J. Mach. Learn. Res. 9, 1823 (2008).
  * Newman [2001] M. E. J. Newman, Clustering and preferential attachment in growing networks, Phys. Rev. E 64, 025102 (2001).
  * Berlusconi  _et al._ [2016] G. Berlusconi, F. Calderoni, N. Parolini, M. Verani, and C. Piccardi, Link prediction in criminal networks: A tool for criminal intelligence analysis, PLoS ONE 11, e0154244 (2016).
  * Guimerà and Sales-Pardo [2009] R. Guimerà and M. Sales-Pardo, Missing and spurious interactions and the reconstruction of complex networks, Proc. Natl. Acad. Sci. USA 106, 22073 (2009).
  * Song  _et al._ [2009] H. H. Song, T. W. Cho, V. Dave, Y. Zhang, and L. Qiu, Scalable proximity estimation and link prediction in online social networks, in  _IMC ’09: Proceedings of the 9th ACM SIGCOMM conference on Internet measurement_ (ACM, New York, NY, USA, 2009) pp. 322–335.
  * Hao [2019] Z. Hao, Link prediction in online social networks based on the unsupervised marginalized denoising model, IEEE Access 7, 54133 (2019).
  * Kumar  _et al._ [2020] A. Kumar, S. S. Singh, K. Singh, and B. Biswas, Link prediction techniques, applications, and performance: A survey, Physica A 553, 124289 (2020).
  * Tamarit [2019] I. Tamarit,  _Ego-centred models of social networks: the social atom_ , Ph.D. thesis, Universidad Carlos III de Madrid (2019).
  * Buijs and Stulp [2022] V. L. Buijs and G. Stulp, Friends, family, and family friends: Predicting friendships of dutch women, Soc. Networks 70, 25 (2022).
  * Grover and Leskovec [2016] A. Grover and J. Leskovec, node2vec: Scalable feature learning for networks, in  _KDD ’16: Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining_ (ACM, 2016) pp. 855–864.
  * Boardman  _et al._ [2012] J. D. Boardman, B. W. Domingue, and J. M. Fletcher, How social and genetic factors predict friendship networks, Proc. Natl. Acad. Sci. USA 109, 17377 (2012).
  * Escribano  _et al._ [2021] D. Escribano, V. D. Martelli, F. J. Lapuente, J. A. Cuesta, and A. Sánchez, Evolution of social relationships between first-year students at middle school: from cliques to circles, Sci. Rep. 11, 11694 (2021).
  * Brañas-Garza  _et al._ [2019] P. Brañas-Garza, P. Kujal, and B. Lenkei, Cognitive reflection test: Whom, how, when, J. Behav. Exp. Econ. 82, 101455 (2019).
  * Brañas Garza  _et al._ [2022] P. Brañas Garza, L. Ductor, and J. Kovárík, The role of unobservable characteristics in friendship network formation (2022).
  * Goodfellow  _et al._ [2016] I. Goodfellow, Y. Bengio, and A. Courville,  _Deep learning_ (MIT Press, Cambridge, MA, USA, 2016) http://www.deeplearningbook.org.
  * Ruiz-Garcia  _et al._ [2019] M. Ruiz-Garcia, A. J. Liu, and E. Katifori, Tuning and jamming reduced to their minima, Phys. Rev. E 100, 052608 (2019).
  * Ruiz-Garcia  _et al._ [2021] M. Ruiz-Garcia, G. Zhang, S. S. Schoenholz, and A. J. Liu, Tilting the playing field: Dynamical loss functions for machine learning, in  _Proceedings of the 38th International Conference on Machine Learning_ , Vol. 139 (2021) pp. 9157–9167.
  * Chawla  _et al._ [2002] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, Smote: synthetic minority over-sampling technique, J. Artif. Intell. Res. 16, 321 (2002).
  * Abadi  _et al._ [2015] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, and S. Ghemawat, Tensorflow: Large-scale machine learning on heterogeneous systems, in  _OSDI’16: Proceedings of the 12th USENIX conference on Operating Systems Design and Implementation_ (ACM, 2015) pp. 265–283.
  * Pedregosa  _et al._ [2011] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, Scikit-learn: Machine learning in python, J. Mach. Learn. Res. 12, 2825 (2011).