#  Mixing-Specific Data Augmentation Techniques for Improved Blind
Violin/Piano Source Separation

Ching-Yu Chiu  Graduate Program of Multimedia Systems and Intelligent
Computing
National Cheng Kung University and Academia Sinica, Taiwan
<EMAIL_ADDRESS>Wen-Yi Hsiao  Yating Music Team
Taiwan AI Labs, Taiwan
<EMAIL_ADDRESS>Yin-Cheng Yeh  Yating Music Team
Taiwan AI Labs, Taiwan
<EMAIL_ADDRESS>Yi-Hsuan Yang  Research Center for IT Innovation, Academia
Sinica, Taiwan
<EMAIL_ADDRESS>Alvin Wen-Yu Su  Dept. CSIE, National Cheng Kung
University, Taiwan
<EMAIL_ADDRESS>
###### Abstract

Blind music source separation has been a popular and active subject of
research in both the music information retrieval and signal processing
communities. To counter the lack of available multi-track data for supervised
model training, a data augmentation method that creates artificial mixtures by
combining tracks from different songs has been shown useful in recent works.
Following this light, we examine further in this paper extended data
augmentation methods that consider more sophisticated mixing settings employed
in the modern music production routine, the relationship between the tracks to
be combined, and factors of silence. As a case study, we consider the
separation of violin and piano tracks in a violin piano ensemble, evaluating
the performance in terms of common metrics, namely SDR, SIR, and SAR. In
addition to examining the effectiveness of these new data augmentation
methods, we also study the influence of the amount of training data. Our
evaluation shows that the proposed mixing-specific data augmentation methods
can help improve the performance of a deep learning-based model for source
separation, especially in the case of small training data.

###### Index Terms:

Music source separation, data augmentation

##  I Introduction

Music source separation, i.e., separating the sources (instruments) involved
in an audio recording, has been a major research topic in the signal
processing community, partly due to its wide downstream applications in music
upmixing and remixing, karaoke, DJ-related applications, and as a pre-
processing tool for other problems [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13,
14, 15, 16]. Its technical difficulty has also been well acknowledged. For
example, the estimation of the number of sources involved in a song remains
challenging [17]. Even when the number of sources is known or given
beforehand, in supervised training we need the _multi-track recordings_ that
provide the ground truth single-instrument tracks (a.k.a., ‘stems’) that
compose a song. Such multi-track recordings are rarely publicly available due
to copyright issues [18, 19]. What are typically available instead are the
_mixed_ versions of the songs, where the multiple tracks have been mixed and
combined into a monaural (i.e., one-channel) or stereo (two-channel)
recording. The lack of available data with ground truth stems not only limits
the development of data-driven methods, but also hinders systematic evaluation
of new methods proposed for the task.

Over the past decades, the main methods for tackling this task can be roughly
classified into two categories: model-based methods and data-centered methods.
As discussed in a recent review paper [8], the performance of model-based
methods could change dramatically when their core assumptions are not met. On
the other hand, data-centered methods rely heavily on the availability of
professionally produced or recorded multi-track data, which is hard to come by
due to copyright issues.

Figure 1:  Flowchart of the baseline random mixing (bottom-left) and the
proposed four augmentation methods (on the right). Given the collected violin
solos and piano solos (in gray block), three types (Silence, Mixing, and
Pairing) of augmentations (in blue blocks) can be applied to produce four
groups of augmented violin/piano chunks (in green buckets). If none of them is
applied, we use the violin/piano chunks (in orange bucket) for the baseline
random mixing. Best viewed in color.

As some medium-scale multi-track datasets have been released in the past few
years, the development of data-driven models grows fast. A data-driven source
separation model is typically a supervised model which is trained by taking an
audio mixture (a monaural or stereo recording) as the input, and aiming to
recover the tracks (i.e., multiple monaural or stereo recordings) that compose
the mixture. In doing so, there have been two main approaches in the
literature. In the first approach, which we refer to as the no data
augmentation approach, the tracks that compose an input mixture are originally
from the same song. In other words, when we have $N_{0}$ multi-track songs, we
would have exactly $N_{0}$ input/output pairs for modeling training. In the
second approach, or random-mixing data augmentation [13, 5], tracks from
different songs are randomly combined to create audio mixtures, leading to
artificial input/output pairs. In such a case, we can have $N\gg N_{0}$
input/output pairs. The downside of this approach is these input mixtures are
not realistic sound mixtures in terms of the tonic, harmonic and rhythmic
relations among the tracks that compose the mixtures. But, for the purpose of
training data-driven models for source separation, the benefit of the
resulting great increase in the number of training data seems to outweigh this
potential concern, as demonstrated in the literature [13, 5].

While there are some other data augmentation methods such as adding noise or
randomly dropout [20, 21], the aforementioned _random-mixing data
augmentation_ , albeit simple, has been shown particularly successful [13].
However, when mixing two tracks, there are actually multiple aspects to
consider [22], suggesting room for the development of more advanced mixing-
specific data augmentation methods for source separation. To our best
knowledge, this has not yet been investigated in the literature. It is
therefore our goal to develop, and to empirically evaluate the effectiveness
of, new data augmentation methods that stem from the random-mixing approach.
We consider in total three types of mixing-specific data augmentation methods
for source separation, as conceptually visualized in Figure 1 and detailed in
Section IV.

Besides, current available data is still not enough for many common musical
instruments, such as the violin. To investigate the benefit of data
augmentation for music source separation in general, we consider in this paper
the separation of violin and piano tracks in a violin piano ensemble, a task
that has rarely been considered in the literature. Moreover, we consider the
case when _no multi-track recordings of violin piano ensembles_ are available
for model training, but instead _only a collection of piano solos and violin
solos_. In such a case, mixing-based data augmentation becomes a major viable
approach.

In sum, the main contribution of this study is to propose a series of data
augmentation/selection approaches that enable _non-paired violin/piano solo
stems_ to approximate features of realistic paired stems, which in turn
facilitate the training of deep learning-based source separation models.

For reproducibility, we share the code, pre-trained models, the audio files of
the ground truth and separated stems (by the ‘Wet’ model; see Section IV) of
the test data publicly at https://github.com/SunnyCYC/aug4mss and
https://sunnycyc.github.io/aug4mss_demo/.

Below, we review related work and the adopted network architecture for source
separation in Section II. Section III describes the training and test data
employed in our implementation. Section IV presents the proposed data
augmentation methods, while Section V talks about the evaluation results.
Finally, we conclude the paper in Section VI.

Figure 2: The network architecture of Open-Unmix [13]. Taking an input mix
spectrogram, the model produces the target spectrogram by multiplying the
input with a full-band mask. The model is based on a three-layer bidirectional
LSTM capable of taking input of arbitrary lengths. The number on the left side
of each component indicates the frequency bin size of the output. The model
crops and only processes the spectrogram under 16kHz, and reconstructs
intermediate product to full band by the last fully connected layer ‘fc3.’

##  II Background

###  II-A Related works

Data augmentation for improving the performance of deep neural networks has
been an important topic in the field of music information retrieval (MIR) in
recent years. Schluter and Grill [21] were one of the earliest to
systematically explore the utility of music data augmentation for singing
voice detection with neural networks. They found pitch shifting combined with
time stretching and random frequency filtering to be quite helpful in reducing
the classification error. Uhlich et al. [5] proposed two neural network
architectures capable of yielding state-of-the-art results for music source
separation at that time, and further boosted the performance through data
augmentation and network blending. Hawthorne et al. [23] experiments with the
use of mixing techniques such as equalization, contrast, and reverberation in
an attempt to make their automatic piano transcription model more robust to
different recording environments and piano qualities. To our knowledge, such
mixing techniques have not been employed in existing work on source
separation.

New neural network architectures for blind music source separation have been
continuously proposed as well. For example, Manilow et al. [15] intellectually
utilized the inherent synergy between transcription and source separation to
improve both tasks using a multi-task learning architecture. Liu and Yang [10]
combined dilated convolution with modified gated recurrent units (GRU) to
extend the receptive field of each dilated GRU unit, enabling their model to
perform better and faster than state-of-the-art models for separating vocals
and accompaniment.

###  II-B Model Architecture

As our focus is on the data augmentation techniques, we adopt an existing
blind source separation framework called Open-Unmix [13] as the backbone
architecture in our work. Open-Unmix is a hybrid convolutional-recurrent
architecture (see Figure 2) that is open source. It takes a fixed-length chunk
of the Short-time Fourier transform (STFT) spectrogram of the mixture as the
input and aims to get the corresponding separated spectrogram of one of the
sources at the output. Although the model is trained on fixed-length chunks,
at testing time it can be applied to spectrograms of arbitrary length. The
parameters of the network is learned by minimizing the difference between the
spectrograms of the ground truth and the separation result, calculated in
terms of mean square error.

The original Open-Unmix model does not deal with the separation of piano and
violin tracks [13], but it is easy to use their code base to train the model
on our piano and violin data. In comparison, the other famous open-source
separation model, called Spleeter [14], has a built-in function to isolate out
the piano track from an input mixture, but it does not provide the script for
retraining the model to isolate out the violin track. In consequence, we
consider the pre-trained Spleeter model released by the authors as the
baseline method for performance comparison, instead of the backbone
architecture of our model.

##  III Datasets

###  III-A Violin/piano Solo as Training Stems

We collected six hours of classical violin solo recordings and six hours of
pop piano solo recordings from the Internet as our training and validation
data. For each instrument, five hours of data is allocated for training and
the remaining one hour for validation. All songs are divided into 10-second
chunks. During training and validation, one chunk will be randomly selected
from each instrument for mixing and serving as the input data.

###  III-B MedleyDB as Evaluation Data

To evaluate the performance of our method on real data, we select multi-track
songs that contain realistic violin and piano stems from the MedleyDB [18, 19]
for evaluation. There are only 16 such songs, highlighting the difficulty of
collecting multi-track recordings for model training. Moreover, after
listening to these songs one-by-one, we discard 10 of them, as there are
severe leakage issues and accordingly the piano/violin stems are not purely
piano/violin. We also note that most of the remaining six songs contain more
than one violin stems. For such songs, we consider the combination of the
piano track and _one_ of the violin tracks as the input mixture, thereby
creating multiple partially realistic mixtures from the same song. All the
other instruments from these songs (e.g., cello) are also excluded. As a
result, we have 16 violin piano ensembles in total for evaluation.

We note that, due to copyright restrictions, we are unfortunately not able to
share the audio files of the training data. Researchers interested in violin
piano separation would have to collect the training data on their own.
However, as mentioned by the end of Section I, we make public the
aforementioned 16 violin piano ensembles so that people can evaluate their
models on the same test set.

TABLE I: Three types of mixing-related data augmentation methods we propose here, and the corresponding parameter settings Mixing | Description | Scale | Range
---|---|---|---
| Contrast | Amount | 1$-$70
| Equalizer 1 | Frequency | 32$-$4,096
| Equalizer 2 | Frequency | 32$-$4,096
| Reverb | Reverb | 1$-$70
| Pink noise | Volume | 0.01$-$0.04
Paring | Description | Threshold | Notes
| Chroma distance | 0.48 | Mean within songs: 0.45
|  |  | Mean across songs: 0.51
| Correlation | 20 | Within songs: 2$-$30
|  |  | Across songs: 0$-$10
Silence | Description | Threshold | Notes
| Reduce silence | 20 | top db

##  IV Proposed Augmentation/Selection Methods

As shown in Figure 1 and Table I, we consider three types of augmentation
methods here. The mixing type augmentation includes factors related to the
common mixing process, such as the use of equalization, contrast, reverb, and
the addition of pink noise. The pairing type methods are designed based on
consideration for the tonic, harmonic and rhythmic relations between the real
paired stems. Lastly, the silence type method is designed for eliminating the
difference of silence duration in violin/piano stems to avoid potential
imbalance of the training data. In what follows, we refer to the unprocessed
original stems as the _original stems_.

###  IV-A Wet Stems

Following [23], we apply the common approaches in the mixing process and also
the addition of pink noise as our first augmentation method. The pink noise is
employed to simulate the background noise seen in real recordings. The
augmentation parameters are shown in Table I. For each original stem, we set a
30% probability to apply a specific process (e.g. equalization). The pink
noise is applied using colorednoise, and the others are applied using a python
package called pysox [24]. In what follows, we also refer to a model trained
with this data augmentation method (i.e., the ‘Wet’ method) as the Wet model.

###  IV-B Chroma Distance-based Pairing

Based on general principles of music theory or psychoacoustics, the paired
stems in music are usually highly coherent in pitch, or are in similar keys.
Therefore, instead of randomly selecting stems for combination, this
augmentation method only picks the stems that have short chroma distance with
one another for combination, in a hope that the resulting mixture would be
more similar to real ensembles. Specifically, we average the chromagram, a
representation of the time-varying intensities of the twelve different pitch
classes, to derive a 12-dimensional chroma feature for each stem, and then
calculate the Euclidean distance between all violin/piano stems. We then need
a threshold for selecting qualified training violin/piano stems. In doing so,
we rely on statistics calculated from the MedleyDB test songs. As shown in
Table I, the mean chroma distance between the violin/piano stems from the same
song of MedleyDB is 0.45, shorter than the mean distance of violin/piano stems
from different songs of MedleyDB. We therefore set the threshold to 0.48. Only
stems with chroma distance lower than this threshold will be selected and
mixed as our training data.

###  IV-C Correlation-based Pairing

Another pairing method is to consider whether the piano stem and violin stem
are _active_ (i.e., non-silent) at the same time; namely, whether they co-
occur. To implement this, we calculate the absolute value of the 2-dimensional
cross correlation between the waveform magnitude (using
scipy.signal.correlate2d) of all the violin/piano stems and set a threshold
for selecting the stems to be combined for training. As shown in Table I, the
cross correlation values for true paired stems in MedleyDB test songs range
from 2–30. We therefore empirically set the threshold to 20.

###  IV-D Silence Removal before Mixing

For the case of violin piano ensemble, we empirically observe sometimes the
activity of the violin part would be too sparse, making the two sounds
unbalanced in the mixture. We consider it worth investigating whether this
would influence the performance of the resulting separation model. Therefore,
we apply librosa.effects.split [25] with ‘top db’ $=$ 20 to remove the silence
part in the training/validation data, before dividing them into 10-second
chunks for mixing. We have also experimented with other values of ‘top db’ and
found 20 works better.

##  V Experiments

###  V-A Experiment Settings & Evaluation Metrics

As discussed in Section I, there are many common instruments that suffer from
the lack of multi-track data. For instruments less common, such as erhu and
suona, even the number of solo recordings is limited. Therefore, we study the
effectiveness of the proposed augmentation methods in two scenarios: a data-
limited one, and a data-rich one.

Specifically, for each instrument and each augmentation method, we train (from
scratch) a separation model using the Open-Unmix architecture and the
corresponding processed stems. In each training epoch, the model will randomly
select $N$ pairs of stems from the pool of processed stems to mix as the
training data. To simulate the data-limited case, we set $N=$ 250 and adopt
only 16 minutes of the training data for each instrument. For the data-rich
case, we use the full data for training and set $N=$ 2000\. After each
training epoch, the model will also select 100 pairs of stems from the
validation pool of processed stems to mix as validation data. The validation
loss is adopted by an early stop mechanism and a learning rate scheduler to
fine-tune the whole training process. For all experiments, an early stop
patience, 140 epochs, is adopted, and the learning rate is set to 0.001. All
the training and testing songs are monaural tracks with a 44,100 hz sampling
rate. The window size and hop length of STFT are 4,096 and 1,024. The
predicted spectrograms are converted back to time-domain waveforms using the
Griffin-Lim algorithm for phase estimation before inverse STFT.

For evaluation metrics, we adopt the signal-to-distortion ratio (SDR), signal-
to-interference ratio (SIR), and signal-to-artifacts ration (SAR), implemented
in the BSS_eval toolkit [1]. Following the convention in SiSEC
(https://sisec.inria.fr/), we report the median values over the testing songs.

We consider the following two methods as the baseline methods. First, we use
the Open-Unmix model to train the separation model using the existing random-
mixing data augmentation method; we refer to this method as the Random method.
Second, we use the official pretrained 5-stem model of Spleeter [14], the
current state-of-the-art for singing voice separation. Specifically, we use
the default ‘mask’-based implementation. Given an input mixture, it generates
as the output separated stems of piano, vocal, drum, bass, and others. From
the result of Spleeter, we sum all but the piano stem as the separated violin
stem.

Training an Open-Unmix based network using any of the proposed data
augmentation method takes around 15 hours on an NVIDIA GeForce GTX 1080 GPU.
At testing time, it takes $\sim$3 mins to complete the separation (for both
the piano and violin) of an 18-min violin piano ensemble, on the same GPU.

###  V-B Results & Discussion

The result of the data-limited case is shown in Table II. The following
observations can be made: First, for piano, all the proposed augmented models
outperform the two baselines in SDR and SAR. In particular, the improvement
made by ‘Correlation’ method is more than 2 dB for both metrics. Second, for
violin, except for the ‘Correlation’ method that still gains improvement in
SDR and SIR, the help of other augmentation methods becomes less obvious. The
performance drop of ‘Wet’ method for violin indicates potential risk for the
randomly applied mixing type method to distort the data under data-limited
scenario. Finally, all the Open-Unmix based models we train greatly outperform
the pre-trained, violin-agnostic Spleeter model in almost all the metrics for
both instruments, except for the piano in terms of SIR. This is not
surprising, as the Spleeter model has not been specifically trained on violin
data. The failure of recognizing the violin also hurts its performance on the
piano.

The result of the data-rich case are shown in Table III. From all the metrics
we can see that the help of augmentations in both instruments becomes less
obvious. For the piano, the baseline ‘Random’ method seems strong enough. For
the violin, the baseline ‘Random’ method is only inferior to the proposed
method with a small margin. This implies that, when the training data is big
enough, the diversity of the data would also increase and overshadow the help
of sophisticated augmentation methods.

TABLE II: The median values of the evaluation metrics (in db) over the 16 MedleyDB songs for the data-limited scenario Piano | Method | SDR | SIR | SAR
---|---|---|---|---
| Spleeter (pretrained) [14] | 5.20 | 17.35 | 2.30
| Random-mixing | 7.43 | 14.38 | 10.52
| Wet | 8.48 | 13.80 | 11.21
| Chroma | 7.47 |  13.09 | 10.91
| Correlation | 9.66 | 13.89 | 13.80
| NonSilence | 8.76 | 13.30 | 11.51
Violin | Method | SDR | SIR | SAR
| Spleeter (pretrained) [14] | 0.28 | 2.05 | 0.22
| Random-mixing | 1.08 | 6.22 | 2.76
| Wet | 0.73 | 3.46 | 3.55
| Chroma | 1.54 | 5.48 | 2.70
| Correlation | 1.56 | 8.11 | 2.31
| NonSilence | 1.27 | 6.10 | 2.69

From Tables II and III, we also see that the separation performance generally
improves along with the increase in the number of training data, which is not
surprising.

The spectrograms of the original and predicted audio of a text song are shown
in Figure 3. It can be seen that, the pre-trained Spleeter model cannot
separate the piano from the violin well. In contrast, despite some visible
residuals, the models trained with the proposed methods work well in
separating the piano and violin. We also see from the result of the violin
that some of the proposed methods (e.g., the ‘wet’ model) can get rid of the
leakage of the piano part and noises in the low frequency bands, while the
baseline methods suffer.

Figure 4 provides another example result. As there is a note offset in the
latter half of the violin, the performance difference among the methods can be
seen more clearly. We can see again that the proposed method suffers less from
the leakage of the piano in the low frequency bands. Besides, the high-
frequency piano residuals can be found to be a bit more in the result of the
baseline ‘random’ model than in the ‘wet’ model. We also note that an extra
low-frequency component (highlighted by the red arrow) in the ground truth
violin in this example. Since it is below the lowest frequency of the violin
sound (i.e., 196 hz), we listened to it and found that it seems to be some
background noise or reverb. The proposed methods can also exclude this low-
frequency part from the separation result.

TABLE III: Evaluation result (in db) for the data-rich scenario Piano | Method | SDR | SIR | SAR
---|---|---|---|---
| Random-mixing |  13.46 | 20.76 | 15.15
| Wet | 11.76 | 22.38 | 13.76
| Chroma | 12.52 | 19.87 | 14.40
| Correlation | 11.37 | 20.80 | 13.48
| NonSilence | 12.82 | 18.85 | 14.98
Violin | Method | SDR | SIR | SAR
| Random-mixing | 3.84 | 17.16 | 4.38
| Wet | 4.48 |  15.00 | 4.67
| Chroma | 3.82 | 16.72 | 4.76
| Correlation |  4.19 | 15.86 | 4.23
| NonSilence | 3.03 | 17.38 | 4.09

##  VI Conclusions and Future works

In this paper, we have proposed and investigated a number of mixing-related
data augmentation methods to facilitate the training of deep learning models
for music source separation. Result demonstrates the effectiveness of the
proposed augmentation methods in the case of small data. When the training
data is big enough, the existing random-mixing based approach [5] is strong
enough. We have also shown that the implemented models outperform the current
available best model, Spleeter [14], in violin/piano separation. We believe
such training and augmentation methods have potential in benefiting other
source separation tasks with limited amount of training data available. In the
future, we are interested in extending our experiments to other popular yet
less investigated instruments (e.g., guitar and saxophone), and in
experimenting with more advanced mixing methods (e.g., applying mixing after
the stems are selected, rather than applying mixing to the stems individually
beforehand as done here).

Figure 3: The spectrograms of a mixture from the MedleyDB (titled
‘MatthewEntwistle_ImpressionsOfSaturn_p0_v1.wav’ in our GitHub repo), the
ground truth violin/piano stems, and the predicted stems by different methods,
trained under the data-rich scenario.

Figure 4: The result of another mixture from the MedleyDB
(‘MatthewEntwistle_AnEveningWithOliver_p0_v6.wav’).

## References

  * [1] E. Vincent, R. Gribonval, and C. Fevotte, “Performance measurement in blind audio source separation,” _IEEE Trans. Audio, Speech, Language Processing_ , vol. 14, no. 4, pp. 1462–1469, July 2006.
  * [2] Y.-H. Yang, “Low-rank representation of both singing voice and music accompaniment via learned dictionaries,” in _Proc. International Society for Music Information Retrieval Conference_ , 2013.
  * [3] P.-K. Jao, L. Su, Y.-H. Yang, and B. Wohlberg, “Monaural music source separation using convolutional sparse coding,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 2016.
  * [4] H. Tachibana, Y. Mizuno, N. Ono, and S. Sagayama, “A real-time audio-to-audio karaoke generation system for monaural recordings based on singing voice suppression and key conversion techniques,” _Journal of Information Processing_ , vol. 24, no. 3, pp. 470–482, 2016.
  * [5] S. Uhlich, M. Porcu, F. Giron, M. Enenkl, T. Kemp, N. Takahashi, and Y. Mitsufuji, “Improving music source separation based on deep neural networks through data augmentation and network blending,” _Proc. IEEE Int. Conf. Acoust. Speech Signal Process_ , pp. 261–265, 2017.
  * [6] A. Jansson, E. Humphrey, N. Montecchio, R. Bittner, A. Kumar, and T. Weyde, “Singing voice separation with deep U-Net convolutional networks,” in _Proc. International Society for Music Information Retrieval Conference_ , 2017.
  * [7] D. Stoller, S. Ewert, and S. Dixon, “Adversarial semi-supervised audio source separation applied to singing voice extraction,” in _Proc. IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2018, pp. 2391–2395.
  * [8] Z. Rafii, A. Liutkus, F. R. Stoter, S. I. Mimilakis, D. Fitzgerald, and B. Pardo, “An overview of lead and accompaniment separation in music,” _IEEE/ACM Trans. Audio Speech Lang. Process._ , vol. 26, no. 8, pp. 1307–1335, 2018.
  * [9] J.-Y. Liu and Y.-H. Yang, “Denoising auto-encoder with recurrent skip connections and residual regression for music source separation,” in _Proc. IEEE Int. Conf. Machine Learning and Applications_ , 2018.
  * [10] ——, “Dilated convolution with dilated GRU for music source separation,” in _Proc. Int. Joint Conference on Artificial Intelligence_ , 2019.
  * [11] G. Meseguer-Brocal and G. Peeters, “Conditioned-U-Net: Introducing a control mechanism in the U-Net for multiple source separations,” in _Proc. International Society for Music Information Retrieval Conference_ , 2019, pp. 159–165.
  * [12] J. H. Lee, H. Choi, and K. Lee, “Audio query-based music source separation,” in _Proc. International Society for Music Information Retrieval Conference_ , 2019, pp. 878–885.
  * [13] F.-R. Stöter, S. Uhlich, A. Liutkus, and Y. Mitsufuji, “Open-Unmix - A reference implementation for music source separation,” _Journal of Open Source Software_ , vol. 4, no. 41, p. 1667, 2019.
  * [14] D. P. Kingma and J. L. Ba, “Spleeter: A fast and state-of-the art music source separation tool with pre-trained models,” in _International Society for Music Information Retrieval Conference, Late-breaking paper_ , 2019\.
  * [15] E. Manilow, P. Seetharaman, and B. Pardo, “Simultaneous separation and transcription of mixtures with multiple polyphonic and percussive instruments,” in _Proc. IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2020, pp. 771–775.
  * [16] T.-H. Hsieh, K.-H. Cheng, Z.-C. Fan, Y.-C. Yang, and Y.-H. Yang, “Addressing the confounds of accompaniments in singer identification,” in _Proc. IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2020\.
  * [17] Y. Luo, Z. Chen, J. R. Hershey, J. L. Roux, and N. Mesgarani, “Deep clustering and conventional networks for music separation: Stronger together,” in _Proc. IEEE International Conference on Acoustics, Speech and Signal Processing_ , 2017, pp. 61–65.
  * [18] R. Bittner, J. Salamon, M. Tierney, M. Mauch, C. Cannam, and J. Bello, “MedleyDB: A multitrack dataset for annotation-intensive MIR research,” _Proc. International Society for Music Information Retrieval Conference_ , pp. 155–160, 2014.
  * [19] R. M. Bittner, J. Wilkins, H. Yip, and J. P. Bello, “Medleydb 2.0 : New data and a system for sustainable data collection,” _Proc. International Society for Music Information Retrieval Conference_ , pp. 2–4, 2016.
  * [20] H. Erdogan and T. Yoshioka, “Investigations on data augmentation and loss functions for deep learning based speech-background separation,” _Proc. INTERSPEECH_ , pp. 3499–3503, 2018.
  * [21] J. Schlüter and T. Grill, “Exploring data augmentation for improved singing voice detection with neural networks,” _Proc. International Society for Music Information Retrieval Conference_ , pp. 121–126, 2015.
  * [22] B. D. Man, “Towards a better understanding of mix engineering,” Ph.D. dissertation, Queen Mary University of London, 2017.
  * [23] C. Hawthorne _et al._ , “Enabling factorized piano music modeling and generation with the MAESTRO dataset,” _Proc. International Conference on Learning Representations_ , pp. 1–12, 2019.
  * [24] R. M. Bittner, E. Humphrey, and J. P. Bello, “Pysox : Leveraging the audio signal processing power of Sox in Python,” _Proc. International Society for Music Information Retrieval Conference_ , pp. 4–6, 2016.
  * [25] B. McFee _et al._ , “Librosa: Audio and music signal analysis in Python,” _Proc. Python in Science Conference_ , pp. 18–24, 2015.


# Predicting the Future of AI with AI:
High-Quality link prediction in an exponentially growing knowledge network

Mario Krenn<EMAIL_ADDRESS>Max Planck Institute for the Science of
Light (MPL), Erlangen, Germany.  Lorenzo Buffoni  Instituto de
Telecomunicações, Lisbon, Portugal.  Bruno Coutinho  Instituto de
Telecomunicações, Lisbon, Portugal.  Sagi Eppel  University of Toronto,
Canada.  Jacob Gates Foster  University of California Los Angeles, USA.
Andrew Gritsevskiy  University of Toronto, Canada.  Cavendish Laboratories,
Cavendish, Vermont, USA.  Institute of Advanced Research in Artificial
Intelligence (IARAI), Vienna, Austria.  Harlin Lee  University of California
Los Angeles, USA.  Yichao Lu  Layer 6 AI, Toronto, Canada.  João P. Moutinho
Instituto de Telecomunicações, Lisbon, Portugal.  Nima Sanjabi  Independent
Researcher, Barcelona, Spain.  Rishi Sonthalia  University of California Los
Angeles, USA.  Ngoc Mai Tran  University of Texas at Austin, USA.  Francisco
Valente  Independent Researcher, Leiria, Portugal.  Yangxinyu Xie  University
of Pennsylvania, USA.  Rose Yu  University of California, San Diego, USA.
Michael Kopp  Institute of Advanced Research in Artificial Intelligence
(IARAI), Vienna, Austria.

###### Abstract

A tool that could suggest new personalized research directions and ideas by
taking insights from the scientific literature could significantly accelerate
the progress of science. A field that might benefit from such an approach is
artificial intelligence (AI) research, where the number of scientific
publications has been growing exponentially over the last years, making it
challenging for human researchers to keep track of the progress. Here, we use
AI techniques to predict the future research directions of AI itself. We
develop a new graph-based benchmark based on real-world data – the
Science4Cast benchmark, which aims to predict the future state of an evolving
semantic network of AI. For that, we use more than 100,000 research papers and
build up a knowledge network with more than 64,000 concept nodes. We then
present ten diverse methods to tackle this task, ranging from pure statistical
to pure learning methods. Surprisingly, the most powerful methods use a
carefully curated set of network features, rather than an end-to-end AI
approach. It indicates a great potential that can be unleashed for purely ML
approaches without human knowledge. Ultimately, better predictions of new
future research directions will be a crucial component of more advanced
research suggestion tools.

Figure 1: Number of papers published per months in the arXiv categories of AI
grow exponentially. The doubling rate of papers per months is roughly 23
months, which might lead to problems for publishing in these fields, at some
point. The categories are cs.AI, cs.LG, cs.NE, and stat.ML.

##  I Introduction and Motivation

The corpus of scientific literature grows at an ever-increasing speed.
Specifically, in the field of Artificial Intelligence (AI) and Machine
Learning (ML), the number of papers every month grows exponentially with a
doubling rate of roughly 23 months (see Fig. 1). Simultaneously, the AI
community is embracing diverse ideas from many disciplines such as
mathematics, statistics, and physics, making it challenging to organize
different ideas and uncover new scientific connections. We envision a computer
program that can automatically read, comprehend and act on AI literature. It
can predict and suggest meaningful research ideas that transcend individual
knowledge and cross-domain boundaries. If successful, it could significantly
improve the productivity of AI researchers, open up new avenues of research,
and help drive progress in the field.

Here, we address this important and challenging vision. New research ideas
often result from drawing novel connections between seemingly unrelated
concepts Evans and Foster (2011); Fortunato  _et al._ (2018); Wang and
Barabási (2021). Therefore, we formulate the evolution of AI literature as a
temporal network modelling task. We created an evolving semantic network
characterizing the content and evolution of the scientific literature in the
field of AI since 1994. The network contains about 64,000 nodes (each
representing a concept used in an AI paper) and 18 million edges that connect
two concepts when they were investigated jointly in a scientific paper.

We use the semantic network as an input to 10 diverse statistical and machine-
learning methods to predict the future evolution of the semantic network with
high accuracy. That is, we can predict which combinations of concepts AI
researchers will investigate in the future. Being able to predict what
scientists will work on is a first crucial step for suggesting new topics that
might have a high impact.

Several of the methods presented in this paper have been contributions to the
Science4Cast competition hosted by IEEE BigData 2021, which ran from August to
November 2021. Broadly, we can divide the methods into two classes: methods
that use hand-crafted network-theoretical features, and those that
automatically learn features. We found that models using carefully hand-
crafted features outperform methods that attempt to learn features
autonomously. This (somewhat surprising) finding indicates a great potential
for improvements of models free of human priors.

Our manuscript has several purposes. First, we introduce a new meaningful
benchmark for AI on real-world graphs. Second, we provide nearly 10 diverse
methods that solve this benchmark. Third, we explain how solving this task
could become an essential ingredient for the big picture goal of having a tool
that could suggest meaningful research directions for scientists in AI or in
other disciplines.111github.com/artificial-scientist-lab/FutureOfAIviaAI

The manuscript is structured in the following way. We first introduce more
background into semantic networks and how they can help to suggest new ideas.
Then we explain how we generate the dataset and some of its network-
theoretical properties. Then we briefly explain the 10 methods that we have
investigated to solve the task. We conclude with a number of important open
questions that could bring us further toward the goal of AI-based suggestions
for research directions.

Figure 2: From arXiv to Science4Cast. We use 143,000 papers in AI and ML
categories on arXiv from 1992 to 2020. From there, we construct a list of
concepts (using RAKE and other NLP tools). Those concepts form the nodes of a
semantic network. The edges are drawn when two concepts occur jointly in the
title or abstract of a paper. In that way, we generate an evolving semantic
network that grows over time as more concepts are investigated together. The
task is to predict, from unconnected nodes (i.e. concepts that have not been
investigated together in the scientific literature), which will be connected
within a few years. In this manuscript, we present 10 diverse statistical and
machine learning methods to solve this challenge.

##  II Semantic Networks

The goal here is to extract knowledge from the scientific literature that can
subsequently be processed by computer algorithms. At first glance, a natural
first step would be to use the features of a large language model (such as
GPT3 Brown  _et al._ (2020), Gopher Rae  _et al._ (2021), MegaTron Smith  _et
al._ (2022) or PaLM Chowdhery  _et al._ (2022)) from the text of each article
to extract concepts automatically. However, those methods still struggle in
reasoning capabilities Kojima  _et al._ (2022); Zhang  _et al._ (2022), thus
it is not yet directly clear how these models can be used for identifying and
suggesting new ideas and concept combinations.

An alternative approach has been pioneered by Rzhetsky and colleagues Rzhetsky
_et al._ (2015). They have shown how knowledge networks (or semantic networks)
in biochemistry can be created from co-occurring concepts in scientific
papers. The nodes in their network correspond to scientific
concepts—concretely, the names of individual biomolecules. The nodes are
linked when a paper mentions both of the corresponding biomolecules in its
title or abstract. Taking millions of papers into account leads to an evolving
semantic network that captures the history of the field. Using supercomputer
simulations, non-trivial statements about the collective behaviour of
scientists can be extracted, which allows for the suggestions of alternative
and more efficient research behaviour Foster  _et al._ (2015). Of course, by
creating a semantic network from concept co-occurrences, only a tiny amount of
knowledge is extracted from each paper. However, if this process is repeated
for a large dataset of papers, the resulting network captures nontrivial and
actionable content.

The idea to build up a semantic network of a scientific discipline was then
applied and extended in the field of quantum physics Krenn and Zeilinger
(2020). There, the authors (including one of us) built a network of more than
6,000 quantum physics concepts. The authors formulate the task of predicting
new research trends and connections for the first time as an ML task. The task
was to identify which concept pairs, which have never been discussed jointly
in the scientific literature, have a high probability to be investigated in
the near future. This prediction task was phrased as one component for
personalized suggestions of new research ideas.

###  II.1 Link Prediction in Semantic Networks

Here we formulate the predictions of future research topics as a link
prediction task in an exponentially growing semantic network in the field of
AI. Two nodes that do not share an edge have not been mentioned together in
the title or abstract of an existing scientific paper. Here, the goal is to
predict which unconnected nodes will be connected in the future—that is,
determine which scientific concepts that have not been researched yet will be
jointly researched in the future.

Link prediction is a very common problem in computer science that can be
solved with classical metrics and features as well as machine learning
techniques. From the network theory side, several works have studied local
motif-based methods Liben-Nowell and Kleinberg (2007); Albert and Albert
(2004); Zhou  _et al._ (2009); Kovács  _et al._ (2019); Muscoloni  _et al._
(2018), often based on path-counting, while other methods have studied more
global features using linear optimization Pech  _et al._ (2019), global
perturbations Lü  _et al._ (2015) and stochastic block models Guimerà and
Sales-Pardo (2009). Other machine-learning works have tried to optimize over a
combination of hundred of predictors Ghasemian  _et al._ (2020). Further
discussion on these methods is available in a recent review on link prediction
Zhou (2021).

In Krenn and Zeilinger (2020), this task was solved by computing 17 hand-
crafted features of the evolving semantic network. In the Science4Cast
competition, the goal was to find more precise methods for link-prediction
tasks in semantic networks (a semantic network of AI that is 10 times larger
than the one in Krenn and Zeilinger (2020)). Specifically, on the one hand, we
would like to determine which features are useful; on the other hand, we would
also like to know whether this task can be solved efficiently without hand-
crafted features. Here, we present results for both questions.

###  II.2 Potential for Idea Generation in Science

The long-term goal of predictions and suggestions in semantic networks is to
provide new ideas to individual researchers. In a way, we hope to build a
creative artificial muse in science Krenn  _et al._ (2022). We can bias or
constrain the model to give research topics that are related to the research
interest of individual scientists, or a pair of scientists to suggest topics
for collaborations in an interdisciplinary setting. Important future questions
concern the discovery of impactful and surprising suggestions, and suggestions
that give more context than two scientific concepts.

##  III Generation and Analysis of the Dataset

###  III.1 Dataset Construction

We use papers that are published on arXiv in the categories cs.AI, cs.LG,
cs.NE, and stat.ML, from 1992 to 2020, to create a dynamic semantic network.
The nodes stand for computer science and in particular artificial intelligence
concepts. We create the list of concepts from the title and abstracts of all
of the 143,000 papers. We use Rapid Automatic Keyword Extraction (RAKE) to
create candidate concepts Rose  _et al._ (2010), and normalize the list using
standard NLP techniques and other self-created methods. Ultimately, this leads
to a list of 64,719 concepts.

These concepts form the nodes of the semantic network. The edges are drawn
when two concepts co-appear in a title or abstract of a paper. Each edge has a
time stamp, which is the publication date of the paper in which the concepts
co-appear. Multiple edges with different time-stamps between two concepts are
very common, as concept pairs can co-appear in many papers with different
publication dates. As edges have time stamps, the entire semantic network is
evolving in time. The workflow is depicted in Fig. 2.

###  III.2 Network-Theoretical Analysis

Figure 3: Node degrees follow heavy-tail distribution due to the hubs. Nodes
with the largest (466,319) and smallest (2) non-zero degrees correspond to
neural network and video compression technique, respectively. The most common
non-zero degree is 64. 1,247 nodes with zero degrees are not shown in this
plot, and both axes are in log scale.

We start by analyzing the degree distribution of the published semantic
network. The network has 64,719 nodes and 17,892,352 unique undirected edges,
which implies a mean node degree of about 553. However, the network contains
many hub nodes that significantly exceed this mean degree, demonstrated by the
heavy-tail degree distribution in Fig. 3. For example, the ten highest node
degrees (and their corresponding concepts) are 466,319 (neural network),
198,050 (deep learning), 195,345 (machine learning), 169,555 (convolutional
neural network), 159,403 (real world), 150,227 (experimental result), 127,642
(deep neural network), 115,334 (large scale), 89,267 (high dimension), and
84,956 (high dimensional).

Figure 4: The network became more connected over the years. Primary (left,
blue) vertical axis: Number of connected components with more than one node.
Secondary (right, orange) vertical axis: Number of nodes in the largest
connected component. For example, the network in 2019 comprises of one large
connected component with 63,472 nodes and 1,247 isolated nodes, i.e. nodes
with no edges. On the other hand, the 2001 network has 19 connected components
with size greater than one, the largest of which has 2,733 nodes.

To investigate whether this complex network is scale-free, we fit a power-law
curve to the degree distribution $p(k)$ using Alstott  _et al._ (2014), and
the software fit $p(k)\propto k^{-2.28}$ for degree $k\geq 1672$.
Nevertheless, the degree distribution of real complex network do not always
follow perfect power-laws and power-laws with exponential cut-offs are often a
better fit than pure power-laws Fenner  _et al._ (2007).

A recent work Broido and Clauset (2019) empirically showed that log-normal
distributions fit most real-world networks as well as or better than power
laws, and confirmed that pure “scale-free networks are rare”. In light of that
result, we used likelihood ratio tests to compare the power law fit with
alternative distributions. The likelihood ratio tests from Alstott  _et al._
(2014) suggested that truncated power law ($p$-value: 0.0031), lognormal
($p$-value: 0.0045), and lognormal positive ($p$-value: 0.015) fit the data
better than power law, while exponential ($p$-value: 3e-10) and stretched
exponential ($p$-value: 6e-05) were worse. We could not conclude whether
truncated power law, lognormal, or lognormal positive best describe the data
with $p$-value $\leq 0.1$.

Next, we discuss changes in the network connectivity over time. While the
degree distributions maintained a heavy tail over the years, the ordering of
the nodes inside the heavy tail changed, likely in response to the popularity
trends in the field. The nodes with most connections (and the year they became
so) are decision tree (1994), machine learning (1996), logic program (2000),
neural network (2005), experimental result (2011), machine learning (2013),
and finally, back to neural network (2015).

Furthermore, the network grew more connected over time according to connected
component analysis in Fig. 4. Groups that were previously separated became
connected, i.e. number of connected components decreased, while the largest
group grew bigger. The trajectory of the mid-sized connected components may
reveal interesting trends about their topics. Take image processing for
instance. A connected component of the following 4 nodes appeared in 1999:
brightness change, planar curve, local feature, and differential invariant. In
2000, 3 more nodes joined the group: similarity transformation, template
matching, and invariant representation. Then in 2006, a paper that discusses
both support vector machine and local feature merged this mid-size group of
nodes to the largest connected component.

Figure 5: Centralization of Concepts. The fraction of nodes (concepts) that
corresponds to the fraction of edges (connections). Cumulative histogram of
edges per node, up to a given year (1999,2004,2009,2014). The graph was
created by going over the edges list and adding to each year only the edges
and nodes that are dated before the year (hence the 2014 plot contains all the
nodes (concepts) in papers before 2014). The nodes are arranged by increasing
degrees. The plot is a cumulative graph; hence the y value in the x=80, is the
fraction of edges contributed by all the nodes in and below the 80s percentile
of degrees.

Another trend that emerges from the semantic network is an increase in
centralization over time, with fewer percentage nodes (concepts) contributing
larger fraction edges (concepts combination) over the years. This trend seems
to be consistent across the entire period of the dataset. It can be seen from
the histogram in Fig. 5 that the fraction of edges corresponding to the
highest degree nodes (most connected) increases over the years, while the
fraction of edges corresponding to the least connected nodes decreases. This
trend is also consistent with the decrease in the average clustering
coefficient over time (average clustering coefficient by year: 1999: 0.919,
2004: 0.844, 2009: 0.773, 2013: 0.650), implying most nodes are less likely to
be connected with each other and more likely to be connected to a few high-
degree central nodes. This trend might be explained by the fact that the AI
community has been focusing on a few methods (e.g. deep learning) which have
grown to dominate the field, compared to more diverse approaches in the 90s
and 2000s. An alternative explanation is the use of more consistent
terminology.

Figure 6: Receiver operating characteristic curve (ROC) for computing the Area
under the Curve (AUC). Random Predictions get the result right in half of the
cases, therefore their ROC curve is a diagonal with an AUC=0.5 (orange). A
model that has learned some properties of the dataset has a $\text{AUC}>0.5$
(blue). Figure 7: The Science4Cast benchmark: Link predictions in an
exponentially growing semantic network. Here we show the AUC values for
different models that use machine learning techniques (ML), hand-crafted
network features (NF) or a combination thereof. The left plot shows results
for the prediction of a single new link (i.e., $w=1$), the right one shows
results for the prediction of new triple links $w=3$. The task is to predict
$\delta=[1,3,5]$ years into the future, with cutoff values $c=[0,5,25]$. We
sort the models by the the results for the task ($w=1,\delta=3,c=0$), which
was the task in the Science4Cast competition. Data points that are not shown
have a AUC below 0.6 or are not computed due to computational costs. Note that
the prediction of new triple edges can be performed nearly determinstically.
It will be interesting to understand the origin of this quasi-deterministic
pattern in AI research.

###  III.3 Problem Formulation: Predictions in an exponentially growing
semantic network

The concrete task is to predict which two nodes $v_{i}$ with degrees
$d(v_{i})\leq c$ that do not share an edge in the year (2021-$\delta$) will
have $w$ edges in the year 2021. We use $\delta=1,3,5$, $c=0,5,25$ and
$w=1,3$. Note that $c=0$ is an interesting special case in which the node does
not have any edge associated to it yet in the initial year. Thus, the model
does not have any information about the node yet; the task there is to predict
which nodes will be connected to entirely new edges. The task $w=3$ goes
beyond simple link prediction, and asks which uninvestigated concept pair will
be studied together in at least 3 papers.222One interesting alternative task
is the prediction of the fastest growing links, which one could denote as
trend prediction.

In the task, we provide a list of 10 million unconnected nodes pairs (each
node having a degree$\leq$c) of the year (2021-$\delta$), and the goal is to
sort this list from highest to lowest probability that in 2021 they will have
at least $w$ edges.

For the evaluation we use the ROC curve Fawcett (2004); see Fig. 6 for
details. The ROC curve is created by plotting the true positive rate against
the false positive rate at various threshold settings. Our evaluation metric
is the commonly used metric Area under the Curve (AUC) of the ROC curve. One
advantage of AUC over mean-square-error is its independence of the data
distribution. Specifically, in our case, where the two classes are highly
asymmetrically distributed (with only about 1-3% of newly connected edges),
and the distribution changing over time, the AUC provides a meaningful and
interpretation. For perfect predictions, AUC=$1$, while random predictions
give AUC=$0.5$. It gives the percentage that a random true element is higher
ranked than a random false one.

##  IV AI-based Solutions

We now demonstrate how to solve this task with numerous different methods,
from pure statistical approaches to hand-crafted features (NF) as an input of
a neural network, to ML models that can work without hand-crafted features.
All results are shown in Fig. 7. The most powerful methods (those with the
highest prediction quality measured by the AUC metric) take advantage of NF,
which are the inputs to an ML model. Surprisingly, using purely network
theoretical features without machine learning works competitively. Pure ML
methods were not yet able to outperform those that use hand-crafted features.
It remains an important open challenge how to solve this task without relying
on hand-crafted features. While the prediction of new links can reach an AUC
of up to 93%, we find that the prediction of links that are generated at least
three times can be solved with AUC¿99.5%. Understanding this apparently quasi-
deterministic pattern in AI research will be an interesting target for follow-
up research.333We have performed numerous additional tests to exclude data
leakage in the benchmark dataset, overfitting or data duplication both in the
set of articles and the set of concepts.

###  IV.1 M1: Features+ML

The solution of team oahciy is based on a blend of a tree-based gradient
boosting approach and a graph neural network approach Lu (2021). Extensive
feature engineering was conducted to capture the centralities of the nodes,
the proximity between node pairs, and their evolution over time. The
centrality of a node is captured by the number of neighbours and the PageRank
score Brin and Page (1998), while the proximity between a node pair is derived
using the Jaccard index. We refer the reader to Lu (2021) for the list of all
features and their feature importance.

The tree-based gradient boosting approach uses the Light Gradient Boosting
Machine (LightGBM) Ke  _et al._ (2017) and applies heavy regularization to
combat overfitting due to the scarcity of positive samples. The graph neural
network approach employs a time-aware graph neural network to learn node
representations on dynamic semantic networks.

###  IV.2 M2: Features+ML

The method proposed by Team HashBrown assumes that the probability that nodes
$u$ and $v$ form an edge in the future is a function of the node features
$f(u)$, $f(v)$, and some edge feature $h(u,v)$. We chose node features $f$
that capture popularity at the current time $t_{0}$ (such as degree,
clustering coefficient Holland and Leinhardt (1971); Watts and Strogatz
(1998), and PageRank Brin and Page (1998)). We also use these features’ first
and second time-derivatives to capture the evolution of the node’s popularity
over time. After variable selection during training, we chose $h$ to consist
of the HOP-rec score Yang  _et al._ (2018); Lin (2021) and a variation of the
Dice similarity score Sorensen (1948) as a measure of similarity between
nodes. In summary, we use 31 node features for each node, and two edge
features, which gives $31\times 2+2=64$ features in total. These features are
then fed into a small multilayer perceptron (MLP) (5 layers, each with 13
neurons) with ReLU activation.

Cold start is the problem that some nodes in the test set do not appear in the
training set. Our strategy for a cold start is imputation. We say a node $v$
is seen if it appeared in the training data, and unseen otherwise; similarly,
we say that a node is born at time $t$ if $t$ is the first time stamp where an
edge linking this node has appeared. The idea is that an unseen node is simply
a node born in the future, so its features should look like a recently born
node in the training set. If a node is unseen, then we impute its features as
the average of the features of the nodes born recently. We found that with
imputation during training, the test AUC scores across all models consistently
increased by about 0.02. For a complete description of this method, we refer
the reader to Tran and Xie (2021).

###  IV.3 M3: Features+ML

This approach, detailed in Sanjabi (2021), uses hand-crafted node features
that have been captured in multiple time snapshots (e.g. every year) and then
uses an LSTM to benefit from learning the time dependencies of these features.
The final configuration uses two main types of features: node features
including degree and degree of neighbours, and edge features including common
neighbours. And to balance the training data the same number of positive and
negative instances have been randomly sampled and combined.

One of the goals was to identify features that are very informative with a
very low computational cost. We found that the degree centrality of the nodes
is the most important feature, and the degree centrality of the neighbouring
nodes and the degree of mutual neighbours gave us the best tradeoff. As all of
the extracted features’ distributions are highly skewed to the right, meaning
most of the features take near zero values, using a power transform like Yeo-
Johnson Yeo and Johnson (2000) helps to make the distributions more Gaussian
which boosts the learning. Finally, for the link prediction task, we saw that
LSTMs perform better than fully connected neural networks.

###  IV.4 M4: pure Features

The following two methods are based on a purely statistical analysis of the
test data and are explained in detail in Moutinho  _et al._ (2021).

Preferential Attachment – In the network analysis we concluded that the growth
of this dataset tends to maintain a heavy-tailed degree distribution, often
associated with scale-free networks. As mentioned before the $\gamma$-value of
the degree distribution is very close to $2$, suggesting that preferential-
attachment Barabási (2013) is likely the main organizational principle of the
network. As such, we implemented a simple prediction model following this
procedure. Preferential-attachment scores in link prediction are often
quantified as

$s_{ij}^{\text{PA}}=k_{i}\cdot k_{j}.$ (1)

with $k_{i,j}$ the degree of nodes $i$ and $j$. However, this assumes the
scoring of links between nodes that are already connected to the network, that
is $k_{i,j}>0$, which is not the case for all the links we must score in the
dataset. As a result, we define our preferential attachment model as

$s_{ij}^{\text{PA}}=k_{i}+k_{j}.$ (2)

Using this simple model with no free parameters we could score new links and
compare them with the other models. Immediately we note that preferential
attachment outperforms some learning-based models, even if it never manages to
reach the top AUC, but it is extremely simple and with negligible
computational cost.

Common Neighbours – We explore another network-based approach to score the
links. Indeed, while the preferential attachment model we derived performed
well, it uses no information about the distance between $i$ and $j$, which is
a popular feature used in link prediction methods Zhou (2021). As such we
decided to test a method known as Common Neighbours Liben-Nowell and Kleinberg
(2007). If we define $\Gamma(i)\cap\Gamma(j)$ as the set of common neighbours
between nodes $i$ and $j$. We can easily score the nodes with

$s_{ij}^{\text{CN}}=|\Gamma(i)\cap\Gamma(j)|$ (3)

the intuition being that nodes which share a larger number of neighbours are
more likely to be connected than distant nodes that do not share any.

Evaluating this score for each pair $(i,j)$ on the dataset of unconnected
pairs, which can be computed as the second power of the adjacency matrix,
$A^{2}$, we obtained an AUC which is sometimes higher than preferential
attachment and sometimes lower than it but is still consistently quite close
with the best learning-based models.

###  IV.5 M5: Features + ML

This method is based on Valente (2021) with a modification disclosed in the
VI.3. First, 10 groups of first-order graph features are extracted to get some
neighbourhood and similarity properties from each pair of nodes: degree
centrality of nodes, pair’s total number of neighbours, common neighbours
index, Jaccard coefficient, Simpson coefficient, geometric coefficient, cosine
coefficient, Adamic-Adar index, resource allocation index, and preferential
attachment index. They are obtained for three consecutive years to capture the
temporal dynamics of the semantic network, leading to a total of 33 features.
Second, principal component analysis (PCA) Jolliffe and Cadima (2016) is
applied to reduce the correlation between features, speed up the learning
process and improve generalization, which results in a final set of 7 latent
variables. Lastly, a random forest classifier is trained (using a balanced
dataset) to estimate the likelihood of new links between the AI concepts.

###  IV.6 M6: Features+ML

The baseline solution for the Science4Cast competition was closely related to
the model presented in Krenn and Zeilinger (2020). It uses 15 hand-crafted
features of a pair of nodes $v_{1}$ and $v_{2}$ (Degrees of $v_{1}$ and
$v_{2}$ in the current year and previous two years, these are six properties.
The number of shared neighbours in total of $v_{1}$ and $v_{2}$ in the current
year and previous two years are six properties. The number of shared
neighbours between $v_{1}$ and $v_{2}$ in the current year and the previous
two years, these are 3 properties). These 15 features are the input of a
neural network with four layers (15, 100, 10, and 1 neurons), intending to
predict whether the nodes $v_{1}$ and $v_{2}$ will have $w$ edges in the
future. After the training, the model computes the probability for all 10
million evaluation examples. This list is sorted and the AUC is computed.

###  IV.7 M7: end-to-end ML (Transformers)

This model, which is detailed in Lee  _et al._ (2021), does not use any
handcrafted features but learns them in a completely unsupervised manner. To
do so, we extract various snapshots of the adjacency matrix through time,
capturing graphs in the form of $\bm{A}_{t}$ for $t=1994,\ldots,2019$. We then
embed each of these graphs into 128-dimensional Euclidean space via Node2vec
Grover and Leskovec (2016); Liu and Krishnan (2021). For each node $u$ in the
semantic graph, we extract different 128-dimensional vector embeddings
$\bm{n}_{u}(\bm{A}_{1994}),\ldots,\bm{n}_{u}(\bm{A}_{2019})$.

Transformers have performed extremely well in natural language processing
tasks Vaswani  _et al._ (2017), thus we apply them to learn the dynamics of
the embedding vectors. We pre-train a transformer to help classify node pairs.
For the transformer, the encoder and decoder had 6 layers each; we used 128 as
the embedding dimension, 2048 as the feedforward dimension and 8-headed
attention. This transformer acts as our feature extractor. Once we pre-train
our transformer, we add a 2-layer ReLU network with hidden dimension $128$ as
a classifier on top.

###  IV.8 M8: end-to-end ML (auto node embedding)

The most immediate way one can apply machine learning to this problem is by
automating the detection of features. Quite simply, the baseline solution M6
is modified such that instead of 15 hand-crafted features, the neural network
is instead trained on features extracted from a graph embedding. In our
approach, we use the ProNE embedding Zhang  _et al._ (2019), which is based on
sparse matrix factorizations modulated by the higher-order Cheeger inequality
Bandeira  _et al._ (2012), as well as Node2Vec Grover and Leskovec (2016). We
use the implementations provided in the nodevectors Python package Ranger
(2021).

The embeddings learn a 32-dimensional representation for each node; hence,
each edge representation is normalized to a single point in $[0,1]^{64}$, and
the concatenated features are the input of a neural network with two hidden
layers of size 1000 and 30, respectively. Similarly to M6, the model is then
tasked with computing the probability for the evaluation examples, which lets
us determine the ROC.

##  V Extensions and Future Work

Creating an AI that can suggest research topics to human scientists is highly
ambitious and challenging. The present work of link prediction for a temporal
network to draw connections between existing concepts is only the first step.
We point out several extensions and future works that are directly relevant to
the overarching goal of AI for AI.

High-quality predictions without feature engineering – Surprisingly, given a
graph with already extracted concepts as nodes and edges plotting the time
evolution of joint appearance of these concepts in publications, the most
powerful methods all used carefully hand-crafted features. It will be
interesting to see whether end-to-end deep learning methods can solve tasks
without feature engineering.

Fully automated concept extraction – The concept list at the moment is created
by a purely statistical text analysis using RAKE. The suggestions by RAKE are
then manually inspected and phrases that do not correspond to a concept are
removed. While this process can be partially automated (as RAKE often makes
the same mistakes which can be captured automatically), it is not a scalable
process if one wants to create concept lists for the much larger corpus of
science and engineering. A fully automated natural language processing
algorithm that can extract meaningful concepts with minimal mistakes would be
extremely useful.

Generation of new concepts – Here we predict the emergence of links between
two known concepts. One important question is whether an AI algorithm can
compose words and generate new concepts. Different from the current work that
is mostly supervised, the generation of new concepts is unsupervised, hence
more difficult. One approach to address this question has been presented in
Salatino  _et al._ (2017, 2018). There the authors can detect clusters of
concepts with specific dynamics that indicate the formation of a new concept.
It will be interesting to see how such emerging concepts can be incorporated
into the current framework and used for suggestions for new research topics.

Semantic information beyond concept pairs – At the moment, every article’s
abstract and title are compressed into several links between concept pairs.
This procedure does not represent all information in the article’s abstract
(let alone, the article itself). The more information one can extract from the
article, the more meaningful the predictions and suggestions will be.
Extending the representation of the semantic network to more complex data
structures, such as hypergraphs Battiston  _et al._ (2021) are likely to be
computationally more demanding but could significantly improve the prediction
qualities. It might be also possible to find some ways to decrease the
complexity of the analysis using clever tricks. For example, the authors in
Coutinho  _et al._ (2020) showed that the maximum node and hyperedge cover
problem, two computational NP-hard problems, can be solved in polynomial time
for most of the real-world hypergraphs tested. Whether such tricks exist for
hyperlink prediction is still an open problem. The inclusion of sociological
factors, such as the status of the involved researchers and their affiliations
might help in prediction tasks.

Predictions of scientific success – The prediction of a new link between nodes
in the semantic network means that we predict which concepts scientists will
study for the first time in the future. This prediction however does not say
anything about the potential importance and impact of the new connection. As a
tool for high-quality suggestions, we need to introduce the prediction of a
metric-of-success, for example, estimated citation numbers of the new link or
the rate of citation growth over time. This extension seems reasonable given
that modelling and predictions of citation information in citation networks
(where nodes are papers) is a prominent area of research within the science of
science Liu  _et al._ (2019); Reisz  _et al._ (2022). Adapting these
techniques to semantic networks will be an interesting future research
direction.

Anomaly detections – In a way, predicting the most likely new connection
between concepts does not necessarily directly coincide with the goal of
suggestions of new surprising research directions. After all, those links are
predictable, thus potentially not surprising by themselves. While we believe
that this type of prediction can still be a very useful contribution for
suggestions, there is another way to more directly find surprising
combinations, namely by finding anomalies in the semantic networks. Those are
potential links that have extreme properties in some metrics. There are
powerful deep learning methods for anomaly detection Kwon  _et al._ (2019);
Pang  _et al._ (2021) and their application in the semantic network presented
here might be very interesting. In fact, while scientists tend to study topics
in which they are already directly involved Fortunato  _et al._ (2018); Wang
and Barabási (2021), often higher scientific impact results from the
unexpected combination of more distant domains Rzhetsky  _et al._ (2015),
which foster the search for those surprising and impactful associations.

End-to-end formulation – As outlined above, we necessarily decomposed our goal
of extracting knowledge from the scientific literature into two subtasks:
extracting concepts and building and predicting the evolution of a semantic
network resulting from those concepts. This stands in contrast to the dominant
paradigm in deep learning that emerged over the last decade of so-called ‘end-
to-end’ training based on early spectacular successes Collobert  _et al._
(2011); Krizhevsky  _et al._ (2012); Mnih  _et al._ (2015); Silver  _et al._
(2016). In this paradigm, problems are not broken into sub-problems but solved
directly using deep differentiable architecture components trained via back-
propagation Glasmachers (2017); LeCun  _et al._ (2015). If such an ‘end-to-
end’ solution approach to our goal could be achieved it would be interesting
to see whether it could replicate the success this deep learning paradigm had
in other areas.

Human level machine comprehension – One of the defining goals of the Dartmouth
Summer Research Project on Artificial Intelligence in 1956 was the following:
‘An attempt will be made to find how to make machines use language, form
abstractions and concepts, solve kinds of problems now reserved for humans,
and improve themselves.’ McCarthy  _et al._ (2006). Such an algorithm would be
expected to handle an evolution in concept denotations due to new insights
(i.e. the emergence of the term ‘Gibbs entropy’ to distinguish Boltzmann’s
original concept of thermo-dynamical entropy as opposed to seeing it in the
light of the more general emergent ‘Shannon entropy’ or ‘von Neumann Entropy’)
or due to disputed originality (i.e. Bolai-Lobatchevskian Geometry and
Hyperbolic Geometry are the same concept). An algorithm with such natural
language understanding capabilities would thus be extremely useful to get
closer to our goal. Although large language models and other multimodally
trained language models like CLIP Radford  _et al._ (2021) or CLOOB Fürst  _et
al._ (2021) have achieved outstanding results recently, it is an open question
how much statistically trained natural language models alone could eventually
form concepts and abstractions on a human level Mitchell (2021); LeCun (2022).

##  VI Conclusion

Here we present a new AI benchmark for link prediction in exponentially
growing semantic networks. Several of the solutions have been collected in the
IEEE BigData Competition Science4Cast in fall 2021, and generalized to the
mode diverse tasks presented here. The goal was to boost the capabilities of
predicting future research directions in the field of AI itself, which grows
enormous over the decade. This ability might be an important part of a tool
that gives personalized research suggestions to human scientists in the
future. We find, rather surprisingly, that the prediction of strong new links
(those that are formed three or more times) can be predicted with extremely
high quality (AUC beyond 99%). It will be interesting to investigate this
quasi-deterministic pattern in AI research in more detail. The best methods
used a clever combination of hand-crafted features and machine learning. It
will be interesting whether pure learning methods, without hand-crafted
features, will achieve high-quality results in the future. We also point out a
number of open problems towards the goal of practical, personalized,
interdisciplinary AI-based suggestions for new impactful research direction –
which we believe could become a disruptive tool in the future.

## Appendix

###  VI.1 Model availability

All of the models described above can be found on GitHub. M1, M2, M3, M4, M5,
M6, M7, M8.

###  VI.2 Details on M9

The solution M9 was not part of the Science4Cast competition and therefore not
described in the corresponding proceedings, thus we want to add more details.
We compare the ProNE embedding to Node2Vec, which is also commonly used for
graph embedding problems. The algorithm maps each node of the network to a
point in 32-dimensional space based on a biased random walk procedure, which
is fundamentally parameterized by two variables—$p$, the “return parameter”,
and $q$, the “in-out parameter”. The return parameter determines the frequency
of backtracking in the random walk, while the in-out parameter determines
whether to bias the exploration to nearby nodes or distant nodes. Notably,
these parameters significantly affect how the network is encoded—for instance,
in the BlogCatalog dataset, optimal parameters were $p=0.25,q=0.25$, whereas
for the Wikipedia graph, they were $p=4,q=0.5$ Grover and Leskovec (2016). In
initial experiments, we used the default $p=q=1$ for a 64-dimensional
encoding, before feeding it into the same neural network as for the ProNE
experiment. The higher variance in Node2Vec-based predictions likely has to do
with the method’s significant sensitivity to its hyperparameters. While ProNE
is clearly better suited for a general multi-dataset link prediction problem,
Node2Vec’s parameter sensitivity may help us identify what features of the
network are most important for predicting its temporal evolution.

###  VI.3 Consideration for Model M6

In this manuscript, a modification was performed in relation to the original
formulation of the method Valente (2021): two of the original features,
average neighbor degree and clustering coefficient, were infeasible to extract
for some of the tasks covered in this paper, as their computation can be heavy
for such a very large network, and they were discarded. Due to some
computational memory issues, it was not possible to run the model for some of
the tasks covered in this study, and so those results are missing.

## Acknowledgements

The authors thank IARAI Vienna and IEEE for supporting and hosting the IEEE
BigData Competition Science4Cast. We are specifically grateful to David Kreil,
Moritz Neun, Christian Eichenberger, Markus Spanring, Henry Martin, Dirk
Geschke, Daniel Springer, Pedro Herruzo, Marvin McCutchan, Alina Mihai, Toma
Furdui, Gabi Fratica, Miriam Vázquez, Aleksandra Gruca, Johannes Brandstetter
and Sepp Hochreiter for helping to set up and successfully execute the
competition and the corresponding workshop. The authors thank Xuemei Gu for
creating Fig.2, and Milad Aghajohari and Mohammad Sadegh Akhondzadeh for
helpful comments on the manuscript. The work of HL, RS, and JGF were supported
by grant TWCF0333 from the Templeton World Charity Foundation. HL is
additionally supported by NSF grant DMS-1952339. JPM acknowledges the support
of FCT (Portugal) through scholarship SFRH/BD/144151/2019. BC thanks the
support from FCT/MCTES through national funds and when applicable co-funded EU
funds under the project UIDB/50008/2020, and FCT through the project
CEECINST/00117/2018/CP1495/CT0001. NMT and YX are supported by NSF Grant
DMS-2113468, the NSF IFML 2019844 award to the University of Texas at Austin,
and the Good Systems Research Initiative, part of University of Texas at
Austin Bridging Barriers.

## References

  * Evans and Foster (2011) James A Evans and Jacob G Foster, “Metaknowledge,” Science 331, 721–725 (2011).
  * Fortunato  _et al._ (2018) Santo Fortunato, Carl T Bergstrom, Katy Börner, James A Evans, Dirk Helbing, Staša Milojević, Alexander M Petersen, Filippo Radicchi, Roberta Sinatra, Brian Uzzi, _et al._ , “Science of science,” Science 359, eaao0185 (2018).
  * Wang and Barabási (2021) Dashun Wang and Albert-László Barabási,  _The science of science_ (Cambridge University Press, 2021).
  * Brown  _et al._ (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, _et al._ , “Language models are few-shot learners,” Advances in neural information processing systems 33, 1877–1901 (2020).
  * Rae  _et al._ (2021) Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, _et al._ , “Scaling language models: Methods, analysis & insights from training gopher,” arXiv:2112.11446 (2021).
  * Smith  _et al._ (2022) Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, _et al._ , “Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model,” arXiv:2201.11990 (2022).
  * Chowdhery  _et al._ (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, _et al._ , “Palm: Scaling language modeling with pathways,” arXiv:2204.02311 (2022).
  * Kojima  _et al._ (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa, “Large language models are zero-shot reasoners,” arXiv:2205.11916 (2022).
  * Zhang  _et al._ (2022) Honghua Zhang, Liunian Harold Li, Tao Meng, Kai-Wei Chang, and Guy Van den Broeck, “On the paradox of learning to reason from data,” arXiv:2205.11502 (2022).
  * Rzhetsky  _et al._ (2015) Andrey Rzhetsky, Jacob G Foster, Ian T Foster, and James A Evans, “Choosing experiments to accelerate collective discovery,” Proceedings of the National Academy of Sciences 112, 14569–14574 (2015).
  * Foster  _et al._ (2015) Jacob G Foster, Andrey Rzhetsky, and James A Evans, “Tradition and innovation in scientists’ research strategies,” American Sociological Review 80, 875–908 (2015).
  * Krenn and Zeilinger (2020) Mario Krenn and Anton Zeilinger, “Predicting research trends with semantic and neural networks with an application in quantum physics,” Proceedings of the National Academy of Sciences 117, 1910–1916 (2020).
  * Liben-Nowell and Kleinberg (2007) David Liben-Nowell and Jon Kleinberg, “The link-prediction problem for social networks,” Journal of the American society for information science and technology 58, 1019–1031 (2007).
  * Albert and Albert (2004) István Albert and Réka Albert, “Conserved network motifs allow protein–protein interaction prediction,” Bioinformatics 20, 3346–3352 (2004).
  * Zhou  _et al._ (2009) Tao Zhou, Linyuan Lü, and Yi-Cheng Zhang, “Predicting missing links via local information,” The European Physical Journal B 71, 623–630 (2009).
  * Kovács  _et al._ (2019) István A Kovács, Katja Luck, Kerstin Spirohn, Yang Wang, Carl Pollis, Sadie Schlabach, Wenting Bian, Dae-Kyum Kim, Nishka Kishore, Tong Hao, _et al._ , “Network-based prediction of protein interactions,” Nature communications 10, 1–8 (2019).
  * Muscoloni  _et al._ (2018) Alessandro Muscoloni, Ilyes Abdelhamid, and Carlo Vittorio Cannistraci, “Local-community network automata modelling based on length-three-paths for prediction of complex network structures in protein interactomes, food webs and more,” bioRxiv , 346916 (2018).
  * Pech  _et al._ (2019) Ratha Pech, Dong Hao, Yan-Li Lee, Ye Yuan, and Tao Zhou, “Link prediction via linear optimization,” Physica A: Statistical Mechanics and its Applications 528, 121319 (2019).
  * Lü  _et al._ (2015) Linyuan Lü, Liming Pan, Tao Zhou, Yi-Cheng Zhang, and H Eugene Stanley, “Toward link predictability of complex networks,” Proceedings of the National Academy of Sciences 112, 2325–2330 (2015).
  * Guimerà and Sales-Pardo (2009) Roger Guimerà and Marta Sales-Pardo, “Missing and spurious interactions and the reconstruction of complex networks,” Proceedings of the National Academy of Sciences 106, 22073–22078 (2009).
  * Ghasemian  _et al._ (2020) Amir Ghasemian, Homa Hosseinmardi, Aram Galstyan, Edoardo M Airoldi, and Aaron Clauset, “Stacking models for nearly optimal link prediction in complex networks,” Proceedings of the National Academy of Sciences 117, 23393–23400 (2020).
  * Zhou (2021) Tao Zhou, “Progresses and challenges in link prediction,” Iscience 24, 103217 (2021).
  * Krenn  _et al._ (2022) Mario Krenn, Robert Pollice, Si Yue Guo, Matteo Aldeghi, Alba Cervera-Lierta, Pascal Friederich, Gabriel dos Passos Gomes, Florian Häse, Adrian Jinich, AkshatKumar Nigam, _et al._ , “On scientific understanding with artificial intelligence,” arXiv:2204.01467 (2022).
  * Rose  _et al._ (2010) Stuart Rose, Dave Engel, Nick Cramer, and Wendy Cowley, “Automatic keyword extraction from individual documents,” Text mining: applications and theory 1, 1–20 (2010).
  * Alstott  _et al._ (2014) Jeff Alstott, Ed Bullmore, and Dietmar Plenz, “powerlaw: a python package for analysis of heavy-tailed distributions,” PloS one 9, e85777 (2014).
  * Fenner  _et al._ (2007) Trevor Fenner, Mark Levene, and George Loizou, “A model for collaboration networks giving rise to a power-law distribution with an exponential cutoff,” Social Networks 29, 70–80 (2007).
  * Broido and Clauset (2019) Anna D Broido and Aaron Clauset, “Scale-free networks are rare,” Nature communications 10, 1–10 (2019).
  * Fawcett (2004) Tom Fawcett, “Roc graphs: Notes and practical considerations for researchers,” Machine learning 31, 1–38 (2004).
  * Lu (2021) Yichao Lu, “Predicting research trends in artificial intelligence with gradient boosting decision trees and time-aware graph neural networks,” in  _2021 IEEE International Conference on Big Data (Big Data)_ (IEEE, 2021) pp. 5809–5814.
  * Brin and Page (1998) Sergey Brin and Lawrence Page, “The anatomy of a large-scale hypertextual web search engine,” Computer networks and ISDN systems 30, 107–117 (1998).
  * Ke  _et al._ (2017) Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu, “Lightgbm: A highly efficient gradient boosting decision tree,” Advances in neural information processing systems 30 (2017).
  * Holland and Leinhardt (1971) Paul W Holland and Samuel Leinhardt, “Transitivity in structural models of small groups,” Comparative group studies 2, 107–124 (1971).
  * Watts and Strogatz (1998) Duncan J Watts and Steven H Strogatz, “Collective dynamics of ‘small-world’networks,” nature 393, 440–442 (1998).
  * Yang  _et al._ (2018) Jheng-Hong Yang, Chih-Ming Chen, Chuan-Ju Wang, and Ming-Feng Tsai, “Hop-rec: high-order proximity for implicit recommendation,” in  _Proceedings of the 12th ACM Conference on Recommender Systems_ (2018) pp. 140–144.
  * Lin (2021) Bo-Yu Lin, “Ogb collab project,” https://github.com/brucenccu/OGB_collab_project (2021).
  * Sorensen (1948) Th A Sorensen, “A method of establishing groups of equal amplitude in plant sociology based on similarity of species content and its application to analyses of the vegetation on danish commons,” Biol. Skar. 5, 1–34 (1948).
  * Tran and Xie (2021) Ngoc Mai Tran and Yangxinyu Xie, “Improving random walk rankings with feature selection and imputation science4cast competition, team hash brown,” in  _2021 IEEE International Conference on Big Data (Big Data)_ (IEEE, 2021) pp. 5824–5827.
  * Sanjabi (2021) Nima Sanjabi, “Efficiently predicting scientific trends using node centrality measures of a science semantic network,” in  _2021 IEEE International Conference on Big Data (Big Data)_ (IEEE, 2021) pp. 5820–5823.
  * Yeo and Johnson (2000) In-Kwon Yeo and Richard A Johnson, “A new family of power transformations to improve normality or symmetry,” Biometrika 87, 954–959 (2000).
  * Moutinho  _et al._ (2021) João P Moutinho, Bruno Coutinho, and Lorenzo Buffoni, “Network-based link prediction of scientific concepts–a science4cast competition entry,” in  _2021 IEEE International Conference on Big Data (Big Data)_ (IEEE, 2021) pp. 5815–5819.
  * Barabási (2013) Albert-László Barabási, “Network science,” Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 371, 20120375 (2013).
  * Valente (2021) Francisco Valente, “Link prediction of artificial intelligence concepts using low computational power,” in  _2021 IEEE International Conference on Big Data (Big Data)_ (2021) pp. 5828–5832.
  * Jolliffe and Cadima (2016) Ian T. Jolliffe and Jorge Cadima, “Principal component analysis: a review and recent developments,” Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 374, 20150202 (2016).
  * Lee  _et al._ (2021) Harlin Lee, Rishi Sonthalia, and Jacob G Foster, “Dynamic embedding-based methods for link prediction in machine learning semantic network,” in  _2021 IEEE International Conference on Big Data (Big Data)_ (IEEE, 2021) pp. 5801–5808.
  * Grover and Leskovec (2016) Aditya Grover and Jure Leskovec, “node2vec: Scalable feature learning for networks,” in  _Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining_ (2016) pp. 855–864.
  * Liu and Krishnan (2021) Renming Liu and Arjun Krishnan, “PecanPy: a fast, efficient and parallelized Python implementation of node2vec,” Bioinformatics 37, 3377–3379 (2021), https://academic.oup.com/bioinformatics/article-pdf/37/19/3377/40556655/btab202.pdf .
  * Vaswani  _et al._ (2017) Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin, “Attention is all you need,” ArXiv abs/1706.03762 (2017).
  * Zhang  _et al._ (2019) Jie Zhang, Yuxiao Dong, Yan Wang, Jie Tang, and Ming Ding, “Prone: Fast and scalable network representation learning,” in  _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_ (International Joint Conferences on Artificial Intelligence Organization, 2019) pp. 4278–4284.
  * Bandeira  _et al._ (2012) Afonso S. Bandeira, Amit Singer, and Daniel A. Spielman, “A cheeger inequality for the graph connection laplacian,”  (2012).
  * Ranger (2021) Matt Ranger, “nodevectors,” https://github.com/VHRanger/nodevectors (2021).
  * Salatino  _et al._ (2017) Angelo A Salatino, Francesco Osborne, and Enrico Motta, “How are topics born? understanding the research dynamics preceding the emergence of new areas,” PeerJ Computer Science 3, e119 (2017).
  * Salatino  _et al._ (2018) Angelo A Salatino, Francesco Osborne, and Enrico Motta, “Augur: forecasting the emergence of new research topics,” in  _Proceedings of the 18th ACM/IEEE on joint conference on digital libraries_ (2018) pp. 303–312.
  * Battiston  _et al._ (2021) Federico Battiston, Enrico Amico, Alain Barrat, Ginestra Bianconi, Guilherme Ferraz de Arruda, Benedetta Franceschiello, Iacopo Iacopini, Sonia Kéfi, Vito Latora, Yamir Moreno, _et al._ , “The physics of higher-order interactions in complex systems,” Nature Physics 17, 1093–1098 (2021).
  * Coutinho  _et al._ (2020) Bruno Coelho Coutinho, Ang-Kun Wu, Hai-Jun Zhou, and Yang-Yu Liu, “Covering problems and core percolations on hypergraphs,” Phys. Rev. Lett. 124, 248301 (2020).
  * Liu  _et al._ (2019) Hanwen Liu, Huaizhen Kou, Chao Yan, and Lianyong Qi, “Link prediction in paper citation network to construct paper correlation graph,” EURASIP Journal on Wireless Communications and Networking 2019, 1–12 (2019).
  * Reisz  _et al._ (2022) Niklas Reisz, Vito D P Servedio, Vittorio Loreto, William Schueller, Márcia R Ferreira, and Stefan Thurner, “Loss of sustainability in scientific work,” New Journal of Physics 24, 053041 (2022).
  * Kwon  _et al._ (2019) Donghwoon Kwon, Hyunjoo Kim, Jinoh Kim, Sang C Suh, Ikkyun Kim, and Kuinam J Kim, “A survey of deep learning-based network anomaly detection,” Cluster Computing 22, 949–961 (2019).
  * Pang  _et al._ (2021) Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel, “Deep learning for anomaly detection: A review,” ACM Computing Surveys (CSUR) 54, 1–38 (2021).
  * Collobert  _et al._ (2011) Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa, “Natural language processing (almost) from scratch,” Journal of machine learning research 12, 2493–2537 (2011).
  * Krizhevsky  _et al._ (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in neural information processing systems 25 (2012).
  * Mnih  _et al._ (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, _et al._ , “Human-level control through deep reinforcement learning,” nature 518, 529–533 (2015).
  * Silver  _et al._ (2016) David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, _et al._ , “Mastering the game of go with deep neural networks and tree search,” nature 529, 484–489 (2016).
  * Glasmachers (2017) Tobias Glasmachers, “Limits of end-to-end learning,” in  _Asian conference on machine learning_ (PMLR, 2017) pp. 17–32.
  * LeCun  _et al._ (2015) Yann LeCun, Yoshua Bengio, and Geoffrey Hinton, “Deep learning,” nature 521, 436–444 (2015).
  * McCarthy  _et al._ (2006) John McCarthy, Marvin L Minsky, Nathaniel Rochester, and Claude E Shannon, “A proposal for the dartmouth summer research project on artificial intelligence, august 31, 1955,” AI magazine 27, 12–12 (2006).
  * Radford  _et al._ (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, _et al._ , “Learning transferable visual models from natural language supervision,” in  _International Conference on Machine Learning_ (PMLR, 2021) pp. 8748–8763.
  * Fürst  _et al._ (2021) Andreas Fürst, Elisabeth Rumetshofer, Viet Tran, Hubert Ramsauer, Fei Tang, Johannes Lehner, David Kreil, Michael Kopp, Günter Klambauer, Angela Bitto-Nemling, _et al._ , “Cloob: Modern hopfield networks with infoloob outperform clip,” arXiv:2110.11316 (2021).
  * Mitchell (2021) Melanie Mitchell, “Abstraction and analogy-making in artificial intelligence,” Annals of the New York Academy of Sciences 1505, 79–101 (2021).
  * LeCun (2022) Yann LeCun, “A path towards autonomous machine intelligence,” openreview preprint (2022).