# Leveraging Linear Independence of Component Classifiers: Optimizing Size and
Prediction Accuracy for Online Ensembles

Enes Bektas
Bilkent Information Retrieval Group
Bilkent University
Ankara
<EMAIL_ADDRESS>
&Fazli Can
Bilkent Information Retrieval Group
Bilkent University
Ankara
<EMAIL_ADDRESS>

###### Abstract

Ensembles, which employ a set of classifiers to enhance classification
accuracy collectively, are crucial in the era of big data. However, although
there is general agreement that the relation between ensemble size and its
prediction accuracy, the exact nature of this relationship is still unknown.
We introduce a novel perspective, rooted in the linear independence of
classifier’s votes, to analyze the interplay between ensemble size and
prediction accuracy. This framework reveals a theoretical link, consequently
proposing an ensemble size based on this relationship. Our study builds upon a
geometric framework and develops a series of theorems. These theorems clarify
the role of linear dependency in crafting ensembles. We present a method to
determine the minimum ensemble size required to ensure a target probability of
linearly independent votes among component classifiers. Incorporating real and
synthetic datasets, our empirical results demonstrate a trend: increasing the
number of classifiers enhances accuracy, as predicted by our theoretical
insights. However, we also identify a point of diminishing returns, beyond
which additional classifiers provide diminishing improvements in accuracy.
Surprisingly, the calculated ideal ensemble size deviates from empirical
results for certain datasets, emphasizing the influence of other factors. This
study opens avenues for deeper investigations into the complex dynamics
governing ensemble design and offers guidance for constructing efficient and
effective ensembles in practical scenarios.

_K_ eywords Big data stream $\cdot$ Ensemble learning $\cdot$ Ensemble design

##  1 Introduction

In the era of big data, where the volume and complexity of information have
reached unprecedented levels, the need for highly accurate classification
models gave rise to ensemble methods. As data generated by technological
devices continues to exceedingly grow, relying solely on individual
classifiers to classify vast amounts of data has diminished effectiveness.
Ensembles, which use a set of classifiers to create a greater classifier that
is better than each individual component classifier in the ensemble, offer a
solution by using the collective intelligence of multiple models [1, 2, 3, 4,
5]. The ensemble methods are used in several areas including statistics,
machine learning, pattern recognition and knowledge discovery in databases
[6].

The two most used methods for ensemble creation are weighting and meta-
combination [7]. Weighting methods assign a weight to each classifier, which
determines the strength of that classifier in the ensemble. Meta-combination
techniques utilize meta-learning, where classifiers learn from training data
and their own classifications on training data to improve prediction accuracy
[1, 7].

One crucial aspect of building an ensemble is determining the appropriate
number of classifiers to include. Even though utilizing more classifiers is
the main idea of ensembles, increasing the ensemble size comes with costs in
terms of memory and time requirements. Several studies have investigated how
the ensemble size can be determined. Bax suggested that any odd number of
classifiers may be optimal [8]. Oshiro et al. stated that adding more
classifiers beyond a certain threshold did not yield significant improvements
[9]. Latinne et al. used the McNemar test to limit the number of classifiers
in an ensemble [10]. Pietruczuk et al. determine ensemble size by deciding if
a new classifier increases the accuracy rate for the whole data stream [11].
Hernàndez-Lobato et al. uses statistics to determine ensemble size [12]. Hu
uses rough sets theory and database operations for constructing ensemble [13].

The contributions of this study are: (1) By examining the literature on
ensemble size and focusing on ensembles using weighting methods, this article
aims to explain the relationship between ensemble size and prediction
accuracy, and also as an outcome, (2) suggests a method to determine the
optimum number of classifiers in an ensemble. Preliminary work on this topic
was presented in [14], [15], [16], [17]. [18]. Section 2 introduces the
preliminary work. We used ’performance’ and ’prediction accuracy’
interchangeably.

##  2 Related Works

This paper focuses on a specific weighting method based on a geometric
framework introduced by Wu and Crestani for information retrieval systems
[15]. Bonab and Can apply this framework to ensemble classification using
weighted majority voting, where each classifier’s vote is treated as a vector
[14]. Wu and Ding extend the framework to dataset-level classification,
further exploring its applicability [16]. Ding et al. apply this framework for
human activity recognition [17].

Although this paper depends on a specific ensemble creation method, the
outcomes can also be generalized.

Table 1: Definition of Symbols Symbol |  Definition
---|---
D = {$I_{1},I_{2},...$} |  The data stream
$I_{k}$ |  An instance of data stream
m |  Number of class labels of data stream
E = {$C_{1},C_{2},...,C_{n}$} |  The ensemble with n-number of component classifiers
n |  Number of classifiers in the ensemble
$S_{i}$ |  Vote vector of $i^{th}$ classifier of the ensemble
$S_{ij}$ |  $j^{th}$ component of $i^{th}$ classifier’s vote
$W_{i}$ |  Weight of $i^{th}$ classifier in the ensemble
$o$ |  The ideal vector which is in the form (0,0,…,0,1,0,0,…,0)
$V=W_{1}S_{1}+W_{2}S_{2}+...+W_{n}S_{n}$ |  Vote of the ensemble
$p_{k}$ |  Probability of a classifier to be linearly dependent on the previous k-dimensional space

Let there be a data stream D with m-number of class labels and an ensemble E
with n-number of component classifiers. In this geometric framework, each vote
of a classifier is treated as a vector in m-dimensional space. Each vote is
normalized so that the sum of their components is 1, i.e.,
$\sum_{j=0}^{m-1}(S_{ij})=1$. An ideal vector, $o_{k}$, is defined as the
vector which points to the true class label of an instance $I_{k}$. The
ensemble’s vote is a linear combination of its component classifiers. The loss
function is defined as the Euclidian distance between the ideal vector and the
ensemble’s vote. The weights are calculated in order to minimize this loss
function [15, 14].

For weight assignment, Bonab and Can suggest that the number of classifiers
should be equal to the number of class labels (n = m) [14, 18]. However, Wu
and Crestani proves that adding more classifiers to the ensemble generally
improves performance; even in the worst case, it performs the same as before
[15]. These contrasting findings highlight the need for further investigation
into the optimal ensemble size and the impact of adding more classifiers.

##  3 A Basis to Understand Effect of Linear Dependency

For weight calculation in this framework, we demonstrate that the linear
independence of classifiers in the ensemble plays an important role because:

  * •

In this geometric framework, the ensemble is created by a linear combination
of its component classifiers.

  * •

Linear dependency of the votes of component classifiers determines the span of
the ensemble’s vote.

Theorem 1 demonstrates the importance of the linear independence of component
classifiers.

Theorem 1: If there are m-number of linearly independent votes for an instance
$I_{k}$ of data stream D, then there exists unique $W_{1},W_{2},...,W_{n}$
with $\sum_{i=1}^{n}W_{i}=1$ such that the ensemble vote is equal to the ideal
vector, i.e., V = o.

Proof: Let the number of component classifiers be m (n = m), and all of them
give linearly independent votes for the instance $I_{k}$. Then there exists a
unique solution to the following equation.

$\displaystyle\begin{bmatrix}S_{11}&S_{21}&\---&S_{n1}\\\
S_{12}&S_{22}&\---&S_{n2}\\\ S_{13}&S_{23}&\---&|\\\ |&|&\---&|\\\
|&|&\---&|\\\ S_{1n-1}&S_{2n-1}&\---&S_{nn-1}\\\
S_{1n}&S_{2n}&\---&S_{nn}\end{bmatrix}\begin{bmatrix}W_{1}\\\ W_{2}\\\
W_{3}\\\ |\\\ |\\\ W_{n-1}\\\ W_{n}\end{bmatrix}=\begin{bmatrix}0\\\ |\\\ 0\\\
1\\\ 0\\\ |\\\ 0\\\ \end{bmatrix}=o$ (1)

It follows that

$W_{1}S_{11}+W_{2}S_{21}+...+W_{1}S_{11}=0$ $...+...+...+...=0$
$W_{1}S_{1k}+W_{2}S_{2k}+...+W_{1}S_{1k}=1$ $...+...+...+...=0$
$W_{1}S_{1n}+W_{2}S_{2n}+...+W_{1}S_{1n}=0$

Sum all the rows and group them according to weights

$W_{1}(S_{11}+S_{12}+...+S_{1n})+...+W_{n}(S_{n1}+S_{n2}+...+S_{nn})=1$

Since $\sum_{i=1}^{n}S_{ki}=1$ for all k between 1 and n, it follows that

$W_{1}+...+W_{n}=1$

If there are more than m-number of classifiers (n > m) and m-number of them
gives linearly independent votes, these m classifiers can be assigned to the
ideal weights, and other n-m classifiers can be assigned to 0 weight.

Discussion:  This theorem shows that having an m-number of linearly
independent votes among n votes in for an instance $I_{k}$ is enough to assign
ideal weights.

For one instance $I_{k}$, only the m-number of classifiers may be enough.
Nevertheless, since the votes given by classifiers change according to the
instance $I_{k}$ of data stream D, the linear dependence of the classifier’s
votes changes. We utilize the probability of linear dependency of component
classifiers to calculate the probability of having m-number of linearly
independent votes among n component classifiers.

Definition: Let $p_{k}$ denote the probability that a vote of a classifier is
linearly dependent to a space with k dimensions. Assume that $p_{k}$s are the
same for all classifiers.

Theorem 2:  Given the probabilities $p_{1},...,p_{m-1}$ the probability of
having m linearly independent votes in an ensemble with n classifiers
(n$\geq$m) is:

$(\prod_{i=1}^{m-1}(1-p_{i}))(\sum_{k=0}^{n-m}(\sum_{x_{1}+...+x_{m-1}=k}(\prod_{j=1}^{m-1}(p_{j}^{x_{j}})))$
(2)

where each $x_{i}$ is a natural number between 0 and k.

Proof:  For every vote provided by classifiers, we construct a new set by
adding one vote at a time and calculate the number of linearly independent
vectors (this number is the same as the dimension of the space created by
these votes) in this new set.

For the ensemble to have m number of linearly independent votes, the dimension
of the new set should increase m-1 times, which has the probability
$\prod_{i=1}^{m-1}(1-p_{i})$. An m-number of linearly independent votes can be
achieved with at least m votes. However, if it cannot be achieved with m
votes, some vectors must have not increased the dimension with the probability
of one of the $p_{k}$s. For example, the probability of having an
m-dimensional system with m+1 vectors is (since all of them are independent
events):

$(p_{1}+p_{2}+...+p_{m-1})\prod_{i=1}^{m-1}(1-p_{i})$

By similar logic, the probability of having an m-dimensional system with m+2
vectors is:

$(p_{1}^{2}+p_{1}p_{2}+...+p_{m-2}p_{m-1}+p_{m-1}^{2})\prod_{i=1}^{m-1}(1-p_{i})$

By summing these probabilities up to n, the total probability of having an
m-number of linearly independent votes among the n-number of classifiers is:

$(\prod_{i=1}^{m-1}(1-p_{i}))(\sum_{k=0}^{n-m}(\sum_{x_{1}+...+x_{m-1}=k}(\prod_{j=1}^{m-1}(p_{j}^{x_{j}})))$

where each xi is a natural number between 0 and k.

Figure 1: The probability tree for n-number of classifiers to have m-number of
linearly independent votes. The numbers at the top of the figure represent the
number of classifiers in the current system. D indicates dimension.

Discussion:  This theorem numerically shows how adding a new classifier to the
ensemble affects the probability of having an m-number of linearly independent
votes. However, we want this probability to be as high as possible so that we
are sure there are enough classifiers. For this reason, its limit should be
known.

Theorem 3:  If none of the $p_{k}$s is 1, then the probability of having
m-number of linearly independent votes converges to 1 as n goes to infinity,
i.e.

$\lim_{n\to+\infty}(\prod_{i=1}^{m-1}(1-p_{i}))(\sum_{k=0}^{n-m}(\sum_{x_{1}+...+x_{m-1}=k}(\prod_{j=1}^{m-1}(p_{j}^{x_{j}})))=1$
(3)

Proof:  There are three possible cases for the above formula:
1) At least one of the $p_{k}$s is 1:
In this case the product $\prod_{i=1}^{m-1}(1-p_{i})$ evaluates to 0; hence
the limit is 0.
2) All of the $p_{k}$s are 0.
In this case both
$\sum_{k=0}^{n-m}(\sum_{x_{1}+...+x_{m-1}=k}(\prod_{j=1}^{m-1}(p_{j}^{x_{j}}))$
and $\prod_{i=1}^{m-1}(1-p_{i})$ evaluates to 1; hence the limit is 1.
3) None of the $p_{k}$ is 1, and at least one of them is not 0.
This case will be proven by induction.

Base case: For m = 2, the equation (2) is equal to

$(1-p_{1})(1+p_{1}+p_{1}^{2}+p_{1}^{3}+...)$

Since $0<p_{1}<1$,

$1+p_{1}+p_{1}^{2}+p_{1}^{3}+...=\sum_{i=0}^{\infty}p_{1}=\frac{1}{1-p_{1}}$

Hence,

$(1-p_{1})(1+p_{1}+p_{1}^{2}+p_{1}^{3}+...)=1$

Let the theorem hold for m = k+1, i.e.,

$(1-p_{1})...(1-p_{k})(1+p_{1}+...+p_{k}+p_{1}^{2}+p_{1}p_{2}+...p_{k}^{2}+p_{1}^{3}+...)=1$
$\implies(1+p_{1}+...+p_{k}+p_{1}^{2}+p_{1}p_{2}+...p_{k}^{2}+p_{1}^{3}+...)=\frac{1}{(1-p_{1})...(1-p_{k})}$

Then it is enough to show that

$1+p_{1}+...+p_{k+1}+p_{1}^{2}+p_{1}p_{2}+...p_{k+1}^{2}+p_{1}^{3}+...=\frac{1}{(1-p_{1})...(1-p_{k+1})}$

Let

$1+p_{1}+...+p_{k+1}+p_{1}^{2}+p_{1}p_{2}+...p_{k+1}^{2}+p_{1}^{3}+...=A$

Group the terms in the equation so that the terms with $p_{k+1}$ are gathered

$1+p_{1}+...+p_{k}+p_{1}^{2}+p_{1}p_{2}+...p_{k}^{2}+p_{1}^{3}+...$
$+p_{k+1}+p_{k+1}p_{1}+...+p{k+1}^{2}+p_{k+1}p_{1}^{2}+...$
$=1+p_{1}+...+p_{k}+p_{1}^{2}+p_{1}p_{2}+...p_{k}^{2}+p_{1}^{3}+...$
$+p_{k+1}(1+p_{1}+...+p_{k+1}+p_{1}^{2}+p_{1}p_{2}+...p_{k+1}^{2}+p_{1}^{3}+...)$
$=\frac{1}{(1-p_{1})...(1-p_{k})}+p_{k+1}(A)=A$ $\implies
A=\frac{1}{(1-p_{1})...(1-p_{k+1})}$

Hence,

$(1-p_{1})...(1-p_{k+1})(1+p_{1}+...+p_{k+1}+p_{1}^{2}+p_{1}p_{2}+...p_{k+1}^{2}+p_{1}^{3}+...)=1$

Discussion:  Theorem 2 showed how adding new classifiers affects the
probability of having an m-number of linearly independent vectors. Theorem 3
shows that this probability converges to 1 when n gets larger and larger.

Table 2: Dataset Specifications. The first half is synthetic, the second half is real-life datasets. The calculated number of classifiers with the equation in Theorem 2 is given as Ideal #Classifiers 111Real-life datasets can be found at: https://anonymous.4open.science/r/concept-drift-datasets-scikit-multiflow-1261/README.md Dataset | #Inst. | #Attr. | #Class Labels | Ideal #Classifiers
---|---|---|---|---
RBF2 | 100,000 | 20 | 2 | 5
RBF4 | 100,000 | 20 | 4 | 11
RBF8 | 100,000 | 20 | 8 | 47
RBF16 | 100,000 | 20 | 8 | 274
Airlines | 539,383 | 7 | 2 | 4
Elec | 45,312 | 6 | 2 | 6
Rialto | 82,250 | 27 | 10 | 31
Covtype | 581,012 | 54 | 7 | 93

These three theorems explain the relationship between the number of component
classifiers and ensemble performance. Furthermore, thinking backward for
Theorem 2, we can set a target probability of having m-number of linearly
independent votes, and calculate the minimum number of component classifiers n
to reach that probability. This method provides us with a suggestion for the
ensemble size.

##  4 Experiments

###  4.1 Setup

The experiments are conducted to investigate the relationship between the
number of classifiers and the accuracy rate. Additionally, we calculate the
minimum number of classifiers required using the equation in Theorem 2 with a
target probability set at 0.999. For experiments, we use GOOWE (Geometrically
Optimum and Online-Weighted Ensemble) as it already implements the geometric
framework introduced [19]. Hoeffding Tree is used as the base classifier of
GOOWE [20].

To evaluate the performance of the ensemble, we employ four real-world and
four synthetic datasets. The synthetic datasets are created using the random
RBF (Radial Basis Function) generator available in the scikit-multiflow
library of Python. The specific details of the datasets are given in Table 2.

The probabilities $p_{k}$s (k=1,…,m-1) are experimentally calculated via
Algorithm 1. We extend the GOOWE algorithm so that it also calculates these
probabilities when iterating each vote.

Algorithm 1 Algorithm for Calculating p Values222For reproducability, the
algorithms used for experiments can be found at:
https://anonymous.4open.science/r/GOOWE-Extended-CF41/README.md

1:$p\\_array\leftarrow\text{zero array with length of \\# class labels}$ //
array that stores unsuccessful trials (vote is linearly  // dependent) when
adding new classifier vote to curMatrix.

2:$p\\_total\leftarrow\text{zero array with length of \\# class labels}$ //
will be used to store total occurences of adding new  // vote to curMatrix for
calculating p values.

3:for each instance I in data stream do

4: $voteMatrix\leftarrow\text{votes of component classifiers of ensemble for I
}$

5: $curMatrix\leftarrow\text{zero matrix with same dimensions as voteMatrix}$

6: $prevDim\leftarrow 1$

7: $curDim\leftarrow 1$

8: for i in range(number of rows of voteMatrix) do // Add a row to curMatrix,
and check the rank of it.  // Update p_array and p_total accordingly.

9: $curMatrix[i]\leftarrow voteMatrix[i]$

10: if i == 0 then

11: $continue$ // Since one vector is always linearly independent

12: $prevDim\leftarrow curDim$

13: $curDim\leftarrow\text{rank of curMatrix}$

14: if curDim == precDim then

15: $p\\_array[curDim-1]\leftarrow p\\_array[curDim-1]+1$

16: $p\\_total[prevDim-1]\leftarrow p\\_total[prevDim-1]+1$

17: if curDim == length of p_array then

18: $break$

19:for i in range(length of p_array) do // After this, p_array[i] = $p_{i+1}$

20: if p_total[i] == 0 then

21: $p\\_array[i]\leftarrow 1$

22: else

23: $p\\_array[i]\leftarrow p\\_array[i]/p\\_total[i]$

###  4.2 Results

Table 2 summarizes the specifications of the datasets used in the experiments,
including the ideal number of classifiers required for each dataset. Figure 2
is the plot of Accuracy Rate vs. Ensemble Size for different datasets.

Figure 2: Prediction accuracy of the geometric framework with ensemble size
ranging from 2 to 128. Synthetic datasets are named as
RBF(number_of_class_labels).

We made several observations during our experiments. Firstly, Theorem 3 shows
that equation (2) follows a convergent curve; the effect of adding more
classifiers to the ensemble diminishes as the number of classifiers in the
ensemble increases. This finding is in line with the rate of change of the
accuracy rate, implying that beyond a certain point, adding more classifiers
to the ensemble does not significantly improve performance.

Furthermore, we confirmed that as the number of classifiers of an ensemble
increases, the accuracy rate generally improves. This observation aligns with
the trend predicted by Formula (3): as the number of classifiers increases,
the probability of obtaining a sufficient number of linearly independent votes
approaches 1. However, for most of the datasets, the accuracy rate decreases
after some point, especially for 128 component classifiers. This shows that
there are factors affecting the ensemble performance other than the linear
independency of component classifiers that we want to explore.

Moreover, for datasets with a low number of class labels, the calculated ideal
ensemble size tends to be near the number of class labels. However, as the
number of class labels increases, the ideal ensemble size deviates
significantly from this linear relationship. This suggests that the complexity
of achieving linearly independent votes among classifiers increases non-
linearly with the number of class labels. Also, calculated ideal #classifiers
match with empirical results only for the RBF8 dataset. This again shows the
importance of factors other than the linear independence of component
classifiers.

##  5 Conclusion & Future Work

In this paper, we show the importance of the linear dependence of classifiers
for an ensemble. Using the probability of component classifiers being linear
dependent, we provide a new perspective to understand how the number of
classifiers in an ensemble affects the accuracy rate of that ensemble.
Moreover, we provide a theoretical approach for determining ensemble size. The
experiments we conducted show that, in general, the accuracy rate increases as
the number of classifiers increases. This finding supports our theoretical
framework.

We assumed that probabilities $p_{k}$s are the same for all classifiers in the
ensemble, which means the classifiers are equally indistinguishable from each
other. However, this is not true in real-life scenarios. In the future, we
plan to investigate the individual classifier-level probability system to
develop a more detailed theoretical framework.

## Acknowledgments

This study is partially supported by TÜBİTAK grant no. 122E271.

## References

  * [1] Xibin Dong, Zhiwen Yu, Wenming Cao, Yifan Shi, and Qianli Ma.  A survey on ensemble learning.  Frontiers of Computer Science, 14:241–258, april 2020.
  * [2] Bartosz Krawczyk, Leandro L. Minku, João Gama, Jerzy Stefanowski, and Michał Woźniak.  Ensemble learning for data stream analysis: A survey.  Information Fusion, 37:132–156, 2017.
  * [3] Heitor Murilo Gomes, Jean Paul Barddal, Fabrício Enembreck, and Albert Bifet.  A survey on ensemble learning for data stream classification.  ACM Comput. Surv., 50:23:1–23:36, 03 2017.
  * [4] Thomas G. Dietterich.  Ensemble methods in machine learning.  In Multiple Classifier Systems, pages 1–15, Berlin, Heidelberg, 2000. Springer Berlin Heidelberg.
  * [5] Kamal Ali and Michael Pazzani.  Error reduction through learning multiple descriptions.  Machine Learning, 24, 11 1997.
  * [6] Grigorios Tsoumakas, Ioannis Partalas, and I. Vlahavas.  A taxonomy and short review of ensemble selection.  ECAI 2008, Workshop on Supervised and Unsupervised Ensemble Methods and Their Applications, 01 2008.
  * [7] Lior Rokach.  Ensemble-based classifiers.  Artif. Intell. Rev., 33:1–39, 02 2010.
  * [8] Eric Bax.  Selecting a number of voters for a voting ensemble.  CoRR, abs/2104.11833, 2021.
  * [9] Thais Oshiro, Pedro Perez, and José Baranauskas.  How many trees in a random forest?  volume 7376, 07 2012.
  * [10] Patrice Latinne, Olivier Debeir, and Christine Decaestecker.  Limiting the number of trees in random forests.  volume 2096, pages 178–187, 07 2001.
  * [11] Lena Pietruczuk, Leszek Rutkowski, Maciej Jaworski, and Piotr Duda.  How to adjust an ensemble size in stream data mining?  Information Sciences, 381:46–54, 2017.
  * [12] Daniel Hernández-Lobato, Gonzalo Martínez-Muñoz, and Alberto Suárez.  How large should ensembles of classifiers be?  Pattern Recognition, 46(5):1323–1336, 2013.
  * [13] Xiaohua Hu.  Using rough sets theory and database operations to construct a good ensemble of classifiers for data mining applications.  volume 233-240, pages 233–240, 01 2001.
  * [14] Hamed Bonab and Fazli Can.  Less is more: A comprehensive framework for the number of components of ensemble classifiers.  IEEE Transactions on Neural Networks and Learning Systems, 30(9):2735–2745, 2019.
  * [15] Shengli Wu and Fabio Crestani.  A geometric framework for data fusion in information retrieval.  Information Systems, 50:20–35, 2015.
  * [16] Shengli Wu and Weimin Ding.  A dataset-level geometric framework for ensemble classifiers.  CoRR, abs/2106.08658, 2021.
  * [17] Weimin Ding, Shengli Wu, and Chris Nugent.  A multimodal fusion enabled ensemble approach for human activity recognition in smart homes.  Health Informatics Journal, 29(2):14604582231171927, 2023.  PMID: 37117157.
  * [18] Hamed R. Bonab and Fazli Can.  A theoretical framework on the ideal number of classifiers for online ensembles in data streams.  In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, CIKM ’16, page 2053–2056, New York, NY, USA, 2016. Association for Computing Machinery.
  * [19] Hamed R. Bonab and Fazli Can.  Goowe: Geometrically optimum and online-weighted ensemble classifier for evolving data streams.  ACM Trans. Knowl. Discov. Data, 12(2), Jan 2018.
  * [20] Pedro Domingos and Geoff Hulten.  Mining high-speed data streams.  In Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’00, page 71–80, New York, NY, USA, 2000. Association for Computing Machinery.


[1]Ghazal Kalhor

[1]School of Electrical and Computer Engineering, College of Engineering,
University of Tehran, Tehran, Iran

[2]Tehran Institute for Advanced Studies, Tehran, Iran

# Forecasting Success of Computer Science Professors and Students Based on
Their Academic and Personal Backgrounds

<EMAIL_ADDRESS>Behnam Bahrak<EMAIL_ADDRESS>*  *

###### Abstract

After completing their undergraduate studies, many computer science (CS)
students apply for competitive graduate programs in North America. Their long-
term goal is often to be hired by one of the big five tech companies or to
become a faculty member. Therefore, being aware of the role of admission
criteria may help them choose the best path towards their goals. In this
paper, we analyze the influence of students’ previous universities on their
chances of being accepted to prestigious North American universities and
returning to academia as professors in the future. Our findings demonstrate
that the ranking of their prior universities is a significant factor in
achieving their goals. We then illustrate that there is a bias in the
undergraduate institutions of students admitted to the top 25 computer science
programs. Finally, we employ machine learning models to forecast the success
of professors at these universities. We achieved an RMSE of 7.85 for this
prediction task.

###### keywords:

graduate admission, academia, educational background, computer science,
machine learning, success prediction

##  1 Introduction

North America is a popular destination for students from around the world who
aspire to pursue their graduate studies [1, 2]. According to Slonim et al.
[3], there has been a growing trend in computer science graduate enrollments
at American and Canadian universities. This trend can be attributed to various
factors. Firstly, there is an increasing interest among students in the field
of artificial intelligence within computer science [4]. Additionally, computer
science graduates have access to numerous job opportunities across different
industries [5]. Moreover, the competitive salaries and job security associated
with careers in computer science contribute to this trend [6]. However,
obtaining a master’s or doctoral degree from a university in the USA or Canada
is often a requirement for finding professional jobs in these countries [7].

Several previous studies have examined the criteria for graduate admissions.
In our previous study [8], we analyzed the impact of students’ personal
background, such as gender and home country, on their admissions to computer
science graduate programs at top North American universities. We found
evidence of a nationality bias in the admission process, but no gender bias
was observed. Turner and Hudson [9] identified unintended bias in reference
letters for Black applicants and proposed suggestions for improvement.
Additionally, a student’s statement of purpose plays a vital role in the
application process for admission to elite universities [10].

Furthermore, students’ educational background, including their grade point
average (GPA), research experience, and previous universities, significantly
impact their chances of admission to top-tier universities. High-ranking
universities often set minimum GPA requirements and reject applicants who fall
below the specified cut-off [11]. Moreover, having publications in prestigious
venues is an indicator of research potential, particularly for doctoral
admissions [12]. These criteria are explicitly stated in universities’
application instructions. However, the selectivity of a student’s prior
university is an implicit factor that admissions committees consider when
reviewing applications and making admission decisions [13].

Despite the importance of undergraduate universities in the admissions
process, only a few studies have analyzed their role in gaining admission to
prestigious graduate programs. Lang [14] used regression techniques to assess
the impact of student characteristics, such as gender, race, and the rank of
their bachelor’s institution, on predicting the rank of their graduate
university. The study demonstrated that the rank of a student’s undergraduate
institution is the most influential factor in predicting attendance at leading
universities. Zhang [15] investigated how the quality of a student’s college
influences the quality of graduate programs they are admitted to. The research
confirmed that the quality of a student’s college has a significant positive
impact on their chances of enrolling in graduate programs in the United
States.

Some studies have attempted to predict faculty members’ success in terms of
their h-index. McCarty et al. [16] predicted authors’ h-index based on the
characteristics of their co-author networks, finding that having more co-
authors results in higher h-indexes. Ayaz [17] and Ibanez et al. [18]
forecasted professors’ h-index over time, suggesting that predictions for
shorter periods are more accurate.

Building upon the previous studies, our research examines the influence of
students’ previous universities on their success in the admission process for
prestigious universities. Additionally, we explore the prior universities of
faculty members and compare them with those of their mentees to uncover any
biases. We also employ regression techniques to predict professors’ h-index
based on various factors. For our analysis, we utilize the Advisor Student
Data that we introduced in [8]. It is important to note that the information
related to prior universities in this dataset is crucial but understudied.

Our contributions can be summarized as follows:

  1. 1.

We analyze the distributions of the previous universities/colleges students
and advisors have attended.

  2. 2.

We assess the correlation between the ranking of students’ prior universities
and their chances of getting accepted to high-ranking universities.

  3. 3.

We examine whether professors at top universities obtained their doctoral
degrees from these institutions.

  4. 4.

We conduct a proportion hypothesis test to determine if advisors are more
likely to admit students from their own bachelor’s universities.

  5. 5.

We construct a network of student-advisor prior universities and calculate
various metrics to investigate it.

  6. 6.

We compare the admissions of female and male students in both the PhD and
direct PhD programs.

  7. 7.

We predict professors’ h-index based on their academic and personal
backgrounds.

The remainder of this paper is structured as follows. Section 2 describes the
methods and metrics utilized in this study. Subsequently, in Section 3, we
present our findings and provide their interpretation. Finally, Section 4
concludes the paper and discusses the implications of our results.

##  2 Materials and methods

In this section, we outline the methodology employed in our study, including
the definition of the network constructed based on the dataset and the methods
and metrics utilized.

###  2.1 Prior universities network

We create an undirected weighted network that depicts the relationships
between students and advisors based on their respective previous universities.
In this network, each node represents a prior university of either students or
faculty members. An edge between universities $a$ and $b$, with a weight of
$W$, indicates the presence of $W$ student-advisor relations between students
from university $a$ and advisors from university $b$ (see Figure 1).

Table 1: An example of the dataset for building the prior universities network. Advisor ID | Advisor Previous Universities | Student ID | Student Previous Universities
---|---|---|---
advisor16 | Caltech/MIT | student264 | MIT
advisor214 | Waterloo/U of T/U of T | student3725 | Waterloo/York
advisor214 | Waterloo/U of T/U of T | student3738 | UBC
Figure 1: A prior universities network based on the dataset in Table 1.

###  2.2 Methods

In this part, we provide the definitions of the methods exploited throughout
the paper.

####  2.2.1 Proportion hypothesis test

The proportion hypothesis test compares the proportion of a specific feature
in a population to a reference proportion. It also provides a range of values
that are likely to contain the population ratio [19]. We utilize this method
to assess the presence of a bias towards bachelor’s universities in graduate
admissions.

####  2.2.2 Disparity filter

The disparity filter is a network sparsification technique used to
significantly reduce the number of ties in a network while preserving its
multi-scale nature [20]. In this study, we employ this method to eliminate
insignificant ties from the prior universities network.

####  2.2.3 Louvain community detection

The Louvain community detection algorithm is employed to identify communities
in a network by optimizing modularity. Its objective is to maximize the
difference between the anticipated and actual number of edges within each
community, employing a greedy approach [21]. We utilize this method to
ascertain the communities within the prior universities network.

####  2.2.4 Bootstrapping

Bootstrapping is a resampling technique that involves randomly sampling data
points with replacement to estimate the sampling distribution of a desired
statistic [22]. We employ this method to compute confidence intervals for the
proportions of PhD entries.

####  2.2.5 Linear regression

Linear regression is a statistical method that utilizes multiple explanatory
variables to forecast the result of a response variable. The objective of this
approach is to depict the linear relationship between the explanatory
variables and the response variable [23].

####  2.2.6 Bayesian ridge

The Bayesian ridge regression model incorporates regularization parameters
into the estimation process by introducing uninformative priors over the
hyperparameters of the model [24].

####  2.2.7 Lasso regression

Lasso regression is a statistical technique that combines variable selection
and regularization to improve prediction accuracy and make the resulting
statistical model more interpretable [23].

####  2.2.8 Ridge regression

Ridge regression is a statistical technique used to estimate coefficients in
multiple-regression models, especially when independent variables are strongly
correlated. It helps mitigate the issue of multicollinearity in linear
regression [23].

####  2.2.9 Decision tree

The decision tree is a non-parametric supervised learning technique applied in
regression problems. The objective of this approach is to establish a model
that forecasts the value of a target variable by understanding straightforward
decision rules deduced from the features within the data [23].

####  2.2.10 Random forest

Random forest is a regression technique in ensemble learning. It creates
multiple decision trees during training and produces predictions by averaging
the outcomes of these individual trees. This approach helps mitigate the
tendency of decision trees to overfit their training data [23].

####  2.2.11 K-nearest neighbors

K-nearest neighbors is a non-parametric supervised learning technique in which
the predicted value for an object is the average of the values from its k
closest neighbors [23].

####  2.2.12 Support vector machine

The support vector machine is a supervised learning model that relies on a
subset of the training data. It focuses on data points crucial for the model’s
accuracy while ignoring those that are close to the model’s prediction [23].

####  2.2.13 Neural network

The neural network is a model that minimizes squared errors using methods like
LBFGS or stochastic gradient descent. Its training occurs iteratively, with
partial derivatives of the loss function computed at each step to update the
model parameters [25].

###  2.3 Metrics

In this part, we explain the metrics calculated for the analyses of this
paper.

####  2.3.1 Kendall’s tau coefficient

Kendall’s tau coefficient is a nonparametric measure used to assess the
association between two rankings. It is calculated as follows:

$\tau=\frac{P-Q}{\sqrt{\frac{P+Q+T}{P+Q+U}}},$ (1)

where $P$ represents the number of concordant pairs, and $Q$ represents the
number of discordant pairs. In addition, $T$ and $U$ are the numbers of pairs
tied only on the first and second ranking, respectively [26]. We employ this
test to evaluate the relationship between the ranking of students’ prior
universities and their acceptance to top graduate programs.

####  2.3.2 Authority centrality

The authority centralities of the nodes of a network can be obtained from the
principal eigenvector of the matrix $A^{T}A$, where $A$ is the adjacency
matrix of the network [27]. We compute this metric for the nodes within the
prior universities network to identify the most authoritative universities
based on admission results.

####  2.3.3 Closeness centrality

The closeness centrality of a node in a network is defined as the average
geodesic distance between that node and all other nodes in the network. It is
calculated as follows:

$C(a)=\frac{n-1}{\sum_{b=1}^{n-1}d(b,a)},$ (2)

where $n$ is the number of nodes that $a$ is reachable from, and $d(b,a)$
represents the geodesic distance between nodes $a$ and $b$ [28]. We utilize
this measure to determine the universities that are closer to other
universities within the prior universities network.

####  2.3.4 R-squared

R-squared ($R^{2}$) signifies the fraction of variance in the target variable
explained by the regression model’s variables. This metric indicates the
improvement the regression line offers over the average line, making it a
fundamental tool for comparing different regression models. The formula for
$R^{2}$ is as follows:

$R^{2}=1-\frac{RSS}{TSS},$ (3)

where $RSS$ is sum of squares of residuals, and $TSS$ is the total sum of
squares [29].

####  2.3.5 Adjusted R-squared

Adjusted R-squared is a modified version of $R^{2}$, which decreases as
irrelevant features are added to the model. This metric helps in achieving an
optimal model with the fewest features. The formula for Adjusted $R^{2}$ is as
follows:

$\text{Adjusted }R^{2}=1-\frac{(1-R^{2})(N-1)}{N-p-1},$ (4)

where $R^{2}$ represents the sample R-squared, N is the total sample size, and
p is the number of independent variables [29].

####  2.3.6 Mean absolute error

Mean absolute error (MAE) represents the average absolute value of the
difference between the actual values and the predicted values. Its unit is the
same as the original data, which is one of its advantages. It is also
resistant to outliers. However, its graph is not differentiable. As a result,
it cannot be used alone to calculate the loss function, and different
optimization techniques such as gradient descent must be applied to make it
differentiable. The formula for MAE is as follows:

$MAE=\frac{\sum_{i=1}^{n}|Y_{i}-\hat{Y}_{i}|}{n},$ (5)

where $Y_{i}$ is the predicted value, $\hat{Y}_{i}$ is the true value, and n
is the total number of data points [29].

####  2.3.7 Mean squared error

Mean squared error (MSE) represents the mean square difference between the
actual values and the predicted values. Its graph is derivable; As a result,
it is suitable for calculating the loss function. The fact that its unit is
not the same as the original data is one of its disadvantages. It is also
sensitive to outliers and penalizes them more than other data. The formula for
MSE is as follows:

$MSE=\frac{1}{n}\sum_{i=1}^{n}(Y_{i}-\hat{Y}_{i})^{2},$ (6)

where $Y_{i}$ is the predicted value, $\hat{Y}_{i}$ is the true value, and n
is the total number of data points [29].

####  2.3.8 Root mean squared error

Root mean squared error (RMSE) represents the square root of the difference
between the actual values and the predicted values. Its unit is the same as
the original data unit. However, unlike the MAE error, it is not resistant to
outlier data and can be influenced by them [29].

####  2.3.9 Root mean squared logarithmic error

Root mean squared logarithmic error (RMSLE) is calculated by taking the
logarithm of RMSE. This logarithmic transformation scales down the error,
making it suitable for scenarios where we intend to develop the model without
input data, particularly when other errors might become excessively large
[29].

##  3 Results and discussion

In this section, we outline the steps taken to address our research questions.
We subsequently offer insightful interpretations of our findings.

###  3.1 Previous universities distributions

In this part, we examine the distribution of universities where students and
advisors have previously graduated from. Figures 2 and 3 display the
prevalence of the most commonly represented universities among students and
faculty members, respectively. According to Figure 2, a significant proportion
of students admitted to the top 25 universities have obtained their degrees
from esteemed global computer science institutions. Furthermore, Figure 3
illustrates that professors predominantly hold at least one degree in computer
science from the top 25 universities in North America. We further analyze and
discuss these observations in subsequent parts of the paper.

Figure 2: Distribution of students’ prior universities in the dataset. Figure
3: Distribution of advisors’ prior universities in the dataset.

###  3.2 Relationship between acceptance and ranking

In this part, we examine the relationship between admissions and the ranking
of students’ previous universities. To accomplish this, we sort the previous
universities in descending order based on the number of students in the
dataset who graduated from them with a bachelor’s or master’s degree.
Subsequently, we provide a ranking for the top 30 universities using this
order and compare it with the QS World University Ranking for Computer Science
2022111https://www.topuniversities.com/university-rankings/university-subject-
rankings/2022/computer-science-information-systems by calculating Kendall’s
tau [26]. The correlation between the two rankings is 0.4244, with a p-value
of 0.0010. Based on these findings, we conclude that there is a relatively
strong positive association between the ranking of previous universities and
the number of students admitted to high-ranking North American universities.
In other words, universities with higher rankings are more likely to have a
greater number of students accepted into top North American institutions.
Therefore, in line with previous studies [14, 15], we establish that the
ranking of students’ prior universities is a significant factor in graduate
admissions.

###  3.3 Evaluating faculty members’ PhD university impact

In this part, our objective is to determine if a majority of the advisors in
the dataset obtained their doctoral degrees from one of the top 25 ranked
North American universities. To accomplish this, we calculate the proportion
of faculty members who graduated from these universities with a PhD degree.
Since the conditions of independence and success-failure are met, we can
assume that the sampling distribution of the proportion is approximately
normal, with a mean of 0.5 and a standard deviation of 0.0184. Furthermore,
our point estimate (the observed ratio) is 0.7395. Therefore, our hypothesis
test is formulated as follows:

$\displaystyle\begin{gathered}H_{0}:p_{top\>ranking\>university\>ratio}=0.5\\\
H_{a}:p_{top\>ranking\>university\>ratio}>0.5\end{gathered}$ (9)

Using a z-test, we obtained a p-value of 3.5639e-39, which is significantly
lower than our chosen significance level of 0.05. Therefore, we reject the
null hypothesis. This provides strong evidence to support the conclusion that
the majority of computer science faculty members at high-ranking North
American universities hold a PhD degree from one of these institutions. This
finding is consistent with Crane’s research [30], which demonstrated a higher
likelihood of PhD graduates from prestigious departments being hired as
professors at top universities. Furthermore, Bedeian et al. [31] stated that
the reputation and prestige of a student’s doctoral university directly impact
the perceived quality and reputation of the academic appointment they receive.

###  3.4 Assessing bachelor’s university bias

In this part, our aim is to test whether advisors demonstrate a preference for
accepting students who graduated from their own previous universities. To
achieve this objective, we employ a simulation-based approach similar to the
methodology used to assess nationality bias in [8]. However, since some
individuals may have attended more than one university, we need to preprocess
the dataset to convert the columns representing students’ and advisors’ prior
universities into single-value representations. Specifically, we focus solely
on the universities from which students and advisors obtained their bachelor’s
degrees. It is worth noting that undergraduate institutions are a better
indicator as many students directly apply to PhD programs with a bachelor’s
degree.

In each of the 500 iterations, we generate 9,400 advisor-student pairs, where
the prior university for each component is selected with a probability
equivalent to the observed proportion in the dataset. This simulation yields a
distribution that approximates a normal distribution, with a mean of 0.0088
and a standard deviation of 0.0042, as depicted in Figure 4. The ratio of
advisor-student pairs sharing the same bachelor’s universities in the dataset
is 0.0289. Consequently, we need to assess the likelihood of observing such a
value within the simulated distribution. Therefore, our hypothesis test is
formulated as follows:

$\displaystyle\begin{gathered}H_{0}:p_{common\>bachelor^{\prime}s\>university\>ratio}=0.0088\\\
H_{a}:p_{common\>bachelor^{\prime}s\>university\>ratio}\neq
0.0088\end{gathered}$ (12)

Figure 4: Histogram of advisor-student common bachelor’s university ratio. The
number of simulations is 500.

Employing a z-test, we obtain a p-value of 1.5427e-6, which is lower than our
significance level of 0.05. Therefore, we reject the null hypothesis and
conclude that the data provide strong evidence of a preference for the
bachelor’s university in the acceptance of graduate students.

###  3.5 Exploring prior universities network

In this part, we analyze the advisor-student prior universities network.
Figure 5 presents a subgraph of this network, focusing on the first 50
universities with the highest node degrees. We then apply the disparity filter
algorithm [20] to remove insignificant ties from the network. Additionally, we
identify the communities within the network using the Louvain community
detection algorithm [21], assigning a specific color to each node based on its
community.

As observed, Canadian and Iranian universities are found in the same
community. This result aligns with the fact that many Iranians choose to apply
to Canadian universities due to the streamlined process of becoming a
permanent resident in Canada, which is more efficient compared to the USA
[32]. Moreover, Canada offers more flexible visa and immigration policies for
international students [1]. The orange community predominantly comprises
Chinese, Indian, and South Korean universities, reflecting the expected high
number of international students from these countries studying at American
universities [33]. The third community consists of European universities,
providing further evidence of a nationality bias in student admissions.

Stanford University holds the highest values for both authority and closeness
centrality, indicating its commitment to promoting diversity and
inclusivity222See https://ideal.stanford.edu/about-ideal/diversity-statement.

Figure 5: A subgraph of the prior universities network. The color of nodes
represents their respective community. The size of nodes corresponds to their
authority value, while the size of their label corresponds to their closeness
centrality. The thickness of edges indicates their weight.

###  3.6 Analyzing doctorate admissions

In this part, we examine the distribution of PhD and direct PhD admissions
among female and male students. Figure 6 illustrates that both genders have a
higher proportion of students being admitted to doctoral programs without
obtaining a master’s degree. This observation aligns with the fact that having
a master’s degree is not a requirement for acceptance to computer science
doctoral programs at top North American universities. Furthermore, the limited
availability of full funding positions for master’s students contributes to
the preference of applying directly to PhD programs [34]. Additionally, the
ratio of direct PhD admissions to PhD admissions is higher for women compared
to men. In other words, female students are more likely to be accepted into
PhD programs directly after completing their bachelor’s degree, in comparison
to their male counterparts.

Figure 6: Gender-disaggregated bar plot of PhD and direct PhD students. 95%
confidence intervals shown, with standard errors calculated using
bootstrapping.

###  3.7 Predicting professors’ success

In this part, we employ various machine learning techniques to assess the
impact of professors’ academic and personal backgrounds on their success. Our
success metric is the h-index, and we utilize a range of features, including
academic rank, gender, home country, research fields, previous universities,
citation count, publication count, first paper year, and the nationality
entropy of their research groups as predictive factors. Since machine learning
models require numerical input, we convert categorical features such as home
country and research fields into a one-hot representation. Additionally, we
remove columns with more than 95% zero values to ensure that our predictors
are not excessively sparse, resulting in a final set of 28 columns.

We randomly allocate 80% of the rows for training, reserving the remaining
rows for the test set. To determine the optimal parameters for each model, we
utilize the grid-search technique provided by the scikit-learn library [35].
Furthermore, we implement 5-fold cross-validation to mitigate the risk of
overfitting in our models. Table 2 provides an overview of the evaluation
metrics for forecasting faculty members’ h-index. As shown, the neural network
outperforms other regression models across most of the metrics. In our best
neural network, the activation function of the hidden layers is ReLU, alpha is
set at 0.001, the initial learning rate is 0.001, and the batch size is 100.
The lowest RMSE achieved for this task is 7.89, with the highest adjusted
R-squared of 0.88.

Table 2: Evaluation metrics of faculty members’ h-index prediction regressors. Regressor | $R^{2}$ | Adj. $R^{2}$ | MAE | RMSLE | MSE | RMSE
---|---|---|---|---|---|---
Linear Regression | 0.83 | 0.80 | 7.88 | 0.34 | 104.06 | 10.20
Bayesian Ridge | 0.84 | 0.80 | 7.69 | 0.33 | 99.79 | 9.99
Lasso Regression | 0.84 | 0.81 | 7.71 | 0.33 | 98.51 | 9.93
Ridge Regression | 0.84 | 0.81 | 7.66 | 0.33 | 99.35 | 9.97
Decision Tree | 0.82 | 0.78 | 6.71 | 0.22 | 111.67 | 10.57
Random Forest | 0.89 | 0.87 | 5.38 | 0.18 | 68.23 | 8.26
K-Nearest Neighbors | 0.78 | 0.73 | 7.30 | 0.24 | 138.36 | 11.76
Support Vector Machine | 0.81 | 0.76 | 8.87 | 0.37 | 120.40 | 10.97
Neural Network | 0.90 | 0.88 | 5.78 | - | 61.58 | 7.85

##  4 Conclusion

In this study, we explored the prior universities of professors and students
majoring in computer science at elite institutions in North America. We
examined how an applicant’s educational background influences admission
decisions. Based on our findings, we observed that the selectivity of a
university positively correlates with the number of its students admitted to
top universities. Additionally, we discovered that a majority of professors
themselves have obtained their doctoral degrees from these top 25
universities. Furthermore, we demonstrated a preference among advisors to
admit students who have completed their bachelor’s degree at the same
university. By analyzing the network of prior universities, we identified
distinct communities and observed that Iranian students tend to favor Canadian
institutions, while Indian and Chinese students often choose American
universities. Moreover, we highlighted a growing trend of students applying
directly to doctoral programs without pursuing a master’s degree, particularly
among women. Finally, we demonstrated that neural networks outperform other
regression techniques in predicting the h-index of professors, achieving an
RMSE of 7.85.

The results of this paper can provide valuable insights for computer science
students planning to pursue higher education at prestigious universities in
the USA or Canada. Additionally, admissions committees at top-tier
universities can leverage our findings to foster a more equitable admission
system and promote diversity and inclusivity among their student populations.

## Declarations

### Funding

No funding was received for conducting this study.

### Competing interests

The authors declare that they have no competing interests.

### Ethics approval

Not applicable.

### Consent to participate

Not applicable.

### Consent for publication

Not applicable.

### Availability of data and materials

The dataset analyzed during the current study is available in the Advisor
Student Data repository, https://github.com/kalhorghazal/Advisor-Student-Data.

### Authors’ contributions

Conceptualization, B.B. and G.K.; methodology, G.K.; validation, G.K.; formal
analysis, G.K.; investigation, G.K.; data curation, G.K.; writing—original
draft preparation, G.K.; writing—review and editing, B.B.; visualization,
G.K.; supervision, B.B.; project administration, B.B. All authors have read
and agreed to the published version of the manuscript.

## References

  *   * She and Wotherspoon [2013] She, Q., Wotherspoon, T.: International student mobility and highly skilled migration: A comparative study of canada, the united states, and the united kingdom. SpringerPlus 2(1), 1–14 (2013)
  * Ghai [2015] Ghai, B.: Analysis & prediction of american graduate admissions process. Department of Computer Science, Stony Brook University, Stony Brook, New York (2015)
  * Slonim et al. [2008] Slonim, J., Scully, S., McAllister, M.: Outlook on enrolments in computer science in canadian universities. Information and Communications Technology Council, Government of Canada. Retrieved from http://www. shrc. ca/en/Default. aspx (2008)
  * Hendler [2008] Hendler, J.: Avoiding another ai winter. IEEE Intelligent Systems 23(02), 2–4 (2008)
  * Stepanova et al. [2021] Stepanova, A., Weaver, A., Lahey, J., Alexander, G., Hammond, T.: Hiring cs graduates: What we learned from employers. ACM Transactions on Computing Education (TOCE) 22(1), 1–20 (2021)
  * Säde et al. [2019] Säde, M., Suviste, R., Luik, P., Tõnisson, E., Lepp, M.: Factors that influence students’ motivation and perception of studying computer science. In: Proceedings of the 50th ACM Technical Symposium on Computer Science Education, pp. 873–878 (2019)
  * Wendler et al. [2012] Wendler, C., Bridgeman, B., Markle, R., Cline, F., Bell, N., McAllister, P., Kent, J.: Pathways through graduate school and into careers. Technical report, Educational Testing Service (2012)
  * Kalhor et al. [2023] Kalhor, G., Zeraati, T., Bahrak, B.: Diversity dilemmas: Uncovering gender and nationality biases in graduate admissions across top north american computer science programs. EPJ Data Science 12(1), 1–23 (2023)
  * Newkirk-Turner and Hudson [2022] Newkirk-Turner, B.L., Hudson, T.K.: Do no harm: Graduate admissions letters of recommendation and unconscious bias. Perspectives of the ASHA Special Interest Groups 7(2), 463–475 (2022)
  * Kanojia et al. [2018] Kanojia, D., Wani, N., Bhattacharyya, P.: Is your statement purposeless? predicting computer science graduation admission acceptance based on statement of purpose. arXiv preprint arXiv:1810.04502 (2018)
  * Raghunathan [2010] Raghunathan, K.: Demystifying the american graduate admissions process. StudyMode. com (2010)
  * Michel et al. [2019] Michel, R.S., Belur, V., Naemi, B., Kell, H.J.: Graduate admissions practices: A targeted review of the literature. ETS research report series 2019(1), 1–18 (2019)
  * Posselt [2014] Posselt, J.R.: Toward inclusive excellence in graduate education: Constructing merit and diversity in phd admissions. American Journal of Education 120(4), 481–514 (2014)
  * Lang [1987] Lang, D.: Stratification and prestige hierarchies in graduate and professional education. Sociological Inquiry 57(1), 12–31 (1987)
  * Zhang [2005] Zhang, L.: Advance to graduate education: The effect of college quality and undergraduate majors. The review of higher education 28(3), 313–338 (2005)
  * McCarty et al. [2013] McCarty, C., Jawitz, J.W., Hopkins, A., Goldman, A.: Predicting author h-index using characteristics of the co-author network. Scientometrics 96(2), 467–483 (2013) https://doi.org/10.1007/s11192-012-0933-0
  * Ayaz et al. [2018] Ayaz, S., Masood, N., Islam, M.A.: Predicting scientific impact based on h-index. Scientometrics 114, 993–1010 (2018)
  * Ibáñez et al. [2011] Ibáñez, A., Larrañaga, P., Bielza, C.: Predicting the h-index with cost-sensitive naive bayes. In: 2011 11th International Conference on Intelligent Systems Design and Applications, pp. 599–604 (2011). IEEE
  * Diez et al. [2012] Diez, D.M., Barr, C.D., Cetinkaya-Rundel, M.: OpenIntro Statistics. OpenIntro, Boston, MA, USA (2012)
  * Serrano et al. [2009] Serrano, M.Á., Boguná, M., Vespignani, A.: Extracting the multiscale backbone of complex weighted networks. Proceedings of the national academy of sciences 106(16), 6483–6488 (2009)
  * Blondel et al. [2008] Blondel, V.D., Guillaume, J.-L., Lambiotte, R., Lefebvre, E.: Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment 2008(10), 10008 (2008)
  * DiCiccio and Efron [1996] DiCiccio, T.J., Efron, B.: Bootstrap confidence intervals. Statistical science 11(3), 189–228 (1996)
  * James et al. [2013] James, G., Witten, D., Hastie, T., Tibshirani, R., et al.: An Introduction to Statistical Learning vol. 112. Springer, ??? (2013)
  * Bishop and Nasrabadi [2006] Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning vol. 4. Springer, ??? (2006)
  * Goodfellow et al. [2016] Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT press, ??? (2016)
  * Kendall [1938] Kendall, M.G.: A new measure of rank correlation. Biometrika 30(1/2), 81–93 (1938)
  * Kleinberg [1999] Kleinberg, J.M.: Authoritative sources in a hyperlinked environment. Journal of the ACM (JACM) 46(5), 604–632 (1999)
  * Freeman [2002] Freeman, L.C.: Centrality in social networks: Conceptual clarification. In: Social Network: Critical Concepts in Sociology vol. 1, pp. 238–263. Routledge, London (2002)
  * Bonaccorso [2017] Bonaccorso, G.: Machine Learning Algorithms. Packt Publishing Ltd, ??? (2017)
  * Crane [1970] Crane, D.: The academic marketplace revisited: A study of faculty mobility using the cartter ratings. American journal of Sociology 75(6), 953–964 (1970)
  * Bedeian et al. [2010] Bedeian, A.G., Cavazos, D.E., Hunt, J.G., Jauch, L.R.: Doctoral degree prestige and the academic marketplace: A study of career mobility within the management discipline. Academy of Management Learning & Education 9(1), 11–25 (2010)
  * Woroby [2015] Woroby, T.: Immigration reform in canada and the united states: How dramatic, how different? American Review of Canadian Studies 45(4), 430–450 (2015)
  * Tshibaka [2018] Tshibaka, A.S.: Understanding international students from asia in american universities: Learning and living globalization. Journal of International Students 8(4), 1941–1943 (2018)
  * Pottinger [1999] Pottinger, R.: Choosing a ph. d. program in computer science. XRDS: Crossroads, The ACM Magazine for Students 6(1), 6–13 (1999)
  * Pedregosa et al. [2011] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al.: Scikit-learn: Machine learning in python. the journal of machine learning research 12 (2011)