# Big fish and small ponds: why the departmental h-index should not be used to
rank universities ††thanks: This work was supported in part by the National
Academy of Sciences of Ukraine, Project KPKBK 6541230.

O. Mryglod1,2 Yu. Holovatch1,2 R. Kenna2,3

(1 Institute for Condensed Matter Physics of the National Academy of Sciences
of Ukraine, 1 Svientsitskii St., 79011 Lviv, Ukraine;
2 ${\mathbb{L}}^{4}$ Collaboration & Doctoral College for the Statistical
Physics of Complex Systems, Leipzig-Lorraine-Lviv-Coventry, Europe
3 Centre for Fluid and Complex Systems, Coventry University, Coventry, CV1
5FB, United Kingdom )

###### Abstract

The size-dependent nature of the so-called group or departmental _h-index_ is
reconsidered in this paper. While the influence of unit size on such
collective measures was already demonstrated a decade ago, institutional
ratings based on this metric can still be found and still impact on the
reputations and funding of many research institutions. The aim of this paper
is to demonstrate the fallacy of this approach to collective research-quality
assessment in a simple way, focusing on the _h-index_ in its original form. We
show that randomly reshuffling real scientometric data (varying numbers of
citations) amongst institutions of varying size, while maintaining the volume
of their research outputs, has little effect on their departmental _h-index_.
This suggests that the relative position in ratings based on the collective
_h-index_ is determined not only by quality (impact) of particular research
outputs but by their volume. Therefore, the application of the collective _h-
index_ in its original form is disputable as a basement for comparison at
aggregated levels such as to research groups, institutions or journals. We
suggest a possible remedy for this failing which is implementable in a manner
that is as simple and understandable as the _h-index_ itself.

Keywords: Hirsch index, group h-index, size effects

##  1 Introduction

_Camille No $\hat{u}$s_ is a fictitious character who emerged into the
academic world in 2020 [1] (see also [2]). She was conceived as a “collective
individual” to embody research of the community as a whole, representing a
desire to return to traditional approaches to construction and dissemination
of knowledge “under the control of the academic community”. Her forename is an
amalgam of the collegial “we” (“nous” in French) and the concepts of reason,
mind or intellect (“$\nu$o$\tilde{\nu}$$\zeta$”) from Greek philosophy.
Symbolizing the “deep attachment” to traditional academic values, _Camille_ is
the antithesis of “indicators elaborated by the institutional management of
research”. Standing for the collective and not the individual, she is in
contrast to the increased use of metrics in the management and control of
research. Indeed, this contrast between the collective and the individual is
one that is frequently lost in an increasingly corporatised academic world.
The increasing variety of science-related activities beyond research itself
causes the engagement of more and more professional managers, university
leaders and other professionals who do not have personal academic experience.
Sometimes overly simplified understanding of the underlying processes,
together with a lack of knowledge of statistics and the need to seek higher
positions in league tables in a highly competitive academic world, may lead to
oversimplified interpretation and, thus, misuse of scientometric indicators
and approaches. Tools based on statistics that are designed to measure the
academic performance of the collective are applied to develop targets for the
individual. And tools designed to measure one element of an individual
academic’s achievements are adapted for ranking and therefore funding the
collective. Altogether, this drives human pursuits of knowledge in a manner
very different to the traditions that Camille represents.

Here we revisit one such tool – the departmental _h-index_ calculated in a
straightforward manner. The _h-index_ itself is probably the most popular and,
at the same time, the most debated scientometric indicator [3, 4]. More than
85 variants of it have been developed and many are listed in [5]. Besides
numerous modifications of the index itself, the generalization of this
indicator of individual researchers’ performance to various levels of
aggregation such as groups, institutions, countries, journals or scientific
topics was suggested from the very beginning. The most straightforward
definition of the group _h-index_ is the same as the original one, but
involves all publications by group members in a single pool. The group _h-
indices_ discussed further in this paper are calculated in this way. “Crude
and imperfect” as it is, it is sometimes suggested as an alternative to other
research evaluation frameworks, e.g., the UK’s REF111The _Research Excellence
Framework_ (REF, seehttp://www.ref.ac.uk) is a peer-review based regular
assessment of the quality of research emanating from universities and higher
education institutes (HEI) in the UK., as a means to rank departments in at
least some disciplines. A famous example is that of influential Oxford
academic Dorothy Bishop in Ref. [6]. Other examples include ratings of
Ukrainian Higher Educaton and Research Institutions. These are fully based on
group _h-index_ and are periodically updated, see Refs. [7, 8]. Although
Bishop’s suggestion [6] was countered in the famous Metrics Tide report which
went on to help shape the UK’s research landscape, and although these scores
are not officially implemented into governmetal assessment procedures in
Ukraine, they remain persuasive amongst research managers and some funding
bodies. So, even though they are groundless, they can be eventually perceived
as objective. This is a classic example of the Thomas Theorem in sociology
[9].

We used very simple methods to show that aggregate measures of _h-indices_ are
not suitable indicators of the quality or research emanating from research
groups, departments, or indeed nations. To pitch this at the level of non-
experts, a very simple statistical process was used and explained. We showed
that the collective (departmental or group) h-index, as used widely is to a
large extent an indicator of the _number of outputs generated by a given
institution, a value that itself is a proxy for size_. This may be ascribed to
the simple but mischievous statistical phenomenon of sampling bias; to use an
analogy from angling, if more big fish are desired, use a big net to catch
them; if randomly distributed, there are fewer big fish in small ponds than
there are in big ones. Random sampling from a randomly shuffled pool of
research outputs makes it more likely that highly cited papers are ascribed to
larger universities. If the _h-index_ is to be used in any meaningful way as a
collective metric, it therefore should be normalised according to size.

Table 1 illustrates the persuasive power of the departmental _h-index_. It
lists the top and bottom UK universities ranked according to their
departmental _h-index_ scores in physics as extracted using Scopus data (the
full table is presented as Table 3 below.) As might be expected, the most
prestigious universities such as Oxford and Cambridge are ranked highest and
lesser known universities are ranked lowest. The fact that this table captures
at least a qualitative correlation between prestige and the departmental _h-
index_ suggests the latter as a reasonable tool for measuring research
quality. It reinforces the notion that the _h-index_ is a reasonable way to
rank institutions.

Table 1: Number of publications and values of group _h-indices_ for Higher Education Institutions (HEI) in UK (Subject Area: “Physics”, publication window: 2001–2007, accessed in March 2020).  HEI |  Scopus group _h-index_
---|---
University of Cambridge |  250
University of Oxford |  207
Imperial College London |  203
$\dots$ |  $\dots$
Aberystwyth University |  40
University of the West of Scotland |  31
University of Brighton |  29

The perceived link between the departmental _h-index_ and prestige might be
seen to suggest that the attractiveness and reputations of prestigious
institutions are the very things that draws the highest quality to them; the
more attractive a university is, the better quality people they draw and the
higher quality the research is. It is undoubtedly true that quality people are
drawn to quality institutions but is this really what is being captured by the
departmental _h-index_? If these research outputs were to be re-shuffled and
randomly assigned to various institutions, one might expect a different
picture to emerge — a more egalitarian one. I.e., a person ill-versed in the
notion of bias in statistics might expect that random allocation of all
research outputs of a nation to its various universities would lead to random
rankings of these universities. We did precisely this — we randomly reshuffled
research outputs amongst research institutions and ranked them according to
their resulting departmental _h-indices_. It might be surprising that the
resulting scores were not so “random” as it could be expected.

The remainder of this paper is devoted to explaining this phenomenon as due to
size bias. It is organized as follows. We provide some context to the
extensive nature of the _h-index_ in the next section. In Section
“Combinatorics of scale” we demonstrate the principal danger of usage of
absolute values dealing with statistical predictions for the number of balls
in baskets of varying sizes while _h-index_ is based on absolute values by its
definiton. In the Section titled “Shaking out the correlations” we calculated
the expected group _h-index_ for research units (groups) of different sizes
based on real but reshuffled citation data. This experiment is performed for
two case studies, discussed in two subsections, correspondingly. The resulting
shapes of characteristic curves are discussed. A short summary is presented in
the last section.

##  2 Context: Size matters in scientometrics

The dependency of the _h-index_ at the level of units which size is identified
by the number of publications was discussed already over decade ago [10, 11]
in the context of journals [12], institutions [13, 14], disciplines [15] or
countries [16, 17]. In each case, the growth of _h-index_ with number of
publications $N$ was found to be:

$h\sim N^{\beta},\qquad N\gg 1.$ (1)

The quantity $\beta>0$ was found to be similar for different levels of
aggregations (different units such as journals, institutions, etc.) [13, 15,
14, 12]. The reasons for such universality have been widely discussed, with
the presumption that the distribution of citations falls in the family of
Paretian distributions with power-law exponent $\alpha$, leading to an
estimate $\beta\approx\frac{1}{1+\alpha}$, see [10, 18]. It is interesting to
note, that such universalities were found independently of the manner in which
the input subsets were built:

  * •

by assembling publications published in different years (cumulative process)
[13, 15];

  * •

by considering particular subsets of general collections: e.g., journal
publications related to different countries or publications assigned to
different subject categories [13];

  * •

by considering independent research units of disparate sizes [14, 19].

This paper fits into the above body of work. Its goal is to illustrate the
significant role that research-unit size plays in the collective _h-index_.
The number of published outputs can be considered as a proxy for the size of
research unit due to natural dependency of the number of outputs on the number
of active researchers (to give an example, Pearson correlation coefficient for
the number of documents in Scopus vs corresponding number of authors equals to
0.98 for the first case study considered in this paper, i.e., for Ukrainian
HEIs) 222Of course, the number of publications can serve only as a proxy for
the size of research unit since many factors such as a number of grants or
level of basic financial support can have an impact.. This illustration adds
to a growing body of evidence that the _h-index_ is not a good basis for
straightforward comparison or ranking research groups, institutions or
countries. This doesn’t mean that _h-index_ calculated for any collection of
documents loses meaning. However, the results have to be interpreted in a
proper way. Departmental _h-index_ is not solely a function of quality of
outputs and, therefore, the rating based on such score is not a quality
rating.

By basing our illustration on a very simple combinatorics model, we hope to
make it accessible to a range of scientific managers. Here we solely focus on
the statistical properties the group _h-index_ ; other arguments such as the
innate inconsistency of the group _h-index_ itself as addressed in [20] are
not discussed here.

Our findings lend support to the idea that the assessment of performance of
research groups has to be relative, i.e., made rather in respect to some
benchmark. Similiarly as relations between group h-indices and the numbers of
contributions by different countries are built for several journals in [13],
the corresponding relations can be derived for any collection of cited
publications. Besides the empirically observed relations, their version for
the uncorrelated system can be observed [21, 22].

##  3 Combinatorics of scale

The aim of this section is to explain the phenomenon described above in the
simplest possible manner. To this end we neglect all the emergent effects
discussed elsewhere in the literature, such as peculiarities of local
managements, infrastructures, recruitment strategies, economy of scale, the
Matthew effect and so on. We rather attract attention to the role of the
sample size suggesting a simple intuitive way to separate this factor and
assess its contribution. We imagined having a data set to hand which comprises
of a collection of research outputs (publications) and their citation counts
for each individual unit to be assessed (i.e., for each individual research
group or university). Each unit produced a certain amount of outputs and we
termed the number of published outputs by a given institution its
“productivity”. Productivity contains no information about quality or citation
counts of individual papers. We imagined placing all of these outputs (each of
which has their own citation counts) into single “pot”, “pool” or “basket”. We
then “fish” them out again and distribute them randomly amongst the various
institutions, maintaining the institutions original productivity. In this way
citation counts were randomly distributed. The only difference between
institutions for this purpose was the size of their individual “basket”. Then
we measured the collective or departmental _h-index_ of each institution. Our
challenge then was to determine the effect of this redistribution of wealth on
the departmental _h-indices_. If size plays no role in the _h-index_ , one
would expect rankings of institutions to be randomly changed. The extent to
which this does not happen provides a measure of the extent to which size
matters for the _h-index_. We will see in the next sections that by
maintaining their “basket” size institutions largely maintain their position
in the rankings so that size indeed plays a predominant role. In other words,
our results show that the _h-index_ is not a straightforward way to compare
institutions of different sizes. To provide a simple mathematical basis to
understand this phenomenon, we studied the selection of a given number of
balls of different sorts from a pool of balls. This is a typical task of
combinatorics. We considered $k_{1}$ black and $k_{2}$ white balls in a pool
containing $K=K_{1}+K_{2}$ balls. The number of combinations that contain
exactly $k_{1}$ black and $k_{2}$ white balls is

${\cal
N}_{K_{1},K_{2}}^{k_{1},k_{2}}=\mathrm{C}_{K_{1}}^{k_{1}}\cdot\mathrm{C}_{K_{2}}^{k_{2}},\qquad{\text{where}}\qquad\mathrm{C}_{K}^{k}=\frac{K!}{k!(K-k)!}\,$
(2)

and where $k_{1}+k_{2}=k$ is the size of the basket. Therefore, the
probability of fishing out $k_{1}$ black and $k_{2}$ white balls from a pool
of $K$ balls using a basket of size $k$ is

$P_{K_{1},K_{2}}^{k_{1},k_{2}}=\frac{{C}_{K_{1}}^{k_{1}}\cdot{C}_{K_{2}}^{k_{2}}}{\mathrm{C}_{K}^{k}},$
(3)

where $\mathrm{C}_{K}^{k}$ is the number of all possible combinations
independent of ball color.

Obviously the proportion of black and white balls picked up by a basket of
size $k$ is $K_{1}/K_{2}$ on average (except the cases when basket size is too
small to reach the most probable proportion with integer number of balls). For
example, if 53% of balls in the pool were black, the most probable proportion
of black balls in the basket is also 53%. Fig. 1 illustrates this for baskets
of different sizes and a pool of 4000 balls. Clearly, there is a distribution
of ratios around the average value of 53% and that distribution is less
“granular” as the basket increases in size.

Figure 1: Probability to get a given share ($k_{1}/k$) of black balls from the
pool with the same share ($k_{1}/k=K_{1}/K$) using a basket of size $k$. In
this figure: $K=4000$, $k_{1}/k=K_{1}/K=53\%$, $k=10,20,\dots,100$.

Figure 2: Probability to get $k_{1}$ black balls from the pool with the black
balls share $K_{1}/K$ using a basket of size $k$. In this figure: $K=4000$,
$K_{1}/K=53\%$, $k=10,20,\dots,100$.

Instead of the proportion of black balls, one can explore their absolute
number. In Fig. 2 we plotted the probability to fish out $N$ black balls from
the same pool of 4000 balls (53% of which are black) using baskets of
different sizes. Obviously a basket of size 10 cannot fish out more than 10
black balls and, as $N$ increases the number of balls fished out increases.
becomes obvious, that larger baskets get proportionally more black balls, see

We considered institutions of different sizes as the baskets of different
capacity in the experiments discussed in the next sections. Thus, we had
‘baskets’ (institutions) full of ‘balls’ (papers) of different ‘values’
(citations). Putting ‘balls’ from all ‘baskets’ together, reshuffling and
redistributing them over the ‘baskets‘ again we lose any correlations between
publications and any additional factors remaining solely with the sizes. To
check what would happen, two case studies have been performed, see the next
subsections.

##  4 Shaking out the correlations. Case studies: h-index rankings in Ukraine
and the UK

###  4.1 Case study 1: 40 HEIs in Ukraine

Lists of research Institutions in Ukraine, sorted by group _h-indices_ , are
periodically published by different sources using titles such as as “Rating of
Universities and Research Institutions” [7, 8]. We gathered bibliographic and
citation data for 40 such Higher Education Institutions (HEI’s) from the
Scopus database for May 2019. We do not fix citation or publication windows
and we do not distinguish different types of paper or citations since these
details are not used for the rating scheme either. Likewise, we used
publications from to the full Scopus profile of Institute. The data for the 40
institutions are listed in Table 2.

Table 2: Number of publications for 40 Ukrainian HEIs and the corresponding group _h-index_ values based on Scopus data (accessed in May 2019). For data source, see [23]. HEI |  Number of publications |  Group _h-index_
---|---|---
Taras Shevchenko National University of Kyiv |  17349 |  90
V. N. Karazin Kharkiv National University |  9452 |  70
Ivan Franko National University of L’viv |  6660 |  61
Odessa I.I.Mechnikov National University |  3469 |  61
Yuriy Fedkovych Chernivtsi National University |  3437 |  61
National Technical University of Ukraine “Igor Sikorsky Kyiv Polytechnic Institute” |  7748 |  54
Donetsk State Medical University |  1255 |  46
National Technical University Kharkiv Polytechnic Institute |  3775 |  43
Oles Honchar Dnipro National University |  3671 |  43
Danylo Halytsky Lviv National Medical University |  1025 |  42
Lviv Polytechnic National University |  6338 |  42
Sumy State University |  2339 |  39
Vasyl Stefanyk Precarpathian National University |  688 |  38
Uzhgorod National University |  2245 |  37
Ukrainian State Chemical Technology University |  1120 |  36
State Establishment Dnipropetrovsk Medical Academy of Health Ministry of Ukraine |  275 |  35
The Bohdan Khmelnytsky National University of Cherkasy |  455 |  35
V.I. Vernadsky Crimean Federal University |  2622 |  34
Bogomolets National Medical University |  741 |  33
National University of Kyiv-Mohyla Academy |  512 |  32
Vasyl’ Stus Donetsk National University |  1817 |  32
National Aerospace University “Kharkiv Aviation Institute” |  1303 |  30
Kharkiv National University of Radio Electronics |  3030 |  29
Lesya Ukrainka Eastern European National University |  771 |  29
Kharkiv National Medical University |  596 |  27
National University of Life and Environmental Sciences of Ukraine |  872 |  26
Donetsk National Technical University |  1336 |  25
Sevastopol State University |  1128 |  24
National Aviation University |  2033 |  21
National University of Food Technologies of Ukraine |  559 |  21
Kyiv National University of Technologies and Design |  486 |  20
Odessa National Polytechnic University |  915 |  20
K. Ushynsky South Ukrainian Pedagogical University |  306 |  19
Donbass State Engineering Academy |  377 |  18
Ukrainian National Forestry University |  240 |  18
Khmelnytsky National University |  419 |  17
Kryvyi Rih National University |  429 |  17
Odessa State Medical University |  387 |  15
Pridneprovskaya State Academy of Building and Architecture |  150 |  14
Volodymyr Dahl East Ukrainian National University |  503 |  13

We randomly re-allocated all outputs across the 40 institutions (over 90
thousand papers333including repetitions for the cases of papers co-authored by
researchers from different institutions), keeping their productivities fixed.
The reshuffling was performed as follows:

  * •

the number of publications for each HEI $N$ was recorded;

  * •

all the publications, labelled by their citation counts, were put in one pool;

  * •

all the publications were redistributed among HEIs randomly maintaining their
original numbers of publications;

  * •

the group _h-index_ for each HEI was calculated.

After 200 reshuffligns we had the set of _h-indices_ based on randomly
distributed data and repeatedly calculated for each HEI. The outcomes are
plotted in Fig. 3. There, the horizontal axis represents the number of
publications generated by each of 40 institutions. The vertical axis for the
black data are the real institional _h-indices_. The grey data points are the
corresponding values after reshuffling. One can observe a strong correlation
between _h-indices_ and the number of publications for real and randomised
data. An average Spearman correlation coefficient of $\approx 0.77$ indicates
significant correlation between ratings based on reshuffled data and the
original one.

Figure 3: Values of group _h-indices_ plotted against group sizes for 40
Ukrainian HEIs. Reshuffled data are presented in grey and real data are in
black. Both data sets tend to follow a power-law dependency, Eq. (1). The
solid red line is a fit to all reshuffled data put in one bag implying
$\beta\approx 0.395$ while the broken blue line is for real data with
$\beta\approx 0.338$. Both results are statistically significant for
$\alpha=0.01$.

If follows from Fig. 3 that the reshuffled values can be approximated by a
line on a log-log scale with the slope close to $\beta\approx 0.4$, see (1).
The corresponding value of $\beta$ estimated for real data is similar. This
value was found for a number of real data collections with $10^{2}<N<10^{5}$
[13].

###  4.2 Case study 2: 41 research groups in Physics in the UK

While in the previous case study we applied the reshuffling procedure to data
from different institutions, irrespective of the field of science they
represent, in case study 2 we analyse data from the same field of science. To
this end, we used Scopus data for 41 research groups working in the field of
Physics and submitted to the UK’s last Research Assessment Exercise 444 The
_Research Assessment Exercise_ (RAE, see http://www.rae.ac.uk/) was the
predecessor of _Research Excellence Framework_ (REF).. Publications were
limited to the years 2001–2007 (see Table 3) because that was the timeframe to
which RAE2008 pertained. It also leads to results comparable with others
published before (e.g., [24]).

Table 3: Number of publications and values of group _h-indices_ for 41 HEIs in UK submitted to RAE2008 the UOA (unit of assessment) categorised as “Physics”. The group _h-index_ values are based on Scopus data accessed in March 2020, see [25]. HEI |  Number of publications [2001-2007]; ’Physics’ |  Scopus _h-index_
---|---|---
University of Cambridge |  9602 |  250
University of Oxford |  8129 |  207
Imperial College London |  7186 |  203
University of Durham |  3208 |  178
University College London |  4842 |  165
University of Edinburgh |  2947 |  156
University of Manchester |  5588 |  156
University of Southampton |  4274 |  150
University of Bristol |  3104 |  149
University of Sheffield |  3389 |  136
University of Birmingham |  2918 |  133
University of St Andrews |  2003 |  130
University of Nottingham |  2609 |  128
University of Leeds |  2687 |  126
University of Glasgow |  2604 |  125
Queen Mary, University of London |  2090 |  124
University of Liverpool |  2966 |  123
University of Sussex |  1398 |  117
University of Leicester |  1571 |  114
Queen’s University Belfast |  1932 |  108
The University of Warwick |  2226 |  99
Cardiff University |  1382 |  98
University of Bath |  1268 |  92
University of Exeter |  1411 |  92
University of Surrey |  2360 |  92
Liverpool John Moores University |  616 |  91
University of Strathclyde |  2115 |  90
Lancaster University |  1243 |  87
Loughborough University |  1455 |  83
Heriot-Watt University |  1358 |  79
Royal Holloway University of London |  664 |  78
University of Hertfordshire |  680 |  78
King’s College London |  1117 |  77
University of York |  1211 |  69
Swansea University |  753 |  68
Keele University |  585 |  63
University of Kent |  578 |  61
University of Central Lancashire |  283 |  47
Aberystwyth University |  250 |  40
University of the West of Scotland |  176 |  31
University of Brighton |  117 |  29

We again randomly re-allocated outputs to different institutions, taking their
sizes into account through the volume of outputs they generate (over 95
thousand papers). These original data and their randomly reshuffled
counterparts are presented in Fig. 4. The notation here is the same as in Fig.
3. The remarkable similarity between the black and grey data points indicates
that the departmental _h-indices_ emanating from research outputs generated by
selected universities is not significantly different to departmental _h-
indices_ that would emanate by random re-allocation of those outputs. One can
notice that actuall data is even closer to reshuffled comparing to the
previous case study: the Spearman correlation coefficient between real and
reshuffled _h-index_ values is about $0.94$ on average. This is possibly due
to disciplinary homogeneity of publication data.

Figure 4: Values of group _h-indices_ vs group sizes for 41 HEIs in the UK
submitted to RAE2008 (UOA “Physics”): for the reshuffled data (grey color) and
for the real one (black symbols). Both data tend to follow a power-law
dependency, Eq. (1). The solid red line is a fit to all reshuffled data put in
one bag implying $\beta\approx 0.42$ while the broken blue line is for real
data with $\beta\approx 0.46$. Both results are statistically significant for
$\alpha=0.01$.

To summarize, Figs. 3 and 4 suggest (nonlinear) correlation between group _h-
indices_ and number of research outputs. Moreover, independently of the way in
which empirical data are collected, the same phenomenon occurs and similar
values of exponent are found. This can be also seen in Fig. 5, where the sizes
of research groups are not taken from real data but derived from power-law or
random distributions. The shape of characteristic curve is also not dependent
on this choice. One can see that the manner in which group _h-index_ depends
on the number of outputs is primarily determined by another input parameter –
citations distribution. This conclusion is in agreement with theoretical
findings from Refs.[10, 18].

Figure 5: Values of group _h-indices_ vs group sizes: real data for 41 HEIs in
the UK submitted to RAE2008 (UOA “Physics”) are shown by black symbols;
results for reshuffled citations using real group sizes are shown by grey
squares; reshuffling results for modified distributions of groups sizes: (i)
following power-law (magenta diamonds) or (ii) random with the bottom limit
defined (blue stars).

##  5 Discussion

The _h-index_ is an attractive metric, widely used for research assessments at
different levels of aggregation by science managers. The reason for this is
that it is easy to calculate the _h-index_ for any set of publications: the
algorithm is simple and an interpretation is fast. However, such haste can be
dangerous since it obfuscates the principal peculiarity of the _h-index_ : it
is neither an absolute, nor a specific indicator. The _h-index_ depends on the
number of sources (publications) by definition and therefore it is dependent
on the number of producers (authors) of these sources. Because the _h-index_
is not normalized per capita it cannot meaningfully compare groups of
different sizes. While several theoretical models were designed to make it
possible to estimate size-corrected values of the _h-index_ , normalisation
procedures did not gain traction in numerous managerial quarters, perhaps
because of the loss of simplicity. This paper attracts attention to this
problem by providing a clear, and hopefully convincing, illustration of the
size-dependency of departmental (or group) _h-indices_. Using an approach
widely known in the natural sciences in circumstances where systems are
investigated to identify principal dependencies, we reshuffled citation data
to redistribute it between research units (institutions, departments). The
correlations which originate from special hiring strategies, better level of
infrastructure or invested funds, and other factors, were lost during the
reshuffling. The only difference between research units left was size. The
reshuffled _h-indices_ were found to be only slightly different from their
original counterparts. Our results confirmed that group _h-index_ lend
advantage to larger groups simply due to their size because of statistical
reasons.

In section 3 we presented a toy model to illustrate the intuitively obvious
fact that the larger a basket is, the larger the number of balls of certain
kind can be picked out of it. Next, we considered the set of research
publications as a collection of balls of many different colors (indicated
different citation scores). They were distributed among the institutions of
different sizes represented by the baskets of varying capacity. Moreover, the
papers of some kind are frequent while the others (i.e., highly cited, the
most favoured ones) are rare, that follows from the known shapes of real
citations distributions, e.g., see [18]. Putting ‘balls’ from all ‘baskets’
together, reshuffling and redistributing them over the ‘baskets‘ again, we
lost any correlations between publications and any additional factors
remaining solely with the sizes of collections.

Putting ‘balls’ from all ‘baskets’ together, reshuffling and redistributing
them over the ‘baskets‘ again, we lost any correlations between publications
and any additional factors remaining solely with the sizes of collections.
This allowed us to separate the factor of scale and assess its impact on
resulting collective _h-index_. To check this for real data, we performed two
case studies. (i) The collection of papers (balls) with different citation
counts (colors) published by Ukrainian HEIs of different sizes (backets) was
analysed in section 4.1. (ii) Another case study described in section 4.2
based on the collection on publications by the set of research groups from
universities and higher education institutes (HEI) in the UK in the field of
Physics (here the limited publication window is considered). In both cases it
was demonstrated that larger HEIs have higher chances to get better scores
even for randomly reshuffled data. This indicates that the _h-index_
calculated for group output in a straighforward way is an ineffective measure
of group research impact.

If the _h-index_ is to be used for comparison as a collective metric, results
presented here demonstrate that it should be normalised according to size. Our
approach opens a way to achieve that; the size-dependent averages of the
reshuffled group _h-index_ scores shown in Figs. 3 and 4 present obvious
benchmarks against which to compare cores of research groups of given sizes.
In other words, rather than comparing the raw collective _h-index_ score of
one university to that of another (of different size), one can compare this
simple metric to the average score coming from reshuffled data for groups of
commensurate size. This has the added advantage of environment-specific
comparisons — the benchmark curve for Fig. 3 differs from that in Fig. 4 so
normalisation can be tailored to specific national environments (Ukraine and
the UK in this instance) and, desirably, disciplines. We hope to develop and
improve this normalisation approach in follow-on research and present it in a
manner accessible to all.

To conclude, it is natural to expect that larger research units demonstrate
better performance due to many factors: more competitive environment and
greater chance to get excellent experts from the larger pool of candidates,
better infrastructure and ability to attract more funds, etc. However, it is
worth noting that large institutions have also additional advantage due to
sampling bias. Our work provides a means to separate and describe this effect.
This is another contribution to the discussion of group size as crucial factor
in measurements of research quality. Moreover, for some practical tasks, e.g.,
allocating of funds to support the pockets of research excellence, the size of
research units becomes one of dominant factors, as it was shown in our
previous works, e.g., see [26, 27]. We believe that our work contributes to
the promotion of the contextualized scientometrics that is not targetted on
providing the universal a priori list of indicators to be used but on choosing
indicators proper for each particular task based on the understanding [28]. In
this context, group _h-index_ is not good or bad per se, but it has to be used
and interpreted taking into account its peculiarities, e.g., size-dependency.

## Acknowledgements

The authors thank to Bertrand Berche and Ihor Mryglod for useful discussions.
OM thanks to Serhii Nazarovets for inspiration which came from the previous
joint work.

## References

  * [1] Cogitamus laboratory (2020).
  * [2] Krasnytska, M., Berche, B., Holovatch, Y., and Kenna, R., Ising model with variable spin/agent strengths, _J. Phys.: Complexity_ 1 (2020) 035008\.
  * [3] Hirsch, J. E., An index to quantify an individual’s scientific research output, _Proceedings of the National academy of Sciences_ 102 (2005) 16569–16572.
  * [4] Alonso, S., Cabrerizo, F. J., Herrera-Viedma, E., and Herrera, F., h-index: A review focused in its variants, computation and standardization for different scientific fields, _Journal of informetrics_ 3 (2009) 273–289.
  * [5] Schubert, A. and Schubert, G., All along the h-index-related literature: A guided tour, in _Springer Handbook of Science and Technology Indicators_ (Springer, 2019), pp. 301–334.
  * [6] Bishop, D., An alternative to REF2014? (2014).
  * [7] Bibliometrics of Ukrainian science (2021): Ukrainian Universitites and Academic Institutions are ranked according to their departmental h-index based on Google Scholar data and calculated in 2021 (in Ukrainian; automatic translation can be found here).
  * [8] Rating of Ukrainian universities based on Scopus data (2021): Ukrainian universities are ordered according to their departmental h-index based on Scopus data and calculated in 2021 (5th column); 6th column is for the same indicator calculated in 2020 (in Ukrainian; automatic translation can be found here).
  * [9] Bornmann, L. and Marx, W., Thomas theorem in research evaluation, _Scientometrics_ (2020) 1–3.
  * [10] Glänzel, W., On the h-index-a mathematical approach to a new measure of publication activity and citation impact, _Scientometrics_ 67 (2006) 315–321.
  * [11] Egghe, L. and Rousseau, R., An informetric model for the hirsch-index, _Scientometrics_ 69 (2006) 121–129.
  * [12] Schubert, A. and Glänzel, W., A systematic analysis of hirsch-type indices for journals, _Journal of Informetrics_ 1 (2007) 179–184.
  * [13] Molinari, J.-F. and Molinari, A., A new methodology for ranking scientific institutions, _Scientometrics_ 75 (2008) 163–174.
  * [14] Sypsa, V. and Hatzakis, A., Assessing the impact of biomedical research in academic institutions of disparate sizes, _BMC Medical Research Methodology_ 9 (2009) 33.
  * [15] Kinney, A., National scientific facilities and their science impact on nonbiomedical research, _Proceedings of the National Academy of Sciences_ 104 (2007) 17943–17947.
  * [16] Babić, D., Kutlača D., Živković, L., Štrbac, D. and Semenčenko, Evaluation of the quality of scientific performance of the selected countries of southeast europe, _Scientometrics_ 106 (2016) 405–434.
  * [17] Montazerian, M., Zanotto, E. D., and Eckert, H., A new parameter for (normalized) evaluation of h-index: countries as a case study, _Scientometrics_ 118 (2019) 1065–1078.
  * [18] Glänzel, W., Moed, H. F., Schmoch, U., and Thelwall, M., _Springer Handbook of Science and Technology Indicators_ (Springer Nature, 2019).
  * [19] Ye, F., An investigation on mathematical models of the h-index, _Scientometrics_ 81 (2009) 493–498.
  * [20] Waltman, L. and Van Eck, N. J., The inconsistency of the h-index, _Journal of the American Society for Information Science and Technology_ 63 (2012) 406–415.
  * [21] Crespo, J. A., Ortuño-Ortín, I., and Ruiz-Castillo, J., The citation merit of scientific publications, _PloS one_ 7 (2012).
  * [22] Meyers, M. A. and Quan, H. The use of the h-index to evaluate and rank academic departments. _Journal of Materials Research and Technology_ , 6(4), 304–311 (2017).
  * [23] Mryglod, O., Ukrainian dataset (2021), 10.6084/M9.FIGSHARE.13625918.V1.
  * [24] Mryglod, O., Kenna, R., Holovatch, Y., and Berche, B., Absolute and specific measures of research group excellence, _Scientometrics_ 95 (2013) 115–127.
  * [25] Mryglod, O., UK dataset (2021), 10.6084/M9.FIGSHARE.13625930.V1.
  * [26] Kenna, R. and Berche, B., Critical mass and the dependency of research quality on group size, _Scientometrics_ 86 (2011) 527–540.
  * [27] Kenna, R. and Berche, B. Managing research quality: critical mass and optimal academic research group size. _IMA Journal of Management Mathematics_ 23(2) (2012) 195–207.
  * [28] Waltman, L. (2019). Quantitative literacy for responsible research policy. Inaugural lecture by Ludo Waltman as newly appointed Professor of Quantitative Science Studies, Leiden (The Netherlands). Retrieved from https://www.cwts.nl/news.


# Hybrid Paradigm-based Brain-Computer Interface for Robotic Arm Control
††thanks: This work was partly supported by Institute of Information &
Communications Technology Planning & Evaluation (IITP) grant funded by the
Korea government (MSIT) (No. 2017-0-00432, Development of Non-Invasive
Integrated BCI SW Platform to Control Home Appliances and External Devices by
User’s Thought via AR/VR Interface; No. 2017-0-00451, Development of BCI based
Brain and Cognitive Computing Technology for Recognizing User’s Intentions
using Deep Learning; No. 2019-0-00079, Artificial Intelligence Graduate School
Program(Korea University))

Byeong-Hoo Lee  Dept. Brain and Cognitive Engineering
Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>Jeong-Hyun Cho  Dept. Brain and Cognitive Engineering
Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>Byoung-Hee Kwon  Dept. Brain and Cognitive Engineering
Korea University
Seoul, Republic of Korea
<EMAIL_ADDRESS>
###### Abstract

Brain-computer interface (BCI) uses brain signals to communicate with external
devices without actual control. Particularly, BCI is one of the interfaces for
controlling the robotic arm. In this study, we propose a knowledge
distillation-based framework to manipulate robotic arm through hybrid paradigm
induced EEG signals for practical use. The teacher model is designed to decode
input data hierarchically and transfer knowledge to student model. To this
end, soft labels and distillation loss functions are applied to the student
model training. According to experimental results, student model achieved the
best performance among the singular architecture-based methods. It is
confirmed that using hierarchical models and knowledge distillation, the
performance of a simple architecture can be improved. Since it is uncertain
what knowledge is transferred, it is important to clarify this part in future
studies.

Keywords—brain-computer interface, electroencephalogram, knowledge
distillation, deep learning

##  I Introduction

Motion imagery (MI) [1, 2, 3] is a widely used paradigm in brain-computer
interfaces (BCIs), which induces meaningful brain signals through imagination
that operates without actual movements. Since MI is related to movements, some
studies on controlling external devices using MI-based BCI have been conducted
[4, 5, 6]. Speech imagery (SI) is related to imagining speaking without
actually speaking. Therefore SI has the advantage of being able to intuitively
generate brain signals [7, 8, 9]. Considering the characteristics of the two
paradigms, using these paradigms together would be advantageous for a
practical BCI system. In this study, we designed hybrid paradigm-based BCI
that uses MI and SI to control robotic arm. Specifically, MI is in charge of
the robotic arm movements, and SI is in charge of communication.

Numerous studies have been conducted to improve SI and MI classification
performance. For MI classification, Ang et al. [10] designed filter bank that
contains several spatial patterns to select features by searching optimal
filters for classification called FBCSP. Schirrmeister et al. [11] designed
convolutional neural network (CNN) which was inspired by pipeline of FBCSP.
The experimental results showed that shallow architecture is advantageous for
MI classification. Lawhern et al. [12] applied depth-wise convolution and
separable convolution that can avoid overparameterization. Fahimi et al. [13]
proposed a generative adversarial network-based framework to generate
additional EEG signals for performance improvement. For speech imagery
classification, DaSalla $et\ al.$[14] introduced common spatial pattern (CSP)
for SI classification. Torres-Garcia $et\ al.$ [9] designed fuzzy inference to
find Pareto front for selecting optimal channels. Nguyen $et\ al.$ [7] showed
that Riemannian manifold is efficient to classify long and short words. Their
experiment showed complexity of word is the key factor of SI classification.

In this paper, we proposed a hybrid paradigm-based BCI to efficiently control
robotic arm. Furthermore, we applied knowledge distillation for compressing
the model parameters that reduces computational complexity of the system.
Teacher model is composed of three convolutional neural network (CNN)
architectures that hierarchically classifies EEG signals. Student model is a
singular CNN architecture. We assume that hierarchical classification has
advantages in hybrid paradigm BCI, and through knowledge distillation, even a
singular structure shows satisfactory classification performance. For
evaluation, two datasets were used; SI and MI datasets of BCI competition
2020111$http://brain.korea.ac.kr/bci2021$. The results of the experiment
demonstrated that hybrid paradigm-based BCI is efficient for practical use
through the knowledge distillation.

Figure 1: Framework of the proposed method. The teacher model is learned using
half the samples on the dataset. The student model is transferred knowledge
from the teacher model through soft label and distillation loss. At this time,
the teacher model is also trained using whole training set. In robotic arm
control system, only student model operates as a decoding model.

Figure 2: Pipeline of the knowledge distillation. During student model
training, teacher model transfer knowledge to student model until the loss
converge.  TABLE I: Design choices of the proposed model. Details of student
model is described in sub section $B.StudentModel$

Parameter | Shared module | MI module | SI module
---|---|---|---
Input | Raw EEG | Feature input | Feature input
| (1, 1, 24, 1000) | (1, 36, 1, 288) | (1, 36, 1, 288)
Hidden layer | Conv2D: 36, 72 | Conv2D: 72, 144, 288 | Conv2D: 72, 144, 288
| AvgPool: (1,3) | AvgPool: (1,3) | AvgPool: (1,3)
| Stride: (1,3) | Stride: (1,3) | Stride: (1,3)
Activation | ELU | ELU | ELU
| Last layer: Softmax | Last layer: Softmax | Last layer: Softmax
Optimizer | AdamW | AdamW | AdamW

##  II Methods

We propose a singular CNN structure-based robotic arm control system that
decodes MI and SI-induced EEG signals. Basically, the EEG signals have no
underlying ground truth of EEG signals and intricate and non-stationary
signals characteristics that make it difficult to classify the MI and SI
induced EEG signals. Moreover, since MI and SI induce EEG signals in different
brain regions, decoding MI and SI-induced EEG signals is difficult using only
singular architecture. Therefore, we applied the knowledge distillation
technique by setting up a teacher model that classifies signals hierarchically
and a student model that has a singular CNN architecture. Overall flow chart
of the proposed method is described in Fig.1.

###  II-A Teacher Model

Teacher model consists of three CNN architectures inspired by [15]; shared
module, MI module, and SI module. The shared module consisted of two
convolution layers to classify paradigm. The first layer performs temporal and
spatial convolution by reducing channel dimension into single. The second
layer performs convolution and categorizes paradigm defining it as MI or SI.
According to prediction, MI or SI module utilizes the features from the second
layer as input to conduct classifications for each paradigm.

MI and SI modules exploit features to improve classification accuracy,
specializing in SI and MI classification. Both SI and MI modules are composed
of three convolution-pooling layers. During the model training two modules
received features from the shared module regardless of the shared module’s
prediction. Through this, one module learns corresponding paradigm signals
while the other module learns the wrong cases at the same time. Therefore, the
modules are specialized in handling each paradigm EEG signals. Detailed design
choices are described in Table I.

To train teacher model, we designed loss function that is composed of three
loss terms [16] defined as follows:

$loss(L_{sh},L_{M},L_{S})=L_{sh}+p_{M}L_{M}+p_{S}L_{S}$ (1)

where ${p_{M}}$ and ${p_{S}}$ are predictions of MI and SI for shared module,
respectively. ${L_{sh}}$, ${L_{M}}$ and ${L_{S}}$ are cross-entropy loss [17]
of the shared module, MI module and SI module, respectively, which are defined
as follow:

$\begin{split}L_{sh}=-\sum_{c=1}^{2}y_{sh}\log{\hat{y}_{sh}}\\\
L_{M}=-\sum_{c=1}^{M}y_{m}\log{\hat{y}_{m}}\\\
L_{S}=-\sum_{c=1}^{N}y_{s}\log{\hat{y}_{s}}\end{split}$ (2)

where ${y_{sh}}$, ${y_{m}}$, and ${y_{s}}$ are label of shared module, MI
module, and SI module, respectively. On the other hand, ${\hat{y}_{sh}}$,
${\hat{y}_{m}}$, and ${\hat{y}_{s}}$ are prediction of the shared module, MI
module, and SI module, respectively. $c$ denotes class, and the shared module
conducts binary classification because it is a paradigm classifier.

###  II-B Student Model

Three convolution-pooling layers composed of a student model. Basically, it
has the same structure as the shared module, and we added a layer with a 144
size convolution filter. It was designed to reduce the interference time in
robotic arm control system. The purpose of the student model is to distill the
knowledge of the teacher model and utilize it for training to classify hybrid
paradigm induced EEG signals even it is a simple structure.

###  II-C Knowledge Distillation

Since the teacher model is a hierarchical structure, the computational cost is
high. Therefore, for daily life BCI-based robotic arm control, a decoding
model need to show robust performance while the computational cost is small.
As mentioned above, we designed a student model with a singular architecture
that is guided by the teacher model through knowledge distillation to achieve
robust classification performance. To this end, we combined cross-entropy loss
with distillation loss [18] which is defined as

$L=\sum_{(x,y)\in D}L_{KD}(S(x,\theta_{S},\tau),T(x,\theta_{T},\tau))+\lambda
L_{CE}(\hat{y_{S}},y)$ (3)

where S, T, $x$, and $y$ denote student and teacher model, input, and true
label. $\theta$, $\tau$, and $\hat{y_{S}}$ denote model parameter,
temperature, and student model output. $T$ makes the output of the lower input
larger and the output of the larger input smaller to maximize the benefits of
using soft labels. Pipeline of the knowledge distillation is described in
Fig.2.

##  III Results and Discussions

We used BCI competition 2020 dataset composed of MI and SI dataset. MI dataset
is composed of dataset that contains 150 trials and 3 classes (cylindrical,
lumbrical, and spherical grasp). On the other hand, 300 trials and 5 classes
consist of SI dataset. It contains 60 training trials and 10 trials of
validation set and test set, respectively. Hence, MI dataset contains 150
trials including all datasets, we randomly selected 80 trials from the MI
dataset for this study to avoid data imbalance problem. As the evaluation was
conducted in leave-one-subject-out manner. Twenty four channels were selected
because MI and SI datasets share 24 channels (Fp1 Fp2 F7 F3 Fz F4 T8 CP5 CP1
CP2 CP6 O2 AF7 AF3 AF4 AF8 C1 C2 C6 TP7 PO3 POz PO4 PO8). Evaluation was
conducted with 0.001 learning rate, 5 iteration patience, 32 batch size, and
200 training epoch. AdamW [19] optimizer with weight decay (0.01) was applied
for the experiment. Experimental environment was conducted as a subject
independent manner on an Intel 3.60 Core i7 9700 K CPU with 32 GB of RAM,
NVIDIA TITAN V GPU, CUDA/Cudnn, and Python version 3.9 with PyTorch version
1.9.

TABLE II: Comparison of evaluation results. Reported accuracy was calculated leave-one-subject-out manner. The upper subscripts 1 and 2 denote deep and shallow ConvNet [11], respectively. RF denotes random forest. The highest numbers are bold. | Paradigm | Class
---|---|---
Model | Accuracy (std) | Accuracy (std)
CSP+RF [20] | 68.18 (3.52) | 23.61 (6.89)
CSP+SVM [10] | 67.74 (6.89) | 22.98 (6.12)
CSP+LDA [10] | 70.05 (2.64) | 25.41 (9.10)
FBCSP [10] | 74.57 (6.65) | 45.47 (5.27)
EEGNet [12] | 84.64 (4.12) | 54.56 (4.30)
ConvNet1 [11] | 82.57 (3.33) | 52.52 (4.02)
ConvNet2 [11] | 86.98 (6.26) | 55.46 (4.72)
Shared module | 83.61 (3.81) | -
Teacher Model | - | 61.27 (4.03)
Student Model | - | 57.36 (5.29)

CSP is a method of maximizing the variance of classes based on covariance
matrices and minimizing the variance of other classes. Relatively small
numbers of data lead to overfitting and performance degradation. Random
forest, support vector machines, and linear discriminant analysis were
selected as classifiers for CSP. Table II shows that CSP-based methods show
lower accuracy (around 70%) than CNN architectures (around 81.95%) in paradigm
classification. For class classification, their average accuracy was around
29.47% which is 30% lower than CNN-based methods (56.23%). The dataset
consists of two paradigm-induced EEG signals, and these dataset properties are
expected to be a hindrance to the mechanism of CSP. Additionally, the small
number of training samples can be one of the reasons of performance
degradation.

Four CNN-based methods [11, 12] including the proposed method were used.
EEGNet [12] and shallow ConvNet [11] showed similar performance in both
paradigm and class classification. Although shallow ConvNet was designed to
extract band power features, which are known as MI-specific features, it
showed the best accuracy in paradigm classification. Compared to paradigm
classification, CNN-based methods dropped around 30% in class classification.
The teacher model was designed to decode hybrid paradigm induced EEG signals
through hierarchical classification and achieved the highest performance
(61.27%) although shared module showed only 83.61%. However, student model
which was trained through knowledge distillation achieved second highest
accuracy. It recorded the best performance among the singular CNN
architectures. Through the results, it confirmed that the hierarchical
architecture improves the performance of a singular architecture through
knowledge distillation. Results are described in Table II.

However, it was not revealed how the student model was influenced by the
knowledge of the teacher model through knowledge distillation. Furthermore,
further experiments are needed on whether simply using soft labels can
efficiently use the performance of student models. Nevertheless, considering
that student models show higher performance than single-structure-based
methods, knowledge distillation can be one way to improve the performance of
hybrid paradigm-based BCIs.

##  IV Conclusion and Future Works

In this paper, we proposed a knowledge distillation based-framework that was
designed to decode hybrid paradigm induced EEG signals for robotic arm
control. Teacher model was designed to hierarchically classify input data.
Student model was trained under the guidance of teacher models through soft
labels and distillation loss function. According to the results, knowledge
distillation improves student model performance even it is a singular
architecture. However, additional experiments are needed on how the student
model was influenced by the teacher model through the distillation of
knowledge. Thus, our future work is to ensure that the hierarchical model
transfers efficient knowledge to a singular architecture through the
distillation of knowledge.

##  V Acknowledgement

The author would like to thank K. Yin and H.-B. Shin for help with the
discussion of the data analysis.

## References

  * [1] G. Pfurtscheller and C. Neuper, “Motor imagery and direct brain-computer communication,” _Proc. IEEE_ , vol. 89, no. 7, pp. 1123–1134, 2001.
  * [2] C. Neuper, M. Wörtz, and G. Pfurtscheller, “ERD/ERS patterns reflecting sensorimotor activation and deactivation,” _Prog. Brain Res._ , vol. 159, pp. 211–222, 2006.
  * [3] J.-H. Jeong, J.-H. Cho, K.-H. Shim, B.-H. Kwon, B.-H. Lee, D.-Y. Lee, D.-H. Lee, and S.-W. Lee, “Multimodal signal dataset for 11 intuitive movement tasks from single upper extremity during multiple recording sessions,” _GigaScience_ , vol. 9, no. 10, p. giaa098, Oct. 2020.
  * [4] K.-T. Kim, H.-I. Suk, and S.-W. Lee, “Commanding a brain-controlled wheelchair using steady-state somatosensory evoked potentials,” _IEEE Trans. Neural Syst. Rehabil. Eng._ , vol. 26, no. 3, pp. 654–665, 2016.
  * [5] C. I. Penaloza and S. Nishio, “BMI control of a third arm for multitasking,” _Sci. Robot._ , vol. 3, no. 20, p. eaat1228, 2018.
  * [6] B. Edelman, J. Meng, D. Suma, C. Zurn, E. Nagarajan, B. Baxter, C. Cline, and B. He, “Noninvasive neuroimaging enhances continuous neural tracking for robotic device control,” _Sci. Robot._ , vol. 4, no. 31, p. eaaw6844, 2019\.
  * [7] C. H. Nguyen, G. K. Karavas, and P. Artemiadis, “Inferring imagined speech using EEG signals: a new approach using Riemannian manifold features,” _J. Neural Eng._ , vol. 15, no. 1, p. 016002, Dec. 2017.
  * [8] X. Tian, J. M. Zarate, and D. Poeppel, “Mental imagery of speech implicates two mechanisms of perceptual reactivation,” _Cortex_ , vol. 77, pp. 1–12, 2016.
  * [9] A. A. Torres-García, C. A. Reyes-García, L. Villaseñor-Pineda, and G. García-Aguilar, “Implementing a fuzzy inference system in a multi-objective EEG channel selection model for imagined speech classification,” _Expert Syst. Appl._ , vol. 59, pp. 1–12, Oct. 2016.
  * [10] K. K. Ang, Z. Y. Chin, H. Zhang, and C. Guan, “Filter bank common spatial pattern (FBCSP) in brain-computer interface,” in _Proc. Int. Jt. Conf. Neural Netw._ , Hong Kong, China, Jun, 2008, pp. 2390–2397.
  * [11] R. T. Schirrmeister, J. T. Springenberg, L. D. J. Fiederer, M. Glasstetter, K. Eggensperger, M. Tangermann, F. Hutter, W. Burgard, and T. Ball, “Deep learning with convolutional neural networks for EEG decoding and visualization,” _Hum. Brain Mapp._ , vol. 38, no. 11, pp. 5391–5420, 2017\.
  * [12] V. J. Lawhern, A. J. Solon, N. R. Waytowich, S. M. Gordon, C. P. Hung, and B. J. Lance, “EEGNet: A compact convolutional neural network for EEG-based brain-computer interfaces,” _J. Neural Eng._ , vol. 15, no. 5, p. 056013, 2018.
  * [13] F. Fahimi, S. Dosen, K. K. Ang, N. Mrachacz-Kersting, and C. Guan, “Generative adversarial networks-based data augmentation for brain-computer interface,” _IEEE Trans. Neural Netw. Learn. Syst._ , 2020.
  * [14] C. S. DaSalla, H. Kambara, M. Sato, and Y. Koike, “Single-trial classification of vowel speech imagery using common spatial patterns,” _Neural Netw._ , vol. 22, no. 9, pp. 1334–1339, Nov. 2009.
  * [15] Z. Yan, H. Zhang, R. Piramuthu, V. Jagadeesh, D. DeCoste, W. Di, and Y. Yu, “HD-CNN: Hierarchical Deep Convolutional Neural Networks for Large Scale Visual Recognition,” in _Proc. IEEE Int. Conf. Comput. Vis._ , 2015, pp. 2740–2748.
  * [16] B.-H. Lee, J.-H. Jeong, K.-H. Shim, and S.-W. Lee, “Classification of high-dimensional motor imagery tasks based on an end-to-end role assigned convolutional neural network,” in _IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP)_. IEEE, 2020, pp. 1359–1363.
  * [17] Z. Zhang and M. Sabuncu, “Generalized cross entropy loss for training deep neural networks with noisy labels,” in _Adv. Neural Inf. Process. Syst. (NIPS)_ , 2018, pp. 8778–8788.
  * [18] G. Hinton, O. Vinyals, J. Dean _et al._ , “Distilling the knowledge in a neural network,” _arXiv preprint arXiv:1503.02531_ , vol. 2, no. 7, 2015\.
  * [19] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” _arXiv preprint arXiv:1711.05101_ , 2017.
  * [20] Y. Qi, “Random forest for bioinformatics,” in _Ensem. mach. learn._ , 2012, pp. 307–323.


# Ross’s second conjecture and supermodular stochastic ordering††thanks: This
preprint has not undergone peer review or any post-submission improvements or
corrections. The Version of Record of this article is published in Queueing
Systems, and is available online at https://doi.org/10.1007/s11134-022-09824-0

Lasse Leskelä111 Department of Mathematics and Systems Analysis, Aalto
University, Otakaari 1, 02015 Espoo, Finland.

(7 February 2022)

##  1 Introduction

In 1978, about eight years before the first issue of _Queueing Systems: Theory
and Applications_ , Sheldon M. Ross presented in [13] two famous conjectures
about the classical single server queue subject to a randomly time-varying
rate arrival rate $\lambda(t)$. Intuition suggests that more autocorrelated
arrival rate processes lead to more bursty arrival patterns, and therefore to
larger workloads and longer waiting times. This question is naturally
approached by modelling the arrival times using a Cox process (doubly
stochastic Poisson process) in which the instantaneous arrival rate
$\lambda(t)$ is a stationary and ergodic random process.

_Ross’s first conjecture_ in [13] states that the mean stationary waiting time
in the system becomes smaller if the Cox arrival process is replaced by a
homogeneous Poisson process with constant rate $\mathbb{E}\lambda(0)$. In
1981, Tomasz Rolski [11] published an elegant proof this conjecture in full
generality. After five more years, Rolski [12] also managed to prove a
complementary upper bound, showing that the mean stationary waiting time in
the system becomes larger if the intensity of the Cox process is replaced by a
random constant $\lambda(0)$. These bounds display two extreme dependence
structures:

  * •

The lower bound corresponds to a fully averaged random environment and leads
to completely independent (Lévy) dependence structure.

  * •

The upper bound corresponds to a frozen or quenched random environment, and
leads to a maximally autocorrelated dependence structure.

_Ross’s second conjecture_ in [13] concerns an arrival process with randomly
time-varying intensity $\lambda_{c}(t)=\lambda(ct)$, parameterised by a
modulation rate $c\in(0,\infty)$ and a stationary and ergodic baseline
intensity $\lambda(t)$. The parameter $c$ corresponds to a modulation of rate
of a random environment, so that $c\to 0$ corresponds to freezing the
environment to a (random) constant, and $c\to\infty$ corresponds to observing
the environment at its ergodic average value. Intuitively, a smaller
modulation rate should correspond to a more autocorrelated intensity process,
more bursty arrival point patters, and longer waiting times. Therefore, Ross
conjectured that the mean stationary waiting time in the queue should be a
decreasing function of the modulation rate $c$.

##  2 Problem statement

It appears that, after more than 40 years, Ross’s second conjecture still
remains not fully solved. This motivates highlighting the following problem.
Consider a single server queue with with a Cox arrival process and IID service
times, where the arrival rate $\lambda_{c}(t)=\lambda(ct)$ is defined in terms
of a modulation rate $c\in(0,\infty)$ and a stationary and ergodic random
baseline intensity $\lambda(t)$ for which the queueing system is stable.
Denote by $w(c)$ the mean stationary workload in the system.

######  Problem 1.

Is it always true that

$c_{1}\leq c_{2}\quad\implies\quad w(c_{1})\geq w(c_{2}),$ (1)

and if not, what is a simple necessary and sufficient condition for
$\lambda(t)$ that is needed for (1) to be valid?

##  3 Discussion

During past decades, the study of the above question has inspired the
development of novel methods, e.g. stochastic dependence orders [10, 14], for
analysing spatial and temporal autocorrelations in depth far greater than with
pairwise correlation coefficients. Key milestones partially answering the
above question include:

  * •

In the original article [13], Ross proved (1) for a special case in which the
baseline intensity is a two-state CTMC with one state being zero.

  * •

In 1991, Chang, Chao, and Pinedo [3] proved (1) in case where the baseline
intensity is a finite-state CTMC with transition rates of form
$q_{ij}=\alpha_{i}>0$ for all $i\neq j$.

  * •

In 1998, Bäuerle and Rolski [1] established (1) in case where the baseline
intensity is a finite-state CTMC for which the transition rate matrix $Q$ and
its time-reversal $Q^{*}$ with entries
$Q^{*}_{ij}=\frac{\pi_{j}}{\pi_{i}}Q_{ji}$ are stochastically monotone with
respect to the strong stochastic order on the real line.

As a consequence of latter milestone, we know that (1) holds whenever the
baseline intensity $\lambda(t)$ is a reversible and stochastically monotone
CTMC, such as a two-state CTMC, or a birth-and-death process. However,
monotonicity in the strong stochastic order is a property of a dynamic system
preserving the ordering of states [5, 6, 7] and as such is not directly
related to autocorrelation properties. If stochastic monotonicity precisely
characterises CTMCs satisfying (1), this would be a fundamental, and perhaps
surprising, discovery.

Bäuerle and Rolski [1] also showed that a sufficient condition for (1) is that
$c\mapsto\lambda_{c}=(\lambda_{c}(t))$ is decreasing in the supermodular
stochastic ordering, in the sense that

$c_{1}\leq c_{2}\implies\lambda_{c_{1}}\geq_{\rm{sm}}\lambda_{c_{2}},$ (2)

where we recall that for stochastic processes $X\leq_{\rm{sm}}Y$ means that
the inequality
$\mathbb{E}\phi(X_{t_{1}},\dots,X_{t_{n}})\leq\mathbb{E}\phi(Y_{t_{1}},\dots
Y_{t_{n}})$ holds for all integers $n\geq 1$, all time instants $t_{k}$, and
all supermodular functions $\phi:\mathbb{R}^{n}\to\mathbb{R}$ for which the
expectations exist [10, 14]. Furthermore, any doubly stochastically monotone
CTMC studied in [1] satisfies (2). These discoveries have inspired impressive
analytical works analysing supermodular regularity of Markov chains [4, 8, 9]
and more recently also spatial random measures and point patterns [2].
Nevertheless, the following problems appears are still unanswered in full
generality to date.

######  Problem 2.

Does there exist a stationary and ergodic CTMC $\lambda=(\lambda(t))$ which is
not stochastically monotone but satisfies (2)?

######  Problem 3.

Describe a simple necessary and sufficient condition for a stationary and
ergodic CTMC $\lambda=(\lambda(t))$ under which (2) holds.

Problems 2 and 3 are appealing because, outside the domain of stochastically
monotone CTMCs, not much seems to be known about them even for state spaces of
size 3. Furthermore, the above problems may also be formulated for discrete-
time Markov chains, either by uniformising the CTMC, or by analysing chains
generated by dampened transition probability matrices $(1-c)I+cP$ with
$c\in(0,1]$ and $P$ representing a baseline Markov chain. To the best of my
knowledge, all of the above problems remain equally open also in the discrete-
time setting.

## References

  * [1] N. Bäuerle and T. Rolski.  A monotonicity result for the workload in Markov-modulated queues.  J. Appl. Probab., 35(3):741–747, 1998.
  * [2] B. Błaszczyszyn and D. Yogeshwaran.  Directionally convex ordering of random measures, shot noise fields, and some applications to wireless communications.  Adv. Appl. Probab., 41(3):623–646, 2009.
  * [3] C.-S. Chang, X. Chao, and M. Pinedo.  Monotonicity results for queues with doubly stochastic Poisson arrivals: Ross’s conjecture.  Adv. Appl. Probab., 23(1):210–228, 1991.
  * [4] T. Hu and X. Pan.  Comparisons of dependence for stationary Markov processes.  Probability in the Engineering and Informational Sciences, 14(3):299–315, 2000.
  * [5] L. Leskelä.  Stochastic relations of random variables and processes.  J. Theor. Probab., 23(2):523–546, 2010.
  * [6] L. Leskelä and M. Vihola.  Stochastic order characterization of uniform integrability and tightness.  Statist. Probab. Lett., 83(1):382–389, 2013.
  * [7] W. A. Massey.  Stochastic orderings for Markov processes on partially ordered spaces.  Math. Oper. Res., 12(2):350–367, 1987.
  * [8] L. E. Meester and J. G. Shanthikumar.  Regularity of stochastic processes: A theory based on directional convexity.  Probability in the Engineering and Informational Sciences, 7(3):343–360, 1993.
  * [9] N. Miyoshi and T. Rolski.  Ross-type conjectures on monotonicity of queues.  Aust. N. Z. J. Stat., 46(1):121–131, 2004.  Festschrift in honour of Daryl Daley.
  * [10] A. Müller and D. Stoyan.  Comparison Methods for Stochastic Models and Risks.  Wiley, 2002.
  * [11] T. Rolski.  Queues with nonstationary input stream: Ross’s conjecture.  Adv. Appl. Probab., 13(3):603–618, 1981.
  * [12] T. Rolski.  Upper bounds for single server queues with doubly stochastic Poisson arrivals.  Mathematics of Operations Research, 11:442–450, 1986.
  * [13] S. M. Ross.  Average delay in queues with non-stationary Poisson arrivals.  J. Appl. Probab., 15(3):602–609, 1978.
  * [14] M. Shaked and J. G. Shanthikumar.  Stochastic Orders.  Springer, 2007.