# A New Graph Node Classification Benchmark: Learning Structure from Histology
Cell Graphs

Claudia Vanea1, 2, Jonathan Campbell1, 2, 3, Omri Dodi4, Liis Salumäe5, Karen
Meir4,
Drorith Hochner-Celnikier4, Hagit Hochner6, Triin Laisk7, Linda M. Ernst,8
Cecilia M. Lindgren 2, Christoffer Nellåker11footnotemark: 1 1, 2
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Nuffield Department of Women’s & Reproductive Health, University of Oxford1
Big Data Institute, University of Oxford2
Visual Geometry Group, University of Oxford3
Faculty of Medicine, Hadassah Hebrew University Medical Center4
Department of Pathology, Tartu University Hospital5
Braun School of Public Health, Hebrew University of Jerusalem6
Institute of Genomics, University of Tartu7
Northshore University HealthSystem, University of Chicago8 Equal contribution.

###### Abstract

We introduce a new benchmark dataset, _Placenta_ , for node classification in
an underexplored domain: predicting microanatomical tissue structures from
cell graphs in placenta histology whole slide images. This problem is uniquely
challenging for graph learning for a few reasons. Cell graphs are large (>1
million nodes per image), node features are varied (64-dimensions of 11 types
of cells), class labels are imbalanced (9 classes ranging from 0.21% of the
data to 40.0%), and cellular communities cluster into heterogeneously
distributed tissues of widely varying sizes (from 11 nodes to 44,671 nodes for
a single structure). Here, we release a dataset consisting of two cell graphs
from two placenta histology images totalling 2,395,747 nodes, 799,745 of which
have ground truth labels. We present inductive benchmark results for 7
scalable models and show how the unique qualities of cell graphs can help
drive the development of novel graph neural network architectures.

##  1 Introduction

The initial graph representation learning benchmarks facilitated the creation
of the first graph neural network (GNN) architectures. _Cora_ , _Citeseer_ ,
_Pubmed_ and _NELL_ [29], [3] drove the development of Graph Convolutional
Network (GCN) [16], _Reddit_ was introduced alongside GraphSAGE [12], and
Graph Attention Network (GAT) [26] achieved state-of-the-art on the protein-
protein interaction (_PPI_) dataset [34]. However, it soon became apparent
that these datasets were not sufficiently complex to challenge GNNs [13, 28].
The datasets had small graphs or a small number of graphs (the largest
dataset, _Reddit_ , had 232,965 nodes), they had potentially misleading train,
validation and test splits [22], and they were so well suited to graph
learning that randomly initialised, untrained models could perform nearly as
well as trained models [27]. In recent years, there has been an influx of
graph benchmark datasets which address these limitations. They are larger and
cover more domains, have realistic train, validation and test splits, and are
challenging enough to have not been ‘solved’ by current GNN architectures [13,
5, 24].

Cell graphs in histology whole slide images (WSIs) are a domain which has been
largely unexplored for graph learning and present a challenge to the graph
learning community [18, 33, 10, 4, 19]. Histology is the study of tissues and
cells under a microscope. A histology WSI is a digitised, high-resolution scan
of thin tissue samples showing individual cells, which can be upwards of 70GBs
per image. All organs are composed of cells that collectively build up the
structures and tissues comprising the body. Consequently, representing organs
as cell graphs is a logical construction closely matching the true physical
system. The value of cell graphs has been highlighted [30, 14, 23], but with
today’s computational power and automated aggregation methods, it can reach
its full potential.

Here, we introduce _Placenta_ , a cell graph from two WSIs as a new benchmark
dataset to facilitate GNN creation. More specifically, we present a node
classification task of predicting one of 9 placenta microanatomical tissue
structures to which cells belong (Appendix A). This problem is challenging
from a graph learning perspective for a number of reasons. Firstly, each WSI
contains a cell graph of >1 million nodes, the node features are defined from
an imbalanced distribution of cell classes, and the tissue labels also have a
high, but expected, class imbalance, from 0.21% of the data to 40.0% (Appendix
B). Secondly, unlike most graph datasets, node clusters are locally homophilic
but spatially distributed throughout the graph, requiring models to understand
same-type node clusters at different locations. Thirdly, as a WSI is only ever
a small sample of the larger structure with arbitrary cut-offs in edges, out-
of-plane cells, and optical and mechanical artefacts, the property of data
incompleteness is inherently a feature of this data domain. Finally, placental
tissue structures grow from one another in a tree-like manner [1, 7], so
singular structures can contain multiple tissue classifications and these
structures will vary in size (30-5,000 $\mu$m2). Models must discriminate
between small changes across small regions within node clusters but have a
large enough receptive field to identify the largest of tissues.

Relevant to the real world domain, developing automated methods which can
quantify the structural tissue units within histology images would be an
invaluable tool for the scientific community and, in future, may have
applications to clinical histopathologic diagnoses and patient care.
Furthermore, this dataset provides a benchmark for developing novel, scalable
GNNs that can adapt to node feature and class label imbalance, that can
understand both large and small node communities distributed throughout the
graph, and that are robust to data incompleteness. These models could be
applied to new domains with similar qualities, such in modelling disease
epidemiology.

Table 1: Dataset characteristics for the intersection graph. | Nodes | Edges | Labels | Homophily | Degree
---|---|---|---|---|---
Cell Graph 1 | 1,159,186 | 2,603,797 | 703,165 | 0.9868 | 5.4925
Cell Graph 2 | 1,236,561 | 2,882,292 | 96,580 | 0.9984 | 5.6618

##  2 Dataset Description

### Overview

At a high level, the learning task is to understand placental tissue
structures as the composition of their cellular communities. Tissue structures
in the placenta are precisely defined and distinguished from one another by
their cellular composition [1]. In a WSI these manifest as spatially clustered
groups of cells, with corresponding tissue structures comprising 10s-10,000s
of cell nodes. Inductive message-passing GNNs, which can aggregate node
neighbourhood information irrespective of graph shapes and sizes are, in
theory, a natural fit for this task.

The dataset we introduce for this task, _Placenta_ , consists of two combined
cell graphs constructed across two healthy, term placenta hematoxylin and
eosin stained WSIs from two institutes (Table 1). Ground truth labels were
created by the primary author and validated by four practising perinatal
pathologists (details in Vanea et al. [25]). See Figure 1 for a high level
visualisation of all cell graph nodes and labelled ground truth nodes and
Figure 2 for a representative region. The use of these images was approved and
the requirement for consent was waived by local ethics committees at both
institutes (approval 0735-18-HMO and 289/T-5).

We recommend a train, validation, test split (Appendix C) which ensures tissue
types have similar distributions across splits. The type of tissue in the
placenta can be influenced by macroscopic location (for example, proximity to
the chorionic or basal plate), so these dataset split regions are chosen to
cover a range of locations and to avoid neighbourhood information leakage.

### Nodes and Node Features

We use a prior two-stage deep learning pipeline to identify the centre point
of all nuclei across the WSI and classify those nuclei as belonging to one of
11 cell types. The nucleus coordinates define the node coordinates in the
graph and the 64-dimension embeddings of the penultimate layer of the cell
classifier model are used for the node features. Full details of this pipeline
can be found in Vanea et al. [25].

### Edges and Edge Features

We explore three edge-building algorithms which act as a proxy for cellular
interaction: KNN [6], Delaunay Triangulation [11], and the intersection of the
two (Appendix D). Unless stated otherwise, we use the intersection graph with
k=5 for all experiments.

Edge construction in the context of cell graphs are representations of the
approximate biological likelihood that nearby cells interact as part of the
same structure. The Euclidean distance-based edge construction we have
explored is one method for this, but alternative approaches for constructing
the network are worth considering. We leave this as an open question for the
community.

Figure 1: Cell graph nodes across from whole slide images (left) with
corresponding labelled ground truth tissue nodes (right). Black tissue nodes
are unlabelled. Figure 2: A 1600x1200 pixel region showing (top left) the raw
histology image, (top right) overlaid cell nodes, (bottom left) the
constructed intersection cell graph, and (bottom right) corresponding ground
truth tissue nodes.

##  3 Experiments

### Experimental Setup

We first construct a cell graph across all cells in both WSIs. Nodes are
masked as either unlabelled, training, validation, or test nodes and only
nodes in the training set are used for back propagation. Both cell graphs are
combined into one disjointed graph, allowing the sampler to treat it as a
dataset of one graph. We restrict our experiments to scalable GNNs as the full
graph cannot fit into GPU memory. During training, for GraphSAGE [12],
ClusterGCN [5], GraphSAINT [31], ShaDow [32] and SIGN [9], node embeddings are
sampled or precomputed according to original implementations. For models
introduced without a sampling technique (GAT [26] and GATv2 [2]), we use
ClusterGCN subgraph sampling (ClusterGAT and ClusterGATv2). During inference,
all neighbourhood nodes are used for aggregation. We additionally compare
against two baselines, an MLP on node embeddings and a randomly initialised
GraphSAGE.

### Hyperparameters

Models are trained for inductive learning fully-batched to 1000 epochs with
early stopping on validation data (evaluated every 100 epochs), weighted
cross-entropy loss, Adam optimiser [15], and a 0.001 learning rate. Models
have 256 hidden units per layer, a batch normalisation layer following each
graph convolutional layer (except for ShaDow), and at most 16 graph
convolutional layers each with a 0.5 dropout and relu activations. For
GraphSage, ClusterGAT, and ClusterGATv2, 16 layer models could not fit into
GPU memory even with reduced sampling and batch sizes and, as such, fewer
layers were used. Further hyperparameter details can be found in the
supplement (Appendix E).

Note that the goal is not to find the best performing architecture for this
dataset, but instead to show the problems existing architectures have on
histology cell graphs. As such, we explore hyperparameters to the extent that
we believe a fair comparison can be made between models but we do not perform
an exhaustive search. This has the added benefit of avoiding overfitting
hyperparameters to these two placenta WSIs, which only represent a small facet
of the histology domain.

### Hardware, Software, and Libraries

Experiments were run on one NVIDIA RTX8000 on an internal cluster. Code was
written in Python (v3.7.2) using PyTorch (v1.9.0)[20] and PyTorch Geometric
(v2.0.4)[8]. WSI visualisation and ground truth annotations were made using
QuPath (v0.3.1)[21]. Code, cell graph dataset, data splits, and experiment
instructions may be found at (https://github.com/Nellaker-group/placenta)
under a MIT licence.

##  4 Results

Table 2 shows the test mean accuracies and standard deviations of models over
5 random weight initializations. The randomly initialised baseline performs
equivalent to random chance (ROC AUC 0.5) and GNN models outperform the MLP
baseline. We found ClusterGAT and ClusterGATv2 were unstable to train with
more layers and this might contribute to their lower performance score. All
scalable GNN architectures: GraphSAGE, ClusterGCN, GraphSAINT, ShaDow, and
SIGN are within 2% mean accuracy of each other, with no models surpassing 65%
accuracy. These results highlight the need for new GNN approaches for cell
graphs in histology.

Table 2: Mean model performance and standard deviations over 5 random weight initialisation. Bold indicates the best performing model for that metric. | Accuracy | Top 2 Accuracy | ROC AUC
---|---|---|---
Random Baseline | 5.56$\pm$6.24 | 14.88$\pm$11.35 | 0.459$\pm$0.057
MLP Baseline | 47.98$\pm$0.79 | 75.22$\pm$0.92 | 0.750$\pm$0.003
GraphSAGE-mean | 64.88$\pm$0.43 | 88.94$\pm$0.38 | 0.883$\pm$0.005
ClusterGCN | 64.24$\pm$1.21 | 88.26$\pm$0.82 | 0.882$\pm$0.006
GraphSAINT-rw | 63.94$\pm$0.23 | 87.86$\pm$0.15 | 0.895$\pm$0.002
SIGN | 64.77$\pm$0.43 | 88.32$\pm$0.42 | 0.886$\pm$0.002
ShaDow | 63.04$\pm$0.77 | 86.88$\pm$0.74 | 0.863$\pm$0.008
ClusterGAT | 58.07$\pm$0.61 | 83.43$\pm$0.96 | 0.851$\pm$0.002
ClusterGATv2 | 57.07$\pm$0.65 | 83.21$\pm$0.55 | 0.854$\pm$0.005

In the supplement, we describe features which the models struggled with and we
take a fast, high-performing model (ClusterGCN) to explore some of these in
more detail (Appendix F, G, H). We show how class imbalance affects
performance, how model size leads to overfitting, and the importance of a
large enough receptive field. We additionally compare model performance using
all edge-building algorithms (Appendix D), using one hot encoded cell type
node features (Appendix I), and using random node data splits (Appendix C).

##  5 Limitations

Our cell graph dataset represents a small facet of the histology domain; two
placenta WSIs from two institutes. WSIs from more placentas and other
laboratories will be needed to validate the generalisability of models
designed using this data. Likewise, the dataset is placenta-specific, so
whilst there is a benefit to using the weights of placenta models for
pretraining, the development of GNN tissue classifiers for other organs will
require organ-specific cell and tissue ground truth data. Cross- or multi-
organ self-supervised approaches could be developed from our dataset but it is
not something we have explored. Regarding model comparison, we have purposely
limited the scope of our hyperparameter search. It is possible that a
different combination of hyperparameters and/or additional sampling,
normalisation, and regularisation, would have yielded better performance. We
release all data necessary for building cell graphs across our WSIs but not
the raw images or pipeline needed to derive the cell features from scratch as
this was out of scope for this work. Finally, it should be noted that
uncertainty in ground truth annotations may limit model performance on the
dataset, however label uncertainty is inherent to the biological problem and
unavoidable in this domain.

##  6 Conclusion

We introduce a new inductive node classification benchmark dataset,
classifying microanatomical tissue structures from cell graphs across placenta
histology WSIs. We show how GNN architectures perform on this task and
highlight the qualities unique to this data domain which present a challenge
for existing models. We hope that this dataset facilitates the creation of new
model architectures to overcome these challenges.

## Acknowledgments and Disclosure of Funding

Claudia Vanea and Jonathan Campbell are supported by the EPSRC Center for
Doctoral Training in Health Data Science (EP/S02428X/1). Cecilia M. Lindgren
is supported by the Li Ka Shing Foundation, NIHR Oxford Biomedical Research
Centre, Oxford, NIH (1P50HD104224-01), Gates Foundation (INV-024200), and a
Wellcome Trust Investigator Award (221782/Z/20/Z). Triin Laisk is funded by
the European Regional Development Fund and the programme Mobilitas Pluss
(MOBTP155) and the Estonian Research Council grant PSG776. The computational
aspects of this research were supported by the Wellcome Trust Core Award Grant
Number 203141/Z/16/Z and the NIHR Oxford BRC. The views expressed are those of
the author(s) and not necessarily those of the NHS, the NIHR or the Department
of Health.

## References

  * Benirschke et al. [2012] K. Benirschke, G. J. Burton, and R. N. Baergen.  Architecture of Normal Villous Trees.  In K. Benirschke, G. J. Burton, and R. N. Baergen, editors, _Pathology of the Human Placenta_ , pages 101–144. Springer, Berlin, Heidelberg, 2012.  ISBN 978-3-642-23941-0.  doi: 10.1007/978-3-642-23941-0_7.  URL https://doi.org/10.1007/978-3-642-23941-0_7.
  * Brody et al. [2022] S. Brody, U. Alon, and E. Yahav.  How Attentive are Graph Attention Networks?, Jan. 2022.  URL http://arxiv.org/abs/2105.14491.  arXiv:2105.14491 [cs].
  * Carlson et al. [2010] A. Carlson, J. Betteridge, B. Kisiel, B. Settles, E. R. H. Jr, and T. M. Mitchell.  Toward an Architecture for Never-Ending Language Learning.  _AAAI Press_ , page 8, 2010.
  * Chen et al. [2020] R. J. Chen, M. Y. Lu, J. Wang, D. F. K. Williamson, S. J. Rodig, N. I. Lindeman, and F. Mahmood.  Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis.  _IEEE transactions on medical imaging_ , PP, Sept. 2020.  ISSN 1558-254X.  doi: 10.1109/TMI.2020.3021387.
  * Chiang et al. [2019] W.-L. Chiang, X. Liu, S. Si, Y. Li, S. Bengio, and C.-J. Hsieh.  Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks.  In _KDD’19_ , May 2019.  doi: 10.1145/3292500.3330925.  URL https://arxiv.org/abs/1905.07953v2.
  * Eppstein et al. [1997] D. Eppstein, M. S. Paterson, and F. F. Yao.  On Nearest-Neighbor Graphs.  _Discrete & Computational Geometry_, 17(3):263–282, Apr. 1997.  ISSN 1432-0444.  doi: 10.1007/PL00009293.  URL https://doi.org/10.1007/PL00009293.
  * Ernst and Carreon [2019] L. M. Ernst and C. K. Carreon.  Placenta.  In L. M. Ernst, E. D. Ruchelli, C. K. Carreon, and D. S. Huff, editors, _Color Atlas of Human Fetal and Neonatal Histology_ , pages 399–424. Springer International Publishing, Cham, 2019.  ISBN 978-3-030-11425-1.  doi: 10.1007/978-3-030-11425-1_36.  URL https://doi.org/10.1007/978-3-030-11425-1_36.
  * Fey and Lenssen [2019] M. Fey and J. E. Lenssen.  Fast Graph Representation Learning with PyTorch Geometric.  _arXiv:1903.02428 [cs, stat]_ , Apr. 2019.  URL http://arxiv.org/abs/1903.02428.  arXiv: 1903.02428.
  * Frasca et al. [2020] F. Frasca, E. Rossi, D. Eynard, B. Chamberlain, M. Bronstein, and F. Monti.  SIGN: Scalable Inception Graph Neural Networks.  In _ICML 2020_ , Apr. 2020.  URL https://arxiv.org/abs/2004.11198v3.
  * Gadiya et al. [2019] S. Gadiya, D. Anand, and A. Sethi.  Histographs: Graphs in Histopathology.  _arXiv:1908.05020 [cs, eess]_ , Aug. 2019.  URL http://arxiv.org/abs/1908.05020.  arXiv: 1908.05020.
  * Guibas et al. [1992] L. J. Guibas, D. E. Knuth, and M. Sharir.  Randomized incremental construction of Delaunay and Voronoi diagrams.  _Algorithmica_ , 7(1):381–413, June 1992.  ISSN 1432-0541.  doi: 10.1007/BF01758770.  URL https://doi.org/10.1007/BF01758770.
  * Hamilton et al. [2017] W. L. Hamilton, R. Ying, and J. Leskovec.  Inductive Representation Learning on Large Graphs.  June 2017.  URL https://arxiv.org/abs/1706.02216v4.
  * Hu et al. [2021] W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec.  Open Graph Benchmark: Datasets for Machine Learning on Graphs.  _arXiv:2005.00687 [cs, stat]_ , Feb. 2021.  URL http://arxiv.org/abs/2005.00687.  arXiv: 2005.00687.
  * Javed et al. [2020] S. Javed, A. Mahmood, M. M. Fraz, N. A. Koohbanani, K. Benes, Y.-W. Tsang, K. Hewitt, D. Epstein, D. Snead, and N. Rajpoot.  Cellular community detection for tissue phenotyping in colorectal cancer histology images.  _Medical Image Analysis_ , 63:101696, July 2020.  ISSN 1361-8415.  doi: 10.1016/j.media.2020.101696.  URL https://www.sciencedirect.com/science/article/pii/S136184152030061X.
  * Kingma and Ba [2014] D. P. Kingma and J. Ba.  Adam: A Method for Stochastic Optimization.  Dec. 2014.  URL https://arxiv.org/abs/1412.6980v9.
  * Kipf and Welling [2017] T. N. Kipf and M. Welling.  Semi-Supervised Classification with Graph Convolutional Networks, Feb. 2017.  URL http://arxiv.org/abs/1609.02907.  arXiv:1609.02907 [cs, stat].
  * Lacoste et al. [2019] A. Lacoste, A. Luccioni, V. Schmidt, and T. Dandres.  Quantifying the Carbon Emissions of Machine Learning, Nov. 2019\.  URL http://arxiv.org/abs/1910.09700.  arXiv:1910.09700 [cs].
  * Lu et al. [2020] W. Lu, S. Graham, M. Bilal, N. Rajpoot, and F. Minhas.  Capturing Cellular Topology in Multi-Gigapixel Pathology Images.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 1049–1058, Seattle, WA, USA, June 2020. IEEE.  ISBN 978-1-72819-360-1.  doi: 10.1109/CVPRW50498.2020.00138.  URL https://ieeexplore.ieee.org/document/9150693/.
  * Lu et al. [2022] W. Lu, M. Toss, M. Dawood, E. Rakha, N. Rajpoot, and F. Minhas.  SlideGraph+: Whole slide image level graphs to predict HER2 status in breast cancer.  _Medical Image Analysis_ , 80:102486, Aug. 2022.  ISSN 1361-8415.  doi: 10.1016/j.media.2022.102486.  URL https://www.sciencedirect.com/science/article/pii/S1361841522001335.
  * Paszke et al. [2019] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala.  PyTorch: An Imperative Style, High-Performance Deep Learning Library, Dec. 2019.  URL http://arxiv.org/abs/1912.01703.  arXiv:1912.01703 [cs, stat].
  * Peter Bankhead [2017] Peter Bankhead.  QuPath: Open source software for digital pathology image analysis | Scientific Reports, 2017.  URL https://www.nature.com/articles/s41598-017-17204-5.
  * Shchur et al. [2019] O. Shchur, M. Mumme, A. Bojchevski, and S. Günnemann.  Pitfalls of Graph Neural Network Evaluation.  _arXiv:1811.05868 [cs, stat]_ , June 2019.  URL http://arxiv.org/abs/1811.05868.  arXiv: 1811.05868.
  * Sirinukunwattana et al. [2018] K. Sirinukunwattana, D. Snead, D. Epstein, Z. Aftab, I. Mujeeb, Y. W. Tsang, I. Cree, and N. Rajpoot.  Novel digital signatures of tissue phenotypes for predicting distant metastasis in colorectal cancer.  _Scientific Reports_ , 8(1):13692, Sept. 2018\.  ISSN 2045-2322.  doi: 10.1038/s41598-018-31799-3.  URL https://www.nature.com/articles/s41598-018-31799-3.
  * Szklarczyk et al. [2019] D. Szklarczyk, A. L. Gable, D. Lyon, A. Junge, S. Wyder, J. Huerta-Cepas, M. Simonovic, N. T. Doncheva, J. H. Morris, P. Bork, L. J. Jensen, and C. Mering.  STRING v11: protein–protein association networks with increased coverage, supporting functional discovery in genome-wide experimental datasets.  _Nucleic Acids Research_ , 47(D1):D607–D613, Jan. 2019.  ISSN 0305-1048.  doi: 10.1093/nar/gky1131.  URL https://doi.org/10.1093/nar/gky1131.
  * [25] C. Vanea, J. Džigurski, V. Rukins, O. Dodi, S. Siigur, L. Salumäe, K. Meir, W. T. Parks, D. Hochner-Celnikier, A. Fraser, H. Hochner, T. Laisk, L. M. Ernst, C. M. Lindgren, and C. Nellåker.  HAPPY: A deep learning pipeline for mapping cell-to-tissue graphs across placenta histology whole slide images.  _In Preparation_.
  * Veličković et al. [2018a] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio.  Graph Attention Networks, Feb. 2018a.  URL http://arxiv.org/abs/1710.10903.  arXiv:1710.10903 [cs, stat].
  * Veličković et al. [2018b] P. Veličković, W. Fedus, W. L. Hamilton, P. Liò, Y. Bengio, and R. D. Hjelm.  Deep Graph Infomax, Dec. 2018b.  URL http://arxiv.org/abs/1809.10341.  arXiv:1809.10341 [cs, math, stat].
  * Wu et al. [2019] F. Wu, T. Zhang, A. H. d. Souza Jr., C. Fifty, T. Yu, and K. Q. Weinberger.  Simplifying Graph Convolutional Networks, June 2019.  URL http://arxiv.org/abs/1902.07153.  arXiv:1902.07153 [cs, stat].
  * Yang et al. [2016] Z. Yang, W. W. Cohen, and R. Salakhutdinov.  Revisiting Semi-Supervised Learning with Graph Embeddings, May 2016.  URL http://arxiv.org/abs/1603.08861.  arXiv:1603.08861 [cs].
  * Yener [2017] B. Yener.  Cell-Graphs: Image-Driven Modeling of Structure-Function Relationship.  _Communications of the ACM_ , 60(1):74–84, 2017\.  URL https://cacm.acm.org/magazines/2017/1/211111-cell-graphs/fulltext.
  * Zeng et al. [2020] H. Zeng, H. Zhou, A. Srivastava, R. Kannan, and V. Prasanna.  GraphSAINT: Graph Sampling Based Inductive Learning Method.  In _arXiv:1907.04931 [cs, stat]_ , Feb. 2020.  URL http://arxiv.org/abs/1907.04931.  arXiv: 1907.04931.
  * Zeng et al. [2022] H. Zeng, M. Zhang, Y. Xia, A. Srivastava, A. Malevich, R. Kannan, V. Prasanna, L. Jin, and R. Chen.  Decoupling the Depth and Scope of Graph Neural Networks, Jan. 2022.  URL http://arxiv.org/abs/2201.07858.  arXiv:2201.07858 [cs].
  * Zhou et al. [2019] Y. Zhou, S. Graham, N. A. Koohbanani, M. Shaban, P.-A. Heng, and N. Rajpoot.  CGC-Net: Cell Graph Convolutional Network for Grading of Colorectal Cancer Histology Images.  _arXiv:1909.01068 [cs, eess]_ , Sept. 2019.  URL http://arxiv.org/abs/1909.01068.  arXiv: 1909.01068.
  * Zitnik and Leskovec [2017] M. Zitnik and J. Leskovec.  Predicting multicellular function through multi-layer tissue networks.  _Bioinformatics_ , 33(14):i190–i198, July 2017\.  ISSN 1367-4803.  doi: 10.1093/bioinformatics/btx252.  URL https://doi.org/10.1093/bioinformatics/btx252.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes] We claim to introduce a new dataset
benchmark in histology cell graphs and show how existing GNN models perform on
this data.

    2. (b)

Did you describe the limitations of your work? [Yes] See Section 5.

    3. (c)

Did you discuss any potential negative societal impacts of your work? [Yes]
See Appendix K.

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [N/A]

    2. (b)

Did you include complete proofs of all theoretical results? [N/A]

  3. 3.

If you ran experiments…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]
See Section 3 and at (https://github.com/Nellaker-group/placenta).

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes] See Section 2, 3 and Appendix C, E.

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [Yes] See Section 4. Experiments were run with 5
random seeds.

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 3
and Appendix L.

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes] We
existing GNN models and code from PyTorch Geometric and we cite the creators

    2. (b)

Did you mention the license of the assets? [Yes] See Section 3.

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [Yes] See Section 2 and at (https://github.com/Nellaker-group/placenta).

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [Yes] See Section 2. Data was obtained without consent
with approval from local ethics committees for research use.

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [Yes] See Section 2 and 5.

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]

##  Appendix A Placenta Cell and Tissue Types

11 placenta cells used in node features:

  * •

SYN: Syncytiotrophoblast

  * •

CYT: Cytotrophoblast

  * •

KNT: Syncytial Knot

  * •

EVT: Extravillus Trophoblast

  * •

FIB: Fibroblast

  * •

HOF: Hofbauer Cell

  * •

VEN: Vascular Endothelial

  * •

VMY: Vascular Myocyte

  * •

MES: Mesenchymal Cell

  * •

MAT: Maternal Decidua

  * •

WBC: Leukocyte

9 placenta tissue types used as node labels:

  * •

Sprout: Villus Sprout

  * •

TVilli: Terminal Villi

  * •

MIVilli: Mature Intermediate Villi

  * •

SVilli: Stem Villi

  * •

AVilli: Anchoring Villi

  * •

Chorion: Chorionic Plate

  * •

Maternal: Basal Plate and Septum

  * •

Fibrin: Fibrin

  * •

Avacular: Avascular Villi

##  Appendix B Cell Types in Node Features and Ground Truth Tissue
Distributions

Figure 3: Cell type proportions in node features (top), tissue label
proportions in all labelled data (middle), and distribution of tissue classes
within dataset splits (bottom).

##  Appendix C Data Splits and the Effect of Random Node Splits

Figure 4: Validation region (left) and test region (right). Table 3: Distribution of class labels across dataset splits. Class Label | Training | Validation | Test
---|---|---|---
Sprout | 15,545 (50%) | 9,066 (29%) | 6,441 (21%)
TVilli | 159,464 (50%) | 84,052 (27%) | 72,402 (23%)
MIVilli | 80,954 (51%) | 39,296 (25%) | 37,332 (24%)
SVilli | 132,138 (55%) | 58,658 (25%) | 48,568 (20%)
AVilli | 5,769 (64%) | 1,634 (18%) | 1,654 (18%)
Chorion | 46,584 (84%) | 4,942 (9%) | 3,937 (7%)
Maternal | 13,329 (75%) | 1,029 (6%) | 3,359 (19%)
Fibrin | 14,195 (54%) | 7,047 (27%) | 5,010 (19%)
Avacular | 891 (49%) | 526 (29%) | 392 (22%)
Figure 5: Accuracy curves during training using random node splits of
70%/15%/15%. Validation performance is identical to training performance using
the same inference sampling method, showing information leakage across splits.

##  Appendix D Edge Construction and Graph Differences

Table 4: Graph Characteristics of KNN, Delaunay, and intersection graphs. | Nodes | Edges | Labels | Homophily | Degree
---|---|---|---|---|---
Intersection Graph 1 | 1,159,186 | 2,603,797 | 703,165 | 0.9868 | 5.4925
Intersection Graph 2 | 1,236,561 | 2,882,292 | 96,580 | 0.9984 | 5.6618
KNN Graph 1 | 1,159,186 | 8,398,932 | 703,165 | 0.9854 | 8.2455
KNN Graph 2 | 1,236,561 | 8,870,066 | 96,580 | 0.7286 | 8.1732
Delaunay Graph 1 | 1,159,186 | 3,477,526 | 703,165 | 0.9522 | 6.9999
Delaunay Graph 2 | 1,236,561 | 3,709,652 | 96,580 | 0.7217 | 6.9999
Figure 6: From top left to bottom right, the raw histology, histology with cell overlay, Delaunay Triangulation edges, KNN edges, intersection edges, and ground truth tissue labels. Table 5: ClusterGCN performance with a KNN graph where k=5 and a Delaunay graph. | Accuracy | Top 2 Accuracy | ROC AUC
---|---|---|---
KNN | 0.654 | 0.894 | 0.903
Delaunay | 0.676 | 0.898 | 0.916

##  Appendix E Hyperparameter Details for Each Model Architecture

Weighted cross entropy weights: [1, 0.85, 0.9, 10.5, 0.8, 1.3, 5.6, 3, 77]

Additional hyperparameter details and any changes from the default
hyperparameters reported in Section 3 are reported below.

### MLP

16 Linear layers, 51,200 batch size.

### GraphSAGE-mean

12 SAGEConv layers due to GPU out of memory error with larger models, 32,000
batch size, 10 neighbours per hop, mean aggregation.

### ClusterGCN

16 SAGEConv layers, 200 batch size, and 400 max subgraph size.

### GraphSAINT-rw

16 GraphConv layers, in-node normalisation, 32,000 batch size, 30 steps, 500
sample coverage, random walk sampler.

### SIGN

16 Linear layers, 51,200 batch size, SIGN transform with 16 hops.

### ShaDow

8 SAGEConv layers due to performance drop with larger models, no batch norm
layers for the same reason, 4000 batch size, 5 neighbours per hop, hop depth
of 6. This smaller receptive field is due to a RAM memory error with a higher
depth or more neighbours.

### ClusterGAT

2 GATConv layers due to GPU out of memory error and performance drop with
larger models, 400 batch size, 400 max subgraph size, 4 attention heads with
0.25 dropout.

### ClusterGATv2

2 GATv2Conv layers due to GPU out of memory error and performance drop with
larger models, 200 batch size, 400 max subgraph size, 4 attention heads with
0.25 dropout.

##  Appendix F The Effect of Class imbalance on Performance

Figure 7: Confusion matrix with values proportional to class imbalance (left),
confusion matrix per number of nodes (middle), and precision-recall curves
with corresponding AUC-PR values (right). The dataset is highly imbalanced and
models perform worse on many minority classes.

##  Appendix G The Effect of Model Size on Overfitting to Training Data

Figure 8: ClusterGCN with 128 hidden units (left), 256 hidden units (middle),
and 512 hidden units (right). Too few hidden units and the model does not
learn the training data, too many and the model overfits.

##  Appendix H The Effect of Receptive Field Sizes

Table 6: ClusterGCN performance with varied receptive fields. Receptive fields are varied by changing cluster subgraph sizes and the number of model layers. | Accuracy | Top 2 Accuracy | ROC AUC
---|---|---|---
800 Subgraph 32 Layers | 0.647 | 0.874 | 0.893
400 Subgraph 16 Layers | 0.642 | 0.883 | 0.882
200 Subgraph 8 Layers | 0.623 | 0.871 | 0.876
100 Subgraph 4 Layers | 0.594 | 0.849 | 0.856
Figure 9: Reducing receptive field sizes (left to right) leads to confused
predictions inside large structures. The yellow tissue structure (SVilli) in
these images is incorrectly labelled as the green tissue structure (Chorion).

##  Appendix I The Effect of Cellular Node Features

Table 7: ClusterGCN performance when trained and evaluated with one hot encoded cell types. Performance is marginally worse than training with 64-dim feature vectors. | Accuracy | Top 2 Accuracy | ROC AUC
---|---|---|---
One Hot Cell Types | 0.628 | 0.871 | 0.833

##  Appendix J Visualisation of Model Predictions

Figure 10: From top to bottom, MLP, GraphSAGE-mean, ClusterGCN, and
GraphSAINT-rw tissue predictions across both WSIs.

Figure 11: From top to bottom, SIGN, ShaDow, ClusterGAT, and ClusterGATv2
tissue predictions across both WSIs.

##  Appendix K Potential Negative Societal Impacts

The development of similar models in the context of histological analyses
could, if in future fully realised and implemented across healthcare service
settings, negatively impact the work of pathologists who do not have access to
the models: “AI won’t replace radiologists, but radiologists who use AI will
replace radiologists who don’t,” (Curtis Langlotz, 2019, Nature, Innovations
In). As with the uptake of any new technology, there would be a ‘growing
pains’ period which could increase the time between tissue sample collection
and diagnosis. Slides will have to be digitised and model inference will have
to be integrated, causing a shift in existing clinical and histological
preparation workflows. We must also consider that any models developed for
histology analysis could cause serious patient harm if, in the extreme case,
they are incorrectly validated and quality controlled, or they have an
undetected bias which influences diagnoses. If we assume models perform as
well or better than clinicians, but their outputs are not clearly expressed in
clinical language and/or there is a lack of trust in these automated,
uninterpretable methods, then there may be resulting uncertainty in diagnosis
and consequently higher referral rates. Conversely, if automated histology
analysis becomes popular, for organs unlike the placenta which require
biopsies to be sampled, the rate of biopsies could harmfully increase.
Finally, the publication of this dataset could facilitate the development of
new GNN architectures that themselves could confer negative impacts in other
domains.

##  Appendix L CO2 Emission Related to Experiments

Experiments were conducted using a private infrastructure, which has a carbon
efficiency of 0.432 kgCO2eq/kWh. An approximate cumulative of 200 hours of
computation was performed on hardware of type RTX 8000 (TDP of 260W).

Total emissions are estimated to be 22.46 kgCO2eq of which 0 percents were
directly offset.

Estimations were conducted using the MachineLearning Impact calculator
presented in [17].


# Selecting applicants based on multiple ratings:
Using binary classification framework as an alternative to inter-rater
reliability

František Bartoš1,2, Patrícia Martinková1
1 Institute of Computer Science of the Czech Academy of Sciences, Prague,
Czech Republic
2 University of Amsterdam, Amsterdam, Netherlands

###### Abstract

Inter-rater reliability (IRR) has been the prevalent quality and precision
measure in ratings from multiple raters. However, applicant selection
procedures based on ratings from multiple raters usually result in a binary
outcome. This final outcome is not considered in IRR, which instead focuses on
the ratings of the individual subjects or objects. In this work, we outline
how to transform the selection procedures into a binary classification
framework and develop a quantile approximation which connects a measurement
model for the ratings with the binary classification framework. The quantile
approximation allows us to estimate the probability of correctly selecting the
best applicants and assess error probabilities when evaluating the quality of
selection procedures using ratings from multiple raters. We draw connections
between the inter-rater reliability and the binary classification metrics,
showing that binary classification metrics depend solely on the IRR
coefficient and proportion of selected applicants. We assess the performance
of the quantile approximation in a simulation study and apply it in an example
comparing the reliability of multiple grant peer review selection procedures.

Keywords: Error rate; mixed-effect models; rating, type I error, type II
error.

##  1 Introduction

Applicant selection procedures typically use ratings to assess applicant
quality, but they generally result in a binary outcome: an applicant is either
hired or refused, a grant proposal is either funded or dismissed, and an
article is either accepted or rejected [[, e.g.,]]cicchetti1991reliability.
However, the most commonly used metric for evaluating the quality of selection
procedures, inter-rater reliability (IRR), does not focus on the outcome of
selection procedures: whether the best applicants, grants proposals, or
articles (further referred to as applicants) were selected.111This is a
different goal than measuring _agreement_ of multiple raters performing binary
decisions using e.g., Cohen’s Kappa [7], that, again, measures agreement and
not whether the best applicants were selected. Instead, IRR focuses on scores
or ratings of the applicants which form an important but not the final step of
the selection procedures.

This is, of course, not a novel observation and the disconnect between
evaluating raters’ agreement and whether the best applicants were selected was
noted earlier [[, e.g.,]]kraemer1991we, nelson1991process, mayo2006peering. In
this work, we try to seal this gap by developing a quantile approximation that
links the ratings’ measurement model to the expected classification
probabilities of selecting the best applicants. Subsequently, the selection
procedures can be characterized as binary classification and evaluated via
well-known, easy-to-interpret, and tangible metrics such as sensitivity or
false positive/negative rates. Furthermore, the binary classification
framework allows researchers and stakeholders to evaluate and improve
selection procedures while incorporating the costs of incorrect decisions,
increasing the number of raters or modifying the rating procedure.

While our approach is related to univariate classification under measurement
error, this specific use case is different. In comparison to typical
classification tasks which aim to separate subjects into different categories
[9], we assume the existence of a single continuous latent trait measured by
the ratings. Then, we select a proportion of the best applicants defined by
the latent trait and evaluate the (miss)classification probabilities due to
the measurement error contained in the observed ratings.

The paper proceeds as follows: First, we focus on the proposed methods: we
define a minimal measurement model for ratings in the selection procedures, we
characterize the selection procedures in terms of their outcome – the binary
classification, we propose a quantile approximation connecting the measurement
model with the binary classification, and we show the relationship between IRR
and the binary classification metrics. Second, we evaluate the quality of the
quantile approximation in a simulation study, comparing the empirical true
positive rate to the estimated true positive rate. Third, we demonstrate the
outlined methodology on an example of multiple grant peer review, where we
compare multiple selection procedures in terms of their false positive rate
and show how how it changes with increasing the number of raters and IRR. We
close with a discussion of limitations and extensions.

##  2 Methods

###  2.1 Measurement Model

We consider a minimal selection procedure that evaluates $N$ applicants with
the goal of estimating their latent ability $\gamma$. In the selection
process, each applicant is rated $J$ times, resulting in a vector of observed
scores $y_{ij}$ for $i=1,\dots N$ and $j=1,\dots,J$. The resulting measurement
model can be written as

$y_{ij}=\gamma_{i}+\epsilon_{ij},$ (1)

with the usual assumptions that (a) the measurement error $\epsilon_{ij}$ of
each applicant is normally distributed with mean zero and variance
$\sigma^{2}_{\epsilon}$ and (b) the latent ability $\gamma$ is normally
distributed with mean $\mu$ and variance $\sigma^{2}_{\gamma}$.

The quality of the selection procedure is usually measured as inter-rater
reliability (sometimes also referred to as single-rater IRR), which is the
inter-class correlation coefficient [[, ICC(1,1)]]mcgraw1996forming,
shrout1979intraclass

$\text{IRR}_{1}=\frac{\sigma_{\gamma}^{2}}{\sigma_{\gamma}^{2}+\sigma_{\epsilon}^{2}}.$

When the final decisions are based on the average of $J$ raters, the multiple-
rater IRR can be calculated using the Spearmann-Brown formula as

$\text{IRR}_{J}=\frac{\sigma_{\gamma}^{2}}{\sigma_{\gamma}^{2}+\frac{\sigma_{\epsilon}^{2}}{J}}=\frac{J\,\text{IRR}_{1}}{J\,\text{IRR}_{1}+1-\text{IRR}_{1}}.$

###  2.2 Binary Classification

The endpoint of the selection procedure is, however, not estimating the latent
abilities $\gamma_{i}$ per se but selecting $k$ applicants with the highest
latent ability $\gamma$. Consequently, we can divide the applicants into two
groups: high ability applicants $A$ that consist of $k$ applicants with the
highest latent ability $\gamma$ and “not high ability applicants” $\neg A$
that consist of the remaining $N-k$ applicants. Denoting the selected
applicants by $S$ and the remaining applicants by $\neg S$, we can
characterize the selection procedure by probabilities of the different types
of (miss)classification (Table 1).

| Selected ($S$) | Not Selected ($\neg S$) |
---|---|---|---
High Ability Applicants ($A$) | $P(S\cap A)$ | $P(\neg S\cap A)$ | $P(A)$
Not High Ability Applicants ($\neg A$) | $P(S\cap\neg A)$ | $P(\neg S\cap\neg A)$ | $P(\neg A)$
| $P(S)$ | $P(\neg S)$ | $1$
Table 1: Overview of the selection procedure as a binary classification.

Since the unconditional probability of an applicant being selected
corresponds, by definition, to the unconditional probability of an applicant
having high ability, $P(S)=P(A)=\nicefrac{{k}}{{N}}$, the whole classification
process is determined by the probability of correctly selecting the high
ability applicants $P(S\cap A)$, i.e., by the probability of true positives
classifications (Table 2).

| Selected ($S$) | Not Selected ($\neg S$) |
---|---|---|---
High Ability Applicants ($A$) | $P(S\cap A)$ | $\nicefrac{{k}}{{N}}-P(S\cap A)$ | $\nicefrac{{k}}{{N}}$
Not High Ability Applicants ($\neg A$) | $\nicefrac{{k}}{{N}}-P(S\cap A)$ | $P(S\cap A)+\nicefrac{{(N-2k)}}{{N}}$ | $1-\nicefrac{{k}}{{N}}$
| $\nicefrac{{k}}{{N}}$ | $1-\nicefrac{{k}}{{N}}$ | $1$
Table 2: Simplification of the binary classification in selection procedures.

Subsequently, if we knew the probability of true positive classification, we
could use standard metrics for evaluating binary classification [[, e.g.,]pp.
14-33]pepe2003statistical. Since the off-diagonal probabilities of the
classification outcomes are equal (Table 2), many commonly specified metrics
become equal or simplify to a constant. For example, true positive rate (TPR,
aka sensitivity) and positive predictive value (PPV, aka precision) are equal,

$\text{TPR}=\text{PPV}=\frac{P(S\cap A)}{\nicefrac{{k}}{{N}}},$ (2)

and true negative rate (TNR, aka specificity) is equal to
$\nicefrac{{1}}{{2}}$. We can also compute the false positive rate (FPR,
corresponding to type I error rate),

$\text{FPR}=\frac{\nicefrac{{k}}{{N}}-P(S\cap A)}{\nicefrac{{k}}{{N}}},$ (3)

or false negative rate (FNR, corresponding to type II error rate),

$\text{FNR}=\frac{\nicefrac{{k}}{{N}}-P(S\cap A)}{\nicefrac{{(N-k)}}{{N}}}.$
(4)

Alternatively, the classification probabilities can be used with a utility
function specifying the cost of each type of error [[, e.g.,]]metz1978basic.

###  2.3 Quantile Approximation

The probability of true positive classification is, unfortunately, not
directly estimable from the observed scores $y_{ij}$, unless we know the true
latent ability $\gamma_{i}$ of each applicant. To deal with this hindrance, we
propose a quantile approximation that allows us to estimate the true positive
classification probability indirectly. Specifically, we leverage
distributional assumptions of the measurement model (Equation 1), our ability
to estimate the populational parameters of the measurement model, and one
peculiarity of the selection procedure: the fact that we set the unconditional
probability of the applicant with high ability, $P(A)$ and the unconditional
probability of the applicant selected, $P(S)$, when designing the selection
procedure.

In selection procedures the high ability applicants are defined by their
ranking on the latent ability $\gamma$. However, we only measure the observed
scores $y_{ij}$ when performing the selection. According to the measurement
model, the observed scores $y_{ij}$ are an imprecise reflection of the latent
abilities $\gamma_{i}$ which is disturbed by the measurement error
$\sqrt{\sigma_{\epsilon}}$. The $J$ observed scores of each applicant are
usually aggregated into a mean score $\bar{y}_{i}$ with the resulting
measurement error $\sigma_{\epsilon}/\sqrt{J}$. Importantly, the latent
abilities $\gamma$ and the random variable of the aggregated scores $\bar{Y}$
are jointly distributed with bivariate normal density:

$\begin{bmatrix}\gamma_{i}\\\ \bar{y}_{i}\end{bmatrix}\sim
N\left(\begin{bmatrix}\mu\\\
\mu\end{bmatrix},\begin{bmatrix}\sigma_{\gamma}^{2}&\sigma_{\gamma}^{2}\\\
\sigma_{\gamma}^{2}&\sigma_{\gamma}^{2}+\sigma_{\epsilon}^{2}/J\end{bmatrix}\right),$
(5)

[[, e.g.,]pp. 258-259]searle2006variance.

Since the goal of selection procedures is selecting the high ability $k$
applicants and since the desirability of applicants is defined by their latent
ability $\gamma$, the high ability applicants are defined as those with the
highest $k$ latent ability scores $\gamma$. Then, we can think about a cut-
score on the latent ability $\gamma_{c}$ that separates the high ability
applicants from the remaining applicants (with the high ability applicants
satisfying the condition $\gamma_{i}>\gamma_{c}$). Similarly, since the
selection procedure is performed with the observed aggregated scores
$\bar{y}_{i}$ and since we select the high ability $k$ applicants according to
their observed aggregated scores $\bar{y}_{i}$, the selected applicants are
defined as those with the highest $k$ observed scores $\bar{y}_{i}$. Then,
again, we can think about a cut-score on the observed aggregated scores
$\bar{y}_{c}$ that separates the high ability applicants from the remaining
applicants (with the high ability applicants satisfying the condition
$\bar{y}_{i}>\bar{y}_{c}$).

In any particular data set, the cut-scores $\gamma_{c}$ and $\bar{y}_{c}$ are
dependent on the actual latent abilities $\gamma_{i}$ and observed aggregated
scores $\bar{y}_{i}$ of applicants participating in the given selection
procedure. However, under the assumption of the measurement model, we can
approximate the cut-scores with marginal quantile functions of the joint
distribution (Equation 2.4), yielding the quantile approximated cut-scores

$\displaystyle\tilde{\gamma}_{c}$
$\displaystyle=\mu+\sigma_{\gamma}\Phi^{-1}\left(\frac{N-k}{N}\right),$ (6)
$\displaystyle\tilde{\bar{y}}_{c}$
$\displaystyle=\mu+\sqrt{\sigma_{\gamma}^{2}+\sigma_{\epsilon}^{2}/J}\,\Phi^{-1}\left(\frac{N-k}{N}\right),$

with $\Phi$ corresponding to a cumulative distribution function of a standard
normal distribution.

Subsequently, the true positives classification probability in Table 2,
defining the binary classification metrics, can be approximated using the
latent ability $\gamma$, its imperfect measurement via the random variable of
aggregated observed scores $\bar{Y}$, and the quantile approximated cut-scores
$\tilde{\gamma}_{c}$ and $\tilde{\bar{y}}_{c}$,

$P(S\cap A)\approx P(\gamma>\tilde{\gamma}_{c},Y>\tilde{\bar{y}}_{c}).$

In other words, the probability of an applicant being amongst both the high
ability and selected applicants can be approximated by the probability of both
having latent ability higher than the approximated cut-score on the latent
ability and having the observed aggregated score higher than the approximated
aggregated cut-score.

Finally, the true positives classification probability can be approximated
solely with the populational parameters and integrating over the joint
bivariate normal density

$P(\gamma>\tilde{\gamma}_{c},Y>\tilde{\bar{y}}_{c})=\mathop{{{\int}}}_{\tilde{\gamma}_{c}}^{\infty}\mathop{{{\int}}}_{\tilde{\bar{y}}_{c}}^{\infty}N\left(\begin{bmatrix}\gamma\\\
\bar{Y}\end{bmatrix}\mathop{{{{\mid}}}}\begin{bmatrix}\mu\\\
\mu\end{bmatrix},\begin{bmatrix}\sigma_{\gamma}^{2}&\sigma_{\gamma}^{2}\\\
\sigma_{\gamma}^{2}&\sigma_{\gamma}^{2}+\sigma_{\epsilon}^{2}/J\end{bmatrix}\right)\,d\bar{Y}\,d\gamma.$
(7)

###  2.4 Relationship between IRR and Binary Classification

We can further utilize the quantile approximation to draw connections between
IRR and the true positives classification probability of the binary
classification.

We subtract the grand mean $\mu$ from both the latent abilities $\gamma$ and
the observed aggregated scores $\bar{Y}$ and standardize the random variables
by total variance of the observed aggregated scores
$\sqrt{\sigma_{\gamma}^{2}+\sigma_{\epsilon}^{2}/J}$, transforming the latent
abilities into
$\zeta=(\gamma-\mu)/\sqrt{\sigma_{\gamma}^{2}+\sigma_{\epsilon}^{2}/J}$ and
the observed aggregated scores into
$\bar{Z}=(\bar{Y}-\mu)/\sqrt{\sigma_{\gamma}^{2}+\sigma_{\epsilon}^{2}/J}$.
Subsequently, the bivariate normal density of the transformed abilities and
the random variable of transformed observed aggregated scores simplifies to

$\begin{bmatrix}\zeta_{i}\\\ \bar{z}_{i}\end{bmatrix}\sim
N\left(\begin{bmatrix}0\\\
0\end{bmatrix},\begin{bmatrix}\text{IRR}_{J}&\text{IRR}_{J}\\\
\text{IRR}_{J}&1\end{bmatrix}\right),$

and yields the quantile approximated cut-scores of the transformed abilities
and observed aggregated measures

$\displaystyle\tilde{\zeta}_{c}$
$\displaystyle=\text{IRR}_{J}\Phi^{-1}\left(\frac{N-k}{N}\right),$
$\displaystyle\tilde{\bar{z}}_{c}$
$\displaystyle=\Phi^{-1}\left(\frac{N-k}{N}\right).$

Therefore, the true positives classification probability can be directly
estimable with only $\text{IRR}_{J}$ and the proportion of selected applicants
$k$,

$P(\zeta>\tilde{\zeta}_{c},Z>\tilde{\bar{z}}_{c})=\mathop{{{\int}}}_{\tilde{\zeta}_{c}}^{\infty}\mathop{{{\int}}}_{\tilde{\bar{z}}_{c}}^{\infty}N\left(\begin{bmatrix}\zeta\\\
\bar{Z}\end{bmatrix}\mathop{{{{\mid}}}}\begin{bmatrix}0\\\
0\end{bmatrix},\begin{bmatrix}\text{IRR}_{J}&\text{IRR}_{J}\\\
\text{IRR}_{J}&1\end{bmatrix}\right)\,d\bar{Z}\,d\zeta.$

This allows us to retrospectively evaluate selection procedures in the binary
classification framework without access to the primary data.

##  3 Simulation

We conducted a simulation study to assess the quality of the quantile
approximation to the true positives classification probability using binary
classification. Specifically, we compared the $P(S\cap A)$ based on the
empirical (true) classification labels comparing the simulated abilities
$\gamma_{i}$ vs the aggregated ability estimates $\bar{y}_{i}$ with the true
positives classification probability obtained by quantile approximation
$P(\gamma>\tilde{\gamma}_{c},Y>\tilde{\bar{y}}_{c})$ based on
$\hat{\sigma}_{\gamma}^{2}$ and $\hat{\sigma}_{\epsilon}^{2}$ estimates from a
linear mixed model.

We simulated data from the measurement model defined by Equation (1) with
settings inspired by [16]. We manipulated the inter-class correlation
coefficient, $\text{IRR}_{1}=\\{0.15,0.30,0.45\\}$ (fixing the overall
variance $\sigma^{2}$ to 1),222Whereas the measurement model is defined in
terms of the grand mean $\mu$ and variances $\sigma_{\gamma}^{2}$ and
$\sigma_{\epsilon}^{2}$, Equation (2.4) shows that IRR is the only relevant
quantity for the quantile approximation number of applicants,
$N=\\{100,300,1000\\}$, and the number of ratings, $J=\\{3,5,10\\}$. We
replicated each simulation condition 1000 times and estimated the variance
parameters $\sigma_{\gamma}^{2}$ and $\sigma_{\epsilon}^{2}$ using linear
models implemented in the lme4 R package [3]. In the remainder of the section,
we only discuss results based on $N=100$ applicants but see (Appendix A) for
similar results with $N=300$ and $N=1000$.

Figure 1: Mean empirical true positive classification probabilities for 100
applicants (full red line) vs the mean quantile approximated true positive
classification probabilities (dashed blue line) across all possible
proportions of selected applicants ($x$-axis). The diagonal shows the maximum
true positive classification probability for a given proportion of selected
applicants. Results are presented for different inter-rater reliability
coefficients (rows) and the number of ratings (columns).

Figure 1 visualizes the mean empirical true positive classification
probabilities $P(S\cap A)$ (full red line) vs the mean quantile approximated
true positive classification probabilities
$P(\tilde{\gamma}>G_{k},\gamma>G_{k})$ (dashed blue line) across all possible
proportions of selected applicants ($\nicefrac{{k}}{{N}}$, $x$-axis). The
results are presented for different inter-rater reliability coefficients
($\text{IRR}_{1}$, rows), the number of ratings ($J$, columns), and 100
applicants. The $x$-axis corresponds to the proportion of selected applicants
($\nicefrac{{k}}{{N}}$). The diagonal red dotted line corresponds to the
maximum probability of the true positive classification probability at a given
proportion of selected applicants. The mean of the quantile approximation
closely resembles the mean of the true positive classification probabilities –
making the two lines essentially indistinguishable – for all but the smallest
number of raters and the lowest $\text{IRR}_{1}$ (which resulted in a slight
positive bias for a low proportion of selected applicants and a slight
negative bias for a high proportion of selected applicants). However, the
performance of the approximation improved with increasing number of ratings
and inter-rater reliability as well as with the increasing number of
applicants.

Figure 2: Root mean square error of quantile approximation to the true
positive classification probabilities for 100 applicants across all possible
proportions of selected applicants ($x$-axis). Results are presented for
different inter-rater reliability coefficients (rows) and the number of
ratings (columns).

Importantly, the quantile approximation estimates the expected classification
probabilities for a selection procedure with given populational
characteristics. In other words, the quantile approximation might provide
noisy estimates of the true values for any particular selection procedure,
especially when the classification probabilities dependen on a small number of
selected applicants. Figure 2 visualizes the root mean square error (RMSE) of
the quantile approximation to the true positive classification probabilities
across all possible proportions of selected applicants ($\nicefrac{{k}}{{N}}$,
$x$-axis). The results are presented for different inter-rater reliability
coefficients ($\text{IRR}_{1}$, rows), number of ratings ($J$, columns), and
100 applicants. The $x$-axis corresponds to the proportion of selected
applicants ($\nicefrac{{k}}{{N}}$). Note that the true positive classification
probability is bounded at the endpoints – it is zero if no candidate is
selected and it is one if all candidates are selected. As such, the RMSE of
the true positive classification probability is necessarily higher above the
middle proportion of selected applicants since it can attain the widest range
of values. We, again, see a high RMSE for a condition with a low number of
ratings and the inter-rater reliability ($J=3$ and $\text{IRR}_{1}$) with an
odd shape at the maximum proportion of selected applicants (due to the bias).
However, the RMSE of the approximation, again, improves with the increasing
number of ratings and inter-rater reliability as well as with the increasing
number of applicants.

###  3.1 Confidence and Prediction Intervals

Dependency on any particular selection procedure can be further illustrated by
visualizing the empirical vs quantile approximated results for a single trial.
For example, consider the false positive rate, the probability an applicant is
selected although they are not part of the high ability group (a
transformation of the previously summarized true positive classification
probability, Equation 3). Figure 3 compares the quantile approximated false
positive rate (full blue line) to a random simulation’s empirical false
positive rate (full black line) with $J=5$, $\text{IRR}_{1}=0.30$, and
$N=100$.

Figure 3: Example of quantile approximated estimate of the false positive rate
(full blue line), the 95% confidence interval (blue dashed line), 95%
prediction interval (blue dotted line), and the empirical false positive rate
(full black line) from a random simulation with 5 ratings,
$\text{IRR}_{1}=0.30$, and 100 applicants.

The empirical false positive rate wildly oscillates between 0 and 1,
especially when considering the selection of only a small number of applicants
($x$-axis). These wild oscillations, accompanied by the extremely wide 95%
prediction interval (blue dotted line; based on quantile function of binomial
distribution), are an inherent property of summarizing the proportion of
successes of a binomial outcome with a small number of trials. In contrast,
95% confidence intervals of the false positive rate (blue dashed lines) which
quantifies the uncertainty of the false positive rate _estimate_ (based on
non-parametric bootstrap) is wider in the lower proportion of the selected
candidates and shorter around the endpoints. This is a result of the false
positive rate estimate being bounded at the endpoints, with higher uncertainty
in the lower proportion of selected applications – as it is a transformation
of the true positive classification probability.

##  4 Example: Grant Peer Reviews

We illustrate the methodology by estimating binary classification metrics for
several grant peer reviews. See Table 3 for characteristics of the grant peer
reviews as summarized in Table 1 of [11] and extended by the results reported
therein.333We focus only on studies that estimated IRR using all applicants to
prevent bias from the restricted range [11]. It is important to note that the
reported values are averages, often aggregated across disciplines and years
(sometimes accompanied by a range of values). The results should, therefore,
be taken as an illustration showcasing the possible interpretation and
inferences rather than an evaluation of the funding agencies’ grant review
process.

Study | Proposals | $\text{IRR}_{1}$ | $J$ | Selected
---|---|---|---|---
[6] | 150 NSF & COSPUP (1985) | $0.18-0.37$ |  $4.24$ & $4.04$ | $51\%^{1}$
[21] | 8329 FWF (1999-04) | $0.26$ | $2.82$ | $32\%-53\%$
[5] | 260 AIBS (2009-11) | $0.14-0.41$ | $2$ | $5\%-11\%$
[11] | 72 AIBS (2014-17) | $0.37$ | $3$ | $34\%-38\%$
[11] | 2076 NIH (2014-16) | $0.34$ | $2.79$ | $18\%^{2}$

Table 3: Overview of the grant peer reviews

  * •

1 Based on Table 46 from [8, p. 140].

  * •

2 Based on [10].

  * •

COSPUP = Committee on Science and Public Policy of the National Academy of
Sciences, NSF = National Science Foundation, FWF = Austrian Science Fund, AIBS
= American Institute of Biological Sciences, NIH = National Institutes of
Health.

Figure 4 visualizes the false positive rate estimates for each grant peer
review procedure (different colors) based on quantile approximation. Thin full
lines visualize the computed false positive rate across the range of possible
proportions of selected applicants, thick full lines (and points) highlight
the ranges (and values) of the actual proportion of selected applicants. We
see the estimated false positive rate ranges from $0.75$ for the lowest
proportion of selected applicants ($5\%$), lowest inter-rater reliability
($0.14$), and two rates in the AIBS 2009-11 grant proposals data to $18\%$ for
the highest proportion of selected applicants ($0.51$), high inter-rater
reliability ($0.37$) and more than four raters in the NSF & COSPUP (1985)
grant proposals data.

Figure 4: Estimates of the false positive rate for grant peer review
procedures based on quantile approximation for published estimates of IRR and
number of raters $J$ of individual studies (different colors). Thin full lines
visualize the estimated false positive rate across the range of possible
proportions of selected applicants, and thick full lines (and points)
highlight the ranges (and values) of the actual proportion of selected
applicants. For NSF & COSPUP and AIBS, the lower $\text{IRR}_{J}$ estimate,
resulting in a higher false positive rate, is shown as dotted lines / empty
circles, whereas the higher $\text{IRR}_{J}$ estimate is shown as full lines /
full circles.

It is important to highlight that most of the differences in false positive
rate can be ascribed to the difference in the proportion of selected
applicants. Although the false positive rates differ by 58 percentage points
across the example data sets (37 percentage points excluding the lower IRR
estimates of NSF & COSPUP and AIBS), comparing the grant proposal selection
procedures at an equal proportion of selected applicants (across the
$5\%-53\%$ range of selected applicants) reduces the maximum difference to 30
percentage points (15 percentage points excluding the lower IRR estimates of
NSF & COSPUP and AIBS; both at $5\%$ of selected applicants).

###  4.1 In Detail Assessment of NIH (2014-16)

We can take a closer look at the NIH data set of [11] where the reported
$\text{IRR}_{1}=0.34$ was accompanied by 95% CI $[0.31,0.37]$. The
corresponding estimate of the false positive rate is $0.397$, and as in the
simulation study, we can transform the 95% CI of $\text{IRR}_{1}$ to obtain
the 95% CI interval $[0.380,0.416]$ and the 95% prediction interval
$[0.331,0.465]$ for the false positive rate. We can also compute the false
negative rate estimate (i.e., type II error rate, Equation 4) of $0.087$, 95%
CI $[0.083,0.091]$.

Figure 5: Change in false positive rate when increasing increasing the number
of raters for different levels of $\text{IRR}_{1}$ (left) or increasing the
$\text{IRR}_{1}$ for different number of raters (right) for a selection
procedure based on NIH (2014-16) grant peer review. The original trial
selected $18$% out of $2076$ applicants with $J=2.79$ raters (black line in
the right panel) and $\text{IRR}_{1}=0.34$ (black line in the left panel).
Join increase of $\text{IRR}_{1}$ and the number of raters is depicted in
orange and red color.

We might consider redesigning the selection procedure to achieve lower false
positive rate. This can be achieved by increasing the number of raters (left
panel of Figure 5) or modifying the rating protocol to improve the raters
agreement $\text{IRR}_{1}$ (right panel of Figure 5).444The false positive and
false negative rates are both based on the true classification probability and
proportion of selected applicants – they cannot be optimized independently
unless we consider changing the proportion of selected applicants as well.
Even though both changes lead to an increase in the overall inter-rater
reliability, $\text{IRR}_{J}$, and consequently lower false positive rate,
each change corresponds to a different modification of the peer review process
(i.e., hiring more raters vs training the raters).

The black line in Figure 5 visualizes changes in the false positive rate when
increasing the number of raters (left) or the $\text{IRR}_{1}$ (right). The
added orange and red lines show an additional change in the false positive
rate when altering both factors simultaneously. We see that for a selection
procedure resulting in the selection of 18% applicants, the false positive
error rate remains relatively high even when increasing both the number of
raters and $\text{IRR}_{1}$. Similar analysis can further incorporate the cost
of each modification and weight it against the benefit of decreasing the false
positive rate of the selection procedure.

##  5 Discussion

We outlined an approach for evaluating ratings based selection procedures in a
binary classification framework. This approach allows researchers and
stakeholders to assess the quality of selection procedures by linking rating
agreement to the actual selection decisions. As a result, the quality of
selection procedures does not need to be evaluated only via a coefficient of
the inter-rater agreement but also via the desired false (true) positive /
negative rates and their associated costs. These results can be used in more
complex utility functions also featuring costs of increasing the number of
raters, modifying the rating guidelines, and changing the proportion of
selected applicants.

The approach is based on a minimal measurement model, commonly used for
assessing inter-rater reliability, and linked to the binary decisions via a
quantile approximation. We showed how to compute the probability of correct
classification, other binary classification metrics, and showed how the binary
classification relates to inter-rater reliability via the quantile
approximation relying only on the number of ratings and proportion of selected
applicants.

We evaluated the quantile approximation in a simulation study and found little
bias. Importantly, the estimated classification probabilities and resulting
binary classification metrics correspond to the expectation of the selection
procedure. The actual classification probabilities for any given selection
procedure might be far from the expectations, especially when considering only
a small proportion of selected applicants – as the results are dependent on a
few binary events that are inherently noisy.

We showed the methodology in an example comparing false positive rates across
multiple grant peer reviews. The results of our example should be interpreted
with great caution. First, we based our results on averages across different
fields that often vary in the inter-rater reliability, number of raters per
proposal, and the number of selected applicants. Second, the grant agencies
often base the funding decisions on a combination of the overall rating score
and additional information (which is not included in our calculation). While
acknowledging the limitations, the results still indicate a relatively high
false positive rate, especially in selection procedures selecting a small
portion of the applicants.

The outlined approach only evaluates the reliability of selecting the high
ability applicants due to the measurement error, while assuming that the
rating process is valid and indeed measuring the relevant latent abilities
[4]. Furthermore, one might argue against optimizing raters’ agreement as the
disagreement between raters might indicate that raters were selected on
diverse bases and focused on different aspects of applicants [[,
e.g.,]]bailar1991reliability, hargens1991referee, kiesler1991confusion,
lee2012kuhnian.

The outlined approach could be further expanded in multiple directions. First,
we used a minimal measurement model for assessing the inter-rater reliability
of continuous ratings. However, each parameter of the measurement model might
differ across groups of applicants [[, e.g.,]]martinkova2018disparities,
mutz2012heterogeneity, bartos2019testing, martinkova2022assessing. Additional
variance components might be used for modeling dependencies in the data [[,
e.g., the effect of raters]]martinkova2018disparities. Different applicants
might be rated a different number of times, the measured rankings might not be
assumed to be normally distributed [[, e.g., see]for joint modeling of
continuous and ordinal data]pearce2022unified, and the latent abilities might
follow a different distribution as well. All these possibilities can be
included in the measurement model and further propagated through the quantile
approximation, either by using a mixture of distributions for the latent
abilities based on the proportion of groups and group specific parameters,
specifying a different type of distributions and measurement error for the
observed rankings, and different types of distributions for the latent
abilities.

Second, we focused on classification probabilities and the resulting binary
classification metrics themselves. Under the specified measurement model, most
erroneous classifications happen close to the selection boundary.
Consequently, evaluating selection procedures solely via a false positive rate
does not reflect the loss associated with the degree of the errors. A further
extension might consider weighting the classification probabilities by a
utility function accounting for the degree of the error (i.e., mistakenly
refusing an applicant who is right above the classification threshold is less
costly than refusing an applicant much higher on the latent ability).

## Acknowledgement

The study was funded by the Czech Science Foundation grant number 21-03658S.
We are thankful to Julius Pfadt and Elena Erosheva for helpful comments and
suggestions on an earlier version of the manuscript.

## Supplementary material

The analysis and simulation scripts are available at https://osf.io/674fk/.

## References

  * [1] John C Bailar  “Reliability, fairness, objectivity and other inappropriate goals in peer review”  In _Behavioral and Brain Sciences_ 14.1 Cambridge University Press, 1991, pp. 137–138
  * [2] František Bartoš, Patrícia Martinková and Marek Brabec  “Testing heterogeneity in inter-rater reliability”  In _The Annual Meeting of the Psychometric Society_ , 2019, pp. 347–364  Springer  URL: https://doi.org/10.1007/978-3-030-43469-4_26
  * [3] Douglas Bates, Martin Mächler, Ben Bolker and Steve Walker  “Fitting Linear Mixed-Effects Models Using lme4”  In _Journal of Statistical Software_ 67.1, 2015, pp. 1–48  DOI: 10.18637/jss.v067.i01
  * [4] Robert F Bornstein  “The predictive validity of peer review: A neglected issue”  In _Behavioral and Brain Sciences_ 14.1 Cambridge University Press, 1991, pp. 138–139
  * [5] Afton S Carpenter et al.  “A retrospective analysis of the effect of discussion in teleconference and face-to-face scientific peer-review panels”  In _BMJ Open_ 5.9 British Medical Journal Publishing Group, 2015, pp. e009138  URL: http://dx.doi.org/10.1136/bmjopen-2015-009138
  * [6] Domenic V Cicchetti  “The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation”  In _Behavioral and Brain Sciences_ 14.1 Cambridge University Press, 1991, pp. 119–135  URL: https://doi.org/10.1017/S0140525X00065675
  * [7] Jacob Cohen  “A coefficient of agreement for nominal scales”  In _Educational and Psychological Measurement_ 20.1 Sage Publications Sage CA: Thousand Oaks, CA, 1960, pp. 37–46  URL: https://doi.org/10.1177/001316446002000104
  * [8] Stephen Cole, Leonard Rubin and Jonathan R Cole  “Peer review in the National Science Foundation: Phase one of a study”  ERIC, 1978  URL: https://doi.org/10.17226/20041
  * [9] Richard O Duda and Peter E Hart  “Pattern classification”  John Wiley & Sons, 2006
  * [10] Elena A Erosheva et al.  “NIH peer review: Criterion scores completely account for racial disparities in overall impact scores”  In _Science Advances_ 6.23 American Association for the Advancement of Science, 2020, pp. eaaz4868  DOI: 10.1126/sciadv.aaz4868
  * [11] Elena A Erosheva, Patrícia Martinková and Carole J Lee  “When zero may not be zero: A cautionary note on the use of inter-rater reliability in evaluating grant peer review”  In _Journal of the Royal Statistical Society: Series A (Statistics in Society)_ 184.3, 2021, pp. 904–919  DOI: 10.1111/rssa.12681
  * [12] Lowell L Hargens  “Referee agreement in context”  In _Behavioral and Brain Sciences_ 14.1 Cambridge University Press, 1991, pp. 150–151
  * [13] Charles A Kiesler  “Confusion between reviewer reliability and wise editorial and funding decisions”  In _Behavioral and Brain Sciences_ 14.1 Cambridge University Press, 1991, pp. 151–152
  * [14] Helena Chmura Kraemer  “Do we really want more “reliable” reviewers?”  In _Behavioral and Brain Sciences_ 14.1 Cambridge University Press, 1991, pp. 152–154  URL: https://doi.org/10.1017/S0140525X00065870
  * [15] Carole J Lee  “A Kuhnian critique of psychometric research on peer review”  In _Philosophy of Science_ 79.5 University of Chicago Press Chicago, IL, 2012, pp. 859–870
  * [16] Patrícia Martinková, Dan Goldhaber and Elena A Erosheva  “Disparities in ratings of internal and external applicants: A case for model-based inter-rater reliability”  In _PLoS One_ 13.10 Public Library of Science San Francisco, CA USA, 2018, pp. e0203002  DOI: 10.1371/journal.pone.0203002
  * [17] Patrı́cia Martinková, František Bartoš and Marek Brabec  “Assessing inter-rater reliability with heterogeneous variance components models: Flexible approach accounting for contextual variables”  In _arXiv_ , 2022  URL: https://arxiv.org/abs/2207.02071v1
  * [18] Nancy E Mayo et al.  “Peering at peer review revealed high degree of chance associated with funding of grant applications”  In _Journal of Clinical Epidemiology_ 59.8 Elsevier, 2006, pp. 842–848  URL: https://doi.org/10.1016/j.jclinepi.2005.12.007
  * [19] Kenneth O McGraw and Seok P Wong  “Forming inferences about some intraclass correlation coefficients.”  In _Psychological Methods_ 1.1 American Psychological Association, 1996, pp. 30  URL: https://doi.org/10.1037/1082-989X.1.1.30
  * [20] Charles E Metz  “Basic principles of ROC analysis”  In _Seminars in Nuclear Medicine_ 8.4, 1978, pp. 283–298  Elsevier  URL: https://doi.org/10.1016/s0001-2998(78)80014-2
  * [21] Rüdiger Mutz, Lutz Bornmann and Hans-Dieter Daniel  “Heterogeneity of inter-rater reliabilities of grant peer reviews and its determinants: A general estimating equations approach”  In _PLoS One_ 7.10 Public Library of Science San Francisco, USA, 2012, pp. e48509  URL: https://doi.org/10.1371/journal.pone.0048509
  * [22] Linda D Nelson  “The process of peer review: Unanswered questions”  In _Behavioral and Brain Sciences_ 14.1 Cambridge University Press, 1991, pp. 158–159
  * [23] Michael Pearce and Elena A Erosheva  “A unified statistical learning model for rankings and scores with application to grant panel review”  In _arXiv preprint arXiv:2201.02539_ , 2022  URL: https://arxiv.org/abs/2112.02539
  * [24] Margaret Sullivan Pepe  “The statistical evaluation of medical tests for classification and prediction”  Oxford University Press, USA, 2003
  * [25] Shayle R Searle, George Casella and Charles E McCulloch  “Variance components”  John Wiley & Sons, 2006
  * [26] Patrick E Shrout and Joseph L Fleiss  “Intraclass correlations: Uses in assessing rater reliability.”  In _Psychological Bulletin_ 86.2 American Psychological Association, 1979, pp. 420  URL: https://doi.org/10.1037/0033-2909.86.2.420

## Appendix A: Further Simulation Results

Figure 6: Mean empirical true positive classification probabilities for 300
applicants (full red line) vs the mean quantile approximated true positive
classification probabilities (dashed blue line) across all possible
proportions of selected applicants ($x$-axis). The diagonal shows the maximum
true positive classification probability for a given proportion of selected
applicants. Results are presented for different inter-rater reliability
coefficients (rows) and the number of ratings (columns). Figure 7: Mean
empirical true positive classification probabilities for 1000 applicants (full
red line) vs the mean quantile approximated true positive classification
probabilities (dashed blue line) across all possible proportions of selected
applicants ($x$-axis). The diagonal shows the maximum true positive
classification probability for a given proportion of selected applicants.
Results are presented for different inter-rater reliability coefficients
(rows) and the number of ratings (columns). Figure 8: Root mean square error
of quantile approximation to the true positive classification probabilities
across all possible proportions of selected applicants ($x$-axis). Results
shown for 300 applicants and are presented for different inter-rater
reliability coefficients (rows) and the number of ratings (columns). Figure 9:
Root mean square error of quantile approximation to the true positive
classification probabilities across all possible proportions of selected
applicants ($x$-axis). Results shown for 1000 applicants and are presented for
different inter-rater reliability coefficients (rows) and the number of
ratings (columns).