# Benchmarking the Impact of Noise on Deep Learning-based Classification of
Atrial Fibrillation in 12-Lead ECG

Theresa Bender Corresponding Author: Theresa Bender, University Medical Center
Göttingen, Robert-Koch-Str. 40, 37075 Göttingen, Germany,
<EMAIL_ADDRESS>Philip Gemke  Ennio Idrobo-Avila
Henning Dathe  Dagmar Krefting  Nicolai Spicher  Department of Medical
Informatics, University Medical Center Göttingen, Göttingen, Germany  DZHK
(German Centre for Cardiovascular Research), partner site Göttingen,
Göttingen, Germany

###### Abstract

Electrocardiography analysis is widely used in various clinical applications
and Deep Learning models for classification tasks are currently in the focus
of research. Due to their data-driven character, they bear the potential to
handle signal noise efficiently, but its influence on the accuracy of these
methods is still unclear. Therefore, we benchmark the influence of four types
of noise on the accuracy of a Deep Learning-based method for atrial
fibrillation detection in 12-lead electrocardiograms. We use a subset of a
publicly available dataset (PTB-XL) and use the metadata provided by human
experts regarding noise for assigning a signal quality to each
electrocardiogram. Furthermore, we compute a quantitative signal-to-noise
ratio for each electrocardiogram. We analyze the accuracy of the Deep Learning
model with respect to both metrics and observe that the method can robustly
identify atrial fibrillation, even in cases signals are labelled by human
experts as being noisy on multiple leads. False positive and false negative
rates are slightly worse for data being labelled as noisy. Interestingly, data
annotated as showing baseline drift noise results in an accuracy very similar
to data without. We conclude that the issue of processing noisy
electrocardiography data can be addressed successfully by Deep Learning
methods that might not need preprocessing as many conventional methods do.

###### keywords:

Deep Learning, Electrocardiogram, Atrial Fibrillation, Noise

, , , , ,

## Introduction

Electrocardiograms (ECGs) are recordings of the electrical activity of the
heart and are frequently used in emergency and in-patient care. However,
different types of noise, either stemming from the patient’s behaviour (e.g.
motion) or the devices (e.g. power line interference), can be introduced
during measurement. The presence of noise leads to a twofold problem: It
impedes detection of anomalies leading to false findings and alarms [1] and,
if the signal-to-noise ratio (SNR) reaches a certain level, detecting
diagnostically-relevant features becomes impossible [2].

One class of features with high clinical importance are the so-called
”fiducial points”, i.e. the center, on- and offsets of ECG waves such as the
QRS complex and the P-/T-wave. They are used for segmenting heartbeats into
meaningful intervals [3] and by doing so allow for arrhythmia detection.
Atrial fibrillation (AF) is the most prevalent arrhythmia which is
characterized by uncoordinated electrical impulses in the atrium and might
lead to severe cardiovascular issues, such as stroke or heart failure.
Analyzing the interval in a heartbeat where a P-wave is expected is crucial
for AF classification as its absence indicates a lack of sinoatrial node
activity and is thereby a sign for AF [4]. However, so-called fibrillatory
waves might occur, mimicking P-waves, impeding the assessment of sinoatrial
node activity.

Many state-of-the-art algorithms for ECG classification are based on
extracting semantic features derived from human expert knowledge, such as
fiducial points. However, as these algorithms tend to wrong results in case of
noise [5], various denoising strategies [6] have been proposed. In contrast,
algorithms from the field of deep learning (DL) were explored for ECG
classification tasks recently [7, 8]. Instead of semantic features, they are
based on agnostic features derived from fully-automatic correlation analysis
between input ECGs and output classes in an end-to-end fashion. These models
are based on the underlying premise that training and test datasets are
stemming from the same distribution, which is often their pitfall in case of
dataset shifts (variant devices, users, noise). Although initial studies
indicate a better robustness to noise [9], it remains unclear to which extend
it affects these models.

Thereby, in this work we benchmark the accuracy of a state-of-the-art pre-
trained DL model for 12-lead ECG classification regarding its susceptibility
to different types of noise. We use the publicly available PTB-XL dataset
which contains annotations for several categories of noise made by human
technical experts and compare the model’s accuracy w.r.t. type of noise.

## Methods

We analyze a subset of the PTB-XL dataset containing 12-lead ECGs of 10 second
length [10]. It contains all 1,514 ECGs annotated as showing AF (label in PTB-
XL: AFIB) and we add the first $2,000$ normal ECGs (NORM) as healthy controls.
For each signal, we use a qualitative and a quantitative method to estimate
SNR.

### SNR based on annotations

($\textrm{SNR}_{\textrm{\small a}}$) For each ECG we determine the number of
noisy leads using the columns baseline_drift, static_noise, burst_noise and
electrodes_problems provided in the PTB-XL metadata. In the majority of cases,
they contain the name of a single lead (e.g. “aVL”), multiple leads (“I,aVR”)
or ranges (e.g. “I-III”). Using a custom script, we convert this information
to numeric values ranging from $0$ to $12$ for each type of noise. The labels
“alles” (all) and “noisy recording” are converted to $12$. We remove ECGs
associated with other labels as they are of a more qualitative nature (e.g.
“leicht” (light)). In this way, for each signal a qualitative, unit-less,
linear SNR measure is computed, ranging from $0$ (no noise reported) to
$12*4=48$ (all leads are affected by all types of noise). As shown in Tbl. 1,
we use this information to split the dataset in ECGs without (”w/o”) a noise
label and ECGs with (”w/”) a noise label.

It has to be underlined that a value of zero does not have to mean that there
is no noise, it just reflects that there is a potential for a noise-free ECG.
The authors of PTB-XL also indicated that missing annotations in case of
artifacts or false annotations in case of noise-free signals might occur.
However, they concluded that the metadata bears the potential for ECG quality
assessment [11].

Table 1: Properties of subset extracted from PTB-XL (left) and results of DL-based AF classification (right). ECGs are grouped according to annotations: In case there is one or more noise label in the metadata, an ECG is assigned to ”w/”, else to ”w/o”. FP and FN denote False Positive and False Negative, respectively. Noise Label | AF | Healthy controls |  | Noise Label | DL: FP | DL: FN
---|---|---|---|---|---|---
w/o | 1,097 | 1,581 |  | w/o | 0.04 % | 3.96 %
w/ | 417 | 419 |  | w/ | 0.24 % | 7.06 %

### Measured SNR

($\textrm{SNR}_{\textrm{\small m}}$) Due to the limitations of the manual
annotations and as they are only available for $22\%$ of the PTB-XL database
[11], we additionally use a quantitative SNR measure for each signal. We
compute the Fourier Transform of the signals as well as the ratio of energies
in two frequency bands as proposed in [12]. Based on the expected heart rates
during AF, we define the “signal” frequency band ranging from $40$ to $150$
beats-per-minute ($0.66$ to $2.5$ Hz) and define the “noise” frequency band as
$<40$ and $>150$ beats-per-minute. By scaling with $10\log 10$, we arrive at
an SNR expressed in logarithmic decibel scale (dB).

### DL classification

ECG data is classified with a pre-trained model by Ribeiro et al. [7]. The
model is a residual network and was trained on more than two million ECGs that
were acquired within a Brazilian telehealth network. It outputs independent
probabilities for six abnormalities, but we limit our analysis to AF. We use a
threshold defined by the authors111https://github.com/antonior92/automatic-
ecg-diagnosis/blob/master/generate_figures_and_tables.py, commit 89f929d, line
121.

### Data analysis

We analyze the subset regarding differences between ECGs with and without
noise labels for i) their distribution of $\textrm{SNR}_{\textrm{\small m}}$
and $\textrm{SNR}_{\textrm{\small a}}$ as well as ii) the accuracy of DL
classification of each noise category. For ii) we compared the noisy
recordings ($\textrm{SNR}_{\textrm{\small a}}>0$) with randomly drawn signals
from equally sized control groups ($\textrm{SNR}_{\textrm{\small a}}=0$).

## Results

Fig. 1 shows the distribution of $\textrm{SNR}_{\textrm{\small a}}$ and
$\textrm{SNR}_{\textrm{\small m}}$ values on the left and right side. The
majority of ECGs with noise labels has less than $15$ with the maximum being
$29$. This shows that even in the duration of $10$ seconds, different data
quality issues per lead may occur. $\textrm{SNR}_{\textrm{\small m}}$ values
are occurring in the range of $[-33.03,-7.78]$ dB with no clear difference
between ECGs with and without noise labels.

Tbl. 1 (right) shows FP and FN rates of AF classification w.r.t. the existence
of noise labels. FP is worsened by $0.2\%$ and FN by $3.1\%$ in case ECGs are
annotated with noise labels. Tbl. 2 shows the DL accuracy for each type of
noise compared to the same number of ECGs but randomly drawn $100$ times from
data without noise labels. ECGs with baseline drift or electrode problems are
classified more accurately in comparison to random ECG signals without noise
annotations, whereas ECGs with annotated burst and static noise reveal worse
performance.

## Discussion

In general, the DL model can robustly classify AF, even in case ECGs are
labelled by human experts as having multiple leads influenced by noise.
Interestingly, in presence of baseline drift or electrode problems, accuracy
is not deteriorated, but within one standard deviation compared to signals
without noise labels. As a limitation, it has to be underlined that
annotations are non-complete [11] and the subset contains only six signals
annotated with electrode problems.

Figure 1: Distribution of values of both SNR metrics with (grey) and without (blue) noise labels. Table 2: DL accuracy w.r.t. the four types of noise. The variable $n$ represents the number of signals with the given label (w/). For comparison to signals without a label (w/o), $n$ ECGs are randomly drawn $100$ times and accuracy is given as mean $\pm$ standard deviation. Label Type |  1.8cmBaseline Drift
($n=305$) |  1.8cmStatic Noise
($n=478$) |  1.8cmBurst Noise
($n=156$) |  2.5cmElectrode Problems ($n=6$)
---|---|---|---|---
w/o | $96.8\%\pm 0.9\%$ | $96.8\%\pm 0.7\%$ | $96.9\%\pm 1.3\%$ | $96.3\%\pm 8.0\%$
w/ | $97.7\%$ | $94.6\%$ | $94.9\%$ | $100.0\%$

As the DL model can be assumed as a ”black box”, we can only speculate about
the reasons for this behaviour. It could be explained by partial
misinterpretation of baseline drift or static noise as P-waves. As we could
show in previous work [13], the DL model was trained such that P-waves and
R-peaks have a high relevance, similar to human perception, while numerous
other features influence its decision. This multi-factor decision process
could be robust to different kinds of noise, but this requires its presence
during training. A shift between training and test datasets is always an issue
for DL models. To mitigate this effect is has been suggested to intentionally
include noise during training [9]. The model used in this work was trained on
$2,000,000$ non-public ECGs.

However, since the distribution of $\textrm{SNR}_{\small\textrm{m}}$ looks
visually similar with or without noise labels,
$\textrm{SNR}_{\small\textrm{a}}$ might not be optimal for quality assessment
on its own. A ”no noise” label, explicitly identifying ECGs without data
quality issues, and more labels in general would be a valuable addition for
future experiments.

## Conclusion

Results show that the DL model is able to detect AF in 12-lead ECGs with high
accuracy, even in the presence of data quality issues according to human
experts. We conclude that the difficulty of processing noisy ECGs can be
addressed by end-to-end DL models based on agnostic features. In contrast to
conventional methods based on semantic features, they might not require
preprocessing methods for achieving high accuracy. However, more experiments
with larger and more diverse datasets should be the subject of future work.

## References

  * [1] Festag S, Spreckelsen C. Semantic Anomaly Detection in Medical Time Series. Stud Health Technol Inform 2021; 278:118–25, doi: 10.3233/SHTI210059.
  * [2] Apandi ZFM, Ikeura R, Hayakawa S, Tsutsumi S. An Analysis of the Effects of Noisy Electrocardiogram Signal on Heartbeat Detection Performance. Bioengineering 2020; 7(2), doi: 10.3390/bioengineering7020053.
  * [3] Spicher N, Kukuk M. Delineation of Electrocardiograms Using Multiscale Parameter Estimation. IEEE J Biomed Health Inform 2020; 24(8):2216–29, doi: 10.1109/JBHI.2019.2963786.
  * [4] Kreimer F, Aweimer A, Pflaumbaum A, Mügge A, Gotzmann M. Impact of P-wave indices in prediction of atrial fibrillation-Insight from loop recorder analysis. Ann Noninv Electrocard 2021; 26(5):e12854, doi: 10.1111/anec.12854.
  * [5] Kumar P, Sharma VK. Detection and classification of ECG noises using decomposition on mixed codebook for quality analysis. Healthc Technol Lett 2020; 7(1):18–24, 10.1049/htl.2019.0096.
  * [6] Mir HY, Singh O. ECG denoising and feature extraction techniques – a review. Journal of Medical Engineering & Technology 2021; 45(8):672–84, doi: 10.1080/03091902.2021.1955032.
  * [7] Ribeiro AH, Ribeiro MH, Paixão GMM, Oliveira DM, Gomes PR, Canazart JA et al. Automatic diagnosis of the 12-lead ECG using a deep neural network. Nat Commun 2020; 11(1):1760, doi: 10.1038/s41467-020-15432-4.
  * [8] Attia ZI, Harmon DM, Behr ER, Friedman PA. Application of artificial intelligence to the electrocardiogram. Eur Heart J 2021; 42(46):4717–30, doi: 10.1093/eurheartj/ehab649.
  * [9] Venton J, Harris PM, Sundar A, Smith NAS, Aston PJ. Robustness of convolutional neural networks to physiological electrocardiogram noise. Philos Trans A Math Phys Eng Sci 2021; 379(2212):20200262, doi: 10.1098/rsta.2020.0262.
  * [10] Wagner P, Strodthoff N, Bousseljot R-D, Kreiseler D, Lunze FI, Samek W et al. PTB-XL, a large publicly available electrocardiography dataset. Sci Data 2020; 7(1):154, doi: 10.1038/s41597-020-0495-6.
  * [11] Strodthoff N, Wagner P, Schaeffter T, Samek W. Deep Learning for ECG Analysis: Benchmarks and Insights from PTB-XL; 2020 Apr 28, doi: 10.48550/arXiv.2004.13701.
  * [12] Haan G de, Jeanne V. Robust pulse rate from chrominance-based rPPG. IEEE Trans Biomed Eng 2013; 60(10):2878–86, doi: 10.1109/TBME.2013.2266196.
  * [13] Bender T, Beinecke JM, Krefting D, Müller C, Dathe H, Seidler T et al. Analysis of a Deep Learning Model for 12-Lead ECG Classification Reveals Learned Features Similar to Diagnostic Criteria; 2022, doi: 10.48550/arXiv.2211.01738.


Given the importance of getting calibrated predictions and reliable uncertainty estimations, various post-hoc calibration methods have been developed for neural networks on standard multi-class classification tasks.
However, these methods are not well suited for calibrating graph neural networks (GNNs), which presents unique challenges such as accounting for the graph structure and the graph-induced correlations between the nodes.
In this work, we conduct a systematic study on the calibration qualities of GNN node predictions.
In particular, we identify five factors which influence the calibration of GNNs: general under-confident tendency, diversity of nodewise predictive distributions, distance to training nodes, relative confidence level, and neighborhood similarity.
Furthermore, based on the insights from this study, we design a novel calibration method named Graph Attention Temperature Scaling (GATS), which is tailored for calibrating graph neural networks.
GATS incorporates designs that address all the identified influential factors and produces nodewise temperature scaling using an attention-based architecture.
GATS is accuracy-preserving, data-efficient, and expressive at the same time.
Our experiments empirically verify the effectiveness of GATS, demonstrating that it can consistently achieve state-of-the-art calibration results on various graph datasets for different GNN backbones.[Source code available at <https://github.com/hans66hsu/GATS>]

§ INTRODUCTION

Graph-structured data, such as social networks, knowledge graphs and internet of things, have wide-spread presence and learning on graphs using neural networks has been an active area of research. For node classification on graphs, a wide range of graph neural network (GNN) models, including GCN [Kipf and Welling, 2017], GAT [Veličković et al., 2018] and GraphSAGE [Hamilton et al., 2017], have been proposed to achieve high classification accuracy.

This said, high accuracy is not the only desideratum for a classifier. Especially, reliable uncertainty estimation is crucial for applications like safety critical tasks and active learning. Neural networks are known to produce poorly calibrated predictions that are either overconfident or under-confident [Guo et al., 2017, Wang et al., 2021]. To mitigate this issue a variety of post-hoc calibration methods [Guo et al., 2017, Kull et al., 2019, Zhang et al., 2020, Tomani et al., 2021, Gupta et al., 2021] have been introduced over the last few years for calibrating neural networks on standard multi-class classification problems. However, calibration of GNNs, in the context of node classification on graphs, is currently still an underexplored topic. While it is possible to apply existing calibration methods designed for multi-class classification to GNNs in a nodewise manner, this does not address the specific challenges of node classification on graphs. Especially, node predictions in a graph are not i.i.d. but correlated, and we are tackling a structured prediction problem [Nowozin and Lampert, 2011]. A uniform treatment when calibrating node predictions would fail to account for the structural information from graphs and the non i.i.d. behavior of node predictions.

Our contribution.
In this work, we focus on calibrating GNNs for the node classification task [Kipf and Welling, 2017, Veličković et al., 2018].
First, we aim at understanding the specific challenges posed by GNNs by conducting a systematic study on the calibration qualities of GNN node predictions. Our study reveals five factors that influence the calibration performance of GNNs: general under-confident tendency, diversity of nodewise predictive distributions, distance to training nodes, relative confidence level, and neighborhood similarity.
Second, we develop Graph Attention Temperature Scaling (GATS) approach, which is designed in a way that accounts for the aforementioned influential factors.
GATS generates nodewise temperatures that calibrate GNN predictions based on the graph topology.
Third, we conduct a series of GNN calibration experiments and empirically verify the effectiveness of GATS in terms of calibration, data-efficiency, and expressivity.

§ RELATED WORK

For standard multi-class classification tasks, a variety of post-hoc calibration methods have been proposed in order to make neural networks uncertainty aware: temperature scaling (TS) [Guo et al., 2017], ensemble temperature scaling (ETS) [Zhang et al., 2020], multi-class isotonic regression (IRM) [Zhang et al., 2020], Dirichlet calibration [Kull et al., 2019], spline calibration [Gupta et al., 2021], etc. Additionally, calibration has been formulated for regression tasks [Kuleshov et al., 2018]. More generally, instead of transforming logits after training a classifier, a plethora of methods exists that modify either the model architecture or the training process itself. This includes methods that are based on Bayesian paradigm [Hernandez-Lobato and Adams, 2015, Blundell et al., 2015, Gal and Ghahramani, 2016, Maddox et al., 2019, Wen et al., 2018], evidential theory [Sensoy et al., 2018], adversarial calibration [Tomani and Buettner, 2021] and model ensembling [Lakshminarayanan et al., 2017]. One common caveat of these methods is the trade-off between accuracy and calibration, which oftentimes do not go hand in hand.
Post-hoc methods like temperature scaling, on the other hand, are accuracy preserving. They ensure that the per node logit rankings are unaltered.

Calibration of GNNs is currently a substantially less explored topic. Nodewise post-hoc calibration on GNNs using methods developed for the multi-class setting has been empirically evaluated by Teixeira et al., 2019. They show that these methods, which perform uniform calibration of nodewise predictions, are unable to produce calibrated predictions for some harder tasks. Wang et al., 2021 observe that GNNs tend to be under-confident in contrast to the majority of multi-class classifiers, which are generally overconfident [Guo et al., 2017]. Based on their findings, Wang et al., 2021 propose the CaGCN approach, which attaches a GCN on top of the backbone GNN for calibration. Some approaches improve the uncertainty estimation of GNNs by adjusting model training. This includes Bayesian learning approaches [Zhang et al., 2019, Hasanzadeh et al., 2020] and methods based on the evidential theory [Zhao et al., 2020, Stadler et al., 2021].

§ PROBLEM SETUP FOR GNN CALIBRATION

We consider the problem of calibrating GNNs for node classification tasks:
given a graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$, the training data consist of nodewise input features $\{x_i\}_{i\in\mathcal{V}} \in \mathcal{X}$ and ground-truth labels $\{y_i\}_{i\in\mathcal{L}}\in \mathcal{Y}=\{1,\dots, K\}$ for a subset $\mathcal{L} \subset \mathcal{V}$ of nodes, and the goal is to predict the labels $\{y_i\}_{i\in\mathcal{U}}\in \mathcal{Y}$ for the rest of the nodes $\mathcal{U} = \mathcal{V} \setminus \mathcal{L}$.
A graph neural network tackles the problem by producing nodewise probabilistic forecasts $\hat{p}_i$.
These forecasts yield the corresponding label predictions $\hat{y}_i:=\argmax_y \hat{p}_i(y)$ and confidences $\hat{c}_i:=\max_{y} \hat{p}_i(y)$.
The GNN is calibrated when its probabilistic forecasts are reliable, e.g., for predictions with confidence $0.8$, they should be correct $80\%$ of the time.
Formally, a GNN is perfectly calibrated [Wang et al., 2021] if

\begin{equation}
\forall c \in [0,1], \quad \mathbb{P}(y_i = \hat{y}_i|\hat{c}_i=c) = c.
\end{equation}

In practice, we quantify the calibration quality with the expected calibration error (ECE) [Pakdaman Naeini et al., 2015, Guo et al., 2017]. We follow the commonly used definition from Guo et al., 2017 which uses a equal width binning scheme to estimate calibration error for any node subset $\mathcal{N} \subset \mathcal{V}$: the predictions are regrouped according to $M$ equally spaced confidence intervals, i.e. $(B_1,\dots,B_M)$ with $B_m = \{j\in \mathcal{N} \,|\, \frac{m-1}{M} < \hat{c}_j \leq \frac{m}{M} \}$,
and the expected calibration error of the GNN forecasts is defined as

\begin{align}
&\text{ECE} = \sum_{m=1}^{M}\frac{|B_m|}{|\mathcal{N}|} \Big| \text{acc}(B_m) - \text{conf}(B_m) \Big|, \text{ with } \\
&\text{acc}(B_m)=\frac{1}{|B_m|}\sum_{i\in B_m} \mathbf{1}(y_i=\hat{y}_i) \text{ and }
\text{conf}(B_m)=\frac{1}{|B_m|}\sum_{i\in B_m}\hat{c}_i.
\end{align}

§ FACTORS THAT INFLUENCE GNN CALIBRATION

To design calibration methods adapted to GNNs, we need to figure out the particular factors that influence the calibration quality of GNN predictions. For this we train a series of GCN [Kipf and Welling, 2017] and GAT [Veličković et al., 2018] models on seven graph datasets: Cora [Sen et al., 2008], Citeseer [Sen et al., 2008], Pubmed [Namata et al., 2012], Amazon Computers [Shchur et al., 2018], Amazon Photo [Shchur et al., 2018], Coauthor CS [Shchur et al., 2018], and Coauthor Physics [Shchur et al., 2018]. We summarize the dataset statistics in Appendix <ref>. Details about model training are provided in Appendix <ref> for reproducibility. To compare with the standard multi-class classification case, we additionally train ResNet-20 [He et al., 2016] models on the CIFAR-10 image classification task [Krizhevsky and Hinton, 2009] as a reference.

Our experiments uncover five decisive factors that affect the calibration quality of GNNs.
In the following we discuss them in detail.

§.§ General under-confident tendency

Reliability diagrams of GCN models trained on various graph datasets. We see a general tendency of under-confident predictions (plots above the diagonal) except the Physics dataset. This is in contrast to the overconfident behavior of multi-class image classification using CNNs (in gray).

Starting with a global perspective, we notice that GNNs tend to produce under-confident predictions. In Figure <ref> we plot the reliability diagrams [Niculescu-Mizil and Caruana, 2005] for results on different graph datasets using GCN. Similar to Wang et al., 2021, we see a general trend of under-confident predictions for GNNs. This is in contrast to the standard multi-class image classification case which has overconfident behavior. Also, it is interesting to see that this under-confident trend can be more or less pronounced depending on the dataset. For Coauthor Physics, the predictions are well calibrated and have no significant bias.

Results using GAT models lead to similar conclusions and are provided in Appendix <ref>.

§.§ Diversity of nodewise predictive distributions

Entropy distributions of GCN predictions on graph datasets. Compared to the standard classification case, GNN predictions tend to be more dispersed, reflecting their disparate behaviors.

Contrary to the standard multi-class case, GNN outputs can have varying roles depending on their positions in the graph, which means that their output distributions could exhibit dissimilar behaviors. This is empirically evident in Figure <ref>, where we visualize the entropy distributions of GCN output predictions v.s. the standard multi-class results (GAT results are available in Appendix <ref>). We see that the entropies of GNN outputs have more spread-out distributions, which indicates that they have distinct roles and behaviors in graphs.

In terms of GNN calibration, this observation implies that uniform node-agnostic adjustments like temperature scaling [Guo et al., 2017] might be insufficient for GNNs, whereas nodewise adaptive approaches could be beneficial.

§.§ Distance to training nodes

Nodewise calibration error of GCN results depending on the minimum distance to training nodes. We observe that training nodes and their neighbors tend to be better calibrated.

A graph provides additional structural information for its nodes. One insightful feature is the minimum distance to training nodes. We discover that nodes with shorter distances, especially the training nodes themselves and their direct neighbors, tend to be better calibrated.

To evaluate the calibration quality nodewise, we propose the nodewise calibration error, which is based on the binning scheme used to compute the global expected calibration error (ECE) [Pakdaman Naeini et al., 2015, Guo et al., 2017]: for each node, we find its corresponding bin depending on its predicted confidence, and the calibration error of this bin is assigned to be its nodewise calibration error.

Using this nodewise metric, in Figure <ref> we visualize the influence of minimum  distance to training nodes on the nodewise calibration quality (c.f. Appendix <ref> for GAT results). We see that nodes close to training ones typically have lower nodewise calibration error. This suggests that minimum distance to training nodes can be useful for GNN calibration.

§.§ Relative confidence level

Nodewise calibration error of GCN results depending on the relative confidence level. We observe that nodes which are less confident than their neighbors tend to have worse calibration.

Another important structural information is the neighborhood relation. We find out that the relative confidence level $\delta \hat{c}_i$ of a node $i$, i.e., the difference between the nodewise confidence $\hat{c}_i$ and the average confidence of its neighbors

\begin{equation}
\delta \hat{c}_i = \hat{c}_i - \frac{1}{|n(i)|}\sum_{j \in n(i)} \hat{c}_j
\label{eq:delta_ci}
\end{equation}

has an interesting correlation to the nodewise calibration quality. In Figure <ref> we show the relation between the relative confidence level of a node and its nodewise calibration error (c.f. Appendix <ref> for GAT results). Especially, We observe that nodes which are less confident than their neighbors tend to have worse calibration, and it is in general desirable to have comparable confidence level w.r.t. the neighbors.
For GNN calibration, the relative confidence level $\delta\hat{c}_i$ can be a useful node feature to consider.

§.§ Neighborhood similarity

Nodewise calibration error of GCN results depending on the node homophily. Nodes with strongly agreeing neighbors tend to have significantly lower calibration errors.

Furthermore, we find that different neighbors tend to introduce distinct influences. For assortative graphs which are the focus of this work, we find out that calibration of nodes are affected by node homophily, i.e., whether a node tends to have the same label prediction as its neighbors. For a node with $n_a$ agreeing neighbors and $n_d$ disagreeing ones, we measure the node homophily as

\begin{equation}
\text{Node homophily} = \log \Big( \frac{n_a + 1}{n_d + 1} \Big),
\end{equation}

where positive values indicate greater ratio of agree neighbors and vice versa.

Figure <ref> summarizes the variation of nodewise calibration error w.r.t. the node homophily for different graph datasets (c.f. Appendix <ref> for GAT results). We find out that nodewise calibration errors tend to decrease significantly for nodes with strongly agreeing neighbors. This suggests that neighborhood predictive similarity should be considered when doing GNN calibration.

§ GRAPH ATTENTION TEMPERATURE SCALING (GATS)

Based on the findings in Section <ref>, we design a new post-hoc calibration method, named Graph Attention Temperature Scaling (GATS), which is tailored for GNNs.

§.§ Formulation and design of GATS

To obtain a calibration method that is adapted to the graph structure $\mathcal{G} = (\mathcal{V}, \mathcal{E})$ and reflects the observed influential factors in Section <ref>, the graph attention temperature scaling approach extends the temperature scaling [Guo et al., 2017] method to produce a distinct temperature $T_i$ for each node $i \in \mathcal{V}$. $T_i$ is then used to scale the uncalibrated nodewise output logits $z_i$ and produce calibrated node predictions $\hat{p}_i$

\begin{equation}
\forall i \in \mathcal{V}, \quad \hat{p}_i = \softmax\Big( \frac{z_i}{T_i} \Big).
\end{equation}

Illustration of graph attention temperature scaling (GATS) for a graph with four nodes where node 2 is a training node and node 1 is the target node which aggregates information from the other three nodes. GATS incorporates the five factors discussed in Section <ref> and produces nodewise temperatures $T_i$ using the graph structure and an attention mechanism.

Formulation of $T_i$.
The nodewise temperature $T_i$ should address the five factors discussed in Section <ref>. We achieve this via the following considerations:

* We introduce a global bias parameter $T_0$ to account for the general under-confident tendency;

* To tackle the diverse behavior of node predictions, we learn a nodewise temperature contribution $\tau_i$ based on the predicted nodewise logits $z_i$;

* To incorporate the relative confidence w.r.t. neighbors, we introduce $\delta \hat{c}_i$ from Eq. <ref> as an additional contribution term scaled by a learnable coefficient $\omega$;

* To model the influence of neighborhood similarity, we use an attention mechanism [Vaswani et al., 2017] to aggregate neighboring contributions $\tau_j$ with attention coefficients $\alpha_{i,j}$ depending on the output similarities between the neighbors $i$ and $j$;

* Distance to training nodes is used to introduce a nodewise scaling factor $\gamma_i$ to adjust the node contribution and the aggregation process. It is learnable for training nodes and their direct neighbors and fixed to $1$ for the rest:

\begin{equation}
\gamma_i = \begin{cases}
\gamma_t, &\text{ if } i \text{ is a training node} \\
\gamma_n, &\text{ if } i \text{ is a neighbor of training node} \\
1, &\text{ otherwise}
\end{cases}, \quad \gamma_t, \gamma_n \text{ learnable parameters}.
\end{equation}

Putting together the above components, the nodewise temperature $T_i$ has the following expression:

\begin{equation}
\forall i \in \mathcal{V}, \quad T_i = \frac{1}{H} \sum_{h=1}^H \softplus \big( \omega\ \delta \hat{c}_i + \sum_{j \in \hat{n}(i)} \alpha_{i,j} \ \gamma_j\ \tau^h_j \big) + T_0.
\label{eq:gats_t}
\end{equation}

Here we have a multi-head formulation where $h$ indicates the $h$-th attention head, and $\hat{n}(i)$ denotes the neighbors of node $i$ including self-loop. We uses 8 heads ($H=8$), which works well in practice.  (c.f. Section <ref>.)

Defining $\tau^h_i$. Nodewise contribution $\tau^h_i$ are computed as the outputs of parameterized linear layers $\phi^h(\cdot;\theta)$ which take transformed nodewise output logits $\tilde{z}_i$ as input

\begin{equation}
\forall i \in \mathcal{V}, \forall h \in [1, H], \quad \tau^h_i = \phi^h(\tilde{z}_i; \theta^h) = (\theta^h)^{\top} \ \tilde{z}_i.
\end{equation}

The transformed nodewise output logits $\tilde{z}_i$ are produced as follows: we first normalize the original logits $z_i$ to range $[0, 1]$, then sort the classwise logits for each node. This makes the linear layers $\phi^h$ focus on the general logit distributions rather than class predictions. A similar idea has been explored by Rahimi et al., 2020, where they show that intra order-preserving functions improve the model calibration. Here we find out that this sorting-based transformation helps GATS to learn useful representations of nodewise contributions $\tau^h_i$. (c.f. Section <ref>.)

Defining $\alpha_{i,j}$. The attention coefficients $\alpha_{i,j}$ are defined based on the neighbor similarity, which is determined by the inner product between rescaled nodewise logits $z_i / \gamma_i$ and $z_j / \gamma_j$. Inspired by Veličković et al., 2018, we compute the attention coefficients $\alpha_{i,j}$ as follows:

\begin{equation}
(\alpha_{i,j})_{j \in \hat{n}(i)} = \softmax_{j \in \hat{n}(i)} \Big( \leakyrelu \Big( \frac{1}{\gamma_i\ \gamma_j} z_i^{\top}\ z_j \Big) \Big).
\label{eq:gats_alpha}
\end{equation}

§.§ Calibration properties of GATS

Zhang et al., 2020 propose three desiderata for calibration methods: accuracy-preserving, data-efficient, and expressive. GATS fulfills all of them:

* GATS is accuracy-preserving: since all node predictions are scaled by inverse temperatures $1 / T_i$ which are positive scalars, the order of output logits is preserved;

* GATS is data-efficient. It is a parametric calibration model with $C\cdot H+4$ learnable parameters ($T_0, \omega, \gamma_t, \gamma_n, (\theta^h)_{1 \leq h \leq H}$). According to Zhang et al., 2020, parametric methods are already data-efficient;

* GATS is expressive, as it produces nodewise temperatures $T_i$ adapted to the graph structure.

Experiments in Section <ref> empirically confirm the data-efficiency and expressivity of GATS.

§.§ Comparison with CaGCN

It is interesting to compare our proposed GATS approach to the CaGCN method proposed by Wang et al., 2021, since both approaches aim at calibrating GNNs, and both make use of the graph structure to produce nodewise temperatures. While CaGCN uses a GCN to generate nodewise temperatures straightforwardly, GATS uses an attention mechanism which differentiates the influence from various neighbors. GATS also integrates a series of careful designs following the insights from the study in Section <ref>. Experiments in Section <ref> shows that GATS tends to produce better calibration results compared to CaGCN.

§ EXPERIMENTS

To evaluate the performance of GATS on GNN calibration and understand the effects of its designs, we conduct a series of experiments for baseline comparison and ablation study.
We use two representative GNNs: GCN and GAT, which are trained on the seven aforementioned graph datasets plus a larger graph, CoraFull [Bojchevski and Günnemann, 2018], for post-hoc calibration.

We display the ECE results with $M=15$ bins.
We use the expected calibration error (ECE) [Pakdaman Naeini et al., 2015, Guo et al., 2017] with $M=15$ bins as an evaluation metric, and follow an experimental protocol similar to Kull et al., 2019, Kull et al., 2017:
For all the experiments, we randomly split the labeled/unlabeled ($15\%$/$85\%$) data five times, and use three-fold internal cross-validation of the labeled data to train the GNNs and the calibrators. We also utilize five random initializations, resulting in 75 total runs for each experiment. We provide detailed experimental settings in Appendix <ref>.

§.§ Performance comparison

We benchmark GATS against existing baselines on a variety of GNN calibration tasks.
We compare GATS with the following baselines:

* Temperature scaling (TS) [Guo et al., 2017] simply uses a global temperature to scale the logits.

* Vector scaling (VS) [Guo et al., 2017] scales the logits separately over the class dimension and additionally introduces a classwise bias for the recalibrated output logits.

* Ensemble temperature scaling (ETS) [Zhang et al., 2020] learns a mixture of uncalibrated, TS-calibrated, and uniform probabilistic outputs.

* GCN as a calibration function (CaGCN) [Wang et al., 2021] is specifically designed for calibrating GNNs. It uses a GCN on top to generate nodewise temperatures.

Additionally, we also report the ECEs of uncalibrated predictions as a reference. Among the above baselines, TS, VS, and ETS are calibration methods designed for standard classification cases and operate on nodes uniformly. CaGCN on the other hand performs separate nodewise adjustments and uses the graph structure, similar to our proposed GATS approach.

For the post-hoc calibration experiments, we fix the weight of the trained GNN backbones and adjust the parameters of the calibration methods on the validation set. Negative log-likelihood is chosen as the objective for the calibration process. We provide details of method configurations and calibration settings in Appendix <ref>. Table <ref> summarizes the calibration results.

GNN calibration results in terms of ECE (in percentage, lower is better) of GATS and other baseline methods on various graph datasets. Overall, GATS achieves state-of-the-art performance, getting the best results in most scenarios. Also, all best results are achieved by methods that consider the graph structure. This shows the need for dedicated methods to tackle GNN calibration.

2*Cora     GCN    13.04$\pm$5.22    3.92$\pm$1.29    4.36$\pm$1.34    3.79$\pm$1.35    5.29$\pm$1.47    3.64$\pm$1.34

GAT    23.31$\pm$1.81    3.69$\pm$0.90    3.30$\pm$1.12    3.54$\pm$1.01    4.09$\pm$1.06    3.18$\pm$0.90

2*Citeseer     GCN    10.66$\pm$5.92    5.15$\pm$1.50    4.92$\pm$1.44    4.65$\pm$1.69    6.86$\pm$1.41    4.43$\pm$1.30

GAT    22.88$\pm$3.53    4.74$\pm$1.47    4.25$\pm$1.48    4.11$\pm$1.64    5.75$\pm$1.31    3.86$\pm$1.56

2*Pubmed     GCN    7.18$\pm$1.51    1.26$\pm$0.28    1.46$\pm$0.29    1.24$\pm$0.30    1.09$\pm$0.52    0.98$\pm$0.30

GAT    12.32$\pm$0.80    1.19$\pm$0.36    1.00$\pm$0.32    1.20$\pm$0.32    0.98$\pm$0.31    1.03$\pm$0.32

2*Computers     GCN    3.00$\pm$0.80    2.65$\pm$0.57    2.70$\pm$0.63    2.58$\pm$0.70    1.72$\pm$0.53    2.23$\pm$0.49

GAT    1.88$\pm$0.82    1.63$\pm$0.46    1.67$\pm$0.52    1.54$\pm$0.67    2.03$\pm$0.80    1.39$\pm$0.39

2*Photo     GCN    2.24$\pm$1.03    1.68$\pm$0.63    1.75$\pm$0.63    1.68$\pm$0.89    1.99$\pm$0.56    1.51$\pm$0.52

GAT    2.02$\pm$1.11    1.61$\pm$0.63    1.63$\pm$0.69    1.67$\pm$0.73    2.10$\pm$0.78     1.48$\pm$0.61

2*CS     GCN    1.65$\pm$0.92    0.98$\pm$0.27    0.96$\pm$0.30    0.94$\pm$0.24    2.27$\pm$1.07    0.88$\pm$0.30

GAT    1.40$\pm$1.25    0.93$\pm$0.34    0.87$\pm$0.35    0.88$\pm$0.33    2.52$\pm$1.04    0.81$\pm$0.30

2*Physics     GCN    0.52$\pm$0.29    0.51$\pm$0.19    0.48$\pm$0.16    0.52$\pm$0.19    0.94$\pm$0.51    0.46$\pm$0.16

GAT    0.45$\pm$0.21    0.50$\pm$0.21    0.52$\pm$0.20    0.50$\pm$0.21    1.17$\pm$0.42    0.42$\pm$0.14

GCN    6.50$\pm$1.26    5.54$\pm$0.43    5.76$\pm$0.42    5.38$\pm$0.49    5.86$\pm$2.52    3.76$\pm$0.74

GAT    4.73$\pm$1.39    4.00$\pm$0.50    4.17$\pm$0.43    3.89$\pm$0.56    6.55$\pm$3.69    3.54$\pm$0.63

Overall, we observe that GATS consistently produces well-calibrated predictions for all graph datasets and GNN backbones. Except for the GAT model trained on Pubmed (3rd best) and the GCN model trained on Amazon Computers (2nd best), GATS achieves the highest calibration quality in all cases.

Also, it is interesting to see that for all cases the best result is achieved by methods which use the graph structure and produce adapted adjustments for different nodes. This demonstrates the necessity of designing calibration methods that address the unique challenges posed by GNN calibration.

Although CaGCN can get the best results for Pubmed using GAT and Amazon Computers using GCN, we see that its performance is rather unstable for different scenarios, and sometimes it can even produce worse calibration results than the uncalibrated baseline. Using their proposed margin-based loss did not help in our settings. We suspect that CaGCN might have an overly complex architecture for the task, and it cannot differentiate neighborhood influences with the common normalized adjacency matrix. Our proposed GATS model does not have this issue. It has consistent and good calibration performance in all cases.

We also observe that the results tend to have high variations, since GNN backbones tend to predict highly varying results when trained with different initial weights and random splits [Shchur et al., 2018]. We ensure the reliability of the results by averaging over a total of 75 runs with various initial weights and random splits for each case.

§.§ Data-efficiency and expressivity of GATS

ECEs (in percentage) on CoraFull for ETS, CaGCN, and GATS using various amounts of calibration data. We see that GATS is data-efficient and expressive for GNN calibration.

Furthermore, we analyze the data-efficiency and the expressivity of GATS for GNN calibration. For this we reuse the GNN models trained on the CoraFull dataset, and consider the influence of calibration sample size on the GNN calibration performance. For comparison we also report the corresponding results using ensemble temperature scaling and CaGCN. Figure <ref> visualizes the results with GCN backbone. The results for GAT backbone are summarized in Appendix <ref>.

Overall, we see that GATS is both data-efficient and expressive. It requires few calibration samples to get decent calibration performance. This is in contrast to CaGCN which needs more than $5\%$ of nodes for calibration to get acceptable results. Compared to ETS, GATS is more expressive and has a considerably lower calibration error for CoraFull, which is a large graph dataset.

§.§ Ablation study

To empirically analyze the effect of various GATS design choices, we conduct a series of ablation study experiments in this section. Overall, we notice that all designs are advantageous and removing any of them leads to a general decrease in performance.

Ablation study results in terms of ECE (in percentage) for various GATS designs. Overall, all designs are beneficial and removing any of them leads to worse results in general.

w/o $T_0$
w/o $\gamma_i$
w/o $\delta \hat{c}_i$
w/o attention
w/o sorting

GCN    3.72$\pm$1.20    3.80$\pm$1.51    3.71$\pm$1.18    3.63$\pm$1.48    4.35$\pm$1.77    3.64$\pm$1.34

GAT    3.25$\pm$1.00    3.46$\pm$1.00    3.24$\pm$0.89    3.69$\pm$0.96    4.18$\pm$1.70    3.18$\pm$0.90
GCN    5.50$\pm$1.76    4.73$\pm$1.45    4.49$\pm$1.30    4.95$\pm$1.56    5.87$\pm$1.99    4.43$\pm$1.30

GAT    3.56$\pm$1.73    4.39$\pm$1.46    3.87$\pm$1.55    4.81$\pm$1.54    4.99$\pm$2.34    3.86$\pm$1.56
GCN    2.20$\pm$0.88    1.60$\pm$0.64    1.54$\pm$0.52    1.59$\pm$0.67    1.68$\pm$0.61    1.51$\pm$0.52

GAT    2.37$\pm$1.01    1.53$\pm$0.63    1.47$\pm$0.62    1.62$\pm$0.66    1.77$\pm$0.70    1.48$\pm$0.61

Effect of global bias $T_0$.
We consider the effect of global bias $T_0$ by comparing to a GATS variant without it (i.e., setting $T_0=0$ in Eq. (<ref>)). Its results are collected in column “w/o $T_0$” of Table <ref>. Overall we see that a learnable bias $T_0$ is beneficial in most cases.

Effect of nodewise scaling factor $\gamma_i$.
The nodewise scaling factors $\gamma_i$ provide custom adjustment for training nodes and their neighbors. Removing its influence can be done by fixing all $\gamma_i$ to one in Eqs. (<ref>) and (<ref>). The results of the variant without $\gamma_i$ are recorded in column “w/o $\gamma_i$” of Table <ref>. They are worse in general, suggesting that nodewise scaling factors $\gamma_i$ are indeed helpful.

Effect of relative confidence level $\delta \hat{c}_i$.
To evaluate the impact of introducing nodewise relative confidence level $\delta \hat{c}_i$, we create a GATS variant where $\omega$ in Eq. (<ref>) is fixed to zero, which effectively removes the influence of the relative confidence level $\delta \hat{c}_i$. In column “w/o $\delta \hat{c}_i$” of Table <ref> we have the results corresponding to this variant, which is in general slightly worse than the standard GATS which includes $\delta \hat{c}_i$.

Effect of attention-based aggregation.
To understand the role played by the attention mechanism, we create a GATS variant which completely removes the attention related term ($\sum_{j \in \hat{n}(i)} \alpha_{i,j} \ \gamma_j\ \tau^h_j$) from  Eq. (<ref>). Its performance is shown in column “w/o attention” of Table <ref>. Again, we observe that removing the attention-based component tends to worsen the calibration performance.

Effect of logit sorting. GATS uses normalized and sorted logits $\tilde{z}_i$ as input to generate nodewise temperature contributions $\tau_i^h$. And we find that this sorting transform is essential for learning good representations of $\tau_i^h$. In column “w/o sorting” of Table <ref> we have the results for GATS variants which uses the logits without sorting to compute $\tau_i^h$. And we observe that this deteriorates the calibration performance.

Calibration results (ECE in percentage) of GATS models with different numbers of attention heads. 8 attention heads are sufficient for optimal GNN calibration results.

5cNumber of Heads

GCN    3.95$\pm$1.45    3.75$\pm$1.42    3.79$\pm$1.45    3.66$\pm$1.33    3.50$\pm$1.25

GAT    3.48$\pm$1.22    3.49$\pm$1.18    3.43$\pm$1.15    3.20$\pm$0.90    3.31$\pm$0.74
GCN    4.74$\pm$1.43    4.77$\pm$1.66    4.70$\pm$1.52    4.43$\pm$1.30    4.42$\pm$1.04

GAT    4.53$\pm$1.66    4.57$\pm$1.55    4.20$\pm$1.50    3.86$\pm$1.56    4.20$\pm$1.54
GCN    1.51$\pm$0.54    1.54$\pm$0.59    1.52$\pm$0.58    1.51$\pm$0.52    1.50$\pm$0.51

GAT    1.52$\pm$0.61    1.52$\pm$0.65    1.53$\pm$0.60    1.48$\pm$0.61    1.50$\pm$0.60

Effect of attention head count $H$.
Finally, we analyze the influence of multi-head count $H$ on the GNN calibration results. For this we run a series of experiments using GATS models with 1, 2, 4, 8, and 16 attention heads. The results are collected in Table <ref>. We see that for GCN backbones, GATS models with more attention heads tend to get better results. However, for GAT backbones, using 16 heads results in worse performance compared to 8 heads. Accounting also for the fact that doubling the attention head count effectively doubles the computational requirements, 8 attention heads is a decent general setting for GATS.

§ CONCLUSION

In this work, we tackle the GNN calibration problem. We conduct a systematic study to analyze the calibration properties of GNNs predictions.
Our study reveals five influential factors and manifests the unique challenges raised by GNN calibration.
Based on the insights from our studies, we propose a novel calibrator, GATS, which accounts for the identified factors and is tailored for calibrating GNNs.
GATS is accuracy-preserving, data-efficient, and expressive at the same time.
Our experiments demonstrate that GATS achieves state-of-the-art performance for GNN calibration on various graph datasets and for different GNN backbones.

Our work focuses on the node classification tasks for assortative graphs, where neighbors tend to agree with each other.
It is thus important to realize that the validity of the conclusions from Section <ref> is limited to the assortative case, and might no longer hold for disassortative graphs [Zhu et al., 2020, Pei et al., 2020].
It can be an interesting future work to conduct similar studies for GNN calibration in the heterophilous case, especially when more established GNN architectures are available.
More generally, devising calibration methodologies for other graph learning tasks such as link prediction [Zhang and Chen, 2018] and graph classification [Errica et al., 2020] could also be an interesting direction for future research.

This work was supported by the Munich Center for Machine Learning (MCML) and by the ERC Advanced Grant SIMULACRON. The authors would like to thank Nikita Araslanov for proofreading and helpful discussions, as well as the anonymous reviewers for their constructive feedback.

[Blundell et al., 2015]
C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra.
Weight uncertainty in neural network.
In ICML, 2015.

[Bojchevski and Günnemann, 2018]
A. Bojchevski and S. Günnemann.
Deep gaussian embedding of graphs: Unsupervised inductive learning
via ranking.
In ICLR, 2018.

[de Haan, 1999]
P. de Haan.
On the use of density kernels for concentration estimations within
particle and puff dispersion models.
Atmospheric Environment, 330 (13):0
2007–2021, 1999.
ISSN 1352-2310.

[Errica et al., 2020]
F. Errica, M. Podda, D. Bacciu, and A. Micheli.
A fair comparison of graph neural networks for graph classification.
In ICLR, 2020.

[Fey and Lenssen, 2019]
M. Fey and J. E. Lenssen.
Fast graph representation learning with PyTorch Geometric.
In ICLR Workshop on Representation Learning on Graphs and
Manifolds, 2019.

[Gal and Ghahramani, 2016]
Y. Gal and Z. Ghahramani.
Dropout as a bayesian approximation: Representing model uncertainty
in deep learning.
In ICML, 2016.

[Guo et al., 2017]
C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger.
On calibration of modern neural networks.
In ICML, 2017.

[Gupta et al., 2021]
K. Gupta, A. Rahimi, T. Ajanthan, T. Mensink, C. Sminchisescu, and R. Hartley.
Calibration of neural networks using splines.
In ICLR, 2021.

[Hamilton et al., 2017]
W. Hamilton, Z. Ying, and J. Leskovec.
Inductive representation learning on large graphs.
In NeurIPS, 2017.

[Hasanzadeh et al., 2020]
A. Hasanzadeh, E. Hajiramezanali, S. Boluki, M. Zhou, N. Duffield,
K. Narayanan, and X. Qian.
Bayesian graph neural networks with adaptive connection sampling.
In ICML, 2020.

[He et al., 2016]
K. He, X. Zhang, S. Ren, and J. Sun.
Identity mappings in deep residual networks.
In ECCV, 2016.

[Hernandez-Lobato and Adams, 2015]
J. M. Hernandez-Lobato and R. Adams.
Probabilistic backpropagation for scalable learning of bayesian
neural networks.
In ICML, 2015.

[Kingma and Ba, 2014]
D. P. Kingma and J. Ba.
Adam: A method for stochastic optimization.
In ICLR, 2014.

[Kipf and Welling, 2017]
T. N. Kipf and M. Welling.
Semi-supervised classification with graph convolutional networks.
In ICLR, 2017.

[Kraft, 1988]
D. Kraft.
A software package for sequential quadratic programming.
Technical report, DLR German Aerospace Center, 1988.

[Krizhevsky and Hinton, 2009]
A. Krizhevsky and G. Hinton.
Learning multiple layers of features from tiny images.
Technical report, University of Toronto, 2009.

[Kuleshov et al., 2018]
V. Kuleshov, N. Fenner, and S. Ermon.
Accurate uncertainties for deep learning using calibrated regression.
In ICML, 2018.

[Kull et al., 2017]
M. Kull, T. M. S. Filho, and P. Flach.
Beyond sigmoids: How to obtain well-calibrated probabilities from
binary classifiers with beta calibration.
Electronic Journal of Statistics, 110 (2):0
5052 – 5080, 2017.

[Kull et al., 2019]
M. Kull, M. Perello Nieto, M. Kängsepp, T. Silva Filho, H. Song, and
P. Flach.
Beyond temperature scaling: Obtaining well-calibrated multi-class
probabilities with dirichlet calibration.
In NeurIPS, 2019.

[Lakshminarayanan et al., 2017]
B. Lakshminarayanan, A. Pritzel, and C. Blundell.
Simple and scalable predictive uncertainty estimation using deep
In NIPS, 2017.

[Liu and Nocedal, 1989]
D. C. Liu and J. Nocedal.
On the limited memory bfgs method for large scale optimization.
MATHEMATICAL PROGRAMMING, 45:0 503–528, 1989.

[Maddox et al., 2019]
W. J. Maddox, P. Izmailov, T. Garipov, D. P. Vetrov, and A. G. Wilson.
A simple baseline for bayesian uncertainty in deep learning.
In NeurIPS, 2019.

[Namata et al., 2012]
G. Namata, B. London, L. Getoor, and B. Huang.
Query-driven active surveying for collective classification.
In ICML Workshop, 2012.

[Niculescu-Mizil and Caruana, 2005]
A. Niculescu-Mizil and R. Caruana.
Predicting good probabilities with supervised learning.
In ICML, 2005.

[Nowozin and Lampert, 2011]
S. Nowozin and C. H. Lampert.
Structured learning and prediction in computer vision.
Foundations and Trends® in Computer Graphics
and Vision, 60 (3–4):0 185–365, 2011.

[Oliver et al., 2018]
A. Oliver, A. Odena, C. A. Raffel, E. D. Cubuk, and I. Goodfellow.
Realistic evaluation of deep semi-supervised learning algorithms.
In NeurIPS, 2018.

[Pakdaman Naeini et al., 2015]
M. Pakdaman Naeini, G. Cooper, and M. Hauskrecht.
Obtaining well calibrated probabilities using bayesian binning.
In AAAI, 2015.

[Paszke et al., 2019]
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen,
Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito,
M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and
S. Chintala.
Pytorch: An imperative style, high-performance deep learning library.
In NeurIPS. 2019.

[Pei et al., 2020]
H. Pei, B. Wei, K. C.-C. Chang, Y. Lei, and B. Yang.
Geom-GCN: Geometric Graph Convolutional Networks.
In ICLR, 2020.

[Rahimi et al., 2020]
A. Rahimi, A. Shaban, C. Cheng, R. Hartley, and B. Boots.
Intra order-preserving functions for calibration of multi-class
neural networks.
In NeurIPS, 2020.

[Scott, 1992]
D. W. Scott.
Multivariate density estimation.
Wiley, 1992.

[Sen et al., 2008]
P. Sen, G. Namata, M. Bilgic, L. Getoor, B. Gallagher, and T. Eliassi-Rad.
Collective classification in network data.
AI Mag., 290 (3):0 93–106, 2008.

[Sensoy et al., 2018]
M. Sensoy, L. M. Kaplan, and M. Kandemir.
Evidential deep learning to quantify classification uncertainty.
In NeurIPS, 2018.

[Shchur et al., 2018]
O. Shchur, M. Mumme, A. Bojchevski, and S. Günnemann.
Pitfalls of graph neural network evaluation.
arXiv preprint arXiv:1811.05868, 2018.

[Stadler et al., 2021]
M. Stadler, B. Charpentier, S. Geisler, D. Zügner, and S. Günnemann.
Graph posterior network: Bayesian predictive uncertainty for node
In NeurIPS, 2021.

[Teixeira et al., 2019]
L. Teixeira, B. Jalaian, and B. Ribeiro.
Are graph neural networks miscalibrated?
In ICML Workshop, 2019.

[Tomani and Buettner, 2021]
C. Tomani and F. Buettner.
Towards trustworthy predictions from deep neural networks with fast
adversarial calibration.
In AAAI, 2021.

[Tomani et al., 2021]
C. Tomani, S. Gruber, M. E. Erdem, D. Cremers, and F. Buettner.
Post-hoc uncertainty calibration for domain drift scenarios.
In CVPR, 2021.

[Vaswani et al., 2017]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,
L. Kaiser, and I. Polosukhin.
Attention is all you need.
In NIPS, 2017.

[Veličković et al., 2018]
P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and
Y. Bengio.
Graph attention networks.
In ICLR, 2018.

[Wang et al., 2021]
X. Wang, H. Liu, C. Shi, and C. Yang.
Be confident! towards trustworthy graph neural networks via
confidence calibration.
In NeurIPS, 2021.

[Wen et al., 2018]
Y. Wen, P. Vicol, J. Ba, D. Tran, and R. Grosse.
Flipout: Efficient pseudo-independent weight perturbations on
arXiv preprint arXiv:1803.04386, 2018.

[Zhang et al., 2020]
J. Zhang, B. Kailkhura, and T. Y. Han.
Mix-n-match : Ensemble and compositional methods for uncertainty
calibration in deep learning.
In ICML, 2020.

[Zhang and Chen, 2018]
M. Zhang and Y. Chen.
Link prediction based on graph neural networks.
In NeurIPS, 2018.

[Zhang et al., 2019]
Y. Zhang, S. Pal, M. Coates, and D. Üstebay.
Bayesian graph convolutional neural networks for semi-supervised
In AAAI, 2019.

[Zhao et al., 2020]
X. Zhao, F. Chen, S. Hu, and J. Cho.
Uncertainty aware semi-supervised learning on graph data.
In NeurIPS, 2020.

[Zhu et al., 2020]
J. Zhu, Y. Yan, L. Zhao, M. Heimann, L. Akoglu, and D. Koutra.
Beyond homophily in graph neural networks: Current limitations and
effective designs.
In NeurIPS, 2020.

§ CHECKLIST

* For all authors...

* Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

* Did you describe the limitations of your work?
c.f. Section <ref>.

* Did you discuss any potential negative societal impacts of your work?

* Have you read the ethics review guidelines and ensured that your paper conforms to them?

* If you are including theoretical results...

* Did you state the full set of assumptions of all theoretical results?

* Did you include complete proofs of all theoretical results?

* If you ran experiments...

* Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

* Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

* Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

* Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

* If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

* If your work uses existing assets, did you cite the creators?

* Did you mention the license of the assets?

* Did you include any new assets either in the supplemental material or as a URL?

* Did you discuss whether and how consent was obtained from people whose data you're using/curating?

* Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

* If you used crowdsourcing or conducted research with human subjects...

* Did you include the full text of instructions given to participants and screenshots, if applicable?

* Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

* Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

§ EXPERIMENTAL SETTINGS

§.§ Dataset statistics

We consider eight real-world graph datasets including citation networks Cora [Sen et al., 2008], Citeseer [Sen et al., 2008], Pubmed [Namata et al., 2012], Coauthor CS [Shchur et al., 2018], Coauthor Physics [Shchur et al., 2018], and CoraFull [Bojchevski and Günnemann, 2018] together with Amazon co-purchase networks Computers [Shchur et al., 2018] and Photo [Shchur et al., 2018].
Table <ref> summarizes their statistics.

Benchmark dataset statistics.

Nodes    2,708    3,327    19,717    13,752    7,650    18,333    34,493    19,793

Edges    10,556    9,104    88,648    491,722    238,162    163,788    495,924    126,842

Features    1,433    3,703    500    767    745    6,805    8,415    8,710

Classes    7    6    3    10    8    15    5    70

Homophily    82.52%    70.62%    79.24%    78.53%    83.65%    83.20%    91.53%    58.61%

We report the homophily index proposed by Pei et al., 2020, which provides a global view of the neighborhood similarity for a graph. Given a graph $\mathcal{G} = (\mathcal{V} , \mathcal{E})$, the homophily is defined as

\begin{equation}
\mathcal{H}(\mathcal{G})=\frac{1}{|\mathcal{V}|}\sum_{i\in \mathcal{V}}\frac{\text{Number of node }i\text{'s neighbors who have the same label as } i}{\text{Number of }i\text{'s neighbors}}.
\label{eq:homo}
\end{equation}

§.§ Details of model training setup

We follow the setting of Shchur et al., 2018 to define GCN [Kipf and Welling, 2017] and GAT [Veličković et al., 2018] models.
Both models consist of 2 layers and the hidden dimension is fixed to 64.
For the multi-head layer in GAT, the number of attention heads is fixed to 8 with 8 hidden units per head.
We implement the models and training pipelines in PyTorch [Paszke et al., 2019] and PyTorch Geometric [Fey and Lenssen, 2019].
All models are trained for a maximum of 2000 epochs, using early stopping with a patience of 100 epochs.
We choose Adam [Kingma and Ba, 2014] as the optimizer with initial learning rate 0.01. We add a weight decay of 5e-4 for Cora, Citeseer, and Pubmed, and 0 for the rest.

We use stratified sampling to randomly select 15% of the nodes as observed set, mask out the output labels of the rest 85% of the nodes for test prediction, and ensure that the nodes with the same label are split proportionally.

Following Kull et al., 2019, Kull et al., 2017, we further divide the labeled set with three-fold cross-validation.
The bigger portions (10%) are used as training sets and the rest (5%) are used as validation sets.
The GNN models (GCN and GAT) are trained on the training set, then used to predict the masked-out test set.
Figure <ref> illustrates the aforementioned data partition in our experiments.
In total, we use 5 random data splits, three-fold cross-validation for each split, and 5 random model initializations per data partition, resulting in 75 total runs for each experiment.

at (1.5,0.75)5 splits, 5 initialization;
at (1,0.25)15%;
at (4,0.25)85%;
at (1,-0.25)Labeled;
at (4,-0.25)Unlabeled;

(0,0) – (6,0) – (6,0.5) – (0,0.5) – (0,0);
(2,0) – (2,0.5);

at (1.5,-1)3-fold cross-validation;
(0,-1.75) – (6,-1.75) – (6,-1.25) – (0,-1.25) – (0,-1.75);
(2,-1.75) – (2,-1.25);
(1.2,-1.75) – (1.2,-1.25);
at (0.6,-1.5)2/3;
at (1.6,-1.5)1/3;
at (4,-1.5)85%;
at (0.6,-2)Train;
at (1.6,-2)Val;
at (4,-2)Test;

Data partition schema for graph datasets.

§.§ Details of model calibration setup

We compare GATS with four scaling-based calibrators: temperature scaling (TS) [Guo et al., 2017], vector scaling (VS) [Guo et al., 2017], ensemble temperature scaling (ETS) [Zhang et al., 2020] , CaGCN [Wang et al., 2021].
The calibrators are trained on the validation set using the negative log-likelihood (NLL) loss and validated on the training set for early stopping and hyperparameter search.
The optimizer configuration and the training schedule are the same as Section <ref>.
We observe TS and VS using Adam with weight decay 0 achieves better performance than using L-BFGS [Liu and Nocedal, 1989] in the original implementation[<https://github.com/gpleiss/temperature_scaling>].
For ETS, we follow the official implementation that uses Sequential Least SQuares Programming (SLSQP) [Kraft, 1988].
For CaGCN, we use a two-layer GCN with 16 hidden units and choose the hyperparameters following the original paper.
For GATS, we utilize one message passing layer and
initialize $T_0$, $\gamma_t$, and $\gamma_n$ to 1 and $\omega$ to 0.
We find the best hyperparameter using cross-validation.
Table <ref> shows the search space for GATS hyperparameters.

Hyperparameter search space for GATS.

Search space

Weight decay    0, 1e-3, 5e-3, 1e-2, 5e-2, 1e-1, 2e-1, 3e-1

Initial $T_0$    1, 1.5

§ ADDITIONAL PLOTS

Here we include additional plots which shows the corresponding factors influencing the calibration of GAT models (c.f. Section <ref>). Overall, we reach the same conclusions as the GCN case.

§.§ General under-confident tendency for GAT

Figure <ref> summarizes the GAT results.
We see a general tendency of under-confident predictions (plots above the diagonal) except for the Physics dataset, which differs from the overconfident behavior of multiclass image classification using CNNs.

Reliability diagrams of GAT models trained on various graph datasets.

§.§ Diversity of node distributions for GAT

Figure <ref> shows the GAT results. Compared to the standard classification case, predictions of GAT also tend to be more spread out.

Entropy distributions of GAT predictions on graph datasets.

§.§ Effect of distance to training nodes for GAT

GAT result are shown in Figure <ref>. We also see that training nodes and their neighbors tend to be better calibrated.

Nodewise calibration error of GAT results w.r.t the minimum distance to training nodes.

§.§ Relative confidence level for GAT

The plots for the GAT case are shown in Figure <ref>. Similar to the GCN case, we observe that nodes which are less confident than their neighbors tend to be poorly calibrated and it is in general desirable to have a comparable confidence level w.r.t. the neighbors.

Nodewise calibration error of GAT results depending on the relative confidence level.

§.§ Neighborhood similarity for GAT

Figure <ref> shows tha GAT results. Analogue to the GCN case, nodes with strongly agreeing neighbors tend to have lower calibration errors.

Nodewise calibration error of GCN results depending on the node homophily.

§ ADDITIONAL CALIBRATION RESULTS

This section includes supplementary results with additional metrics and calibration methods.

§.§ Results using additional metrics

Variants of ECE
Even though ECE [Pakdaman Naeini et al., 2015, Guo et al., 2017] is the most commonly used metric for measuring calibration, it has some limitations:
(1) ECE only considers top-1 probabilistic output but can not reflect classwise calibration.
(2) The binning-based estimator [Pakdaman Naeini et al., 2015] is dependent on the choice of binning scheme.

To alleviate these disadvantages, we evaluate the trained calibrators using the classwise-ECE [Kull et al., 2019] and the kernel density estimation (KDE) based KDE-ECE [Zhang et al., 2020] which is a binning-free metric:

* Classwise-ECE measures the gap between the classwise average prediction $\text{conf}(B_{m,k})$ and the actual frequency of that class $\text{freq}(B_{m,k})$ in $M$ equally spaced bins across all classes $k \in K$:

\begin{align}
\text{Classwise-ECE} = \frac{1}{K}\sum_{k}^{K}\sum_{m}^{M}&\frac{|B_{m,k}|}{|\mathcal{N}|}|\text{freq}(B_{m,k}) - \text{conf}(B_{m,k})| \\
\text{freq}(B_{m,k})&=\frac{1}{|B_{m,k}|}\sum_{i\in B_{m,k}} \mathbf{1}(y_i=k) \\
\text{conf}(B_{m,k})&=\frac{1}{|B_{m,k}|}\sum_{i\in B_{m,k}}\hat{p}_{i,k}
\end{align}

where $\hat{p}_{i,k}$ denotes the probability of predicting class $k$ for sample $i$ in bin $B_{m,k}$.
We compute classwise-ECE using $M=15$ bins in our implementation.

* Instead of using a binning-based estimator, KDE-ECE uses a kernel function $K_h$ to estimate the accuracy $\tilde{\pi}(c)$ given confidence prediction $c$ and the marginal density function $f(c)$ of the predictive confidence:

\begin{align}
\text{KDE-ECE} = \int&\left| \tilde{\pi}(c) - c \right|\tilde{f}(c) dc\\
\tilde{\pi}(c) =& \frac{\sum\limits_{i \in \mathcal{N}}  \mathbf{1}(y_i=\hat{y}_i) K_{h}(c - \hat{c}_{i})}{\sum\limits_{i \in \mathcal{N}} K_{h}(c - \hat{c}_i)}\\
\tilde{f}(c) =& \frac{h^{-1}}{|\mathcal{N}|}\sum_{i \in \mathcal{N}}K_{h}(c - \hat{c}_{i})
\end{align}

where $i\in \mathcal{N} \subset \mathcal{V}$ denotes the evaluated node, and $h$ is the bandwidth of the kernel function.
We follow the official implementation[<https://github.com/zhang64-llnl/Mix-n-Match-Calibration>] of KDE-ECE, where the the Triweight Kernel $K_h(u)=(1/h)\frac{35}{32}(1-(u/h)^2)^3$ [de Haan, 1999] on $[-1,1]$  is chosen as the kernel function and bandwidth is calculated as $h=1.06\sigma |\mathcal{N}|^{-1/5}$  [Scott, 1992] with $\sigma$ being the standard deviation of the confidence.

The classwise-ECEs are summarized in Table <ref>, and the KDE-ECEs are collected in Table <ref>. In general, we observe similar conclusions as in the confidence ECE case (c.f. Section <ref>): Overall GATS achieves the state-of-the-art calibration results.

GNN calibration results in terms of classwise-ECE (in percentage, lower is better).

GCN    4.34$\pm$1.41    2.06$\pm$0.27    2.11$\pm$0.30

GAT    7.24$\pm$0.46

GCN    4.51$\pm$1.86    2.85$\pm$0.40

GAT    8.34$\pm$1.02    3.12$\pm$0.52

GCN    4.96$\pm$0.99

GAT    8.54$\pm$0.49    1.94$\pm$0.31

GCN    0.97$\pm$0.16    0.93$\pm$0.11    0.91$\pm$0.12

GAT    0.83$\pm$0.13

GCN    0.89$\pm$0.22

GAT    0.92$\pm$0.26

GCN    0.39$\pm$0.11

GAT    0.39$\pm$0.15

GCN    0.39$\pm$0.11

GAT    0.39$\pm$0.06

GCN    0.35$\pm$0.03

GAT    0.33$\pm$0.03

GNN calibration results in terms of KDE-ECE (in percentage, lower is better).

GCN    13.35$\pm$5.07    3.21$\pm$1.18    3.56$\pm$1.25

GAT    23.33$\pm$1.79    3.00$\pm$0.80

GCN    10.72$\pm$5.86    4.80$\pm$1.40

GAT    22.86$\pm$3.54

GCN    7.33$\pm$1.48    1.32$\pm$0.27    1.57$\pm$0.38

GAT    12.32$\pm$0.80    1.20$\pm$0.29

GCN    3.05$\pm$0.97     2.60$\pm$0.71    2.68$\pm$0.75

GAT    1.89$\pm$1.00

GCN    2.59$\pm$1.29

GAT    2.25$\pm$1.16

GCN    2.14$\pm$0.98

GAT    1.74$\pm$1.30

GCN    0.94$\pm$0.27

GAT    0.84$\pm$0.10

GCN    6.46$\pm$1.30

GAT    4.76$\pm$1.44

Non calibration metrics
Although not calibration metrics, we also report the results in terms of negative log-likelihood (Table <ref>) and Brier score (Table <ref>) for reference.

GNN calibration results in terms of negative log-likelihood $(\times 10^{-2})$.

GCN    62.90$\pm$5.68     56.37$\pm$3.12

GAT    75.67$\pm$2.37

GCN    90.13$\pm$6.00

GAT    100.90$\pm$6.33

GCN    39.31$\pm$1.47

GAT    46.87$\pm$0.98

GCN    42.96$\pm$1.21

GAT    37.26$\pm$1.53

GCN    28.92$\pm$1.20

GAT    26.83$\pm$1.78

GCN    21.85$\pm$0.74

GAT    24.76$\pm$1.46

GCN    11.95$\pm$0.41

GAT    12.88$\pm$0.41

GCN    143.07$\pm$2.02

GAT    139.77$\pm$2.16

GNN calibration results in terms of Brier score $(\times 10^{-2})$.

GCN    28.68$\pm$2.54

GAT    34.47$\pm$1.21

GCN    42.56$\pm$2.82

GAT    47.42$\pm$3.26

GCN    21.31$\pm$0.71

GAT    25.33$\pm$0.56

GCN    18.57$\pm$0.80

GAT    16.79$\pm$0.80

GCN    11.72$\pm$0.66

GAT    11.52$\pm$0.88

GCN    10.28$\pm$0.27

GAT    11.36$\pm$0.58

GCN    6.13$\pm$0.20

GAT    6.54$\pm$0.18

GCN    52.32$\pm$0.68

GAT    51.73$\pm$0.77

§.§ Results for additional baselines

While in the main paper we focus on “temperature scaling style” methods which directly rescale the output logits, here we compare with the following additional calibration methods which have different principles. These methods are all designed for multi-class classification and do not consider the structural information of the graph.

* Multi-class isotonic regression (IRM) [Zhang et al., 2020] is a multi-class generalization of the non-parametric isotonic regression method;

* Calibration using spline (Spline) [Gupta et al., 2021] fits the calibration function with splines;

* Dirichlet calibration (DIR) [Kull et al., 2019] uses the Dirichlet distribution to model the distribution of probabilistic outputs. It also employs an off-diagonal and intercept regularization (ODIR);

* Order invariant calibration (OI) [Rahimi et al., 2020] is the order-invariant intra order-preserving model. It uses sorted output logits as calibration input and builds up a neural network with special structures to preserve the accuracy and the intra order of the predicted logits.

The authors of spline calibration specify how to calibrate a specific class or a chosen top-$r$ class, and in their implementation[<https://github.com/kartikgupta-at-anu/spline-calibration>] they focus on calibrating the top-$1$ class. However, it is not clear how to adjust the rest of the predictions to ensure valid probabilistic predictions after calibration. We adopt a heuristic which proportionally rescales the non top-$1$ output probabilities so that the calibrated probabilistic output sums up to one. Also, the authors wrongly claimed that calibrating the top-$1$ score “does not alter the classification accuracy” [Gupta et al., 2021].
In practice, the score after calibration might no longer remain top-$1$ and the predictions could be altered.

For Dirichlet calibration, we find out that the scaling factors $\lambda,\mu$ of ODIR affect the performance and need to be tuned depending on the dataset. Thus we do a hyperparameter search for each dataset with search space $(0.01, 0.1, 1, 10, 100, 1000, 10000, 100000)$.

Table <ref> summarizes the calibration results in terms of ECE. Note that we do not include the results for Dirichlet calibration on CoraFull because it fails to calibrate the GNN backbones and significantly deteriorates the predictive accuracies ($< 10 \%$ v.s. $>60\%$ before calibration).

Overall we observe that GATS still achieves the state-of-the-art performance compared to the additional baselines for GNN calibration.

GNN calibration results (ECE, in percentage) with additional baselines.

GCN    13.04$\pm$5.22

GAT    23.31$\pm$1.81

2*Citeseer     GCN    10.66$\pm$5.92

GCN    1.65$\pm$0.92

§.§ Accuracies of calibration methods

Since many baseline calibration methods are not accuracy-preserving, in Table <ref> we additionally report their test accuracies. Accuracy preserving methods (GATS, TS, ETS, CaGCN, OI) have the same accuracies as the uncalibrated case, which is also reported for reference.

Test accuracies of uncalibrated results (identical to those from accuracy-preserving methods) and calibrated predictions from non accuracy-preserving methods.

GCN    82.78$\pm$0.79

GAT    81.98$\pm$0.92

GCN    72.19$\pm$0.82

GAT    72.37$\pm$0.68

GCN    86.40$\pm$0.27

GAT    84.46$\pm$0.34

GCN    88.13$\pm$0.56

GAT    89.05$\pm$0.60

GCN    92.65$\pm$0.38

GAT    92.65$\pm$0.54

GCN    93.33$\pm$0.15

GAT     92.57$\pm$0.25

GCN    95.99$\pm$0.14

GAT    95.70$\pm$0.13

GCN    63.07$\pm$0.50

GAT    63.00$\pm$0.59

§ DATA EFFICIENCY AND EXPRESSIVENESS OF GATS: GAT RESULTS

Figure <ref> shows the results for the GAT case. We see that GATS is also data efficient and expressive when calibrating GAT models.

ECEs (in percentage) on CoraFull with GAT backbone for ETS, CaGCN, and GATS using various amounts of calibration data. Again, We observe that GATS is data-efficient and expressive. Here CaGCN fails to calibrate the GAT model on CoraFull.

§ CAGCN RESULTS DISCUSSION

While the ECEs of CaGCN in its original paper are promising [Wang et al., 2021], we observe that the ECEs of CaGCN are often unstable and sometimes even worse than that of the uncalibrated model in our experiments.
One possible reason is that we use a different splitting from the CaGCN paper, where they follow a fixed splitting from Kipf and Welling, 2017.
A significant difference is that the splitting from Kipf and Welling, 2017 has more validation nodes than the training nodes.
This differs from typical real-world applications, where the larger fold would often be used to train a good classifier [Oliver et al., 2018], and only the smaller fold is available for fitting the calibrator.

In our splitting, the validation sets of Cora and Citeseer are substantially smaller than those in Kipf and Welling, 2017. We observe that CaGCN yields suboptimal calibration results (see Section <ref>) and predictions with higher confidence tend to be over-confident in the reliability diagram in Figure  <ref>.
By contrast, the validation set in Pubmed is relatively large since it has more nodes.
We notice that CaGCN achieves competitive results in Pubmed and the confidence-accuracy curve almost lies on the diagonal.
We observe that CaGCN also produces suboptimal calibration results in CoraFull, even though the validation set is large.
We suspect that this is caused by the class imbalance of the CoraFull data. Class imbalance is known to be a challenge for many calibration methods [Teixeira et al., 2019].

Top row: Reliability diagram of CaGCN for GCN trained on Cora, Citeseer, and Pubmed. Botom row: Reliability diagram of GATS for GCN trained on Cora, Citeseer, and Pubmed as a reference. We observe that CaGCN is noticeably over-confident in the high confidence region when the size of the validation set is relatively small (e.g., Cora and Citeseer).

§ GATS WEIGHT VISUALIZATION

GATS learns $\tau^h_i$ from sorted logits. We discover that the absolute value of the learned weights in the linear layer generally follows the ranking of the logits across the class.
That is to say, logits with higher value have stronger influence to $\tau^h_i$. In Figure <ref> we visualize the weights $\theta = (\theta^1, \dots, \theta^H)$ of the linear layers $\phi^h$ in GATS.
Here, it is interesting to see that the weights $\theta^h$ from different heads $h$ have slight variations.
Combining multiple heads in the attention with sorting could be considered as a form of ensemble without the model being overly parameterized.

Visualization of the GATS weight $\theta$ on Cora (7 classes) and CoraFull (70 classes).

§ ANALYSIS OF CORRELATIONS BETWEEN THE FACTORS

In this section we visualize the correlation between the local-view factors: the distance to training nodes, the relative confidence level, and the neighborhood similarity.

Each plot shows how the factor on the y-axis varies when the factor on the x-axis is fixed to a given value. Two factors are independent when we observe a horizontal line in the plot.
As the relative confidence level is a model dependent factor, GCN and GAT will have different correlation plots when it is involved.

§.§ Distance to training nodes – relative confidence level

Figures <ref>, <ref>, <ref>, and <ref> show the correlation plots between the distance to training nodes and the relative confidence level. In Figure <ref> and <ref> we see that regardless of the distance to the training nodes, the averaged relative confidence level stays around zero.

Relative confidence level of GCN results depending on the minimum distance to training nodes.

Minimum distance to training
nodes depending on the relative confidence level of GCN predictions.

Relative confidence level of GAT results depending on the minimum distance to training nodes.

Minimum distance to training
nodes depending on the relative confidence level of GAT predictions.

§.§ Relative confidence level – neighborhood similarity

Figures <ref>, <ref>, <ref>, and <ref> show the correlations between the relative confidence level and the neighborhood similarity. We observe some partial correlation between these two factors, especially in the negative region of the node homophily.

Relative confidence level of GCN predictions depending on the node homophily.

Node homophily depending on the relative confidence level of GCN predictions.

Relative confidence level of GAT predictions depending on the node homophily.

Node homophily depending on the relative confidence level of GAT predictions.

§.§ Distance to training nodes – neighborhood similarity

Figures <ref> and <ref> show the correlation between the distance to training nodes and neighborhood similarity.
Note that these two factors are not model-dependent and thus GCN and GAT share the same results.
We observe that these two factors have a less significant correlation.

Node homophily depending on the minimum distance to training nodes.

Minimum distance to training nodes depending on the node homophily.

§ FACTOR NODE COUNT ANALYSIS

In this section we plot the number of test nodes depending on the three local view factors: distance to training nodes, relative confidence level, and neighborhood similarity.

§.§ Node count for distance to training nodes

Figure <ref> summarizes the node count results of the distance to training nodes.
We see the majority of nodes can be connected to the training nodes by one or two hops.

Number of nodes for the minimum distance to training nodes.

§.§ Node count for relative confidence level

Figure <ref> and <ref> are the node count results of the relative confidence level for GCN and GAT respectively.
We observe that most of the nodes are concentrated around the zero relative confidence level.

Number of nodes for the relative confidence level of GCN predictions.

Number of nodes for the relative confidence level of GAT predictions.

§.§ Node count for neighborhood similarity

Figure <ref> shows that the majority of nodes lie in the positive homophily region.

Number of nodes for the node homophily.