Diverse explainability methods of graph neural networks (GNN) have recently been developed to highlight the edges and nodes in the graph that contribute the most to the model predictions. However, it is not clear yet how to evaluate the correctness of those explanations, whether it is from a human or a model perspective. One unaddressed bottleneck in the current evaluation procedure is the problem of out-of-distribution explanations, whose distribution differs from those of the training data. This important issue affects existing evaluation metrics such as the popular faithfulness or fidelity score. In this paper, we show the limitations of faithfulness metrics. We propose GInX-Eval (Graph In-distribution eXplanation Evaluation), an evaluation procedure of graph explanations that overcomes the pitfalls of faithfulness and offers new insights on explainability methods. Using a retraining strategy, the GInX score measures how informative removed edges are for the model and the EdgeRank score evaluates if explanatory edges are correctly ordered by their importance. GInX-Eval verifies if ground-truth explanations are instructive to the GNN model. In addition, it shows that many popular methods, including gradient-based methods, produce explanations that are not better than a random designation of edges as important subgraphs, challenging the findings of current works in the area. Results with GInX-Eval are consistent across multiple datasets and align with human evaluation.

§ INTRODUCTION

While people in the field of explainable AI have long argued about the nature of good explanations, the community has not yet agreed on a robust collection of metrics to measure explanation correctness. Phenomenon-focused explanations should match the ground-truth defined by humans and are evaluated by the accuracy metric. Model-focused explanations contribute the most to the model predictions and are evaluated by the faithfulness metrics. Because ground-truth explanations are often unknown, faithfulness and its variants are the most common measures of quality. Faithfulness metrics remove or retrain only the important graph entities identified and observe the changes in model outputs. However, this edge masking strategy creates Out-Of-Distribution (OOD) graph inputs, so it is unclear if a high faithfulness score comes from the fact that the edge is important or from the distribution shift induced by the edge removal [Günnemann, 2022].

We propose GInX-Eval, an evaluation procedure of in-distribution explanations that brings new perspectives on GNN explainability methods. Testing two edge removal strategies, we evaluate the impact of removing a fraction $t$ of edges in the graph on the GNN model performance. To overcome the OOD problem of faithfulness metrics, the model is retrained and tested on the reduced graphs at each degradation level. The best explainability methods can identify the graph entities whose removal triggers the sharpest model accuracy degradation. We compare generative and non-generative methods on their GInX score against a random baseline across four real-world graph datasets and two synthetic datasets, all used for graph classification tasks. With this strategy, we show that existing explainers are not better than random in most of the cases. In addition, we show the overall superiority of GNNExplainer, PGMExplainer, and most of the generative methods above gradient-based methods and Occlusion. Our results lead to diverging conclusions from recent studies [Yuan et al., 2023, Agarwal et al., 2022] and again question the use of faithfulness as a standard evaluation metric in explainable AI (xAI). The GInX-Eval framework also proposes the EdgeRank score to assess the capacity of explainers to correctly order edges by their true importance for the model. Finally, GInX-Eval is a useful tool to validate ground-truth explanations provided with some datasets and discover both human- and model-based explanations. Due to the computational cost of re-training, GInX-Eval is not proposed as a systematic evaluation metric but as a tool to throw light on the true informative power of explainers and validate ground-truth explanations.
To summarize our contributions:

* We first show the limitations of faithfulness, namely that (1) it evaluates out-of-distribution explanations, (2) it is inconsistent with the accuracy metric, (3) it leads to divergent conclusions across datasets, and (4) across edge removal strategies. Overcoming those limitations, we propose GInX-Eval (Graph In-distribution eXplanation Evaluation), a new evaluation framework for GNN explainability methods. The GInX score evaluates how informative explanatory edges are to the model and the EdgeRank score assesses if those edges are correctly ordered by their importance.

* We propose a validation protocol of ground-truth explanations using the GInX score. This way, we can measure the degree of alignment between human-based and model-based explanations.

* With GInX-Eval, we now finally demonstrate the true informative power of well-known explainability methods, filter out bad methods, and choose methods that can correctly rank edges.

The rest of this article is organized as follows. Section <ref> discusses the literature on graph neural networks (GNN) explainability evaluation and the OOD problem. Section <ref> presents the limitations of the current evaluation with faithfulness and introduces GInX-Eval, our new in-distribution evaluation procedure, and its two scores, GInX and EdgeRank. Section <ref> presents the experiments that we conducted in detail. Section <ref> summarizes the paper and discusses the future opportunities.

§ RELATED WORK

Evaluation in xAI. To measure the correctness of explanations, a few metrics have been developed. GNN explainability method should satisfy accuracy, faithfulness, stability [27, Yuan et al., 2023, Agarwal et al., 2021, Agarwal et al., 2022], consistency and contrastivity [Yuan et al., 2023], usefulness [Colin et al., 2021]. The two most popular approaches are: (1) measuring accuracy using ground-truth annotations and (2) measuring faithfulness using objective metrics [Chan et al., 2022]. However, the accuracy metric also referred to as plausibility [Li et al., 2022, Longa et al., 2022, Nauta et al., 2022], needs ground-truth explanations and is therefore less universal and more subjective than faithfulness. [Nauta et al., 2022] argues that evaluating the accuracy of an explanation to humans is different from evaluating its correctness. It is not guaranteed that it aligns with its faithfulness [Jacovi & Goldberg, 2020]. According to [Petsiuk et al., 2018], it is preferable to keep humans out of the evaluation to better capture the model's understanding rather than representing the human's view. Faithfulness metrics are therefore the most popular evaluation metrics, but we show later in Section <ref> that they have strong limitations including evaluating out-of-distribution explanations.

Solving the OOD problem. Recent works have proposed to adapt the GNN model or develop robust explainability methods to overcome the out-of-distribution (OOD) problem. [Faber et al., 2020] argue that explanations should stay in the training data distribution and propose CoGe to produce Distribution Compliant Explanation (DCE). [Li et al., 2021] propose a novel out-of-distribution generalized graph neural network. [13] do not remove features but apply small adversarial changes to the feature values. Instead of developing robust methods, [Hooker et al., 2018] evaluate interpretability methods by observing how the performance of a retrained model degrades when removing the features estimated as important. While this retraining strategy circumvents the OOD problem, it has only been developed for CNN models on images to evaluate feature importance. Building on this retraining strategy, we propose a new evaluation procedure for GNN models and introduce an alternative to faithfulness metrics.

§ METHOD

This section highlights the limitations of the popular xAI evaluation procedure using faithfulness metrics and proposes GInX-Eval to overcome those. We can assess the informativeness of explanations for the model with the GInX score and the capacity of methods to correctly order explanatory edges by their importance with the EdgeRank score.

§.§ Preliminaries

Given a well-trained GNN model $f$ and an instance of the dataset, the objective of the explanation task is to identify concise graph substructures that contribute the most to the model's predictions. The given graph can be represented as a quadruplet $G(\mathcal{V},\mathcal{E}, \mathbf{X}, \mathbf{E})$, where $\mathcal{V}$ is the node set, $\mathcal{E}\subseteq \mathcal{V}\times \mathcal{V}$ is the edge set. $\mathbf{X}\in\mathbb{R}^{|\mathcal{V}|\times d_n}$ and $\mathbf{E}\in\mathbb{R}^{|\mathcal{E}|\times d_e}$ denote the feature matrices for nodes and edges, respectively, where $d_n$ and $d_e$ are the dimensions of node features and edge features. In this work, we focus on structural explanation, we keep the dimensions of node and edge features unchanged. Given a well-trained GNN $f$ and an instance represented as $G(\mathcal{V},\mathcal{E},\mathbf{X},\mathbf{E})$, an explainability method generates an explanatory edge mask $M\in\mathbb{R}^{|\mathcal{E}|}$ that is normalized. Furthermore, to obtain a human-intelligible explanation, we transform the edge mask to a sparse array $M_e$ by forcing to keep only the fraction $t$ of the highest values and set the rest of the matrix values to zero.

Edge removal strategies. There are two strategies to select edges in a graph: the hard selection and the soft selection. Hard selection picks edges from the graph so that the number of edges and nodes is reduced. This creates subgraphs that very likely do not lie in the initial data distribution. Soft selection sets edge weights to zero when edges are to be removed. Therefore it preserves the whole graph structure with all nodes and edge indices. Following these two definitions, we define hard and soft explanations.

§.§ Limitations of Faithfulness

Definition. The faithfulness or fidelity scores are the most general quality metrics in the field of GNN explainability. To evaluate the correctness of the explanation, the explanatory subgraph or weighted graph $G_e$ is given as input to the model to compute the fidelity score on the probabilities:

\begin{align}
fid = |p(\hat{y}_{G_e}=y) - p(\hat{y}_{G}=y)| \in [0;1]
\end{align}

where $y$ is the true label for graph $G$ and $\hat{y}_{G}$, $\hat{y}_{G_e}$ the predicted labels by the GNN given $G$ and $G_e$ respectively. The closer $fid$ is to 0, the more faithful the explanation is. The faithfulness score is averaged over the N explanatory graphs $G_{e,i},i\leq N$ as:

\begin{align}
\text{Faithfulness} = 1 - \frac{1}{N} \sum_{i=1}^{N}\left|p(\hat{y}_{G_{e,i}}=y_i) - p(\hat{y}_{G_i}=y_i)\right| \in [0;1]
\end{align}

The metric is normalized and the closer it is to 1, the more faithful the evaluated $N$ explanations are to the initial predictions. The above score corresponds to the $fid-^{prob}$, one of the four forms of the fidelity scores [Yuan et al., 2023], described in Appendix <ref>.

Prior work. While faithfulness metrics are the most popular quality measures independent of ground-truth explanations, they have been recently criticized. Based on a “removal” strategy, i.e., we remove or keep the graph entities estimated as important, faithfulness withdraws some entities by setting them to a baseline value either removing them from the graph or putting their weight to zero. [13] correctly observe that this evaluation procedure favors graph entities that are far away from the baseline. Consequently, methods that focus on highly weighted edges while maybe ignoring low-weight but still essential connections are favored. In addition, truncated graphs after edge removal can lie out of the data distribution used for training the GNN model [Hooker et al., 2018]. In this case, model behavior might differ not because of removing important information but because of evaluating a sample outside the training distribution. The out-of-distribution risk is even larger with graphs because of their discrete nature [Faber et al., 2020].

Out-Of-Distribution Explanations. The biggest limitation of the faithfulness metrics is the so-called OOD problem. The model prediction is altered not necessarily because we keep only the important entities but also because the model lacks knowledge about these new explanatory graphs. Therefore, the faithfulness score does not entirely capture the quality of explanations and is ill-suited to evaluate explainability methods.

§.§ GInX-Eval

To evaluate an explainability method, GInX-Eval first generates explanations as edge masks for all graph instances. From each explanatory mask, we deduce an edge ranking. The most important edges have a weight close to 1 in the mask, while the least important ones have weights close to 0. For the top $t$ fraction of the ordered edge set, we remove the edges following one of the two edge removal strategies. We generate new train and test sets at different degradation levels $t\in [0.1,0.2,...,1]$ where t is the fraction of removed edges. Afterward, the GNN model is retrained from random initialization on the new dataset and evaluated on the new test data. GInX-Eval requires retraining the model on the modified dataset rather than re-scoring the modified graph at inference time like faithfulness evaluation. While being the most computationally expensive aspect of GInX-Eval, it is scalable (see Appendix <ref>) and we argue that it is a necessary step to decouple whether the model’s degradation in performance is due to the loss of informative edges or due to the distribution shift. The objective here is not to provide a computationally efficient evaluation metric but to highlight the limitations of popular evaluation in xAI for GNN and question the superiority of gold standard methods.

A drop in test accuracy when removing edges indicates that those edges were important for the model to make correct predictions. These edges are therefore considered as important as they are the most informative to the model. It is worth noticing that edges might be correlated and those spurious correlations can lead to an absence of accuracy drop when removing the top important edges and then a sudden decrease in accuracy when all correlated edges are removed.

§.§.§ GInX Score

Following this evaluation procedure, we define the GInX score at $t$. It captures how low the test accuracy is after removing the top $t$ edges. Let $G_e^t$ be the hard or soft explanatory graph containing the top $t$ edges according to the explanatory mask $M_e$, we define GInX$(t)$ as:

\begin{align}
\text{GInX}(t) = 1 - \text{TestAcc}(t) = 1 - \frac{1}{N_{test}} \sum_{i=0}^{N_{test}} \mathds{1}(\hat{y}_i^{G\setminus G_e^t}=y_i)
\end{align}

The closer the GInX score is to one, the more informative the removed edges are to the model. Note here that the GInX score at $t$ can be computed following the hard or soft edge removal strategy; however, we show in Appendix <ref> that the GInX score computed with hard edge removal has higher expressiveness.

§.§.§ EdgeRank Score
Based on the GInX score, we can compute the power of explainability methods to rank edges, i.e., to correctly order edges based on their importance. The edge ranking power can be evaluated with the EdgeRank score defined as:

\begin{align}
\text{EdgeRank} = \sum_{t=0,0.1,...,0.8} (1-t)\times(\text{GInX}(t+0.1)-\text{GInX}(t))
\end{align}

A high edge ranking score indicates methods that assign the highest importance weights to the most genuinely informative edges for the model. This is especially important when you try to characterize an explanation and identify fundamental entities within the explanatory substructure.

§ EXPERIMENTAL RESULTS

In the following section, we propose to validate the GInX-Eval procedure and show its superiority over the widely used faithfulness metric. We support our claims with well-chosen experiments.

Experimental setting
Explainability methods were evaluated on two synthetic datasets, BA-2Motifs and BA-HouseGrid, three molecular datasets MUTAG, Benzene and BBBP, and MNISTbin. They all have ground-truth explanations available except for the BBBP dataset. We test two GNN models: GIN [Hu et al., 2020] and GAT [Veličković et al., 2018] because they score high on the selected real-world datasets, with a reasonable training time and fast convergence. For the two synthetic datasets, we only use GIN since the GAT model does not give good accuracy. Further details on the datasets, GNN training parameters, and time are given in Appendix <ref>. We compare non-generative methods, including the heuristic Occlusion [Zeiler & Fergus, 2014], gradient-based methods Saliency [Baldassarre & Azizpour, 2019], Integrated Gradient [Sundararajan et al., 2017], and Grad-CAM [Pope et al., 2019], and perturbation-based methods GNNExplainer [Ying et al., 2019], PGMExplainer [Vu & Thai, 2020] and SubgraphX [Yuan et al., 2021]. We also consider generative methods: PGExplainer [Luo et al., 2020], GSAT [Miao et al., 2022], GraphCFE (CLEAR) [Ma et al., 2022], D4Explainer and RCExplainer [Bajaj et al., 2021]. For more details on the differences between generative and non-generative explainers, we refer the reader to Appendix <ref>. We compare those explainability methods to base estimators: Random, Truth, and Inverse. Random assigns random importance to edges following a uniform distribution. Truth estimates edge importance as the pre-defined ground-truth explanations of the datasets. The Inverse estimator corresponds to the worst-case scenario where edges are assigned the inverted ground-truth weights.

§.§ Inconsistencies with Faithfulness

We compare the rankings of explainability methods according to the faithfulness evaluated on the two types of explanations, Hard Fidelity and Soft Fidelity, respectively. We also evaluate their accuracy with the AUC score, i.e., the area under the ROC curve between the computed explanatory edge mask and the ground-truth edge mask, to stay consistent with previous work [Longa et al., 2022]. The base estimators, i.e., Truth, Random, and Inverse, were evaluated on the full explanatory masks, while the explainability methods were evaluated on the truncated explanations, keeping the top 10 important undirected edges. The reason is that expected explanations are usually sparse: we select top $k$ edges to produce human-intelligible explanatory subgraphs. For the BBBP dataset, the accuracy score is not reported due to the absence of a ground-truth explanation for this dataset.

Rankings of base and non-generative explainability methods according to the faithfulness score computed on soft explanations, the faithfulness score on hard explanations, and the AUC score. The AUC ranking is only reported for datasets with ground-truth explanations.

Claim 1 The faithfulness metric is not consistent with the accuracy score. For the datasets with ground-truth explanations available, i.e., all except BBBP, we can compare the rankings of the methods according to the faithfulness metric, i.e., Hard Fid, and the accuracy metric, i.e., AUC score, and report the results on figure <ref>. We observe a consistent ranking between Hard Fid and AUC score only for the Benzene dataset. For the other four datasets, we always observe that the Truth estimator, whose accuracy is maximal, has a small faithfulness score $\sim 0.5$. In addition, we also observe strong misalignment in the rankings for the rest of the methods based on faithfulness or AUC score. For instance, for MNISTbin, Inverse is by far the best according to the faithfulness score while being the worst explainer by definition on the AUC score. For BA-2Motifs, Random has the highest faithfulness score but can only be 50% accurate by definition.

Claim 2 The evaluation of the explainability methods with the faithfulness metric is not consistent across datasets. Figure <ref> shows no consensus on the top-3 methods across datasets on the soft faithfulness or hard faithfulness score. For instance, we observe that GradCAM and Occlusion have the highest Soft Fid scores for BA-House-Grid, MUTAG, Benzene, and MNISTbin, but not for BA-2Motifs and BBBP where Truth, Random, and GNNExplainer outperform. For Hard Fid, the results are also very heterogeneous among the six datasets.

Claim 3 The faithfulness metric is not consistent across edge removal strategies. On figure <ref>, the top-3 ranking for Soft Fid and Hard Fid is always different except for Benzene dataset. This means that the edge removal strategy influences the model perception: the model does not predict labels only based on the information contained in the explanatory edges but also based on the structure of the given explanations.

§.§ Validation of GInX-Eval Procedure

We validate the GInX-Eval procedure on the BA-HouseGrid synthetic dataset because ground-truth explanations, i.e., house and grid motifs, are very well-defined and class-specific. In the binary classification setting, graphs are labeled 1 if they have grids and 0 if they have house motifs attached to the main Barabási graph. We test three explainability baselines: the Random explainer that assigns values in $[0,1]$ following a uniform distribution, the Truth that assigns ground-truth explanations, and the Inverse estimator that returns the inverse ground-truth explanations and is, therefore, the worst estimator possible.

On Figure <ref>, GInX-Eval distinguishes the three methods because we observe a sharp decrease of the Truth explainer after 10% edge removal, while the Inverse estimator does not degrade the model performance, and the Random baseline starts decreasing after 20% of the edges are removed. Without re-training, all base importance estimators lead to a model performance degradation. Therefore, evaluating without retraining the model cannot reflect the true explainability power of the methods.

Comparison between not retraining the GNN model and GInX-Eval on the BA-HouseGrid dataset. Without re-training, the model's performance also decreases for the worst estimator Inverse where uninformative edges are removed first, preventing a correct evaluation of explainability methods. However, for GInX-Eval where the model is retrained on modified datasets, we observe no test accuracy degradation for the worst-case estimator Inverse.

§.§ Evaluating with GInX-Eval

§.§.§ Overview

Test accuracy of a retrained GIN model on graphs with increasing fraction of removed edges. The removed edges are the most important based on explainability methods, and new input graphs are obtained with the hard selection, i.e., explanatory edges are strictly removed from the graph so that the number of edges and nodes is reduced. For more methods, see Appendix <ref>.

GInX-Eval evaluates to what extent removing explanatory edges degrades the model accuracy. We adopt the hard selection strategy to remove edges. Even if conclusions are similar for both selection strategies (see Appendix <ref> and <ref>), the degradation is of the order of $10^{-1}$ with hard selection versus $10^{-2}$ for soft selection. For visualization purposes, we prefer conveying here results with the hard selection. We refer the reader to Appendix <ref> for additional results with the soft selection.

Figure <ref> shows how the test accuracy diminishes when we remove a fraction $t\in [0.1,...,0.9]$ of the most important edges according to multiple explainability methods. For clarity, we choose to display a smaller set of explainability methods. For more methods, we refer the reader to Appendix <ref>. We first observe that model performance is remarkably robust to graph modification for the four real-world datasets, with a maximum drop of 30% accuracy observed for the Benzene dataset. For the synthetic datasets, removing many edges leads to a random assignment of labels by the model. In real-world datasets, GNN models might be able to capture high-level information even with absent connections.

We note a particularly small performance drop for MNISTbin, i.e., in the order of $10^{-2}$. For this dataset, the GNN model is robust to edge modification. After removing most of the edges from the input graph, the model retains most of the predictive power. The reason might be that node and edge features are more important for the prediction than the graph structure itself for those two datasets.

§.§.§ GInX-Eval of Base estimators

Is the ground-truth explanation meaningful to the model? The Truth and the Inverse edge importance estimators are evaluated on all datasets except BBBP which has no ground-truth available. We observe in figure <ref> that the test accuracy stays constant for Inverse and drops significantly for Truth. We conclude that the explanations generated with Inverse have only uninformative edges for the model, while the ground-truth edges contain crucial information for making correct predictions. GInX-Eval is a useful tool to validate the quality of provided ground-truth explanations of published graph datasets.

Does a random assignment of edge importance informative to the model? For all datasets except Benzene, the Random baseline leads to a similar degradation as the Truth estimator in figure <ref>. There are two reasons for this. First, random explanations contain a few edges present in the ground-truth explanation. Removing just these few edges makes the accuracy drop sharply because of the strong correlations that exist among informative edges. Second, true informative edges might have correlations to some other random edges, so removing edges randomly affects the capacity of the model to correctly learn important patterns in the graph.

Are explanations obtained with graph explainability methods better than a random guess? We observe that a random edge modification removes more informative edges than GradCAM, Integrated Gradient, Occlusion, RCExplainer, and PGExplainer. Therefore, those methods are not better than Random.

GInX-Eval identifies how informative ground-truth explanations are for the model, thus assessing the agreement between model and human-based explanations, and draws attention to how much random explanations are meaningful to the model.

§.§.§ GInX-Eval of Explainability methods

What fraction $t$ of removed edges should I fix to compare the GInX score of multiple explainability methods? Methods produce explanations of different sizes: some methods constrain their explanations to be sparse, while others assign importance weight to almost all edges in the graph. Figure <ref> indicates the heterogeneity of masks generated by different explainability methods. While Truth, PGMExplainer, SubgraphX, and GraphCFE constrain their explanations to be sparse, the rest of the methods include most of the edges in the explanations, assigning a different importance weight to each edge.

Mask sparsity of different explainability methods. A high sparsity indicates an explanatory mask with many zeros and small pre-processing explanatory subgraphs.

BA-2Motifs       BA-HouseGrid     Benzene          MNISTbin            MUTAG
Truth sparsity     $0.216$     $0.065$     $0.175$     $0.235$     $0.039$

Optimal threshold     $0.3$     $0.1$     $0.2$     $0.3$     $0.1$

Truth mask sparsity values for each dataset and the deduced optimal thresholds.

The critical threshold $t^{c}_m$ of a method $m$ is the ratio of non-zero values in the masks. Beyond this critical threshold, we are not evaluating the method anymore but a random assignment of edge importance weights. Therefore, it is crucial to compare methods at a threshold $t$ smaller than the minimum of the methods' critical thresholds. To compare methods, we propose to define the dataset's optimal threshold $t^{*}$ such as $t^* = min_{m\in\mathcal{M}} \{t^{c}_{m}\}$, where $\mathcal{M}$ denotes the set of explainability methods. The optimal threshold corresponds to the threshold closest to the average mask sparsity of ground-truth explanations. In other words, we take as reference the size of ground-truth explanations as the optimal number of informative edges in the graph and compare methods at this sparsity threshold. We compute the optimal thresholds for the six datasets and report them in table <ref>. Only the BBBP dataset has no ground-truth explanation available so we set $t^*=0.3$ to have human-intelligible sparse explanations.

GInX scores at the optimal thresholds. For the BBBP dataset, we define an arbitrary optimal threshold $t=0.3$. For the other datasets, the optimal threshold is estimated based on the explanatory mask sparsity generated by the Truth estimator.

Figure <ref> displays the GInX scores of explainability methods at the optimal threshold defined for each dataset. Except for the Benzene dataset, we observe that gradient-based methods and Occlusion have the smallest GInX scores at the optimal thresholds. Gradient-based methods contain less informative edges than GNNExplainer, PGMExplainer, and generative methods. This contradicts observations made in figure <ref> where gradient-based methods and Occlusion are always better than GNNExplainer and PGMExplainer. GInX-Eval unveils new insights on gradient-based methods that go against recent studies [Yuan et al., 2023, Agarwal et al., 2022]. On the other hand, GNNExplainer, PGMExplainer, GSAT, and D4Explainer have competitive performance with Random and Truth baselines. This proves that generative methods are not necessarily better at capturing meaningful information than non-generative methods.

The GInX score at the optimal threshold helps filter out uninformative methods including gradient-based methods and Occlusion, and shows that methods can generate informative explanations independent of their generative nature.

§.§.§ EdgeRank score of explainability methods

We use the EdgeRank score to evaluate the capacity of explainers to rank edges correctly according to their true informativeness for the model. In figure <ref>, we observe that gradient-based methods and Occlusion are not good at correctly ordering edges by their importance. This is another reason why they should not be used to generate meaningful explanations. We also observe that RCExplainer and PGExplainer which perform well on the GInX score have a low edge ranking power, except for the BA-HouseGrid dataset. These two methods can capture the most informative edges but cannot decide what the relative importance of those important edges is. Finally, PGMExplainer, GNNexplainer, GraphCFE, and D4Explainer have both a high GInX score (see figure <ref>) and a high EdgeRank score, making them the best choice for informative and edge-rank powerful explainers.

With EdgeRank, GInX-Eval indicates what method can better rank edges according to their informative power. This helps to compare methods that already perform well on GInX score.

Edge ranking power of explainability methods.

§ DISCUSSION

This work discusses the pitfalls of faithfulness, one of the most popular metrics in xAI, and the problem of out-of-distribution explanations. Overcoming these limitations, our evaluation procedure GInX-Eval measures the informativeness of explainability methods and their ability to accurately rank edges by their importance for the GNN model. Observing the prediction change, GInX-Eval assesses the impact of removing the generated explanations from the graphs. It gets around the issue of OOD explanations by retraining the GNN model. GInX-Eval is a useful tool to validate the quality of the provided ground-truth explanations. It also demonstrates the poor informativeness of gradient-based methods, contradicting results from recent studies [Yuan et al., 2023, Agarwal et al., 2022] and reproduced in this paper. Combining the GInX and EdgeRank scores, we can filter out uninformative explainability methods and find the optimal ones. Due to the computation cost of re-training, GInX-Eval is not meant to be used systematically but is designed as a validation tool for new metrics. This work paves the way for developing approaches that conform with both human- and model-based explainability.

[Agarwal et al., 2021]
Chirag Agarwal, Marinka Zitnik, and Himabindu Lakkaraju.
Probing GNN explainers: A rigorous theoretical and empirical
analysis of GNN explanation methods.
June 2021.

[Agarwal et al., 2022]
Chirag Agarwal, Owen Queen, Himabindu Lakkaraju, and Marinka Zitnik.
Evaluating explainability for graph neural networks.
August 2022.

[Bajaj et al., 2021]
Mohit Bajaj, Lingyang Chu, Zi Yu Xue, Jian Pei, Lanjun Wang, Peter Cho-Ho Lam,
and Yong Zhang.
Robust counterfactual explanations on graph neural networks.
July 2021.

[Baldassarre & Azizpour, 2019]
Federico Baldassarre and Hossein Azizpour.
Explainability techniques for graph convolutional networks.
CoRR, abs/1905.13686, 2019.

[Chan et al., 2022]
Chun Sik Chan, Huanqi Kong, and Liang Guanqing.
A comparative study of faithfulness metrics for model
interpretability methods.
In Proceedings of the 60th Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers), pp.  5029–5038,
Dublin, Ireland, May 2022. Association for Computational Linguistics.
URL <https://aclanthology.org/2022.acl-long.345>.

[6]
Jialin Chen, Kenza Amara, Junchi Yu, and Rex Ying.
Generative explanation for graph neural network: Methods and
extraction, 540 (12):0 47.

[Colin et al., 2021]
Julien Colin, Thomas Fel, Remi Cadene, and Thomas Serre.
What I cannot predict, I do not understand: A human-centered
evaluation framework for explainability methods.
December 2021.

[Collobert et al., 2011]
Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet.
Torch: A scientific computing framework for luajit.
Proceedings of the Annual Conference on Neural Information
Processing Systems (NIPS), 24:0 237–245, 2011.

[Faber et al., 2020]
Lukas Faber, Amin K Moghaddam, and Roger Wattenhofer.
Contrastive graph neural network explanation.
October 2020.

[Fey & Lenssen, 2019]
Matthias Fey and Jan E. Lenssen.
Fast graph representation learning with PyTorch Geometric.
In ICLR Workshop on Representation Learning on Graphs and
Manifolds, 2019.

[Günnemann, 2022]
Stephan Günnemann.
Graph neural networks: Adversarial robustness.
Graph Neural Networks: Foundations, Frontiers, and
Applications, pp.  149–176, 2022.

[Hooker et al., 2018]
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim.
A benchmark for interpretability methods in deep neural networks.
June 2018.

[13]
Cheng-Yu Hsieh, Chih-Kuan Yeh, Xuanqing Liu, Pradeep Ravikumar, Seungyeon Kim,
Sanjiv Kumar, and Cho-Jui Hsieh.
Evaluation and methods for explanation through robustness analysis.

[Hu et al., 2020]
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande,
and Jure Leskovec.
Strategies for pre-training graph neural networks, 2020.

[Jacovi & Goldberg, 2020]
Alon Jacovi and Yoav Goldberg.
Towards faithfully interpretable nlp systems: How should we define
and evaluate faithfulness?
arXiv preprint arXiv:2004.03685, 2020.

[LeCun et al., 1998]
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.
Gradient-based learning applied to document recognition.
Proceedings of the IEEE, 860 (11):0
2278–2324, 1998.

[Li et al., 2021]
Haoyang Li, Xin Wang, Ziwei Zhang, and Wenwu Zhu.
OOD-GNN: Out-of-Distribution generalized graph neural network.
December 2021.

[Li et al., 2022]
Peibo Li, Yixing Yang, Maurice Pagnucco, and Yang Song.
Explainability in graph neural networks: An experimental survey.
arXiv preprint arXiv:2203.09258, 2022.

[Longa et al., 2022]
Antonio Longa, Steve Azzolin, Gabriele Santin, Giulia Cencetti, Pietro Liò,
Bruno Lepri, and Andrea Passerini.
Explaining the explainers in graph neural networks: a comparative
arXiv preprint arXiv:2210.15304, 2022.

[Luo et al., 2020]
Dongsheng Luo, Wei Cheng, Dongkuan Xu, Wenchao Yu, Bo Zong, Haifeng Chen, and
Xiang Zhang.
Parameterized explainer for graph neural network.
In NeurIPS, 2020.

[Ma et al., 2022]
Jing Ma, Ruocheng Guo, Saumitra Mishra, Aidong Zhang, and Jundong Li.
Clear: Generative counterfactual explanations on graphs.
arXiv preprint arXiv:2210.08443, 2022.

[Miao et al., 2022]
Siqi Miao, Mia Liu, and Pan Li.
Interpretable and generalizable graph learning via stochastic
attention mechanism.
In International Conference on Machine Learning, pp.  15524–15543. PMLR, 2022.

[Nauta et al., 2022]
Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters,
Yasmin Schmitt, Jörg Schlötterer, Maurice van Keulen, and Christin
From anecdotal evidence to quantitative evaluation methods: A
systematic review on evaluating explainable ai.
ACM Computing Surveys, 2022.

[NVIDIA Corporation, 2023]
NVIDIA Corporation.
NVIDIA CUDA Toolkit.
<https://developer.nvidia.com/cuda-toolkit>, 2023.

[Petsiuk et al., 2018]
Vitali Petsiuk, Abir Das, and Kate Saenko.
Rise: Randomized input sampling for explanation of black-box models.
arXiv preprint arXiv:1806.07421, 2018.

[Pope et al., 2019]
Phillip E. Pope, Soheil Kolouri, Mohammad Rostami, Charles E. Martin, and Heiko
Explainability methods for graph convolutional neural networks.
In CVPR, pp.  10772–10781, 2019.

[27]
Benjamin Sanchez-Lengeling, Jennifer Wei, Brian Lee, Emily Reif, Peter Y Wang,
Wesley Wei Qian, Kevin Mc Closkey, Lucy Colwell, and Alexander Wiltschko.
Evaluating attribution for graph neural networks.
Accessed: 2021-11-22.

[Sterling & Irwin, 2015]
Teague Sterling and John J Irwin.
Zinc 15–ligand discovery for everyone.
Journal of chemical information and modeling, 550
(11):0 2324–2337, 2015.

[Sundararajan et al., 2017]
Mukund Sundararajan, Ankur Taly, and Qiqi Yan.
Axiomatic attribution for deep networks.
In ICML, volume 70, pp.  3319–3328, 2017.

[Veličković et al., 2018]
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro
Liò, and Yoshua Bengio.
Graph attention networks, 2018.

[Vu & Thai, 2020]
Minh N. Vu and My T. Thai.
Pgm-explainer: Probabilistic graphical model explanations for graph
neural networks.
In NeurIPS, 2020.

[Wu et al., 2018]
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse,
Aneesh S Pappu, Karl Leswing, and Vijay Pande.
Moleculenet: a benchmark for molecular machine learning.
Chemical science, 90 (2):0 513–530, 2018.

[Ying et al., 2019]
Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec.
Gnnexplainer: Generating explanations for graph neural networks.
In NeurIPS, pp.  9240–9251, 2019.

[Yuan et al., 2021]
Hao Yuan, Haiyang Yu, Jie Wang, Kang Li, and Shuiwang Ji.
On explainability of graph neural networks via subgraph explorations.
ArXiv, 2021.

[Yuan et al., 2023]
Hao Yuan, Haiyang Yu, Shurui Gui, and Shuiwang Ji.
Explainability in graph neural networks: A taxonomic survey.
IEEE Trans. Pattern Anal. Mach. Intell., 450
(5):0 5782–5799, May 2023.

[Zeiler & Fergus, 2014]
Matthew D Zeiler and Rob Fergus.
Visualizing and understanding convolutional networks.
In Computer Vision–ECCV 2014: 13th European Conference,
Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pp.  818–833. Springer, 2014.

§ PREVIOUS EVALUATION FRAMEWORK

§.§ Faithfulness metrics

\[
\begin{minipage}[t]{.49\textwidth}
\centering
\small
\textbf{Remove}
\begin{flalign*}
&fid-^{prob}= \frac{1}{N} \sum_{i=1}^{N}\left|p(\hat{y}_i^{G}=y_i) - p(\hat{y}_i^{G_e}=y_i)\right|\\
&fid-^{acc}= \frac{1}{N} \sum_{i=1}^{N}\left| \mathds{1}(\hat{y}_i^G=y_i)- \mathds{1}(\hat{y}_i^{G_e}=y_i) \right|
\end{flalign*}
\end{minipage}
\vline\hspace{2mm}
\begin{minipage}[t]{.49\textwidth}
\centering
\small
\textbf{Keep}
\begin{align*}
&fid+^{prob}= \frac{1}{N} \sum_{i=1}^{N}\left|p(\hat{y}_i^{G}=y_i) - p(\hat{y}_i^{G\setminus G_e}=y_i)\right|\\
&fid+^{acc}= \frac{1}{N} \sum_{i=1}^{N}\left| \mathds{1}(\hat{y}_i^G=y_i)- \mathds{1}(\hat{y}_i^{G\setminus G_e}=y_i)\right|
\end{align*}
\label{eq:fidelity}
\end{minipage}
\]

Faithfulness or fidelity metrics [Yuan et al., 2023] evaluate the contribution of the produced explanations to the initial prediction, either by giving only the explanation as input to the model (fidelity-) or by removing it from the entire graph and re-run the model (fidelity+). The keep/removing is done according to hard or soft selection (See section <ref>) and the explanatory edge mask. The fidelity scores capture how well an explainable model reproduces the natural phenomenon. The fidelity is measured concerning the ground-truth label $y$. Equations <ref> detail the mathematical expressions of the different fidelity scores. The fidelity scores (+/-) can be expressed either with probabilities ($fid_{+/-}^{prob}$) or indicator functions ($fid_{+/-}^{acc}$). While $fid_{+/-}^{prob}$ metrics are more appropriate for evaluating explanations in the context of regression tasks because they are only based on the predicted probabilities, $fid_{+/-}^{acc}$ metrics use the indicator function and are more suitable for classification problems.

§.§ The Out-Of-Distrbution Problem

Comparison between not retraining the GNN model and GInX-Eval on the MUTAG
dataset when hard removing edges estimated informative by the three base estimators: Truth, Inverse, and Random. New graphs are obtained with the hard selection strategy, i.e., edges are strictly removed from the initial graph structure.

Comparison between not retraining the GNN model and GInX-Eval on the MUTAG
dataset when soft removing edges estimated informative by the three base estimators: Truth, Inverse, and Random. New graphs are obtained with the soft selection strategy, i.e. edges receive a zero weight if they must be removed.

The Inverse estimator is a good indicator of the out-of-distribution problem existence. When you remove uninformative edges and the model is not retrained, the test accuracy should stay constant. But in the case of out-of-distribution explanations, the model has never seen these instances and therefore is not able to predict the correct labels any more. A robust model toward OOD is a model in which test accuracy does not degrade with the Inverse estimator, i.e., as we remove uninformative edges.

In figures <ref> and <ref>, we observe in both edge removal strategies a drop of model performance with the Inverse estimator when the model is not retrained; but no degradation with GInX-Eval retraining. Retraining the GNN model at each degradation level overcomes the OOD problem and enables a robust evaluation.

Remark The degradation of the GNN test accuracy is smaller with retraining than without retraining. For the MUTAG dataset we observe in figure <ref> a drop in performance for the Truth estimator of 0.18 with GInX-Eval versus 0.25 with no retraining at 90% edge removal. The model is remarkably robust to edge modification. After removing a large portion of all edges, the model still learns to make correct predictions and retain almost most of the original predictive power. For instance, for the MUTAG dataset, a random modification of 50% of the edges degrades the accuracy only to 85% accuracy. The model can extract meaningful representations from a small amount of remaining edges, which suggests that many edges are likely redundant or correlated.

§.§ Hard versus Soft Selection

In this section, we explore the effect of the selection strategy on GInX-Eval.
With hard edge removal, the drop in test accuracy is significant, in the order of $10^{-1}$. For the soft edge removal, the drop is very small in the order of $10^{-2}$ or $10^{-3}$ for some datasets. In figure <ref>, we observe for the MUTAG dataset that, after removing 90% of the edges and when the model is retrained, the accuracy drops by 10% for the Truth estimator in the case of soft selection, against 28% with the hard selection on figure <ref>. This suggests that removing informative edges by setting their weights to zeros does not prevent the model from learning from the masked information. For soft selection, the removed informative edges are not fully ignored. On the contrary, with hard selection, the model has no way of capturing the removed information since edges are fully removed from the structure. For this reason, we favor hard edge selection, even though similar effects are observed at different scales and similar conclusions are drawn with both types of selection strategies (See Appendix <ref>).

§.§ Message Passing with edge weights

To understand why the soft selection enables the model to capture masked information and be robust to edge removal, we explain here how the soft selection affects the GNN training.
In a weighted graph, each edge is associated with a semantically meaningful scalar weight. For soft explanations, for instance, the edge weights are importance scores. Graph neural networks integrate the graph topology information in forward computation by the message-passing mechanism.
Edge weights modify the message from node $i$ to node $j$ in different ways according to the type of convolutional layer. For GAT convolutional layers, the attention scores are updated with additional edge-level attention coefficients $\alpha$ that correspond to the edge weights, so that $\alpha=\alpha_{\text{node}} + \alpha_{\text{edge}}$. For GINE convolutional layers, the message is the node vector $x_j$ followed by a ReLU activation. To account for edge importance, the message is modified by adding to the node vector the transformed edge weight $Lin(w_{ij})$. By modifying each message with its corresponding edge weight, the model keeps the whole graph connectivity but learns the importance of the node connections.

§ EXPERIMENTAL DETAILS

§.§ Datasets

We evaluate the explainability methods on both synthetic and real-world datasets used for graph classification tasks.

BA-2Motifs [Luo et al., 2020] is a synthetic dataset with binary graph labels. The house motif and the cycle motif give class labels and thus are regarded as ground-truth explanations for the two classes.

BA-HouseGrid is our new synthetic dataset where we attach house or grid motifs to a Barabási base structure. We choose two distinct motifs, house and grid, where one is not the subgraph of the other, unlike BA-2Motifs with the cycle that is contained in the house motif. The datasets contain 2,000 graphs with balanced BA-Grid and BA-House graphs. The base Barabási graphs contain 80 nodes, and the number of motifs varies between 2 and 5 per graph.

MUTAG is a collection of $\sim$3,000 nitroaromatic compounds and it includes binary labels on their mutagenicity on Salmonella typhimurium. The chemical fragments -NO2 and -NH2 in mutagen graphs are labeled as ground-truth explanations [Luo et al., 2020].

BBBP includes binary labels for over 2,000 compounds on their permeability properties [Wu et al., 2018]. The task
is to predict the target molecular properties. In molecular datasets, node features encode the atom type and edge features encode the type of bonds that connect atoms.

Benzene contains 12,000 molecular graphs extracted from the ZINC15 [Sterling & Irwin, 2015] database and labeled into two classes where the task is to identify whether a given molecule has a benzene ring or not. The ground-truth explanations are the nodes (atoms) comprising the benzene rings, and in the case of multiple benzenes, each of these benzene rings forms an explanation.

MNISTbin contains graphs that are converted from images in MNISTbin [LeCun et al., 1998] using superpixels. In these graphs, the nodes represent the superpixels, and the edges are determined by the spatial proximity between the superpixels. The coordinates and intensity of the corresponding superpixel construct the node features. We reduce the MNISTbin graph dataset for binary classification by selecting only the input of classes 0 and 1.

4cReal-world     2cSynthetic

Datasets     MUTAG     BBBP     Benzene     MNISTbin     BA-2Motifs     BA-HouseGrid

# Graphs     2,951     2,039     12,000     14,780     1,000     2,000

Avg # Nodes     30     24     20.6     63     25     101

Avg # Edges     61     51.9     43.7     486.6     51     817

# Node features     14     9     14     5     1     1

# Edge features     1     3     5     1     1     1

# Classes     2     2     2     2     2     2

GT Explanation     NH$_{2}$, NO$_{2}$     -     Benzene ring     Figure pixels     House, cycle     House, grid

Datasets for graph classification

§.§ GNN Training

We test two GNN models: GIN [Hu et al., 2020] and GAT [Veličković et al., 2018] because they score high on the selected real-world datasets, with a reasonable training time and fast convergence. The network structure of the GNN models for graph classification is a series of 3 layers with ReLU activation, followed by a max pooling layer to get graph representations before the final fully connected layer. We split the train/validation/test with 80/10/10$\%$ for all datasets and adopt the Adam optimizer with an initial learning rate of 0.001. Each model is trained for 200 epochs with an early stop. Each training is always repeated on five different seeds.

We report in Table <ref> the test accuracy of both GIN and GAT models for all six datasets. We report the average and standard error of the experiments run on five different seeds. We observe high test accuracy, i.e., $> 0.8$, for all datasets for the GIN model and all real-world datasets for the GAT model. Only for BA-2Motifs and BA-HouseGrid is the test accuracy of the GAT model not good enough to be tested in the explainability analysis and evaluated with GInX-Eval.

BA-2Motifs       BA-HouseGrid     BBBP             Benzene          MNISTbin            MUTAG
GAT     $0.428$$\pm 0.068$     $0.512$$\pm 0.034$     $0.839$$\pm 0.024$     $0.917$$\pm 0.044$     $0.992$$\pm 0.001$     $0.905$$\pm 0.022$

GIN     $0.988$$\pm 0.016$     $1.00$$\pm 0.000$      $0.835$$\pm 0.035$     $0.999$$\pm 0.001$     $0.995$$\pm 0.002$     $0.907$$\pm 0.016$

Test accuracy of GIN and GAT graph neural networks for the six datasets on graph classification task.

§.§ GInX-Eval computation time

Here we report the time for evaluating with GInX-Eval. GInX-Eval requires 10 GNN trainings if we vary the threshold $t$ from $[0,0.1,...,0.9]$. In table <ref>, we report the average GNN training time for GIN and GAT models for each dataset. In the case of the six datasets tested in this paper, the re-training strategy has a low computational burden. In the case of large-scale datasets, selecting just a representative train/test subset can also speed up GInX-Eval.

BA-2Motifs       BA-HouseGrid     BBBP             Benzene          MNISTbin            MUTAG
GAT     28.1     63.4     39.4     209.1     290.7     79.0

GIN     23.9     53.5     28.8     159.0     225.1     66.8

Training times (s) of GIN and GAT graph neural networks for the six datasets on graph classification task.

§.§ Generative Explainability Methods

Non-generative explainability methods like gradient-based or perturbation-based methods optimize individual explanations for a given instance. They lack a global understanding of the whole dataset as well as the ability to generalize to new unseen instances. To tackle this problem, non-generative methods have been developed. They learn the initial data distribution before generating individual explanations. Therefore, generative methods learn the underlying distributions of the explanatory graphs across the entire dataset, providing a more holistic approach to GNN explanations. The recent study of [6] shows the superiority of generative methods and in particular concerning their generalization capacity and faster inference time. GInX-Eval also proves that generative methods are the only type of methods that can do better than a random guess, and this is consistent across datasets and GNN models.

§.§ Code Implementation

Our code is implemented using torch-geometric 2.3.0 [Fey & Lenssen, 2019] and Torch 1.9.1 with CUDA version 11.1 [Collobert et al., 2011, NVIDIA Corporation, 2023]. The generation of edge masks by the explainability methods and the training of the graph neural networks at each removal threshold are performed on a Linux machine with 1 GPUs NVIDIA RTX A6000 with 10 GB RAM memory. The code is available at <https://anonymous.4open.science/r/GInX-Eval>.

§ ADDITIONAL RESULTS

§.§ More explainability methods

Here, we provide the GInX scores for all the tested methods: three baselines, eight non-generative methods, and five generative ones.

Test accuracy of a retrained GIN model on graphs with increasing fraction of removed edges. The removed edges are the most important based on explainability methods, and new input graphs are obtained with the hard selection, i.e. explanatory edges are strictly removed from the graph so that the number of edges and nodes is reduced.

Why is the performance not dropping linearly? We found two possible reasons for the nonlinear performance decrease when removing important edges. One reason is that edges might be redundant: some correlations exist between them and their effect is neutralized only if all are removed. Another reason is that important edges are not always correctly ordered or have no order of importance, e.g. Truth assigns a weight of 1 to all ground-truth edges. In these cases, the true least important edge among important edges according to the estimator might be removed first, not degrading the performance of the model that is still trained with the most informative edges.

§.§ GInX-Eval with Soft Selection

In the paper, we describe the different selection strategies to remove edges from a graph (see section <ref>) and explain our preference for the hard selection to convey the results in section <ref>. Here, we add the results of GInX-Eval for soft selection. The experimental setting is the same. We display the results for the four real-world datasets.

Compared to the results with hard selection, the degradation here is way smaller, in the order of 0.01, as mentioned in the paper. Instead of observing a drop of accuracy for the best methods, we observe that the accuracy stays constant. The impact of masking informative edges does not prevent the model from learning because the model can still pass the message on the node level. For more details, we refer the reader to the discussion in Appendix <ref>. For the worse methods, the test accuracy increases instead of staying constant. Removing uninformative edges by setting their weight to zero brings the model to better learn the actual class of the graphs. Only for Benzene, the Random and Truth baselines as well as most of the generative methods produce a drop of $\sim 0.07$ model accuracy.

Results on the MNISTbin dataset cannot be interpreted since the degradation is of the order of $10^{-3}$ and Truth and Inverse estimators have the same behavior. For the three other datasets Benzene, MUTAG, and BBBP, figure <ref> confirms the superiority of the non-generative methods GNNExplainer and PGMExplainer, and the generative methods GSAT, GraphCFE, and D4Explainer. Once again GradCAM, Occlusion, and the gradient-based methods Saliency and Integrated Gradients capture the less informative edges, contradicting the previous studies on explainability for GNN [Yuan et al., 2023, Agarwal et al., 2022].

Test accuracy of a retrained GIN model on graphs with increasing fraction of removed edges. The removed edges are the most important based on explainability methods, and new input graphs are obtained with the soft selection, i.e. explanatory edges are masked so that their respective weight is put to zero.

§.§ More GNN models

This section is dedicated to GInX-Eval with Graph Attention Networks (GAT). Because GAT reaches low accuracy with the two synthetic graph datasets, we only show the results for the four real-world datasets. Figure <ref> and <ref> show the test accuracy degradation of GAT models retrained on new graphs obtained after a hard and a soft edge removal strategy, respectively.

Test accuracy of a retrained GAT model on graphs with increasing fraction of removed edges. The removed edges are the most important based on explainability methods, and new input graphs are obtained with the hard selection, i.e. explanatory edges are strictly removed from the graph so that the number of edges and nodes is reduced.

Test accuracy of a retrained GAT model on graphs with increasing fraction of removed edges. The removed edges are the most important based on explainability methods, and new input graphs are obtained with the soft selection, i.e. explanatory edges are masked so that their respective weight is put to zero.


# The Importance of Human-Labeled Data in the Era of LLMs

Yang Liu ByteDance Research
UC Santa Cruz<EMAIL_ADDRESS>
###### Abstract

The advent of large language models (LLMs) has brought about a revolution in
the development of tailored machine learning models and sparked debates on
redefining data requirements. The automation facilitated by the training and
implementation of LLMs has led to discussions and aspirations that human-level
labeling interventions may no longer hold the same level of importance as in
the era of supervised learning. This paper presents compelling arguments
supporting the ongoing relevance of human-labeled data in the era of LLMs.

##  1 Introduction

Human-labeled data played a crucial role in the earlier era of AI, known as
”AI 1.0,” where machine learning models heavily relied on such data Deng et
al. (2009). The celebrated supervised learning framework Vapnik (1999); LeCun
et al. (2015) was designed and developed exactly for this paradigm. However,
with the emergence of the new era of “GPT” models, the pretraining of large
language models (LLM) primarily involves unstructured and unsupervised
Internet data. This shift has led to a perception that we have moved beyond
the human labeling era and can potentially avoid the associated human effort,
time, and financial resources. This development is both exciting and aligns
with the longstanding goal of the weakly-, semi-, and self-supervised learning
community Zhu (2005); Zhou (2018); Gui et al. (2023); Balestriero et al.
(2023).

Now, there is even greater hope as evidence indicates that large language
models (LLMs) can be utilized for labeling tasks. Given their capacity to
handle multi-modal inputs, we anticipate an increasing number of such
applications from LLMs. Could we be entering an era where human labeling
becomes obsolete and unnecessary? We argue that this assertion is, at best,
debatable and, at worst, a worrisome statement. Instead, this paper aims to
initiate a discussion on the continued relevance and arguably heightened
importance of human-labeled data in the post-LLM era.

##  2 Hopes and Dangers

Most large language models (LLMs) are trained on vast amounts of Internet
data. Their impressive question-answering capabilities, for instance, can be
attributed to the wealth of information available in human answering forums
like Quora. Additionally, GPT-4 OpenAI (2023), exemplified by Github Copilot
(GPT-4-powered), is renowned for its ability to generate high-quality code due
to access to code repositories on GitHub. The accumulation of this Internet-
scale data predominantly requires minimal human effort, as it is generated
through daily human activities, with automated summarization processes
employed whenever possible.

Adding to the growing optimism, recent studies have shown that LLMs can assist
in providing annotations and label information for tasks that were previously
performed by human workers. For instance, in the study by Gilardi et al.
(2023), it is demonstrated that ChatGPT outperforms crowd workers recruited
from Amazon Mechanical Turk in simple text classification tasks. The following
case studies reported in Figure 1 further exemplify the effectiveness of
utilizing LLMs for labeling tasks, with an emphasis on engineering efforts to
ensure appropriate prompts:

Figure 1: Examples of using ChatGPT to perform text classification.

Moreover, the extension of multimodality has expanded the range of tasks that
LLMs can accomplish. For instance, LLMs (i.e., Blip Li et al. (2022)) can now
be tasked with identifying relevant objects within a given image (Figure 2).
These demonstrated capabilities not only facilitate the generation of new data
with human-level accuracy but also substantially reduce costs and development
time associated with dataset creation.

Figure 2: Visual question answering of LLMs for object identification on CIFAR
dataset.

Machines generate bad answers and make mistakes too. Prior versions of
unaligned language models do show tendencies for generating hallucinating
content, unreliable answers, content that promotes violent and illegal
behaviors, or that reinforces stereotypical social biases Bai et al. (2022).
This is something we shall further discuss in the next section. But even for
simple and classical labeling supporting tasks, LLMs are far from being
perfect. In Toloka (2023), a recent report has shown that even the most
advanced GPT model underperforms well-trained human annotators in text
labeling. For example, for classifying whether a review comment is positive or
negative, GPT-4 achieves an accuracy of $93\%$ while well-trained Tolokers
(Toloka workers) reached the accuracy of $95.3\%$.

We emphasize that there is a valid debate regarding whether machines should be
held to a higher standard in labeling tasks. For human labeling, we have a
well-established “insecurity” of human-labeled data and a number of “safety”
protocols have been established to make sure the human-generated data meets
certain performance requirements. These efforts include building incentive
mechanisms Liu and Chen (2016); Witkowski et al. (2013), human spot-
checking/auditing mechanisms Shah and Zhou (2015) and automatic error analysis
in human labels Zhu et al. (2022a, 2021b). More sophisticated systems can be
built too. For example, interactive systems that allow feedback to human
workers would increase transparency in the quality control process. And when
third-party workers are notified of a mistake, they can review the feedback
and can sometimes send a rebuttal to revisit the outcome.

Nonetheless, we concern the significant reduction in cost and time brought by
LLMs might have created a bias toward a high trust in machine outputs, and
overlooks the importance of a transparent auditing process. Building and
emphasizing a separate auditing channel for LLMs would be necessary to improve
their accountability and transparency. Furthermore, prior research has
suggested that machines and humans have distinct perspectives and may make
different types of errors Liu et al. (2023a). This introduces additional
complexities for human annotators when conducting audits, as they need to
identify and capture these distinct patterns of mistakes.

##  3 Safety and Regulation Alignments

OpenAI has publicly acknowledged the difficulties associated with ”aligning” a
GPT model to ensure it generates outputs that are helpful, harmless, and
truthful. It is worth noting that human-generated data often contains
dangerous, violence-inciting, and unethical content. As GPT models are trained
on such data, it is not surprising that these issues may arise and should be
expected. To address these challenges, GPT models employ a technique called
reinforcement learning from human feedback (RLHF) Christiano et al. (2017).
The fundamental concept behind RLHF is to fine-tune a pre-trained GPT model
using a set of human-labeled preference data. This data encompasses various
forms of human inputs:

  * $\bullet$

Human preference data over multiple LLMs’ responses: this type of human inputs
is a ranking preference of multiple different responses generated by LLMs;
this ranking data can help further generate pair-wise comparisons.

  * $\bullet$

Sample answers collected from humans as “template answers: when a red team of
human annotators identified a potentially harmful response from an LLM, they
will also pair the question with an exemplary answer written carefully by
human.

There are a couple of challenges in handling this alignment data. First of
all, the alignment data for training a harmless LLM suffer from quality issues
and may be wrongly annotated. Figure 3 shows that the training data published
by Anthropic Ganguli et al. (2022) contains annotation errors. The sample
indeed contains harmful content (negative samples) but is wrongly annotated as
harmless ones (positive samples), which mislead the training and may cause
unsafe results 111The results are obtained using the result reported in Zhu et
al. (2022a) and an opensourced detector docta.ai. .

Figure 3: Human annotation errors from existing LLM alignment data. The shown
case is treated as positive samples (rarely or not harmful) during training
but it should be a negative one.  Figure 4: Image captioning results obtained
from Amazon Mturk.

Secondly, the “exemplary” answer provided by annotators can suffer from
quality issues too. Technically speaking, this human-written answer is nothing
more than a label provided by humans, but it is coming from a rather large and
infinite label space. Therefore we expect the same quality issues can happen.
In Figure 4 we collected captions on Amazon Mturk for a set of images from
Flickr-8k Hodosh et al. (2013) and we observe a clear difference between them
and the gold standard captions (provided by experts with a strict quality
control process). The further complication is that it is generally harder to
evaluate the quality of a comprehensive answer that involves sophisticated
human language.

##  4 Risk Control

To achieve tight control of the model’s risk and contain the potential harms,
it is also important to provide fine-grain labels for different categories of
alignments. The survey paper Weidinger et al. (2021) has identified 21
categories of risks that LLM should attempt to align with. Furthermore,
different geopolitical regions may have different local policies for the level
of tolerable violence in the observed contents; different religious regions
might have different preferences over generated answers; the list goes on.

Figure 5: Label distribution of Anthropic’s red-teaming data.

Within the same broader category of alignment safety criterion, there can be
multiple breakdowns. As Figure 6, for example, the category of “Toxicity” can
include a list of labels such as violent content, emotional comments, and
offensive language.

Figure 6: Fine-grained categories of safety alignment.

Aligning using a single combined dataset lacks the transparency, coverage, and
customization of the LLMs’ risk control ability. In Figure 5, through an
analysis of Anthropic’s data, we do observe an imbalanced distribution of
alignment categories. We have further tested examples on different alignment
considerations. In Figure 7, we see that DialoGPT Zhang et al. (2019), a
variant of the GPT models, performs relatively better with violence-related
questions but can be improved w.r.t. social stereotype biases. Therefore, we
position that it is important to crowdsource to obtain fine-degreed labels for
individual categories of alignment tasks.

Figure 7: Example conversations with DialoGPT.

##  5 Prompt Engineering

The most effective use of LLMs relies on the quality of the prompts. A
carefully designed prompt can unlock the most power of an LLM. For instance,
it has been shown that few-shot prompting via providing an LLM examples can
substantially improve the quality of the answers Brown et al. (2020); Min et
al. (2022); Touvron et al. (2023). In Xie et al. (2023), it is shown that
providing sequential feedback in the prompts can also help LLMs better
understand the users’ demand.

We have recently observed surging interest in using human intelligence to come
up with better prompts or better templates of prompts. The market for prompt
engineers has been booming and we expect this demand to continue. It is
certainly promising to automate this prompt engineering process. Recent works
have explored the possibility of red-teaming an LLM using another language
model to identify useful prompts Perez et al. (2022). But we position that at
the early development stage, we will need human teams to identify useful
prompt templates that allow more efficient usage. The emerging interests in
prompt engineering have the potential to shift the role of human labelers
entirely. Instead of providing the final supervision of a task (e.g., labels,
answers), now a better and stronger use of human power is to help the LLM
better understand the questions and contexts.

##  6 Confidence Calibration

The LLMs tend to be more confident than they should be, especially when the
answers are likely to be wrong or uninformative, or hallucinating OpenAI
(2023). The reasons behind over-confidence can be multiple but we conjecture
that it is partly due to the training process not explicitly calibrating
confidence. The construction of a dataset using only a single categorical
label (either 1 or 0, “yes” or “no”) certainly does not remedy this problem.

Calibrating LLMs’ answer confidence is crucial. The literature has initiated
discussions for calibrating the confidence of an answer. For example, the
literature on conformal prediction proposes a posthoc treatment that uses the
trained classifier to generate a set with multiple predictions to calibrate
the confidence Shafer and Vovk (2008).

Using multiple human annotations altogether is another promising solution to
addressing this issue of illy-calbirated labels. Suppose we are able to
solicit 6 independent human reviewers to review this question and collect the
following answers (1 for being Toxic and 0 for being Non-toxic):

$\text{Raw labels}\rightarrow\texttt{[1,~{}1,~{}1,~{}0,
~{}0,~{}1]}\rightarrow[67\%,33\%]$

We will then be able to claim that the generated answer is 67% likely to
contain toxic information. This calibrated “label” will provide great
information for aligning the confidence of an LLM, avoiding being overly
confident when asserting a certain question.

In a recent paper Wei et al. (2023), it is indeed shown that when the training
labels come from subjective and noisy label sources, keeping them separate,
instead of aggregating them into a single label Liu and Liu (2015); Karger et
al. (2013, 2011), might increase a model’s generalization power. This idea
echoes the necessity of label smoothing Müller et al. (2019); Wei et al.
(2022a) in supervised learning for generalizations but using human annotations
to generate soft labels helps provide more precise, targeted, and calibrated
soft labels that characterize individual instance’s uncertainty. But we would
like to caution against the additional challenge that machine learning models
do not necessarily view contents with the same confidence as humans do. In
Figure 3 of Liu et al. (2023a), we see machines are confident with examples
(measured by agreements between different predictions) that differ from
humans.

##  7 Proper Evaluations

The secure deployment of an LLM relies on comprehensive evaluations.
Conducting a multi-faceted evaluation not only aids in identifying potential
safety concerns and ensuring a low-risk deployment of the model but also acts
as a means to earn users’ trust Papenmeier et al. (2019). Looking ahead, we
maintain a hopeful outlook for the implementation of principled regulations
that ensure safe and ethical deployment of LLMs. Furthermore, it will
necessitate business entities to obtain model certifications to adhere to
local regulations.

Existing efforts have been promoting responsible documentation of dataset
Gebru et al. (2021) and models Mitchell et al. (2019) and we expect these
efforts to continue and extend for LLMs. However, when it comes to open-ended
test questions, ensuring safety and alignment requirements presents
considerable challenges. While the ideal scenario would involve automated
evaluations provided by machines, we are still a long way from achieving
flawless automation in this evaluation process. Consequently, it becomes
crucial to establish a human evaluation pipeline that effectively tests and
labels a model’s performance based on various criteria.

##  8 Challenges and Opportunities

### Quality control of human-labeled data.

Human labels continue to face quality issues and in Section 3 we have
highlighted that this issue persists in building alignment data for LLMs.
Careless annotations will not only drop but also creates a false sense of
security Zhu et al. (2023). This calls for the development of incentive-
compatible data marketplace Liu et al. (2023b); Liu and Chen (2017); Liu et
al. (2020), post-hoc automatic check solutions for providing high-quality
auditing of collected data Zhu et al. (2022a, 2021b); Liu and Liu (2015), as
well as robust learning solutions from noisy supervisions Cheng et al. (2021);
Zhu et al. (2021a).

### Learning from imperfect human supervisions.

Human labels do not scale well. It is hopeful that self- and weakly-supervised
learning techniques can be applied or developed to reduce the load for human
annotations for some of the discussed tasks above. Nonetheless, we want to
caution that these less-supervised learning methods reduce trustworthiness and
loosen risk control. The literature has discussed the potential issues when
applying these approaches, including requirements of assumptions and prior
knowledge Natarajan et al. (2013); Liu and Guo (2020); Wei and Liu (2021),
non-unified benchmarking Wei et al. (2022b), and unequal coverage of different
subpopulations Zhu et al. (2022b); Liu (2021) in the data and different tasks
Zhu et al. (2022c). How to properly implement the idea is worth exploring.

### Transfer learning.

Another idea to improve the efficiency of using human-labeled data is to
develop publicly available and open-source data-supporting pipelines for the
task of safety-aligning an LLM model. An associated technical question is also
can we build transfer learning techniques Weiss et al. (2016); Chen et al.
(2022) to reuse the alignment data resource and transfer the guaranteed safety
properties.

### Comprehensive labeling paradigm.

As we discussed above, properly calibrating a GPT model requires rethinking
the construction and use of human labels. Moving forward, we would desire a
new label collection and storage paradigm for annotations that go beyond
deterministic labels Wei et al. (2023).

### A co-evolving system: decision supporting with Human-in-the-loop.

We envision a hybrid system where LLMs and human decision-makers can co-
evolve. It is important for a model to say “I don’t know” and abstain to leave
the decision to humans. Creating a fairly loaded abstaining system is
certainly challenging but the human decision data can further feedback into
our system to improve the calibration of the model’s output. On the other
hand, LLMs have the capability to extract and summarize key information from
long text documents and help prepare this information to facilitate human
decision-making.

Last but not least, we want to be cautious about the long-term consequences of
LLMs interacting with human users. This issue has been raised in recent
literature on strategic machine learning Hardt et al. (2016); Chen et al.
(2020), performative effects of machine learning models Perdomo et al. (2020);
Liu et al. (2021); Estornell et al. (2021), and designing machine learning for
long-term objectives when their deployments also shift the distributions Raab
and Liu (2021); Zhang et al. (2020); Yin et al. (2023).

## References

  * Bai et al. [2022] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  arXiv preprint arXiv:2204.05862, 2022.
  * Balestriero et al. [2023] Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, Avi Schwarzschild, Andrew Gordon Wilson, Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamed Pirsiavash, Yann LeCun, and Micah Goldblum.  A cookbook of self-supervised learning, 2023.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * Chen et al. [2020] Yiling Chen, Yang Liu, and Chara Podimata.  Learning strategy-aware linear classifiers.  Advances in Neural Information Processing Systems, 33:15265–15276, 2020.
  * Chen et al. [2022] Yatong Chen, Reilly Raab, Jialu Wang, and Yang Liu.  Fairness transferability subject to bounded distribution shift.  Advances in neural information processing systems, 2022.
  * Cheng et al. [2021] Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu.  Learning with instance-dependent label noise: A sample sieve approach.  In International Conference on Learning Representations, 2021.
  * Christiano et al. [2017] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  Advances in neural information processing systems, 30, 2017.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * Estornell et al. [2021] Andrew Estornell, Sanmay Das, Yang Liu, and Yevgeniy Vorobeychik.  Unfairness despite awareness: Group-fair classification with strategic agents.  arXiv preprint arXiv:2112.02746, 2021.
  * Ganguli et al. [2022] Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al.  Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.  arXiv preprint arXiv:2209.07858, 2022.
  * Gebru et al. [2021] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford.  Datasheets for datasets.  Communications of the ACM, 64(12):86–92, 2021.
  * Gilardi et al. [2023] Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli.  Chatgpt outperforms crowd-workers for text-annotation tasks.  arXiv preprint arXiv:2303.15056, 2023.
  * Gui et al. [2023] Jie Gui, Tuo Chen, Qiong Cao, Zhenan Sun, Hao Luo, and Dacheng Tao.  A survey of self-supervised learning from multiple perspectives: Algorithms, theory, applications and future trends.  arXiv preprint arXiv:2301.05712, 2023.
  * Hardt et al. [2016] Moritz Hardt, Nimrod Megiddo, Christos Papadimitriou, and Mary Wootters.  Strategic classification.  In Proceedings of the 2016 ACM Conference on Innovations in Theoretical Computer Science, pages 111–122. ACM, 2016.
  * Hodosh et al. [2013] Micah Hodosh, Peter Young, and Julia Hockenmaier.  Framing image description as a ranking task: Data, models and evaluation metrics.  Journal of Artificial Intelligence Research, 47:853–899, 2013.
  * Karger et al. [2011] David R Karger, Sewoong Oh, and Devavrat Shah.  Iterative learning for reliable crowdsourcing systems.  In Advances in neural information processing systems, pages 1953–1961, 2011.
  * Karger et al. [2013] David R Karger, Sewoong Oh, and Devavrat Shah.  Efficient crowdsourcing for multi-class labeling.  In ACM SIGMETRICS Performance Evaluation Review, volume 41, pages 81–92. ACM, 2013.
  * LeCun et al. [2015] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  nature, 521(7553):436–444, 2015.
  * Li et al. [2022] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In International Conference on Machine Learning, pages 12888–12900. PMLR, 2022.
  * Liu and Chen [2016] Yang Liu and Yiling Chen.  Learning to incentivize: Eliciting effort via output agreement.  International Joint Conferences on Artificial Intelligence, 2016\.
  * Liu and Chen [2017] Yang Liu and Yiling Chen.  Machine-learning aided peer prediction.  In Proceedings of the 2017 ACM Conference on Economics and Computation, pages 63–80, 2017.
  * Liu and Guo [2020] Yang Liu and Hongyi Guo.  Peer loss functions: Learning from noisy labels without knowing noise rates.  In International Conference on Machine Learning, pages 6226–6236. PMLR, 2020.
  * Liu and Liu [2015] Yang Liu and Mingyan Liu.  An online learning approach to improving the quality of crowd-sourcing.  In Proceedings of the 2015 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS ’15, pages 217–230, New York, NY, USA, 2015. ACM.
  * Liu et al. [2020] Yang Liu, Michael Gordon, Juntao Wang, Michael Bishop, Yiling Chen, Thomas Pfeiffer, Charles Twardy, and Domenico Viganola.  Replication markets: Results, lessons, challenges and opportunities in ai replication.  arXiv preprint arXiv:2005.04543, 2020.
  * Liu et al. [2021] Yang Liu, Yatong Chen, Zeyu Tang, and Kun Zhang.  Model transferability with responsive decision subjects.  arXiv preprint arXiv:2107.05911, 2021.
  * Liu et al. [2023a] Minghao Liu, Jiaheng Wei, Yang Liu, and James Davis.  Do humans and machines have the same eyes? human-machine perceptual differences on image classification.  arXiv preprint arXiv:2304.08733, 2023.
  * Liu et al. [2023b] Yang Liu, Juntao Wang, and Yiling Chen.  Surrogate scoring rules.  ACM Transactions on Economics and Computation, 10(3):1–36, 2023\.
  * Liu [2021] Yang Liu.  Understanding instance-level label noise: Disparate impacts and treatments.  In International Conference on Machine Learning, pages 6725–6735. PMLR, 2021.
  * Min et al. [2022] Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer.  Rethinking the role of demonstrations: What makes in-context learning work?  arXiv preprint arXiv:2202.12837, 2022.
  * Mitchell et al. [2019] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru.  Model cards for model reporting.  In Proceedings of the conference on fairness, accountability, and transparency, pages 220–229, 2019.
  * Müller et al. [2019] Rafael Müller, Simon Kornblith, and Geoffrey E Hinton.  When does label smoothing help?  Advances in neural information processing systems, 32, 2019.
  * Natarajan et al. [2013] Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari.  Learning with noisy labels.  In Advances in neural information processing systems, pages 1196–1204, 2013.
  * OpenAI [2023] OpenAI.  Gpt-4 system card.  2023\.
  * Papenmeier et al. [2019] Andrea Papenmeier, Gwenn Englebienne, and Christin Seifert.  How model accuracy and explanation fidelity influence user trust.  arXiv preprint arXiv:1907.12652, 2019.
  * Perdomo et al. [2020] Juan Perdomo, Tijana Zrnic, Celestine Mendler-Dünner, and Moritz Hardt.  Performative prediction.  In International Conference on Machine Learning, pages 7599–7609. PMLR, 2020.
  * Perez et al. [2022] Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving.  Red teaming language models with language models.  arXiv preprint arXiv:2202.03286, 2022.
  * Raab and Liu [2021] Reilly Raab and Yang Liu.  Unintended selection: Persistent qualification rate disparities and interventions.  Advances in Neural Information Processing Systems, 34:26053–26065, 2021.
  * Shafer and Vovk [2008] Glenn Shafer and Vladimir Vovk.  A tutorial on conformal prediction.  Journal of Machine Learning Research, 9(3), 2008.
  * Shah and Zhou [2015] Nihar Bhadresh Shah and Dengyong Zhou.  Double or nothing: Multiplicative incentive mechanisms for crowdsourcing.  Advances in neural information processing systems, 28, 2015.
  * Toloka [2023] Toloka.  GPTs vs. human crowd in real-world text labeling: Who outperforms who?  Towards AI, 2023.  Accessed: 2023-05-29.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * Vapnik [1999] Vladimir N Vapnik.  An overview of statistical learning theory.  IEEE transactions on neural networks, 10(5):988–999, 1999.
  * Wei and Liu [2021] Jiaheng Wei and Yang Liu.  When optimizing $f$-divergence is robust with label noise.  In International Conference on Learning Representations, 2021.
  * Wei et al. [2022a] Jiaheng Wei, Hangyu Liu, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Yang Liu.  To smooth or not? when label smoothing meets noisy labels.  In International Conference on Machine Learning, pages 23589–23614. PMLR, 2022.
  * Wei et al. [2022b] Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu.  Learning with noisy labels revisited: A study using real-world human annotations.  In International Conference on Learning Representations, 2022.
  * Wei et al. [2023] Jiaheng Wei, Zhaowei Zhu, Tianyi Luo, Ehsan Amid, Abhishek Kumar, and Yang Liu.  To aggregate or not? learning with separate noisy labels.  29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023.
  * Weidinger et al. [2021] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al.  Ethical and social risks of harm from language models.  arXiv preprint arXiv:2112.04359, 2021.
  * Weiss et al. [2016] Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang.  A survey of transfer learning.  Journal of Big data, 3(1):1–40, 2016.
  * Witkowski et al. [2013] Jens Witkowski, Yoram Bachrach, Peter Key, and David C. Parkes.  Dwelling on the Negative: Incentivizing Effort in Peer Prediction.  In Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing (HCOMP’13), 2013.
  * Xie et al. [2023] Yutong Xie, Zhaoying Pan, Jinge Ma, Luo Jie, and Qiaozhu Mei.  A prompt log analysis of text-to-image generation systems.  In Proceedings of the ACM Web Conference 2023, pages 3892–3902, 2023.
  * Yin et al. [2023] Tongxin Yin, Reilly Raab, Mingyan Liu, and Yang Liu.  Long-term fairness with unknown dynamics.  arXiv preprint arXiv:2304.09362, 2023.
  * Zhang et al. [2019] Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan.  Dialogpt: Large-scale generative pre-training for conversational response generation.  arXiv preprint arXiv:1911.00536, 2019.
  * Zhang et al. [2020] Xueru Zhang, Ruibo Tu, Yang Liu, Mingyan Liu, Hedvig Kjellstrom, Kun Zhang, and Cheng Zhang.  How do fair decisions fare in long-term qualification?  Advances in Neural Information Processing Systems, 33:18457–18469, 2020.
  * Zhou [2018] Zhi-Hua Zhou.  A brief introduction to weakly supervised learning.  National science review, 5(1):44–53, 2018.
  * Zhu et al. [2021a] Zhaowei Zhu, Tongliang Liu, and Yang Liu.  A second-order approach to learning with instance-dependent label noise.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10113–10123, 2021.
  * Zhu et al. [2021b] Zhaowei Zhu, Yiwen Song, and Yang Liu.  Clusterability as an alternative to anchor points when learning with noisy labels.  In International Conference on Machine Learning, pages 12912–12923. PMLR, 2021.
  * Zhu et al. [2022a] Zhaowei Zhu, Zihao Dong, and Yang Liu.  Detecting corrupted labels without training a model to predict.  In International Conference on Machine Learning, pages 27412–27427. PMLR, 2022.
  * Zhu et al. [2022b] Zhaowei Zhu, Tianyi Luo, and Yang Liu.  The rich get richer: Disparate impact of semi-supervised learning.  In International Conference on Learning Representations, 2022.
  * Zhu et al. [2022c] Zhaowei Zhu, Jialu Wang, and Yang Liu.  Beyond images: Label noise transition matrix estimation for tasks with lower-quality features.  In International Conference on Machine Learning, pages 27633–27653. PMLR, 2022.
  * Zhu et al. [2023] Zhaowei Zhu, Yuanshun Yao, Jiankai Sun, Hang Li, and Yang Liu.  Weak proxies are sufficient and preferable for fairness with missing sensitive attributes.  International Conference on Machine Learning, 2023.
  * Zhu [2005] Xiaojin Zhu.  Semi-supervised learning literature survey.  Technical Report 1530, Computer Sciences, University of Wisconsin-Madison, 2005.