# Adaptive Label Smoothing To Regularize
Large-Scale Graph Training

Kaixiong Zhou
Rice University
<EMAIL_ADDRESS>
&Ninghao Liu
University of Georgia
<EMAIL_ADDRESS>
&Fan Yang
Rice University
<EMAIL_ADDRESS>
&Zirui Liu
Rice University
<EMAIL_ADDRESS>
&Rui Chen
Samsung Research America
<EMAIL_ADDRESS>
&Li Li
Samsung Research America
<EMAIL_ADDRESS>

&Soo-Hyun Choi
Samsung Electronics
<EMAIL_ADDRESS>

&Xia Hu
Rice University
<EMAIL_ADDRESS>
###### Abstract

Graph neural networks (GNNs), which learn the node representations by
recursively aggregating information from its neighbors, have become a
predominant computational tool in many domains. To handle large-scale graphs,
most of the existing methods partition the input graph into multiple sub-
graphs (e.g., through node clustering) and apply batch training to save memory
cost. However, such batch training will lead to label bias within each batch,
and then result in over-confidence in model predictions. Since the connected
nodes with positively related labels tend to be assigned together, the
traditional cross-entropy minimization process will attend on the predictions
of biased classes in the batch, and may intensify the overfitting issue. To
overcome the label bias problem, we propose the adaptive label smoothing (ALS)
method to replace the one-hot hard labels with smoothed ones, which learns to
allocate label confidences from the biased classes to the others.
Specifically, ALS propagates node labels to aggregate the neighborhood label
distribution in a pre-processing step, and then updates the optimal smoothed
labels online to adapt to specific graph structure. Experiments on the real-
world datasets demonstrate that ALS can be generally applied to the main
scalable learning frameworks to calibrate the biased labels and improve
generalization performances.

##  1 Introduction

Large-scale graphs, which are characterized by massive nodes and edges, are
ubiquitous in real-world applications, such as social networks [11, 42, 33]
and knowledge graphs [43, 43]. Although graph neural networks (GNNs) have
shown effectiveness in many fields [25, 40, 45, 9, 47], most of them rely on
propagating messages over the whole graph dataset, and are mainly developed
for relatively small graphs. Such message passing paradigms lead to
prohibitive computation and memory requirements.

Recently, several scalable algorithms of GNNs have been proposed to handle the
large-scale graphs, among which sub-graph sampling methods are dominant in
literature [20, 6, 7, 10, 52]. Specifically, instead of training on the full
graph, the sampling methods sample subsets of nodes and edges to formulate a
sub-graph at each step, which is treated as an independent mini-batch. For
example, Cluster-GCN [10] first clusters the input graph into sub-graph
groups, and then formulates each batch with a fixed number of groups (referred
as batch size) during model training. LGCN [15] samples sub-graphs via breadth
first search, as motivated by small patch cropping on a large image.

Nevertheless, the label bias existing in the sampled sub-graphs could make GNN
models become over-confident about their predictions, which leads to
overfitting and lowers the generalization accuracy [17]. Note that in the
real-world assortative graphs [39], the closely connected nodes are potential
to share the same label or positively related labels. The sub-graph sampling
methods usually assign these related nodes into the same sub-graph and lead to
label bias in a batch, whose node label distribution is significantly
different from the other batches. Taking Cluster-GCN as an example, where the
community with the same node labels is clustered as a sub-graph, the label
distribution variance among batches is dramatic as shown in Figure 1.
Comparing with the traditional deep neural networks trained by uniform batch
[29, 26, 13, 48], the cross-entropy minimization in the biased batch will
severely make GNN model to attend only on the correctness of biased ground-
truth category by producing an extremely large prediction probability. Such
over-confident prediction will overfit on training set (e.g., the decreasing
training loss of Cluster-GCN in Figure 1), but generalizes poorly on the
testing set (e.g., the increasing testing loss). To overcome the over-
prediction and overfitting, label smoothing has been proposed to soften the
one-hot class label by mixing it with a uniform distribution [38]. Through
penalizing the over-confident prediction towards the ground-truth label, the
label smoothing (LS) has been used to improve the generalization performance
across a range of tasks, including image classification [29, 48, 27, 51],
semantic parsing [17], and neural machine translation [28, 16].

However, it is non-trivial to apply the label smoothing to regularize and
adapt to the large-scale graph training from two structural levels: local node
and global graph. First, different from generic machine learning problems
associated with independent data instances, in graph data, it is generally
assumed that class labels of connected nodes are positively related in many
real-world applications [39, 22]. In other words, for a specific local node,
its label prediction highly depends on the label distribution of its
neighbors. The traditional label smoothing, after mixing one-hot hard target
with a uniform distribution, could wrongly regularize nodes in graph data.
Second, considering the global graph, the relevance between different pairs of
labels could vary. For example, in academic networks, civil engineering
researchers tend to collaborate more with ecology researchers than with
physicists [30]. The optimal label smoothing should be conditioned on the the
relevance between the ground-truth label and the related labels. The fixed
label smoothing paradigm, by mixing uniform distribution, will fail to model
such global label relevance specifically to the downstream tasks.

To bridge the gap, in this paper, we develop a simple yet effective adaptive
label smoothing (ALS) method to regularize the representation learning on
large-scale graphs. We aim to answer two research questions. First, given a
local node, how can we estimate its smoothed label that is aware of the
neighborhood structure? Second, how can we learn the global label relevance
for a specific task? Through exploring these questions, we make three
significant contributions as follows.

  * •

We are the first to analyze the label bias problem in the sub-graph sampling
methods for the large-scale graph training. The biased batch training could
make GNN model produce over-confident prediction and overfit on the training
set.

  * •

We present an adaptive label smoothing methods decoupled into the following
stages: a label propagation preprocessing step to aggregate the local
neighborhood label distribution; a label refinement step mapping the
preprocessed neighborhood labels to learn the global smoothed label
adpatively. Our method is very simple and memory efficient, and could be
scaled to the large-scale graph with negligible step to map the desired
smoothed label.

  * •

We propose a label smoothing pacing function to allocate different smoothing
strengths along the training process, in order to avoid the overly
regularization at the beginning.

  * •

The empirical results show that our adaptive label smoothing could relieve the
overfitting issue and yield better node classification accuracies based upon
most scalable learning frameworks.

Figure 1: Left, Middle: The training and testing losses on ogbn-products.
While Plain-$a$ means the original Cluster-GCN trained with batch size of $a$,
ALS-$a$ denotes the Cluster-GCN equipped with ALS. Right: The mean probability
$p_{c}$ of nodes with specific class $c$ within a batch, and the standard
variance of probability $p_{c}$ among batches. Herein we show $c=0$ on ogbn-
products for an example.

##  2 Label Bias, Over-confident Prediction and Overfitting

#### Notations and problem definition.

We denote matrices with boldface capital letters (e.g. $\bm{A}$), vectors with
boldface lowercase letters (e.g., $\bm{y}$ or $\bm{y}_{i}$) and scalars with
lowercase alphabets (e.g., $a$). We use $A_{i,j}$ to index the $(i,j)$-th
element in matrix $\bm{A}$, and use $y_{i,j}$ to represent the $j$-th entity
of vector $\bm{y}_{i}$. In this work, we focus on node classification tasks,
and propose using label smoothing to address the over-confident prediction and
overfitting issue in large-scale graph analysis. A graph is represented by
$G=(\bm{A},\bm{X})$, where $\bm{A}\in\mathbb{R}^{N\times N}$ denotes the
adjacency matrix, $\bm{X}\in\mathbb{R}^{N\times d}$ denotes the feature
matrix, and $N$ is the number of nodes. Each node $i\in\mathcal{V}$ is
associated with a feature vector $\bm{x}_{i}\in\mathbb{R}^{d}$ (indexed by the
$i$-th row in $\bm{X}$) and a one-hot class label
$\bm{y}_{i}\in\mathbb{R}^{C}$, where $C$ is the number of class labels. Given
a training set $\mathcal{V}_{l}$ with labels, the goal is to classify the
nodes in the unlabeled set
$\mathcal{V}_{u}=\mathcal{V}\setminus\mathcal{V}_{l}$ via learning effective
node representations. Let $f_{\theta}$ denote the GNN model, where $\theta$
denotes model parameters. The prediction for a node is
$\hat{\bm{y}}_{i}=f_{\theta}((\bm{A},\bm{X}))\in\mathbb{R}^{C}$. Recalling the
batch training in a large-scale graph, the plain cross-entropy loss in a batch
$\mathcal{B}$ is given by:

$\mathcal{L}^{\mathrm{Plain}}(\theta)=\frac{1}{|\mathcal{B}|}\sum_{i\in\mathcal{B}}H(\bm{y}_{i},\hat{\bm{y}}_{i})=-\frac{1}{|\mathcal{B}|}\sum_{i\in\mathcal{B}}\sum_{c=1}^{C}y_{i,c}\log{\hat{y}_{i,c}},$
(1)

where $H$ is the cross-entropy function. Sub-graphs are sampled to build each
batch, and $\mathcal{B}$ contains the nodes in the sampled sub-graphs for
training. $|\mathcal{B}|$ denotes the number of training nodes in
$\mathcal{B}$.

#### Model analysis.

We empirically study the label bias phenomenon, over-prediction and
overfitting issues of Cluster-GCN models trained on the ogbn-products dataset.
Other sub-graph sampling methods with the similar issue are shown in Appendix.
First, to evaluate the label bias, we define the probability of nodes with
class $c$ in a batch as:
$p_{c}=\frac{\sum_{i\in\mathcal{B}}\mathbbm{1}_{{y}_{i,c}=1}}{|\mathcal{B}|}$,
where $\mathbbm{1}_{{y}_{i,c}=1}$ indicates whether node $i$ belongs to class
$c$. The mean and standard variance of $p_{c}$ among batches are shown in the
right part of Figure 1. We could observe that the standard variance is
significantly larger when batch size is small, while the mean value is
relatively stable. It means that nodes within a small batch tend to belong to
certain classes, instead of evenly distribute across all classes. Such label
bias is inherent in sub-graph sampling methods, such as clustering [10] and
random walk sampling [52], since positively related nodes are more likely to
be selected together in a sampled sub-graph.

We further analyze the training and testing losses of Cluster-GCN in Figure 1.
Different from using uniform batch labels in traditional machine learning, the
label bias in the batches further leads the GNN model to over-confidently
attend (i.e., produce large prediction probabilities) on the ground-truth
classes by minimizing the vanilla cross-entropy loss. The over-confident
prediction overfits the training set and accelerates the decrease of training
loss, but poorly generalizes to the testing set as shown in the increased
testing loss. Comparing with the batch size of $2$, a larger batch size of
$32$ reduces the variance of $p_{c}$ and relieves the label bias to some
extent, which brings a smaller testing loss and better generalization
performance. However, the big batch size would improve computation and memory
costs, which is not inline with the purpose of sub-graph batch training on
large graphs.

#### Label smoothing.

To overcome over-confidence and overfitting, label smoothing has been proposed
to mix the one-hot hard labels with uniform distribution in the image
classification [29, 48, 27, 51] and natural language processing [17, 28]. To
be specific, considering a training node $i$, its smoothed label is given by:
$\bm{y}_{i}^{LS}=(1-\alpha)\bm{y}_{i}+\alpha\bm{1}/C$, where
$\bm{1}/C=[1/C,\dots,1/C]^{\top}\in\mathbb{R}^{C}$ is a uniform distribution
and $\alpha$ is the regularization strength. Then the cross-entropy is given
by:

$\mathcal{L}^{LS}(\theta)=\frac{1}{|\mathcal{B}|}\sum_{i\in\mathcal{B}}H(\bm{y}_{i}^{LS},\hat{\bm{y}}_{i})=\frac{1}{|\mathcal{B}|}\sum_{i\in\mathcal{B}}(1-\alpha)H(\bm{y}_{i},\hat{\bm{y}}_{i})+\alpha
H(\bm{1}/C,\hat{\bm{y}}_{i}).$ (2)

By minimizing $\mathcal{L}^{LS}(\theta)$, we are able to prevent the model
from being over-confident on the ground-truth $\bm{y}_{i}$, via a penalty that
enforces non-nigligible prediction probabilities on the other classes.

##  3 Adaptive Label Smoothing

The above label smoothing fails to adapt to graph data by ignoring two
informative attributes: the label distribution within local neighborhood and
the global label relevance. First, the nodes’ labels are not independently
distributed, and correlate positively to their local neighbors [22, 41, 23,
54, 61]. The vanilla label smoothing assumes that node labels are independent
and identically distributed, and applies the uniform distribution to
regularize representation learning. It could mislead the model prediction to
attend on the negatively related labels. Second, in the overall graph, the
relevance between each pair of labels is different from each other. For
example, the pair-wise collaboration strengths among engineering, ecology, and
physical researchers are unbalanced in the academic networks [30]. In the
hierarchical GNNs, it is commonly assumed that the connections between labeled
communities should be sparse [50, 59], instead of the uniform and full
connection. The vanilla label smoothing with fixed uniform distribution cannot
properly learn the latent global label relevance for the downstream task.

###  3.1 Proposed Techniques

In this work, we propose ALS to calibrate the label bias and regularize the
sub-graph batch training for the large-scale graph. ALS consists of three
parts: (1) a pre-processing step of label propagation to obtain the prior
knowledge of neighborhood label distribution; (2) a label refinement step to
correct the prior knowledge and learn the global label relevance in the
training phase; and (3) a smooth pacing function to gradually schedule the
smooth strength and avoid the overly label regularization.

#### Label propagation.

Based on the expectation that the two connected nodes are likely to have the
same label according to graph homophily, label propagation passes labels
iteratively to learn the label predictions [23, 41, 1, 2]. However, most of
them involve parameters and are expensive to be trained. To scale to large-
scale graphs, we simplify the label propagation by removing the trainable
weights, and conduct it only as a pre-processing step. Specifically, let
$\bm{Y}^{(k)}=[\bm{y}^{(k)}_{1},\cdots,\bm{y}^{(k)}_{N}]^{\top}\in\mathbb{R}^{N\times
C}$ denote the propagated label matrix obtained from the $k$-th iteration of
label propagation. At the $k+1$-th iteration, we update the propagated label
matrix as follows:

$\bm{Y}^{(k+1)}=(1-\beta)\bm{D}^{-1}\bm{A}\bm{Y}^{(k)}+\beta\bm{Y}^{(0)}.$ (3)

For the initial label matrix
$\bm{Y}^{(0)}=[\bm{y}^{(0)}_{1},\cdots,\bm{y}^{(0)}_{N}]^{\top}$, it consists
of one-hot hard labels $\bm{y}_{i}$ for training nodes and zero vectors
otherwise. $\beta$ is residual strength to preserve the initial training
labels, and $\bm{D}$ is the diagonal degree matrix of $\bm{A}$. The label
propagation in Eq. (3) is similar in spirit to [60], but we preserve
$\bm{Y}^{(0)}$ to avoid the overwhelming of training labels. After $K$
iterations of label propagation, we obtain the prior knowledge of neighborhood
label distribution $\bm{Y}^{(K)}$ up to $K$ hops away. Such prior knowledge
provides enough neighborhood information to be refined. Our label propagation
has a good trade-off between the efficiency and effectiveness.

#### Label refinement.

In this step, we aim to refine propagated label matrix $\bm{Y}^{(K)}$.
Specifically, given the propagated label $\bm{y}^{(K)}_{i}\in\mathbb{R}^{C}$
of training node $i$ (i.e., indexed from the $i$-th row of $\bm{Y}^{(K)}$), we
correct it by:
$\bm{y}_{i}^{\mathrm{soft}}=\mathrm{Softmax}(\bm{W}\bm{y}^{(K)}_{i})$.
$\bm{W}\in\mathbb{R}^{C\times C}$ is a trainable matrix. The smoothed label
used to regularize model training is then given by:

$\bm{y}_{i}^{ALS}=(1-\alpha)\bm{y}_{i}+\alpha\bm{y}_{i}^{\mathrm{soft}}.$ (4)

Notably, element $W_{c,j}$ indicates the latent relevance between classes $c$
and $j$, and is shared globally by all nodes over the graph. Considering node
$i$, the real relevance to class $c$ is corrected to be proportional to
$\sum_{j}W_{c,j}{y}^{(K)}_{i,j}$. To well learn the global label relevance in
$\bm{W}$, we jointly train with classification task and compute the batch loss
as follows:

$\begin{array}[]{rl}\mathcal{L}^{ALS}(\theta,\bm{W})&=\frac{1}{|\mathcal{B}|}\sum_{i\in\mathcal{B}}H(\bm{y}_{i}^{ALS},\hat{\bm{y}}_{i})+\gamma\mathrm{KL}(\bm{y}_{i}^{\mathrm{soft}},\bm{1}/C)\\\
&=\frac{1}{|\mathcal{B}|}\sum_{i\in\mathcal{B}}(1-\alpha)H(\bm{y}_{i},\hat{\bm{y}}_{i})+\alpha
H(\bm{y}_{i}^{\mathrm{soft}},\hat{\bm{y}}_{i})+\gamma\mathrm{KL}(\bm{y}_{i}^{\mathrm{soft}},\bm{1}/C),\end{array}$
(5)

where $\mathrm{KL}$ denotes the KL distance of two probability distribution
vectors, and $\gamma$ is a positive hyperparameter. Compared with the
traditional label smoothing in Eq. (2), ALS relaxes the uniform distribution
to learn the optimal soft label $\bm{y}_{i}^{\mathrm{soft}}$ and adapt to the
downstream task. On one hand, parameter $\bm{W}$ is updated to learn the
global label relevance and produce a reasonable $\bm{y}_{i}^{\mathrm{soft}}$.
On the other hand, the KL distance constraint is exploited to avoid
$\bm{y}_{i}^{\mathrm{soft}}$ collapsing into the one-hot hard target
$\bm{y}_{i}$ and guarantee the divergence.

#### Smooth pacing.

Considering the batch training in large-scale graph, the constant smoothing
strength $\alpha$ may overly regularize model at the initial training phase.
Given the randomly initialized parameter $\bm{W}$, the soft label
$\bm{y}_{i}^{\mathrm{soft}}$ will mislead model prediction $\hat{\bm{y}}_{i}$
to attend on the unrelated classes. Motivated from the batch pacing in
curriculum learning [3, 24], we propose a smooth pacing function to gradually
schedule the appropriate smoothing strength $\alpha_{t}$ at the $t$-th epoch.
At the early phase, since the over-confident prediction has not appeared, we
use a small $\alpha_{t}$ to let model learn the correct prediction. With the
ongoing of training, we gradually improve $\alpha_{t}$ to regularize model.
Specially, we consider the following two categories of pacing function: (1) a
linear pacing function of $\alpha_{t}=\min(r\cdot t,\alpha_{\max})$, where $r$
is pacing rate and $\alpha_{\max}$ is the maximum smoothing strength; (2) an
exponential pacing function of $\alpha_{t}=\min(b\cdot\mathrm{exp}(r\cdot
t),\alpha_{\max})$, where $b$ is the initial smoothing strength at epoch
$t=0$. In our ALS, we replace the constant smoothing strength $\alpha$ in Eq.
(5) with $\alpha_{t}$.

###  3.2 Model Analysis

#### Scalability analysis.

Based on the sparse matrix multiplication, the time complexity of label
propagation is $\mathcal{O}(K||\bm{A}||_{0}C)$, where $||\bm{A}||_{0}$ is
number of nonzeros in $\bm{A}$. Since the label propagation is conducted in
pre-processing step, it could scale to the large-scale graph on CPU platforms
with large memory. We thus ignore its memory complexity. The computation of
label refinement mainly lines in the matrix multiplication with $\bm{W}$.
Considering any a backbone network, the extra time complexity is only
$\mathcal{O}(|\mathcal{B}|C^{2})$, and the extra memory complexity is only
$\mathcal{O}(C^{2})$. Therefore, our ALS can augment any scalable algorithms
to handle the large-scale graph.

#### Generalization analysis.

We apply ALS to augment Cluster-GCN and train on dataset ogbn-products. As
shown in Figure 1, comparing with the plain Cluster-GCN, our ALS has larger
training losses but is accompanied with smaller testing losses. In other word,
our model has better generalization performance on testing test by avoiding
the overfitting on training set. Specially, the label bias problem is much
severe in the small batch size of $2$. In this case, the testing loss of plain
Cluster-GCN increases significantly due to the extremely over-confident
prediction and the overfitting on training set. In contrast, our model could
still avoid the over-confident prediction by even increasing the training loss
at the end of training. The label smoothing regularization brings and
maintains a lower testing loss.

#### Comparison to previous work.

Although the label smoothing has been applied in computer vision and natural
language processing [29, 26], it has not been studied to regularize GNN models
for the graph data analytics. The previous GNNs are mainly developed to
process small graph. In this paper, we observe the label bias problem resulted
from the sub-graph batch training in the large-scale graph, and analyze the
over-confident prediction and overfitting issue. Compared with the traditional
label smoothing with uniform distribution, we propose ALS to adapt to the
graph data. We are aware that recently there have been some label smoothing
works to learn the soft label [17, 13], which is similar to our label
refinement module. However, they are not targeted for graph data, missing to
incorporate the graph structure. In the experiments, we empirically
demonstrate that all the three modules in ALS are crucial to regularize the
large-scale graph training.

Another similar line of work is label propagation. Most of previous methods
involve trainable weights and cannot scale to the large-scale graph [23, 41,
1, 2]. For those simple and scalable methods, they either directly use the
propagated labels to predict the testing nodes [60], or concatenate them to
node features as the nodes’ inputs [37]. In this work, we exploit the pre-
processed soft label to regularize the model training. In the experiments, we
empirically show that the label smoothing is a better way to exploit this
prior label knowledge.

##  4 Experiments

In this section, we empirically evaluate the effectiveness of ALS on several
real-world datasets. Overall, we aim to answer four research questions as
follows. Q1: Can ALS effectively regularize the model to obtain better
generalization performance, comparing with the plain model and label smoothing
with uniform distribution? Q2: How does each module of ALS affect its
performance? Q3: How does ALS preform comparing with the other exploitation
ways of prior label knowledge? Q4: How do the hyperparameters influence the
performance of ALS?

###  4.1 Experiment Setup

#### Datasets.

We evaluate our proposed models on $4$ graphs with different scales using node
classification tasks, following the previous large-scale graph representation
learning efforts. These benchmark datasets include Flickr [52], Reddit [20],
ogbn-products and ogbn-mag [21], whose node numbers are 89K, 233K, 2449K and
1940K, respectively. Their data statistics are provided in Appendix.

#### Backbone frameworks.

We mainly evaluate ALS on the scalable backbone frameworks based upon sub-
graph sampling. Since the precomputing methods are another important lines of
scalable graph representation learning, we conduct our method on them to
demonstrate the general effectiveness. For the sub-graph sampling based
methods, we adopt the popular backbone frameworks of GraphSAGE [20], Cluster-
GCN [10] and GraphSAINT [52]. For the pre-computing based methods, we choose
the backbone frameworks of MLP and SIGN [34]. The detailed descriptions of
these five frameworks are provided in Appendix. Note that we aim to
demonstrate the general effectiveness of ALS in improving model generalization
for the diverse scalable learning frameworks, instead of achieving the state-
of-the-art performance on each classification task. Therefore, for each
experiment on benchmark datasets, we conduct and compare three
implementations: the plain scalable model trained with cross-entropy loss in
Eq. (1), the model augmented with conventional label smoothing (LS) as shown
in Eq. (2), and the model augmented with ALS as shown in Eq. (5).

#### Implementation.

We directly use the implementations of the backbone networks either from the
their official repositories or based on the official examples of PyTorch
Geometric. We further implement LS and ALS over each backbone model. For LS
with uniform distribution, we set the constant smoothing strength $\alpha$ as
$0.1$, which is widely applied in regularizing image classification. For our
ALS, we choose the appropriate hyperperameters of residual strength $\beta$
and step $K$ in the label propagation, and also determine the KL distance
constraint $\gamma$ as well as the smooth pacing rate $r$. While the linear
smooth pacing is adopted in the sub-graph sampling methods, the exponential
pacing function is used in the precomputing methods. The detailed choices on
four datasets are shown in Appendix. We study the influences of these
hyperparameters in the experiments, and show that our model is not sensitive
to them within a wide value range.

###  4.2 Experiment Results

#### Generalization improvement by label smoothing.

To provide answers for the research question Q1, Table 1 summarizes the
comprehensive comparisons among the plain model without any label smoothing,
the regularized model with LS, and the regularized model with ALS over each
combination of backbone framework and dataset. It is observed that our ALS can
achieve superior performances in $18$ cases out the 20 in total. Specifically,
compared with the plain frameworks based on sub-graph sampling, both LS and
ALS can generally improve test accuracy. The sampling methods assign connected
nodes possibly with the same label into a sub-graph, which will lead to label
bias within a batch. The label bias will make model over-confidently attend on
the prediction of the ground-truth class, and may mislead model to fall into
local minimums and decrease its generalization ability. While LS uses the
uniform distribution to regularize model’s prediction on other classes, our
proposed ALS calibrates model more accurately by considering the local
neighbors and using the global label refinement to correct the smoothed label.

Compared with the plain precomputing model, our ALS can still generally
improve the test accuracy, although LS tends to deteriorate model performance.
Trained on GeForce RTX 2080 Ti GPU, the official implementations of MLP and
SIGN use a full batch of training nodes. Since the label bias is not a big
concern in such full batch training scenarios, the crude LS over-regularizes
models and further hinders the accurate predictions on the ground-truth class.
Notably, ALS exploits the informatic neighborhood label distribution to
calibrate the model prediction, considering that the connected nodes should be
close in the label space. Furthermore, the label refinement module learns to
correct the smoothed label and jointly trains with the cross-entropy
classification loss, which could adapt to the desired classification task.

Base frameworks | Methods | Flickr | Reddit | ogbn-products | ogbn-mag
---|---|---|---|---|---
GraphSAGE | Plain | 52.12$\pm$0.33 | 96.04$\pm$0.07 | 78.45$\pm$0.35 | 46.70$\pm$0.30
LS | 52.05$\pm$0.42 | 96.23$\pm$0.08 | 78.51$\pm$0.32 | 46.87$\pm$0.55
ALS | 52.34$\pm$0.21 | 96.26$\pm$0.08 | 78.64$\pm$0.46 | 47.06$\pm$0.38
Cluster-GCN | Plain | 49.78$\pm$0.25 | 94.32$\pm$0.21 | 80.16$\pm$0.45 | 37.58$\pm$0.33
LS | 49.98$\pm$0.41 | 95.18$\pm$0.17 | 80.06$\pm$0.23 | 37.69$\pm$0.31
ALS | 50.20$\pm$0.28 | 95.24$\pm$0.11 | 80.78$\pm$0.40 | 37.89$\pm$0.29
GraphSAINT | Plain | 51.43$\pm$0.20 | 95.05$\pm$0.13 | 79.07$\pm$0.33 | 47.67$\pm$0.24
LS | 51.58$\pm$0.23 | 95.02$\pm$0.12 | 79.27$\pm$0.22 | 47.74$\pm$0.38
ALS | 51.74$\pm$0.13 | 95.24$\pm$0.11 | 79.48$\pm$0.44 | 47.94$\pm$0.25
MLP | Plain | 46.36$\pm$0.20 | 70.95$\pm$0.15 | 61.07$\pm$0.21 | 27.16$\pm$0.18
LS | 46.03$\pm$0.93 | 71.61$\pm$0.16 | 61.20$\pm$0.18 | 27.00$\pm$0.22
ALS | 46.43$\pm$0.17 | 71.91$\pm$0.06 | 61.35$\pm$0.10 | 27.13$\pm$0.28
SIGN | Plain | 51.15$\pm$ 0.20 | 96.29$\pm$0.03 | 74.04$\pm$0.12 | 19.61$\pm$0.14
LS | 51.01$\pm$0.31 | 96.50$\pm$0.04 | 71.07$\pm$0.09 | 18.84$\pm$0.15
ALS | 50.84$\pm$0.50 | 96.56$\pm$0.03 | 74.20$\pm$0.09 | 19.77$\pm$0.10
Table 1: Test accuracies in percent of the plain model, the model with LS, and
the model augmented with ALS. The best performance in each study is in bold.

#### Ablation studies.

To demonstrate how each module of ALS affects the generalization performance
and answer the research question Q2, we perform ablation studies over two sub-
graph sampling based backbone networks, i.e., Cluster-GCN and GraphSAINT. In
particular, to study the contribution of label propagation in ALS, we ablate
it and replace soft label $\bm{y}^{(K)}_{i}$ with one-hot hard label
$\bm{y}_{i}$ to compute the smoothed label used for model training. The
smoothed label is then obtained by:
$\bm{y}_{i}^{ALS}=(1-\alpha_{t})\bm{y}_{i}+\alpha_{t}\mathrm{Softmax}(\bm{W}\bm{y}_{i})$.
To ablate the label refinement, we use the soft label $\bm{y}^{(K)}_{i}$
obtained from the label propagation to compute the smoothed label, i.e.,
$\bm{y}_{i}^{ALS}=(1-\alpha_{t})\bm{y}_{i}+\alpha_{t}\bm{y}^{(K)}_{i}$. At the
same time, we remove the KL distance constraint in Eq. (5). To ablate smooth
pacing module, we use a constant smoothing strength $\alpha$ with $0.1$ to
generate smoothed label, i.e.,
$\bm{y}_{i}^{ALS}=0.9*\bm{y}_{i}+0.1*\mathrm{Softmax}(\bm{W}\bm{y}^{(K)}_{i})$.

We summarize the ablation studies on the three modules of ALS in Table 2. It
is observed that the ablation of any module will decrease the test accuracy,
which empirically demonstrate their importances to adapt label smoothing in
regularizing the graph representation learning. Comparing with the ablation of
label propagation, we observe that the removing of label refinement and smooth
pacing extremely damages the performance of ALS. Even with inaccurate prior
knowledge of neighborhood label distribution, the label refinement module
could be supervised to refine the smoothed label correctly to regularize model
prediction. The smooth pacing allocates a smaller smoothing strength
$\alpha_{t}$ at the initial training phase, since the smoothed label is far
from being well refined, and then gradually improves $\alpha_{t}$ to
regularize model from being overfitting.

Backbones | Methods | Flickr | Reddit | ogbn-products | ogbn-mag
---|---|---|---|---|---
| ALS | 50.20$\pm$0.28 | 95.24$\pm$0.11 | 80.78$\pm$0.40 | 37.89$\pm$0.29
Cluster- | w/o label propagation | 50.06$\pm$0.24 | 95.18$\pm$0.15 | 80.61$\pm$0.40 | 37.85$\pm$0.20
GCN | w/o label refinement | 49.97$\pm$0.25 | 95.06$\pm$0.16 | 80.19$\pm$0.41 | 37.62$\pm$0.28
| w/o smooth pacing | 50.09$\pm$0.20 | 95.12$\pm$0.13 | 80.53$\pm$0.62 | 37.87$\pm$0.30
| ALS | 51.74$\pm$0.13 | 95.24$\pm$0.11 | 79.48$\pm$0.44 | 47.94$\pm$0.25
Graph- | w/o label propagation | 51.59$\pm$0.13 | 95.23$\pm$0.08 | 79.31$\pm$0.44 | 47.91$\pm$0.35
SAINT | w/o label refinement | 51.67$\pm$0.19 | 95.17$\pm$0.09 | 79.11$\pm$0.55 | 47.79$\pm$0.39
| w/o smooth pacing | 51.64$\pm$0.24 | 95.13$\pm$0.09 | 79.09$\pm$0.61 | 47.83$\pm$0.24
Table 2: Test accuracies in percent of ALS and its three variants obtained by
ablating specific modules.

#### Comparison of prior label knowledge.

Besides the label smoothing, to scale to the large-scale graph, there are two
other lines of work to exploit the prior knowledge of neighborhood label
distribution $\bm{y}^{(K)}_{i}$. First, the label propagation method uses
$\bm{y}^{(K)}_{i}$ to predict test nodes without any learnable parameters.
Second, one can concatenate $\bm{y}^{(K)}_{i}$ with node features that are
treated as the input for classification model. To answer the research question
Q3, we compare these three different methods to exploit $\bm{y}^{(K)}_{i}$.
Specifically, we implement label smoothing and concatenate $\bm{y}^{(K)}_{i}$
over the GraphSAINT backbone, and directly adopt the proposed label
propagation module.

Table 3 summarizes the test accuracies on the four benchmark datasets, where
ALS generally achieves the superior performances. The label propagation fails
to learn the informatic label prediction on ogbn-mag dataset, since it cannot
combine the neighbor labels effectively without modeling the diverse node/edge
types in heterogeneous graphs. The label propagation cannot adapt the prior
label knowledge to classification tasks without any learnable parameters.
Compared with the input concatenation, our label smoothing approach directly
regularizes the model prediction to avoid the over-confident prediction and
thus obtains better generalization performance.

Methods | Flickr | Reddit | ogbn-products | ogbn-mag
---|---|---|---|---
Label propogation | 50.33 | 92.33 | 73.45 | -
GraphSAINT+Label Input | 51.75 ± 0.22 | 95.10 ± 0.09 | 76.90 ± 0.32 | 47.20 ± 0.29
GraphSAINT+ALS | 51.74$\pm$0.13 | 95.24$\pm$0.11 | 79.48$\pm$0.44 | 47.94$\pm$0.25
Table 3: Test accuracies in percent of different label exploitation methods. Backbone | Methods | obgn-products
---|---|---
C&S | Plain | 84.18$\pm$0.07
LS | 84.22$\pm$0.09
ALS | 84.28$\pm$0.06
Table 4: Test accuracy in percent.

Recently, scalable learning method of C&S is proposed to refine model
prediction in the post-processing step [22], and shows promising performance
on ogbn-products. By training a simple MLP to obtain the initial label
predictions, C&S propagates prediction errors and labels to obtain smoothed
prediction results. To demonstrate that ALS is general to any scalable
learning framework, we use LS and ALS to regularize the training of MLP module
in C&S, and compare test accuracies in Table 4. By regularizing MLP to obtain
better initial label predictions, our proposed ALS can further improve the
test accuracy up to $84.28$, which is the state-of-the-art performance in the
leader board.

#### Hyperparameter studies.

To answer the research question Q4, we conduct experiments with different
values of pacing rate $r$, loss hyperparameter $\gamma$, residual strength
$\beta$, and label propagation step $K$. Figure 2 illustrates the
hyperparameter studies of GraphSAINT+ALS on ogbn-products. In general, within
appropriate value ranges, most of these hyperparameter settings can achieve
test accuracies larger than $79.3\%$, which is much superior than the baseline
GraphSAINT. Specifically, the over-small (or large) pacing rate $r$ damages
the test performance due to the insufficient (or excessive) label smoothing
regularization. The loss hyperparameter $\gamma$ should be large enough (e.g.,
$\gamma\geq 10^{-3}$), so as to avoid the learned smoothed label collapsing
into one-hot hard target and to guarantee its regularization effect. The
superior performance brought by $\beta\leq 0.5$ demonstrates the importance of
neighborhood label distribution in learning the structure-aware smoothed
label. Based on Eq. (3), with a smaller $\beta$, we tend to aggregate
neighborhood labels during the label propagation. Similar to the common GNN
settings, a small value of $K$ is sufficient to aggregate the positively
related neighbors to model the smoothed label correctly.

Figure 2: Hyperparameter studies of $r$, $\gamma$, $\beta$, and $K$ of
GraphSAINT+ALS on ogbn-products.

#### Global label relevance visualization.

We visualize $\mathrm{Softmax}$ transformation of the global label relevance
matrix along each row, i.e., $\mathrm{Softmax}(\bm{W})$. Note that $\bm{W}$ is
learned on backbone framework GraphSAINT and dataset Flickr. As shown in
Figure 3, we observe that each class label has unbalanced relevance strengths
to the other classes. This is in line with our motivation that the pair-wise
label relevance is different from each other, and far away from the uniform
distribution. As demonstrated in the previous experiments, the modeling of
such global label relevance delivers the superior performances comparing with
LS.

##  5 Related Work

Figure 3: Visualization of $\mathrm{Softmax}(\bm{W})$ of ALS on Flickr. X and
Y axis denote node label.

#### Graph neural networks.

GNNs have shown superiority in processing graphs, i.e., data with rich
relational structures. GNNs could be categorized into spectral domain and
spatial domain models. The spectral models [5] extends convolution on images
to graph data by modeling on the spectrum of graph Laplacian. Models designed
from the spatial perspective simplify the spectral models. Spatial models such
as ChebNet [12], GCN [25], GAT [40] and GIN [47], could also be understood
from the message passing perspective. GNNs are playing increasingly crucial
roles in various applications such as recommender systems [49, 8, 55], social
network analysis [14, 58, 56, 57], and biochemical module analysis [18, 59].

#### Scalable graph representation learning.

Two types of methods have been proposed to tackle scalability issue of GNNs,
including sub-graph sampling methods [20, 49, 6, 15, 7, 10, 52] and
precomputing methods [45, 34, 4]. To reduce computation and memory cost, the
sub-graph sampling methods feed GNNs only with a small batch of sub-graphs,
which consist of subsets of nodes and edges. Specifically, ClusterGCN [10]
conducts training on sampled sub-graphs in each batch, but the sub-graphs are
obtained through clustering algorithms. GraphSAINT [52] samples sub-graphs
that are appropriately connected for information propagation, where a
normalization technique is also proposed to eliminate bias. The major
limitation for sub-graph based methods is that distant nodes in the original
graph are unlikely to be fed into the GNNs in the same batch, thus leading to
label bias in the trained models. The precomputing methods of SIGN [34] and
SGC[45] remove trainable weights, and propagate node features over the graph
in advance to store the smoothed features.

#### Label propagation.

Label propagation distributes the observed node labels over the graph
following the connection between nodes [54, 62]. It has been used for semi-
supervised training on graph data [53], where node labels are partially
observed. The assumption behind is that labels and features change smoothly
over the edges of the graph. It has also been proposed to combine feature
propagation with label propagation towards a unified message passing scheme
[36]. Some recent work connects GNNs with label propagation [41, 23] by
studying how labels/features spread over a graph and how the initial
feature/label of one node influences the prediction of another node.

#### Label smoothing.

Label smoothing improves the generalization [38, 32] and robustness [31, 19,
35] of a deep neural network. Label smoothing replaces one-hot labels with
smoothed labels. It has been shown that label smoothing has similar effect as
randomly replacing some of the ground-truth labels with incorrect values at
each mini-batch [46]. [31] proposes reverse cross entropy for gradient
smoothing. It encourages a model to better distinguish adversarial examples
from normal ones in representation space. [44] proposes the graduated label
smoothing method, where high-confidence predictions are assigned with higher
smoothing penalty than low-confidence ones.

##  6 Conclusions

In this paper, we point out the inherent label bias within the sampled sub-
graphs for the mini-batch training of large-scale graph. By minimizing vanilla
cross-entropy loss, we empirically analyze that such label bias will make GNN
model over-confidently predict the ground-truth class and lead to overfitting
issue. To overcome the label bias and the resulted over-confident prediction,
we propose an adaptive label smoothing to replace the one-hot hard target with
smoothed label, which allocates prediction confidence to other classes to
avoid overfitting. Specially, we learn the smoothed label with the prior
knowledge of local neighborhood label distribution and the global label
refinement to adapt to graph data on hand. The experiments show that our
algorithm could generally improve the test performance by relieving the
overfitting on biased labels.

## References

  * [1] R. K. Ando and T. Zhang.  Learning on graph with laplacian regularization.  Advances in neural information processing systems, 19:25, 2007.
  * [2] M. Belkin, P. Niyogi, and V. Sindhwani.  Manifold regularization: A geometric framework for learning from labeled and unlabeled examples.  Journal of machine learning research, 7(11), 2006.
  * [3] Y. Bengio, J. Louradour, R. Collobert, and J. Weston.  Curriculum learning.  In Proceedings of the 26th annual international conference on machine learning, pages 41–48, 2009.
  * [4] A. Bojchevski, J. Klicpera, B. Perozzi, A. Kapoor, M. Blais, B. Rózemberczki, M. Lukasik, and S. Günnemann.  Scaling graph neural networks with approximate pagerank.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2464–2473, 2020.
  * [5] J. Bruna, W. Zaremba, A. Szlam, and Y. LeCun.  Spectral networks and locally connected networks on graphs.  arXiv preprint arXiv:1312.6203, 2013.
  * [6] J. Chen, T. Ma, and C. Xiao.  Fastgcn: fast learning with graph convolutional networks via importance sampling.  arXiv preprint arXiv:1801.10247, 2018.
  * [7] J. Chen, J. Zhu, and L. Song.  Stochastic training of graph convolutional networks with variance reduction.  arXiv preprint arXiv:1710.10568, 2017.
  * [8] L. Chen, L. Wu, R. Hong, K. Zhang, and M. Wang.  Revisiting graph based collaborative filtering: A linear residual graph convolutional network approach.  In Proceedings of the AAAI Conference on Artificial Intelligence, pages 27–34, 2020.
  * [9] M. Chen, Z. Wei, Z. Huang, B. Ding, and Y. Li.  Simple and deep graph convolutional networks.  In International Conference on Machine Learning, pages 1725–1735. PMLR, 2020.
  * [10] W.-L. Chiang, X. Liu, S. Si, Y. Li, S. Bengio, and C.-J. Hsieh.  Cluster-gcn: An efficient algorithm for training deep and large graph convolutional networks.  In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 257–266, 2019.
  * [11] A. Ching, S. Edunov, M. Kabiljo, D. Logothetis, and S. Muthukrishnan.  One trillion edges: Graph processing at facebook-scale.  Proceedings of the VLDB Endowment, 8(12):1804–1815, 2015.
  * [12] M. Defferrard, X. Bresson, and P. Vandergheynst.  Convolutional neural networks on graphs with fast localized spectral filtering.  arXiv preprint arXiv:1606.09375, 2016.
  * [13] Q. Ding, S. Wu, H. Sun, J. Guo, and S.-T. Xia.  Adaptive regularization of labels.  arXiv preprint arXiv:1908.05474, 2019.
  * [14] W. Fan, Y. Ma, Q. Li, Y. He, E. Zhao, J. Tang, and D. Yin.  Graph neural networks for social recommendation.  In The World Wide Web Conference, pages 417–426, 2019.
  * [15] H. Gao, Z. Wang, and S. Ji.  Large-scale learnable graph convolutional networks.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1416–1424, 2018.
  * [16] Y. Gao, W. Wang, C. Herold, Z. Yang, and H. Ney.  Towards a better understanding of label smoothing in neural machine translation.  In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 212–223, 2020.
  * [17] A. Ghoshal, X. Chen, S. Gupta, L. Zettlemoyer, and Y. Mehdad.  Learning better structured representations using low-rank adaptive label smoothing.  In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
  * [18] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl.  Neural message passing for quantum chemistry.  In International conference on machine learning, pages 1263–1272. PMLR, 2017.
  * [19] M. Goibert and E. Dohmatob.  Adversarial robustness via adversarial label-smoothing.  arXiv preprint, 2019.
  * [20] W. L. Hamilton, R. Ying, and J. Leskovec.  Inductive representation learning on large graphs.  arXiv preprint arXiv:1706.02216, 2017.
  * [21] W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec.  Open graph benchmark: Datasets for machine learning on graphs.  arXiv preprint arXiv:2005.00687, 2020.
  * [22] Q. Huang, H. He, A. Singh, S.-N. Lim, and A. R. Benson.  Combining label propagation and simple models out-performs graph neural networks.  arXiv preprint arXiv:2010.13993, 2020.
  * [23] J. Jia and A. R. Benson.  Residual correlation in graph neural network regression.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 588–598, 2020.
  * [24] L. Jiang, D. Meng, Q. Zhao, S. Shan, and A. Hauptmann.  Self-paced curriculum learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
  * [25] T. N. Kipf and M. Welling.  Semi-supervised classification with graph convolutional networks.  arXiv preprint arXiv:1609.02907, 2016.
  * [26] W. Li, G. Dasarathy, and V. Berisha.  Regularization via structural label smoothing.  In International Conference on Artificial Intelligence and Statistics, pages 1453–1463. PMLR, 2020.
  * [27] J. Lienen and E. Hüllermeier.  From label smoothing to label relaxation.  In Proceedings of the 35th AAAI Conference on Artificial Intelligence, AAAI, Online, February 2-9, 2021. AAAI Press, 2021.
  * [28] C. Meister, E. Salesky, and R. Cotterell.  Generalized entropy regularization or: There’s nothing special about label smoothing.  arXiv preprint arXiv:2005.00820, 2020.
  * [29] R. Müller, S. Kornblith, and G. Hinton.  When does label smoothing help?  arXiv preprint arXiv:1906.02629, 2019.
  * [30] M. E. Newman.  Coauthorship networks and patterns of scientific collaboration.  Proceedings of the national academy of sciences, 101(suppl 1):5200–5205, 2004.
  * [31] T. Pang, C. Du, Y. Dong, and J. Zhu.  Towards robust detection of adversarial examples.  arXiv preprint arXiv:1706.00633, 2017.
  * [32] G. Pereyra, G. Tucker, J. Chorowski, Ł. Kaiser, and G. Hinton.  Regularizing neural networks by penalizing confident output distributions.  arXiv preprint arXiv:1701.06548, 2017.
  * [33] B. Perozzi, R. Al-Rfou, and S. Skiena.  Deepwalk: Online learning of social representations.  In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 701–710, 2014.
  * [34] E. Rossi, F. Frasca, B. Chamberlain, D. Eynard, M. Bronstein, and F. Monti.  Sign: Scalable inception graph neural networks.  arXiv preprint arXiv:2004.11198, 2020.
  * [35] C. Shen, Y. Peng, G. Zhang, and J. Fan.  Defending against adversarial attacks by suppressing the largest eigenvalue of fisher information matrix.  arXiv preprint arXiv:1909.06137, 2019.
  * [36] Y. Shi, Z. Huang, W. Wang, H. Zhong, S. Feng, and Y. Sun.  Masked label prediction: Unified message passing model for semi-supervised classification.  arXiv preprint arXiv:2009.03509, 2020.
  * [37] C. Sun and G. Wu.  Scalable and adaptive graph neural networks with self-label-enhanced training.  arXiv preprint arXiv:2104.09376, 2021.
  * [38] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna.  Rethinking the inception architecture for computer vision.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016.
  * [39] J. Tang, H. Gao, X. Hu, and H. Liu.  Exploiting homophily effect for trust prediction.  In Proceedings of the sixth ACM international conference on Web search and data mining, pages 53–62, 2013.
  * [40] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio.  Graph attention networks.  arXiv preprint arXiv:1710.10903, 2017.
  * [41] H. Wang and J. Leskovec.  Unifying graph convolutional neural networks and label propagation.  arXiv preprint arXiv:2002.06755, 2020.
  * [42] J. Wang, P. Huang, H. Zhao, Z. Zhang, B. Zhao, and D. L. Lee.  Billion-scale commodity embedding for e-commerce recommendation in alibaba.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 839–848, 2018.
  * [43] R. Wang, Y. Yan, J. Wang, Y. Jia, Y. Zhang, W. Zhang, and X. Wang.  Acekg: A large-scale knowledge graph for academic data mining.  In Proceedings of the 27th ACM international conference on information and knowledge management, pages 1487–1490, 2018.
  * [44] S. Wang, Z. Tu, S. Shi, and Y. Liu.  On the inference calibration of neural machine translation.  arXiv preprint arXiv:2005.00963, 2020.
  * [45] F. Wu, A. Souza, T. Zhang, C. Fifty, T. Yu, and K. Weinberger.  Simplifying graph convolutional networks.  In International conference on machine learning, pages 6861–6871. PMLR, 2019.
  * [46] L. Xie, J. Wang, Z. Wei, M. Wang, and Q. Tian.  Disturblabel: Regularizing cnn on the loss layer.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4753–4762, 2016.
  * [47] K. Xu, W. Hu, J. Leskovec, and S. Jegelka.  How powerful are graph neural networks?  arXiv preprint arXiv:1810.00826, 2018.
  * [48] Y. Xu, Y. Xu, Q. Qian, H. Li, and R. Jin.  Towards understanding label smoothing.  arXiv preprint arXiv:2006.11653, 2020.
  * [49] R. Ying, R. He, K. Chen, P. Eksombatchai, W. L. Hamilton, and J. Leskovec.  Graph convolutional neural networks for web-scale recommender systems.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 974–983, 2018.
  * [50] R. Ying, J. You, C. Morris, X. Ren, W. L. Hamilton, and J. Leskovec.  Hierarchical graph representation learning with differentiable pooling.  arXiv preprint arXiv:1806.08804, 2018.
  * [51] L. Yuan, F. E. Tay, G. Li, T. Wang, and J. Feng.  Revisiting knowledge distillation via label smoothing regularization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3903–3911, 2020.
  * [52] H. Zeng, H. Zhou, A. Srivastava, R. Kannan, and V. Prasanna.  Graphsaint: Graph sampling based inductive learning method.  arXiv preprint arXiv:1907.04931, 2019.
  * [53] X. Zhang, W. S. Lee, et al.  Hyperparameter learning for graph based semi-supervised learning algorithms.  In NIPS, volume 33, page 101, 2006.
  * [54] D. Zhou, O. Bousquet, T. N. Lal, J. Weston, and B. Schölkopf.  Learning with local and global consistency.  In Advances in neural information processing systems, pages 321–328, 2004.
  * [55] H. Zhou, Q. Tan, X. Huang, K. Zhou, and X. Wang.  Temporal augmented graph neural networks for session-based recommendations.  arXiv preprint, 2021.
  * [56] K. Zhou, X. Huang, Y. Li, D. Zha, R. Chen, and X. Hu.  Towards deeper graph neural networks with differentiable group normalization.  arXiv preprint arXiv:2006.06972, 2020.
  * [57] K. Zhou, X. Huang, D. Zha, R. Chen, L. Li, S.-H. Choi, and X. Hu.  Dirichlet energy constrained learning for deep graph neural networks.  arXiv preprint arXiv:2107.02392, 2021.
  * [58] K. Zhou, Q. Song, X. Huang, and X. Hu.  Auto-gnn: Neural architecture search of graph neural networks.  arXiv preprint arXiv:1909.03184, 2019.
  * [59] K. Zhou, Q. Song, X. Huang, D. Zha, N. Zou, and X. Hu.  Multi-channel graph neural networks.  arXiv preprint arXiv:1912.08306, 2019.
  * [60] X. Zhu and Z. Ghahramani.  Learning from labeled and unlabeled data with label propagation.  arXiv preprint, 2002.
  * [61] X. Zhu, Z. Ghahramani, and J. D. Lafferty.  Semi-supervised learning using gaussian fields and harmonic functions.  In Proceedings of the 20th International conference on Machine learning (ICML-03), pages 912–919, 2003.
  * [62] X. J. Zhu.  Semi-supervised learning literature survey.  arXiv preprint, 2005.

##  Appendix A Appendix

###  A.1 Datasets

The dataset statistics of Flickr [52], Reddit [20], obgn-products [21], and
ogbn-mag [21] are listed in Table 5. Flickr is a social network, where the
nodes represent images and the edges denote the shared properties between two
images. The node classification task in Flickr is to categorize the types of
images. Reddit is a social netowork, where the nodes represent posts in Reddit
forum and the edges indicate the same user comments between two posts. The
node classification task in Reddit is to predict the communities of online
posts based on user comments. ogbn-products is an Amazon product co-purchasing
network, where the nodes represent products sold in Amazon and the edges
indicate the co-purchasing relationships between two products. The node
classification task in ogbn-products is to predict the category of a product.
ogbn-mag is a heterogeneous network extracted from the Microsoft Academic
Graph. It contains four types of entities: papers, authors, institutions, and
fields of study. The directed edges are categorized into four types–an author
is “affiliated with” an institution, an author “writes” a paper, a paper
“cites” a paper, and a paper “has a topic of” a field of study. The node
classification task in ogbn-mag is to predict the venue (conference or
journal) of each entity of paper.

Datasets | # Nodes | # Edges | # Classes | # Features | # Train/Validation/Test
---|---|---|---|---|---
Flickr | 89,250 | 899,756 | 7 | 500 | 0.50 / 0.25 / 0.25
Reddit | 232,965 | 11,606,919 | 41 | 602 | 0.66 / 0.10 / 0.24
ogbn-products | 2,449,029 | 61,859,140 | 47 | 100 | 0.08 / 0.02 / 0.90
ogbn-mag | 1,939,743 | 21,111,007 | 349 | 128 | 0.85 / 0.09 / 0.06
Table 5: Data statistics.

###  A.2 Backbone Frameworks

We evaluate our ALS on two main categories of scalable graph representation
learning frameworks: one is based on the sub-graph sampling and the other one
is based on precomputing. Although we aim at solving the label bias and over-
confident prediction in the sub-graph sampling methods, we show that our
method is general to both of these two scalable backbone frameworks.
Specifically, we adopt the following five representative backbone frameworks:

  * •

GraphSAGE [20] (sub-graph sampling based). It is a node-wise sampling method
to uniformly sample a batch of training nodes and their neighbors of different
orders. The sampled nodes and neighbors construct several sub-graphs to
formulate a batch.

  * •

Cluster-GCN [10] (sub-graph sampling based). It first conducts node clustering
algorithm to partition the input graph into a series of sub-graphs. During the
training phase, each batch is directly formulated by a random subset of
preprocessed sub-graphs.

  * •

GraphSAINT [52] (sub-graph sampling based). Starting from a subset of training
nodes, we choose the random walk sampler (i.e., the most powerful one as
reported in [52]) to sample their neighbors for constructing sub-graphs in one
batch.

  * •

MLP (precomputing based). MLP is widely used to classify nodes based on the
precomputed node features. Herein, MLP directly uses the original node
features, which has been shown to achieve good classification performance in
the graph data. In the precomputing methods, each node can be regarded as an
independent sample, and does not connect to its neighbors. The batch is thus
directly represented by an independent subset of training nodes.

  * •

SIGN [34] (precomputing based). In the preprocessing step, SIGN conducts
message-passing strategy and precomputes node features as: $\hat{\bm{A}}^{l}X$
for $l\in\\{1,\cdots,L\\}$. $\hat{\bm{A}}$ is a normalized adjacency matrix
used in GCN [25]. The precomputed node features of different orders are
concatenated together to augment the original node features. In the training
phase, the batch is constructed by a random subset of training nodes, and
taken as input to the downstream classification model of MLP.

We implement the above scalable backbone frameworks according to the official
examples of Pytorch Geometric111https://github.com/rusty1s/pytorch_geometric.
The basic model hyperparameters are defined in the examples or determined
according to their public literature, including batch size, learning rate,
weight decay, training epochs, hidden units, dropout rate, etc. All of the
sub-graph sampling methods apply a three-layer GNN model, while the
precomputing methods use a three-layer MLP. Following the official examples,
we use full batch training in the precomputing methods.

###  A.3 Implementation Details

We further implement LS and ALS over each backbone model. For LS with uniform
distribution, we set the constant smoothing strength $\alpha$ as $0.1$, which
is widely applied in regularizing image classification. For our ALS, we use
the linear pacing and exponential pacing functions for the sub-graph sampling
methods and precomputing methods, respectively. To have a fair comparison with
LS, we set $\alpha_{\max}=0.1$ in our ALS. For each combination of backbone
framework and dataset, we choose the appropriate hyperperameters of residual
strength $\beta$ and step $K$ in the label propagation, and also determine the
KL distance constraint $\gamma$ as well as the smooth pacing rate $r$. The
detailed hyperparameters involved in ALS are shown in Table 6. Notably,
comparing with the sub-graph sampling methods, we use the negative pacing rate
$r$ in the precomputing methods. Instead of using mini-batch training, the
official examples of MLP and SIGN applies the full batch training. That means
the trainable parametere $W$ and soft label $\bm{y}_{i}^{\mathrm{soft}}$ could
be well updated at the initial training phase. Therefore, we use the
decreasing smoothing strength in the precomputing methods, where the models
are regularized strictly by the difficult smoothed label from the beginning.
At the end of training, the precomputing methods are relaxed to learn the easy
one-hot hard target to improve the test performance. The initial smoothing
strengths $b$ in the exponential pacing function for MLP are: $0.05$ in
Flickr, $0.2$ in Reddit, and $0.08$ in ogbn-products & ogbn-mag. The values of
$b$ for SIGN are: $0.1$ in Flickr, $0.2$ in Reddit, $0.15$ in ogbn-products,
and $0.08$ in ogbn-mag.

Backbones | Flickr | Reddit | ogbn-products | ogbn-mag
---|---|---|---|---
$r$ | $\gamma$ | $\beta$ | $K$ | $r$ | $\gamma$ | $\beta$ | $K$ | $r$ | $\gamma$ | $\beta$ | $K$ | $r$ | $\gamma$ | $\beta$ | $K$
GraphSAGE | 5e-3 | 5e-3 | 0.5 | 2 | 2e-2 | 1e-2 | 0 | 5 | 1e-2 | 1e-3 | 0.1 | 5 | 1e-2 | 1e-6 | 1e-2 | 5
Cluster-GCN | 1e-2 | 1e-2 | 0.1 | 2 | 2e-2 | 1e-2 | 0 | 5 | 5e-3 | 1e-2 | 0 | 5 | 0.1 | 1e-6 | 1e-2 | 5
GraphSAINT | 5e-3 | 5e-3 | 0.5 | 2 | 2e-3 | 1e-3 | 0 | 5 | 1e-2 | 1e-3 | 0.1 | 2 | 2e-3 | 1e-3 | 0.1 | 2
MLP | -2e-2 | 1e-3 | 0.5 | 2 | -1e-3 | 1e-2 | 0 | 5 | -1e-2 | 1e-2 | 0.1 | 10 | -1e-2 | 1e-3 | 0.8 | 2
SIGN | -0.2 | 1e-3 | 0.1 | 2 | -5e-3 | 1e-3 | 0 | 5 | -0.1 | 1e-6 | 0.1 | 5 | -2e-2 | 1e-3 | 0.8 | 2
Table 6: Hyperparameter choices in ALS for each combination study of backbone
and dataset.

###  A.4 Running Environment

All the experiments are implemented with PyTorch, and tested on a machine with
24 Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GB processors, 128GB CPU memory size,
and one GPU of GeForce RTX 3090 with 24 GB memory size.

###  A.5 Label Bias, Over-confident Prediction and Overfitting Observations

As shown in Figure 1, the sub-graph sampling method of Cluster-GCN brings
label bias, and leads to the over-confident prediction and overfitting in
training set. These problems will damage the model’s generalization
performance in testing set. Our ALS learns to replace the one-hot hard target
$\bm{y}_{i}$ with the smoothed label $\bm{y}_{i}^{ALS}$, which could relieve
these three problems to improve the generalization ability. In this section,
we report the training losses, the testing losses, and the label biases of all
the sub-graph sampling methods on ogbn-products, ogbn-mag, and Flickr. We show
the experimental results in Figures 4-11. Note that the batch sizes of
GraphSAGE, Cluster-GCN, and GraphSAINT are defined by the corresponding sub-
graph sampling functions in Pytorch Geometric222https://pytorch-
geometric.readthedocs.io/en/latest/modules/data.html, i.e., NeighborSampler,
ClusterLoader, and GraphSAINTRandomWalkSampler. While the batch sizes of
GraphSAGE and GraphSAINT specify how many training samples per batch to load,
the batch size of Cluster-GCN determines how many clustered sub-graphs to
sample. We make the following empirical observations:

  * •

All the sub-graph sampling methods bring label bias within a batch. It is
shown that the standard deviance of $p_{c}$ is extremely large, which is
compatible with the mean value of $p_{c}$. In other word, the nodes within a
small batch tend to belong to certain classes, and the label distributions
vary dramatically between batches. In general, the smaller the batch size is,
the larger the standard deviance of $p_{c}$ will be.

  * •

Comparing with the plain backbone frameworks, our ALS has larger training
losses. That is because ALS replaces the one-hot hard target $\bm{y}_{i}$ with
the smoothed label $\bm{y}_{i}^{ALS}$, which distributes label confidences to
both ground-truth class and the other classes. By minimizing the regularized
loss in Eq. (5), ALS reduces the model’s prediction probability on the ground-
truth class, and thus increases the training loss. The regularized prediction
probability on the ground-truth class will help the model avoid the over-
confident prediction and the overfitting on the training set.

  * •

Comparing with the plain backbone frameworks, our ALS generally has smaller
testing losses and better generalization ability. Since the model is
regularized to avoid the over-confident prediction, the smooth prediction
probability in ALS is more easier to be generalized to the unseen testing set.

Figure 4: Left, Middle: The training and testing losses upon backbone
GraphSAGE and dataset ogbn-products. While Plain-$a$ means the original
backbone trained with batch size of $a$, ALS-$a$ denotes one equipped with
ALS. Right: The mean probability $p_{c}$ of nodes with specific class $c$
within a batch (i.e., blue square), and the standard variance of probability
$p_{c}$ among batches (i.e., upper and lower bar). Herein we show $c=0$ on
ogbn-products for an example. Figure 5: Left, Middle: The training and testing
losses upon backbone GraphSAINT and dataset ogbn-products. While Plain-$a$
means the original backbone trained with batch size of $a$, ALS-$a$ denotes
one equipped with ALS. Right: The mean probability $p_{c}$ of nodes with
specific class $c$ within a batch (i.e., blue square), and the standard
variance of probability $p_{c}$ among batches (i.e., upper and lower bar).
Herein we show $c=0$ on ogbn-products for an example. Figure 6: Left, Middle:
The training and testing losses upon backbone GraphSAGE and dataset ogbn-mag.
While Plain-$a$ means the original backbone trained with batch size of $a$,
ALS-$a$ denotes one equipped with ALS. Right: The mean probability $p_{c}$ of
nodes with specific class $c$ within a batch (i.e., blue square), and the
standard variance of probability $p_{c}$ among batches (i.e., upper and lower
bar). Herein we show $c=0$ on ogbn-mag for an example. Figure 7: Left, Middle:
The training and testing losses upon backbone Cluster-GCN and dataset ogbn-
mag. While Plain-$a$ means the original backbone trained with batch size of
$a$, ALS-$a$ denotes one equipped with ALS. Right: The mean probability
$p_{c}$ of nodes with specific class $c$ within a batch (i.e., blue square),
and the standard variance of probability $p_{c}$ among batches (i.e., upper
and lower bar). Herein we show $c=0$ on ogbn-mag for an example. Figure 8:
Left, Middle: The training and testing losses upon backbone GraphSAINT and
dataset ogbn-mag. While Plain-$a$ means the original backbone trained with
batch size of $a$, ALS-$a$ denotes one equipped with ALS. Right: The mean
probability $p_{c}$ of nodes with specific class $c$ within a batch (i.e.,
blue square), and the standard variance of probability $p_{c}$ among batches
(i.e., upper and lower bar). Herein we show $c=0$ on ogbn-mag for an example.
Figure 9: Left, Middle: The training and testing losses upon backbone
GraphSAGE and dataset Flickr. While Plain-$a$ means the original backbone
trained with batch size of $a$, ALS-$a$ denotes one equipped with ALS. Right:
The mean probability $p_{c}$ of nodes with specific class $c$ within a batch
(i.e., blue square), and the standard variance of probability $p_{c}$ among
batches (i.e., upper and lower bar). Herein we show $c=0$ on Flickr for an
example. Figure 10: Left, Middle: The training and testing losses upon
backbone Cluster-GCN and dataset Flickr. While Plain-$a$ means the original
backbone trained with batch size of $a$, ALS-$a$ denotes one equipped with
ALS. Right: The mean probability $p_{c}$ of nodes with specific class $c$
within a batch (i.e., blue square), and the standard variance of probability
$p_{c}$ among batches (i.e., upper and lower bar). Herein we show $c=0$ on
Flickr for an example. Figure 11: Left, Middle: The training and testing
losses upon backbone GraphSAINT and dataset Flickr. While Plain-$a$ means the
original backbone trained with batch size of $a$, ALS-$a$ denotes one equipped
with ALS. Right: The mean probability $p_{c}$ of nodes with specific class $c$
within a batch (i.e., blue square), and the standard variance of probability
$p_{c}$ among batches (i.e., upper and lower bar). Herein we show $c=0$ on
Flickr for an example.


# ML-Schema: Exposing the Semantics of Machine Learning with Schemas and
Ontologies

Gustavo Correa Publio
AKSW Group
University of Leipzig, Germany Diego Esteves
SDA Research
University of Bonn, Germany Agnieszka Ławrynowicz
Poznan University of Technology
Poland Panče Panov
Jožef Stefan Institute
Ljubljana, Slovenia Larisa Soldatova
Brunel University, UK
Tommaso Soru
AKSW Group
University of Leipzig, Germany Joaquin Vanschoren
Eindhoven
University of Technology
The Netherlands Hamid Zafar
SDA Research
University of Bonn, Germany

###### Abstract

The ML-Schema, proposed by the W3C Machine Learning Schema Community Group, is
a top-level ontology that provides a set of classes, properties, and
restrictions for representing and interchanging information on machine
learning algorithms, datasets, and experiments. It can be easily extended and
specialized and it is also mapped to other more domain-specific ontologies
developed in the area of machine learning and data mining. In this paper we
overview existing state-of-the-art machine learning interchange formats and
present the first release of ML-Schema, a canonical format resulted of more
than seven years of experience among different research institutions. We argue
that exposing semantics of machine learning algorithms, models, and
experiments through a canonical format may pave the way to better
interpretability and to realistically achieve the full interoperability of
experiments regardless of platform or adopted workflow solution.

##  1 Introduction

Complex machine learning models have recently achieved great successes in many
predictive tasks. Despite their successes, a major problem is that they are
often hard to interpret, which may affect their safeness and the level of
trust of their users. The problem of interpretability is one of the key
research issues in the area of knowledge engineering and the Semantic Web
community, which deals with making the semantics of various phenomena
explicit. In this community, the problem of dealing with trust and
traceability has gained a major interest last years and resulted in proposals
and uptake of models such as the provenance ontology PROV-O [6].

Despite recent efforts to achieve a high level of interoperability of ML
experiments through existing workflow systems [9], metadata repositories [11,
7] and schemata [4, 8, 5], we still run into problems created due to the
existence of different ML platforms: each of those has a specific
conceptualization or schema for representing data and metadata. Figure 1
depicts the complexity of this scenario: (1) and (2) representing the worse
scenario to achieve reproducibility of the experiments; (3) and (4) which -
although defining a known (local) schema - lack interoperability, since they
follow different standards. To reduce this gap, the aforementioned ML
vocabularies and ontologies have been proposed (e.g., MEX and Expose) (5); and
finally (6) which encompasses the higher level of interoperability to allow
fully reproducible experiments.

In line with those efforts to bridge the gap, in this paper, we present the
ML-Schema111ML-Schema: http://ml-schema.github.io/documentation/ [3],
developed within the W3C Machine Learning Schema Community Group222W3C ML-
Schema Community Group: https://www.w3.org/community/ml-schema/. It is a
simply shared schema that provides a set of classes, properties, and
restrictions that can be used to represent and interchange information on
Machine Learning (ML) algorithms, datasets, and experiments. It can be easily
specialized to create new classes and properties and it is also mapped to more
specific ontologies and vocabularies on machine learning [4, 10, 8, 5], for
instance to represent Deep Learning problems, which are naturally harder to
design when compared to supervised approaches. These ontologies, in turn,
contain terms for representing more detailed characteristics and properties of
ML datasets, algorithms, models, and experiments.

Ultimately, we believe that involving such a canonical and standardized model
meta-data descriptors in design of interpretable methods for ML may lead to
better insights into the data and the properties of ML algorithms.

The gap can be further significantly reduced by achieving interoperability
among state-of-the-art (SOTA) schemata of those resources (Figure 1: item 5),
i.e., achieving the horizontal interoperability (Figure 1: item 6). Therefore,
different groups of researchers could exchange SOTA metadata files in a
transparent manner via web services, e.g.: from OntoDM and MEX
(MLSchema.Schema data = mlschema.convert(‘‘myFile.ttl’’,
MLSchema.Ontology.OntoDM, MLSchema.Ontology.MEX)). Furthermore, the canonical
format also directly benefits different environments, such as ML ecosystems
(e.g. OpenML [11]) and ML Metadata Repositories (e.g. WASOTA [7]) which can
benefit on the mappings of a shared standard.

Figure 1: Vertical and Horizontal Interoperability across ML Environments.

##  2 The ML-Schema

In Fig. 2, we depict the classes and the relationships between the classes
representing the ML-Schema. The schema contains classes for representing
different aspects of machine learning. This includes representations data,
datasets and data/dataset characteristics. Next, it includes representations
of algorithms, implementations, parameters of implementations and software.
Furthermore, the schema contains representations of models, model
characteristics and model evaluations. Finally, the schema has the ability to
represent machine learning experiments with different granularity. This
includes representations of runs of implementations of algorithms on the
lowest level and representation of studies at the highest level.

Figure 2: ML-Schema Core classes. Boxes represent classes. Arrows without
filled heads represent properties, arrows with empty heads represent subclass
relations, and arrows with diamonds represent part-of relations.

##  3 Machine Learning ontologies

In this section, we present the relationship of the ML-Schema to other
proposed ontologies, and vocabularies for the domain of machine learning. The
development of ML-Schema was highly influenced from, initially independent,
research of several groups on modeling the domain of machine learning. Due to
this the classes and relations present in the ML-Schema re-appear in the
current ML ontologies and vocabularies. In Table 1, we present the mapping
between the terms present in the ML-Schema and the current ML ontologies and
vocabularies. Below, we describe each of the mentioned ontology/vocabulary.

#### The OntoDM-core ontology.

The OntoDM-core ontology has been designed to provide generic representations
of principle entities in the area of data mining [8]. In one of the
preliminary versions of the ontology, the authors decided to align the
proposed ontology with the Ontology of Biomedical Investigations (OBI) [2] and
consequently with the Basic Formal Ontology (BFO) [1] at the top level, in
terms of top-level classes and the set of relations. That was beneficial for
structuring the domain in a more elegant way and the basic differentiation of
information entities, implementation entities and processual entities. In this
context, the authors proposed a horizontal description structure that includes
three layers: a specification layer, an implementation layer, and an
application layer. The specification layer in general contains information
entities. In the domain of data mining, example classes are data mining task
and data mining algorithm. The implementation layer in general contains
qualities and entities that are realized in a process, such as parameters and
implementations of algorithms. The application layer contains processual
classes, such as the execution of the data mining algorithm.

#### The Exposé ontology.

The main goal of Exposé [10] is to describe (and reason about) machine
learning experiments. It is built on top of OntoDM, as well as top-level
ontologies from bio-informatics. It is currently used in OpenML333OpenML:
http://www.openml.org/, as a way to structure data (e.g. database design) and
share data (APIs). MLSchema will be used to export all OpenML data as linked
open data (in RDF).

#### The DMOP ontology.

The Data Mining OPtimization Ontology (DMOP) [5] has been developed with a
primary use case in meta-mining, that is meta-learning extended to an analysis
of full DM processes. At the level of both single algorithms and more complex
workflows, it follows a very similar modeling pattern as described in the
MLSchema. To support meta-mining, DMOP contains a taxonomy of algorithms used
in DM processes which are described in detail in terms of their underlying
assumptions, cost functions, optimization strategies, generated models or
pattern sets, and other properties. Such a “glass box” approach which makes
explicit internal algorithm characteristics allows meta-learners using DMOP to
generalize over algorithms and their properties, including those algorithms
which were not used for training meta-learners. DMOP also contains sub-
taxonomies of ML models and provides vocabulary to describe their properties
and characteristics, e.g. model structures, model complexity measures,
parameters.

#### The MEX vocabulary.

The MEX vocabulary [4] has been designed to reuse existing ontologies (e.g.,
PROV-O [6]) for representing basic machine learning experiment configuration
and its outcomes. The aim is not to describe a complete data-mining process,
which can be modeled by more complex and semantically refined structures.
Instead, MEX is designed to provide a simple and lightweight vocabulary for
exchanging basic machine learning metadata in order to achieve a high level of
interoperability. Moreover, the schema aims to serve as a basis for data
management of ML outcomes in the context of WASOTA [7]. The principal
components are: mex-algo444http://mex.aksw.org/mex-algo which describes
algorithms and hyperparameters, mex-core555http://mex.aksw.org/mex-core which
is the basis to describe an experiment, its configurations and executions and
mex-perf666http://mex.aksw.org/mex-perf, the layer to map the outcomes (i.e.,
performance measures).

Table 1: Mapping between the terms between the ML-Schema and the different ML/DM ontologies and vocabularies ML-Schema | OntoDM-core | DMOP | Expose | MEX Vocabulary
---|---|---|---|---
Task | Data mining task | DM-Task | Task | mexcore:ExperimentConfiguration
Algorithm | Data mining algorithm | DM-Algorithm | Algorithm | mexalgo:Algorithm
Software | Data mining software | DM-Software | N/A | mexalgo:Tool
Implementation | Data mining algorithm | DM-Operator | Algorithm implementation | N/A
HyperParameter | Parameter | Parameter | Parameter | mexalgo:HyperParameter
HyperParameterSetting | Parameter setting | OpParameterSetting | Parameter setting | N/A
Study | Investigation | N/A | N/A | mexcore:Experiment
Experiment | N/A | DM-Experiment | Experiment | N/A
Run | Data mining alg. execution | DM-Operation | Algorithm execution | mexcore:Execution
Data | Data item | DM-Data | N/A | mexcore:Example
Dataset | DM dataset | DataSet | Dataset | mexcore:Dataset
Feature | N/A | Feature | N/A | mexcore:Feature
DataCharacteristic | Data specification | DataCharacteristic | Dataset specification | N/A
DatasetCharacteristic | Dataset specification | DataSetCharacteristic | Data quality | N/A
FeatureCharacteristic | Feature specification | FeatureCharacteristic | N/A | N/A
Model | Generalization | DM-Hypothesis | Model | mexcore:Model
ModelCharacteristic | Generalization quality | HypothesisCharacteristic | Model Structure, Parameter, … | N/A
ModelEvaluation | Generalization evaluation | ModelPerformance | Evaluation | N/A
EvaluationMeasure | Evaluation datum | ModelEvaluationMeasure | Evaluation measure | mexperf:PerformanceMeasure
EvaluationSpecification | N/A | N/A | N/A | N/A
EvaluationProcedure | Evaluation algorithm | ModelEvaluationAlgorithm | Performance Estimation | N/A

##  4 Conclusions

In this extended abstract, we have presented ML-Schema, a lightweight schema
for modeling ML domain. The ML-Schema aligns more fine-grained ontologies and
vocabularies, some of which contain detailed vocabulary for representing meta-
data on ML models. The vocabulary and axiomatization included in those
resources may be used to make explicit the semantics of ML models, making them
better interpretable for human users.

### Acknowledgments

Gustavo Correa Publio acknowledges the support of the Smart Data Web BMWi
project (GA-01MD15010B) and CNPq Foundation (201808/2015-3). Agnieszka
Ławrynowicz acknowledges the support from the National Science Centre, Poland,
within the grant number 2014/13/D/ST6/02076. Panče Panov acknowledges the
support of the Slovenian Research Agency within the grant J2-9230. Hamid Zafar
acknowledges the EU H2020 grants for the WDAqua (GA no. 642795) project.

## References

  * [1] Robert Arp, Barry Smith, and Andrew D. Spear.  Building Ontologies with Basic Formal Ontology.  The MIT Press, 2015.
  * [2] Anita Bandrowski, Ryan Brinkman, Mathias Brochhausen, Matthew H. Brush, Bill Bug, Marcus C. Chibucos, Kevin Clancy, Mélanie Courtot, Dirk Derom, Michel Dumontier, Liju Fan, Jennifer Fostel, Gilberto Fragoso, Frank Gibson, Alejandra Gonzalez-Beltran, Melissa A. Haendel, Yongqun He, Mervi Heiskanen, Tina Hernandez-Boussard, Mark Jensen, Yu Lin, Allyson L. Lister, Phillip Lord, James Malone, Elisabetta Manduchi, Monnie McGee, Norman Morrison, James A. Overton, Helen Parkinson, Bjoern Peters, Philippe Rocca-Serra, Alan Ruttenberg, Susanna-Assunta Sansone, Richard H. Scheuermann, Daniel Schober, Barry Smith, Larisa N. Soldatova, Christian J. Stoeckert, Jr., Chris F. Taylor, Carlo Torniai, Jessica A. Turner, Randi Vita, Patricia L. Whetzel, and Jie Zheng.  The ontology for biomedical investigations.  PLoS ONE, 11(4):1–19, 04 2016.
  * [3] Diego Esteves, Agnieszka Lawrynowicz, Pance Panov, Larisa N. Soldatova, Tommaso Soru, and Joaquin Vanschoren.  Ml schema core specification.  Technical report, W3C, 10 2016.  http://www.w3.org/2016/10/mls/.
  * [4] Diego Esteves, Diego Moussallem, Ciro Baron Neto, Tommaso Soru, Ricardo Usbeck, Markus Ackermann, and Jens Lehmann.  MEX vocabulary: a lightweight interchange format for machine learning experiments.  In Proceedings of the 11th International Conference on Semantic Systems, SEMANTICS 2015, Vienna, Austria, September 15-17, 2015, pages 169–176, 2015.
  * [5] C. Maria Keet, Agnieszka Lawrynowicz, Claudia d’Amato, Alexandros Kalousis, Phong Nguyen, Raúl Palma, Robert Stevens, and Melanie Hilario.  The data mining optimization ontology.  J. Web Sem., 32:43–53, 2015.
  * [6] Timothy Lebo, Satya Sahoo, Deborah McGuinness, Khalid Belhajjame, James Cheney, David Corsar, Daniel Garijo, Stian Soiland-Reyes, Stephan Zednik, and Jun Zhao.  Prov-o: The prov ontology.  W3C Recommendation, 30, 2013.
  * [7] Ciro Baron Neto, Diego Esteves, Tommaso Soru, Diego Moussallem, Andre Valdestilhas, and Edgard Marx.  Wasota: What are the states of the art?  In SEMANTiCS (Posters, Demos, SuCCESS), 2016.
  * [8] Pance Panov, Larisa N. Soldatova, and Saso Dzeroski.  Ontology of core data mining entities.  Data Min. Knowl. Discov., 28(5-6):1222–1265, 2014.
  * [9] Olga Tcheremenskaia, Romualdo Benigni, Ivelina Nikolova, Nina Jeliazkova, Sylvia E Escher, Monika Batke, Thomas Baier, Vladimir Poroikov, Alexey Lagunin, Micha Rautenberg, and Barry Hardy.  Opentox predictive toxicology framework: toxicological ontology and semantic media wiki-based opentoxipedia.  Journal of Biomedical Semantics, 2012.
  * [10] Joaquin Vanschoren, Hendrik Blockeel, Bernhard Pfahringer, and Geoffrey Holmes.  Experiment databases - A new way to share, organize and learn from experiments.  Machine Learning, 87(2):127–158, 2012.
  * [11] Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo.  Openml: networked science in machine learning.  ACM SIGKDD Explorations Newsletter, 15(2):49–60, 2014.