# Structure-Aware Label Smoothing for Graph Neural Networks

Yiwei Wang1 Yujun Cai2 Yuxuan Liang1 Wei Wang1 Henghui Ding2
Muhao Chen3 Jing Tang4&Bryan Hooi1 1National University of Singapore
2Nanyang Technological University
3Department of Computer Science, USC
4The Hong Kong University of Science and Technology<EMAIL_ADDRESS>{yujun001<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Representing a label distribution as a one-hot vector is a common practice in
training node classification models. However, the one-hot representation may
not adequately reflect the semantic characteristics of a node in different
classes, as some nodes may be semantically close to their neighbors in other
classes. It would cause over-confidence since the models are encouraged to
assign full probabilities when classifying every node. While training models
with label smoothing can ease this problem to some degree, it still fails to
capture the nodes’ semantic characteristics implied by the graph structures.
In this work, we propose a novel SALS (Structure-Aware Label Smoothing) method
as an enhancement component to popular node classification models. SALS
leverages the graph structures to capture the semantic correlations between
the connected nodes and generate the structure-aware label distribution to
replace the original one-hot label vectors, thus improving the node
classification performance without inference costs. Extensive experiments on
seven node classification benchmark datasets reveal the effectiveness of our
SALS on improving both transductive and inductive node classification.
Empirical results show that SALS is superior to the label smoothing method and
enhances the node classification models to outperform the baseline methods.

##  1 Introduction

Node classification is a fundamental machine learning task on graphs Wu et al.
(2019). It supports numerous practical applications, such as learning
molecular fingerprints Kearnes et al. (2016) and predicting entity properties
Schlichtkrull et al. (2018). Since the introduction of the Graph Convolutional
Network Kipf and Welling (2016), graph neural networks (GNNs) have become the
modern tools of choice for node classification.

GNNs devise the ‘message passing’ mechanism, which aggregates the features for
every node from its neighbors in the feed-forward process Kipf and Welling
(2016). ‘Message passing’ offers the inductive bias of reducing the semantic
distances between connected nodes Rong et al. (2019); Wang and Leskovec
(2020). This inductive bias matches the prior knowledge that the edges
naturally imply the connectivity or relatedness between the connected nodes
Hartuv and Shamir (2000), which is the basis of GNNs’ superior performance
Wang and Leskovec (2020).

On the other hand, how to utilize the graph structural information to generate
effective supervision signals remains under-explored. As far as we know, most
of the existing work trains GNNs with the label distribution of one-hot
vectors, namely the hard targets Wu et al. (2019). However, recent research
has found that using the hard targets to train neural networks tends to cause
over-fitting and over-confidence Szegedy et al. (2016); Müller et al. (2019).
To combat these problems, Szegedy et al. (2016) have proposed the label
smoothing (LS) technique that applies the uniform noise to the label
distribution. Although LS can prevent the learned models from being over-
confident, it cannot describe the characteristics of a node in different
classes implied by the graph structures. For example, if node $i$ in class
$y_{i}$ has a neighbor $j$ in class $y_{j}$, node $i$ should be more
semantically close to $y_{j}$ than other classes due to the connectivity
between nodes $i$ and $j$. Equally treating all the classes for every node
overlooks the graph structures and could limit the GNNs’ performance.

The central idea of this paper is to enrich the supervision of GNNs by
incorporating the graph structural information into the label distribution. We
encapsulate this idea in a simple yet effective method, called SALS (Structure
Aware Label Smoothing), to generate more ‘faithful’ soft targets on node
classification. SALS takes the neighborhood labels as the prior distribution
for the label smoothing of every node. It reflects the semantic
characteristics of a target node in different classes implied by the graph
structures. In the semantic space, our SALS reduces the optimum distances from
a target node to its neighbors so as to guide the target node’s representation
toward an appropriate position. This suits the inductive bias of GNNs and
calibrates them by adaptively offering the structure-aware training targets
for every node.

SALS is a general method for node classification that offers improvements by
regularizing GNN models without extra inference costs. We evaluate SALS on
both transductive and inductive node classification tasks using the Citeseer,
Cora, Pubmed London and Getoor (2014), CoraFull Bojchevski and Günnemann
(2018), Coauthor-Physics Shchur et al. (2018), Flickr McAuley and Leskovec
(2012), and Reddit Hamilton et al. (2017) datasets. Qualitatively, SALS learns
more discriminative node representations (see Fig. 6). We also observe
consistent quantitative improvements measured by test accuracy. Overall, SALS
improves the popular GCN Kipf and Welling (2016), ResGCN Li et al. (2019),
GraphSAGE Hamilton et al. (2017), GraphSAINT Zeng et al. (2019), and SIGN
Frasca et al. (2020) models by a significant margin, and enhances them to
outperform the baseline methods.

##  2 Related Work

Graph Neural Networks (GNNs) for node classification have seen a long history
of studies, we thus refer readers to Wu et al. (2019) and Zhou et al. (2018)
for a comprehensive review. The first work that proposes the convolution
operation on graph data is Bruna et al. (2013). More recently, Kipf and
Welling (2016) made breakthrough advancements in the task of node
classification. After Kipf and Welling (2016), numerous GNN methods have been
proposed for better performance on node classification. There are two main
lines of research in this field. The first one is to propose new GNN
architectures to improve the model capacity Veličković et al. (2017); Zhang et
al. (2018); Haonan et al. (2019); Zhuang and Ma (2018); Qu et al. (2019).
Another one is to propose new mini-batch training techniques for GNNs to
enhance their scalability without the loss of effectiveness Hamilton et al.
(2017); Chiang et al. (2019); Zeng et al. (2019). A common design among these
GNN models is the ‘message passing’ mechanism, which matches the
characteristics of graph data that the edges imply the connectivity or
relatedness between nodes.

Our work is orthogonal to the above two lines in the sense that it does not
alter the GCN architecture, or introduce any new mini-batch technique.
Instead, we propose a new label smoothing method that enriches the supervision
of GNNs without introducing extra inference costs. SALS incorporates the rich
graph structures into the supervision of GNNs. We find that the favorable
characteristics of SALS lead to more accurate predictions.

Label smoothing (LS) is first proposed in image classification tasks as a
regularization technique Szegedy et al. (2016), and has been used in many
state-of-the-art models, including computer vision Zoph et al. (2018); Zhang
et al. (2021), and natural language processing Chorowski and Jaitly (2016);
Vaswani et al. (2017). However, the soft targets produced by the LS cannot
describe the structural characteristics of nodes implied by the connectivity.
In contrast, we use the graph structural information to guide the label
smoothing so as to supervise GNNs with the structure-aware label distribution.
As far as we know, our work is the first to devise the label smoothing method
for the graph data.

Figure 1: A node’s neighborhood is informative on describing its own
characteristics. In a athlete graph, every node is an athelete, and every edge
represents a teammate relation. An athlete’s neighbors imply its own
characteristics. For example, Michael Jordan, a famous basketball athlete, is
also professional at baseball, as implied by his neighbors.

##  3 Methodology

In this section, we introduce the technical details of our proposed structure-
aware label smoothing (SALS). Our SALS incorporates the idea of modeling
deterministic data, namely observed class labels, in terms of a set of
probability distributions instead of a hard target. SALS utilizes the rich
graph structure information for supervising graph neural networks on node
classification, so as to produce more ‘faithful’ training targets reflecting
the connectivity or relatedness between the connected nodes. Consequently,
SALS yields the soft target for every node to be adaptive to its neighborhood,
which is shown to be crucial to learning effective node embeddings.

###  3.1 Preliminaries

We hereby introduce the preliminaries by introducing the node classification
problem, graph neural networks, and the original label smoothing technique.

#### Node Classification

Consider a graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$, where $\mathcal{V}$
is the set of nodes and $\mathcal{E}=\\{(i,j)\mid i,j\in\mathcal{V}\\}$ is the
set of edges. The goal of node classification is to learn a mapping
$\mathcal{M}\colon\mathcal{V}\mapsto\mathbb{P}(\mathcal{Y})$, where
$\mathcal{Y}$ is a set of class labels, and $\mathbb{P}(\mathcal{Y})$ is the
space of probability distributions over $\mathcal{Y}$.

Node classification has been mainly addressed by Graph Neural Network (GNN)
based methods in recent literature. The edges in a graph imply the
connectivity or relatedness between the connected nodes Hartuv and Shamir
(2000). Based on this prior knowledge, GNNs utilize the connectivity of nodes
in graph structures to learn the nodes’ representations. GNNs are a kind of
multi-layer neural networks that propagates the nodes’ representations across
edges between different nodes, which is also known as the ‘message passing’
mechanism. GNNs stack multiple trainable layers to achieve message passing
over edges, where every GNN layer updates a target node $i$’s representations
by aggregating the last layer’s representations from the target node’s
neighbors (in the training set) denoted as $\mathcal{N}(i)$.

Figure 2: The gap between the optimum logits on the correct class $y_{i}$ and
another class $c^{\prime}$ with our SALS (see Eq. (6)). ‘Ratio of Neighbors’
denotes the ratio of the neighbors in class $c$. More neighbors in class $c$
corresponds to a smaller gap from the correct class to class $c$ with our
SALS.

Existing work finds that ‘message passing’ reduces the distance in the
embedding space between the connected nodes Rong et al. (2019); Wang and
Leskovec (2020). This property matches the characteristics of the graph data
that every node is correlated to its neighbors. It acts as a useful inductive
bias that supports GNNs’ state-of-the-art performance.

#### Label Smoothing

The classical training is based on the deterministic label, which is
(explicitly or implicitly) treated as a one-hot vector, namely the hard
targets. The recent study Müller et al. (2019) finds that the hard targets are
too extreme to express the normally realistic assumption of a non-
deterministic dependency between the features and labels. Some instances are
semantically close to other instances in the classes different from themselves
Zhang et al. (2021). Instead of using hard labels for training, label
smoothing (LS) replaces the surrogate distribution $q$ with a less extreme
surrogate

$q^{\mathrm{LS}}=(1-\epsilon)q+\epsilon u_{c}$ (1)

as a soft target for the learner, where $u_{c}\in\mathbb{P}(\mathcal{Y})$ is a
uniform distribution among all the classes in $\mathcal{Y}$ and
$\epsilon\in(0,1]$ is a smoothing factor. LS can be seen as an attempt at
presenting the training information in a more ‘faithful’ way: a smoothed
target probability $q^{\mathrm{LS}}$ is arguably more realistic than a
degenerated hard distribution $q$ assigning the full probability mass to a
single class label.

LS obviously leads to less extreme predictions than the hard targets. However,
the adjusted distribution $q^{\mathrm{LS}}$ assigns the uniform noise to all
the classes, which may not reflect the intrinsic characteristics of an
instance to be classified.

###  3.2 Structure-Aware Label Smoothing

For training GNNs, most of existing work minimizes the expected value of the
cross-entropy loss between the true targets $q(c|i)$ and the network’s
prediction $p(c|i)$ in

$H(q,p)=-\sum_{c=1}^{C}q(c|i)\log p(c|i).$ (2)

Minimizing this loss is equivalent to maximizing the expected log-likelihood
of the label that is selected according to the distribution $q(c|i)$, of which
the gradient is bounded between $-1$ and $1$ Szegedy et al. (2016).

Consider the case of the hard target, where $q(y_{i}|i)=1$ and $q(c|i)=0$ for
$\forall c\neq y_{i}$, training GNNs with hard targets has several
limitations. First, it overlooks the correlations between connected nodes. The
edges in a graph naturally imply the connectivity or relatedness between
connected nodes Hartuv and Shamir (2000). Second, using hard targets to
supervise GNNs does not match their inductive bias of reducing the semantic
distances between connected nodes Wang and Leskovec (2020). To classify node
$i$, the ‘message passing’ mechanism of GNNs aggregates the features from
$i$’s neighbors, while its neighbor $j\in\mathcal{N}(i)$ can belong to any
other classes than $y_{i}$. Enforcing GNNs to assign full probabilities as
prediction for connected nodes in different classes ignores their structural
relationship, and is hard to generalize to the unseen nodes.

To address the above issues, intuitively, we aim to find a label distribution
that reveals the semantic characteristics of a node in each class implied by
the graph structures. Consider an example in Fig. 1. In a social network of
athletes where the edges represent the ‘teammate’ relations, the node ‘Michael
Jordan’ is a famous professional basketball athlete, and he is meanwhile an
expert in baseball, which is implied by his neighbors of professional baseball
athletes Mathur et al. (1997). Here, ‘basketball’ and ‘baseball’ should be
assigned a higher probability mass than the other sports, e.g., ‘soccer’, at
which the node ‘Michael Jordan’ is not a professional. It would be beneficial
to consider these genuine ‘preferences’ of a node on different classes implied
by the graph structures for label smoothing.

We propose a method called SALS (Structure-aware Label Smoothing) to mine the
intrinsic semantic characteristics of every node from rich graph structures
for node classification, and produce structure-aware soft targets for training
GNNs. Specifically, we use the labels of a node’s neighbors to mine its
characteristics beyond its own hard target, and use the structural information
to produce the soft targets that are adaptive to the graph structures. SALS
regularizes GNNs for better generalization and makes node representation more
adaptable by leveraging the rich graph structural knowledge for supervision.
For node $i$ with ground-truth labels $y_{i}$, consider the distributions of
its neighbors’ labels $\left\\{y_{j},j\in\mathcal{N}(i)\right\\}$. First, we
analyze the ratio of neighbors (in the training set) of nodes $i$ in different
classes. In particular, let the ratio of node $i$’s neighbors in class $y$ be

$r_{c}(i)\coloneqq\frac{\sum_{j\in\mathcal{N}(i)}{1[y_{j}=c]}}{|\mathcal{N}(i)|}.$
(3)

We define the neighborhood label of node $i$, which is the average label of
$i$’s neighbors in the training set, to be

$\eta(i)\coloneqq\frac{1}{|\mathcal{N}(i)|}\sum_{j\in\mathcal{N}(i)}\delta_{c,y_{j}}=\sum_{y\in\mathcal{Y}}r_{y}(i)\delta_{c,y},$
(4)

where $\delta_{c,y_{j}}$ is Dirac delta, which equals 1 for $c=y_{j}$ and 0
otherwise. We replace the label distribution of $q(c|i)=\delta_{c,y_{i}}$ with

$\displaystyle
q^{\mathrm{SALS}}(c|i)=(1-\epsilon)\delta_{c,y_{i}}+\epsilon\Big{(}\gamma\eta(i)+(1-\gamma)u_{c}\Big{)},$
(5)

which is a mixture of the original hard target $q(c|i)$ and its neighbors’
label distributions, with weights $1-\epsilon$ and $\epsilon$, respectively.
$\epsilon$ is a hyper-parameter for weighting the label smoothing. $\gamma$ is
the ‘factor’ balancing our neighborhood label and the pre-defined uniform
smoothing from LS Müller et al. (2019). Our SALS introduced in Eq. (5) can be
seen as the distribution of node $i$’s label obtained as follows: first, set
it to the ground-truth label $c=y_{i}$; then, with probability $\epsilon$,
replace $y_{i}$ with a sample drawn from the distribution
$\gamma\eta(i)+(1-\gamma)u_{c}$, which is the prior distribution over labels
of node $i$’s neighborhood.

Figure 3: The message passing of GNNs reduce the embedding distances between
the connected nodes. We compare the node embeddings retrieved by GNNs to a
physical equilibrium mode, where edges connecting nodes serve as the rubber
bands that expose explicit constraints to pull the connected nodes together.

Note that SALS achieves desired goals of utilizing the graph structural
information for supervising GNNs. It encourages the target node’s penultimate
layer representation to be close to the template of the correct class, and
meanwhile closer to its neighbors’ class templates than to other incorrect
classes. This matches the prior knowledge that the edges between nodes
represent connectivity or relatedness. Meanwhile, it matches the inductive
bias from ‘message passing’, that the nodes’ features are aggregated along the
edges, by which GNNs tend to give similar predictions to the connected nodes
Wang and Leskovec (2020).

When training GNNs with our SALS, the gap between the optimum logits on the
correct class $y_{i}$ and another class $c^{\prime}$, which minimizes the
cross entropy loss, is

$\displaystyle\begin{split}&(\mathbf{x}_{i}^{T}\mathbf{w}_{y_{i}})^{\star}-(\mathbf{x}_{i}^{T}\mathbf{w}_{c^{\prime}})^{\star}\\\
=&\log\left(\frac{1-\epsilon+\epsilon\gamma
r_{c}(i)+\epsilon(1-\gamma)/|\mathcal{Y}|}{\epsilon\gamma
r_{c}(i)+\epsilon(1-\gamma)/|\mathcal{Y}|}\right),\end{split}$ (6)

where $\mathbf{x}_{i}$ is the final layer representation of node $i$ and
$\mathbf{w}_{c}$ is the template of class $c$. We visualize this gap in Fig. 2
with $\gamma=1$. With more neighbors in class $c^{\prime}$, this gap shrinks,
which validates that our SALS reflects the influence of the neighborhood in
class $c^{\prime}$. When $\epsilon\rightarrow 0$, SALS degrades to the hard
targets. In this case, the gap in Eq. (6) is $\infty$, which is unachievable
given finite values of logits. In contrast, our SALS makes the target gaps be
finite values, which are more ‘practical’ training targets for GNNs, so as to
reduce the over-confidence problems. Overall, our SALS enriches the
supervision signals with graph structural knowledge, which regularizes GNNs to
adapt to complex graph topology.

Figure 4: Our SALS produces the soft targets for every node adaptive to graph
structures. The structure-aware soft targets produced by SALS match the
inductive bias of GNNs’ message passing that reduces the semantic distances
between the connected nodes.

###  3.3 Discussion

In this section, we analyze how our SALS method influences the supervision of
node classification. An interpretation of SALS can be obtained by considering
the cross entropy:

$\displaystyle H\left(q^{\mathrm{SALS}}(c|i),p(c|i)\right)$ $\displaystyle=$
$\displaystyle(1-\epsilon)H\left(q(c|i),p(c|i)\right)+$ (7)
$\displaystyle\epsilon\Big{(}\frac{\gamma}{|\mathcal{N}(i)|}\sum_{j\in\mathcal{N}(i)}H\left(q(c|j),p(c|i)\right)+(1-\gamma)H(u,p)\Big{)}$

Thus, SALS is to add a group of losses
$\left\\{H\left(q(c|j),p(c|i)\right),j\in\mathcal{N}(i)\right\\}$, which
penalizes the deviation of predicted label distribution $p(c|i)$ from the
label distributions of neighbors. Note that this deviation could be
equivalently captured by the KL divergence, which is a measure of how
dissimilar the predicted distribution $p(c|v)$ is to its neighbors’ label
distributions. This is in accordance with the characteristics of the graph
data. In a graph, edges represent the natural connectivity or relatedness
between nodes Hartuv and Shamir (2000). Motivated by this, we use the loss in
Eq. (3.3) to encourage the nodes to have the final-layer representations that
are closer to the class templates of their neighbors.

To intuitively understand the role of our SALS, we draw an analogy with a
physical equilibrium model as shown in Fig. 3. Each node is seen as a
particle, while the supervised classification signals act as the implicit
force pulling the nodes away from the decision boundary. Without edges (Fig.
1(a)), nodes are loosely placed in the embedding space. In contrast, with
message passing, edges act as the rubber bands and expose explicit constraints
on the representations of connected nodes. In an ideal case where edges only
connect nodes with the same label, the message passing will pull nodes within
the same class together, which greatly benefits classification (Fig. 1(b)).

The connected nodes may have different class labels from each other. These
edges pull the nodes of low $r_{y_{i}}$ (see Eq. (3)) towards the decision
boundaries (see Fig. 1(c)). If training with the hard targets, the nodes of
lower $r_{y_{i}}$ have larger losses and gradient magnitudes (on the final
layer), since these nodes are closer to the decision boundary than the nodes
of higher $r_{y_{i}}$. These gradients of higher magnitudes influence the
updating of GNNs’ learnable weights more heavily during training. Enforcing
the nodes of low $r_{y_{i}}(i)$ to be far away from the decision boundary can
lead to the over-confidence problem, and is hard to generalize to unseen
nodes. This is meanwhile conflicted with the graphs’ characteristics that the
connected nodes are semantically close to each other. On the other hand, since
the gradients on the nodes of lower $r_{y_{i}}(i)$ dominate the gradients for
updating GNNs, the optimizer takes relatively lower efforts to form the
discriminative representations for the nodes of higher $r_{y_{i}}(i)$.

In contrast, with our SALS, the soft targets (see Eq. (5)) reflect the
influence of neighbors in different classes. As a result, the gradient
magnitudes of lower $r_{y_{i}}(i)$ are balanced with the nodes of higher
$r_{y_{i}}(i)$. Moreover, if the number of nodes in class $c$ increases, the
influence from the class $c$ improves, and our SALS’s soft can reflect this
change with higher target probability on class $c$. These structure-aware soft
targets align well with the prior knowledge of the graph structures. SALS
matches the inductive bias of graph neural networks and can generalize to the
unseen nodes in a better calibrated manner with more appropriate confidence
targets (see Fig. 7). Overall, our soft targets correspond to the appropriate
positions in the embedding space, which are adaptive to graph structures, and
meanwhile maintain the correctness of classification.

##  4 Experiments

In this section, we evaluate the effectiveness of our SALS by applying it to
various GNN architectures. We report the experimental results under both the
transductive and inductive settings. In the transductive setting, the training
phase has access to the features of all nodes but only the labels of nodes in
the training set. In the inductive setting, neither the features nor labels of
nodes in the validation/testing set are available during training. In
addition, we visualize the learned representations of GNNs with SALS compared
with those from the original GNNs without SALS. Last but not least, we conduct
ablation studies to show the influence of SALS, as well as the sensitivity
with respect to the hyper-parameters.

We use seven benchmark datasets: Cora, Citeseer, Pubmed London and Getoor
(2014), CoraFull Bojchevski and Günnemann (2018), Coauthor-Physics (short as
Coauthor-Phy) Shchur et al. (2018), Flickr McAuley and Leskovec (2012), and
Reddit Zeng et al. (2019) for evaluation. The former three are citation
networks, where each node is a document and each edge is a citation record.
CoraFull is the larger version of the well-known citation network Cora
dataset, where nodes represent publications and edges represents their
citations, and the nodes are labeled based on the research topics. In Flickr,
each node represents one image. An edge is built between two images if they
share some common properties (e.g., same geographic location, same gallery,
etc.). Reddit is collected from an online discussion forum where users comment
in different topical communities. Two posts (nodes) are connected if some
users comment on both posts. Each of them contains an unweighted adjacency
matrix and bag-of-words features. The statistics of these datasets are
summarized in Table 1.

For the hyper-parameters of baselines, e.g., the number of hidden units, the
optimizer, the learning rate, etc., we set them as suggested by their authors.
For the hyper-parameters of our SALS, we set $\epsilon=0.4,\gamma=0.8$ for
SALS by default. Note that this setting holds for all of our experiments
unless otherwise specified.

Table 1: Statistics of the datasets for node classification. Dataset |  $\\#$Nodes |  $\\#$Edges |  $\\#$Classes
---|---|---|---
Cora | 2,708 | 5,429 | 7
Citeseer | 3,327 | 4,732 | 6
Pubmed | 19,717 | 44,338 | 3
CoraFull | 19,793 | 65,311 | 70
Coauthor-Phy | 34,493 | 247,962 | 5
Flickr | 89,250 | 899,756 | 7
Reddit | 232,965 | 11,606,919 | 41
Table 2: Test Accuracy (%) of transductive node classification. We conduct 100 trials with random weight initialization. The mean and standard derivations are reported. Method | Citeseer | Cora | Pubmed | CoraFull | Coauthor-Phy
---|---|---|---|---|---
GCN Kipf and Welling (2016) | 77.1 $\pm$ 1.4 | 88.3 $\pm$ 0.8 | 86.4 $\pm$ 1.1 | 64.5 $\pm$ 1.3 | 96.0 $\pm$ 0.5
GAT Veličković et al. (2017) | 76.3 $\pm$ 0.8 | 87.6 $\pm$ 0.5 | 85.7 $\pm$ 0.7 | 65.3 $\pm$ 0.9 | 96.2 $\pm$ 0.8
JKNet Xu et al. (2018) | 78.1 $\pm$ 0.9 | 89.1 $\pm$ 1.2 | 86.9 $\pm$ 1.3 | 65.0 $\pm$ 1.1 | 95.1 $\pm$ 0.5
LGCN Gao et al. (2018) | 77.5 $\pm$ 1.1 | 89.0 $\pm$ 1.2 | 86.5 $\pm$ 0.6 | 64.6 $\pm$ 1.0 | 95.4 $\pm$ 0.6
GMNN Qu et al. (2019) | 77.4 $\pm$ 1.5 | 88.7 $\pm$ 0.8 | 86.7 $\pm$ 1.0 | 64.9 $\pm$ 1.1 | 95.6 $\pm$ 0.9
ResGCN Li et al. (2019) | 77.9 $\pm$ 0.8 | 88.1 $\pm$ 0.6 | 87.1 $\pm$ 1.2 | 64.8 $\pm$ 1.2 | 95.3 $\pm$ 1.0
GCN + LS Szegedy et al. (2016) | 77.5 $\pm$ 1.3 | 88.5 $\pm$ 0.6 | 86.7 $\pm$ 0.6 | 64.9 $\pm$ 1.1 | 96.2 $\pm$ 0.5
ResGCN + LS Szegedy et al. (2016) | 78.1 $\pm$ 0.9 | 88.3 $\pm$ 0.7 | 87.2 $\pm$ 0.9 | 65.2 $\pm$ 1.3 | 95.6 $\pm$ 1.0
GCN + OLS Zhang et al. (2021) | 77.7 $\pm$ 1.1 | 88.6 $\pm$ 1.0 | 86.8 $\pm$ 0.7 | 65.0 $\pm$ 1.2 | 96.1 $\pm$ 0.7
ResGCN + OLS Zhang et al. (2021) | 78.2 $\pm$ 0.9 | 88.3 $\pm$ 0.6 | 87.3 $\pm$ 0.8 | 65.1 $\pm$ 1.1 | 95.4 $\pm$ 0.9
GCN + SALS (Ours) | 77.8 $\pm$ 1.0 | 88.9 $\pm$ 0.7 | 87.6 $\pm$ 0.6 | 66.2 $\pm$ 1.0 | 96.3 $\pm$ 0.4
ResGCN + SALS (Ours) | 78.4 $\pm$ 0.9 | 88.7 $\pm$ 0.7 | 87.9 $\pm$ 0.8 | 66.5 $\pm$ 1.1 | 95.8 $\pm$ 0.9

###  4.1 Transductive Node Classification

In the transductive settings, we take the popular GNN models of GCN Kipf and
Welling (2016), GAT Veličković et al. (2017), JKNet Xu et al. (2018), LGCN Gao
et al. (2018), GMNN Qu et al. (2019), ResGCN Li et al. (2019), the original
label smoothing (LS), and the recently proposed Online Label Smoothing (OLS)
for image classification Zhang et al. (2021) as the baselines for comparison.
We follow the prior studies Xu et al. (2018) to split nodes in each graph into
60%, 20%, 20% for training, validation, and testing for a fair comparison. We
make 10 random splits and conduct the experiments for 100 trials with random
weight initialization for each split.

We vary the number of layers from 1 to 10 for each model and choose the best
performing number with respect to the validation set. The results are reported
in Table 2. We observe that SALS improves the test accuracy of GCN by 0.9% on
Citeseer, 0.7% on Cora, 1.4% on Pubmed, 2.6% on CoraFull, 0.3% on Coauthor-
Phy, and improves ResGCN by 0.6% on Citeseer, 0.7% on Cora, 0.9% on Pubmed,
2.7% on CoraFull, 0.5% on Coauthor-Phy respectively. As a result, SALS
regularizes GCN and ResGCN to outperform all the baseline methods.

Taking a closer look, we find that given the same GNN model, SALS consistently
produces larger improvements than LS and OLS. The advantages come from our
structure-aware soft targets that utilize the structural information on label
smoothing, while neither LS nor OLS utilizes the connectivity characteristics
of the graph data for supervising GNNs. SALS regularizes GNNs to adapt to the
complex topology for every node at different positions of the graph.

Table 3: Test accuracy (%) of inductive node classification. We report mean and standard derivations of 100 trials with random weight initialization. We implement LS, OLS, and our SALS with GraphSAGE, GraphSAINT, and SIGN. Method | Flickr | Reddit
---|---|---
GraphSAGE | 50.1 $\pm$ 1.1 | 95.3 $\pm$ 0.1
GraphSAGE + LS | 50.2 $\pm$ 0.7 | 95.5 $\pm$ 0.1
GraphSAGE + OLS | 50.5 $\pm$ 0.8 | 95.6 $\pm$ 0.1
GraphSAGE + SALS (Ours) | 51.1 $\pm$ 0.9 | 96.0 $\pm$ 0.1
GraphSAINT | 51.1 $\pm$ 0.2 | 96.6 $\pm$ 0.1
GraphSAINT + LS | 51.4 $\pm$ 0.3 | 96.7 $\pm$ 0.1
GraphSAINT + OLS | 51.2 $\pm$ 0.2 | 96.6 $\pm$ 0.1
GraphSAINT + SALS (Ours) | 51.9 $\pm$ 0.2 | 96.9 $\pm$ 0.1
SIGN | 51.4 $\pm$ 0.1 | 96.8 $\pm$ 0.0
SIGN + LS | 51.5 $\pm$ 0.1 | 96.8 $\pm$ 0.1
SIGN + OLS | 51.7 $\pm$ 0.1 | 96.9 $\pm$ 0.1
SIGN + SALS (Ours) | 52.2 $\pm$ 0.1 | 97.1 $\pm$ 0.1

###  4.2 Inductive Node Classification

In the inductive settings, we follow the existing work to use the datasets
Flickr, Reddit with the fixed partition for evaluation Zeng et al. (2019);
Frasca et al. (2020). These datasets can be too large to be handled well by
the full-batch training. Hence, existing work devises the advanced scalable
GraphSAGE Hamilton et al. (2017) and GraphSAINT Zeng et al. (2019), and SIGN
Frasca et al. (2020) models to deal with the large-scale graphs. We take them
as the baselines for comparison. We implement SALS with GraphSAGE, GraphSAINT,
and SIGN to study whether SALS can improve the performance of GNNs under the
inductive setting.

We vary the number of layers of each method from 1 to 10 for each model and
choose the best performing model with respect to the validation set. The
results are reported in Table 3. We observe that SALS improves the test
accuracy of GraphSAGE by 2.0% on Flickr, 0.7% on Reddit, GraphSAINT by 1.6% on
Flickr, 0.3% on Reddit, and SIGN by 1.6%, 0.3% on Reddit respectively. As a
result, SALS enhances them to outperform the baseline methods.

Overall, the results above demonstrate the effectiveness of SALS to improve a
wide range of GNN models in terms of both transductive and inductive node
classification, and consistently outperforms LS and OLS.

(a) Accumulative Losses

(b) Gradient Magnitudes

Figure 5: The losses and gradient norms on nodes of different $r_{y_{i}}(i)$.
(a) The normalized accumulative classification loss (y-axis) versus the index
of nodes in Citeseer in the ascending order of $r_{y_{i}}(i)$ (x-axis). (b)
The average $\ell_{2}-$norm of gradients on the nodes in a bucket (y-axis)
versus the index of node buckets in Pubmed in the ascending order of
$r_{y_{i}}(i)$ (x-axis). Without our SALS, the nodes of lower $r_{y_{i}}(i)$
dominate the training losses and the gradients magnitudes, while our SALS
balances the losses and gradients.

###  4.3 Visualization and Ablation Study

We conduct a number of visualization and ablation studies to analyze our SALS.
First, we investigate the distribution of the classification losses and the
final-layer gradients on nodes of different $r_{y_{i}}(i)$. In Fig. 5, we sort
the nodes in the training set by the ascending order of $r_{y_{i}}(i)$ as
defined in Eq. (3), i.e., the ratio of neighbors that are in the same class as
the target node. We train a 3-layer GCN with and without SALS on different
datasets and record the node-wise classification losses and gradients after
convergence. In Fig 5(a), we visualize the normalized accumulative losses for
the sorted nodes in the Citeseer dataset, where we linearly normalize the
losses so that the summation of losses on all the nodes is 1. In addition, in
Fig 5(b), we split all the training nodes in the Pubmed dataset into six even
buckets of nodes in the ascending order of $r_{y_{i}}(i)$ and present the
average $\ell_{2}$-norm of gradients in each bucket at the $y$-axis.

(a) GCN

(b) GCN + SALS (Ours)

Figure 6: The learned representations of nodes in the Pubmed dataset
(visualized by t-SNE). Colors denote the ground-truth class labels. The node
representations given by GCN with our SALS are more discriminative than those
given by the original GCN.

Without our SALS, the nodes of smaller $r_{y_{i}}(i)$ contributes more to the
classification loss and the gradient magnitudes, since GNNs reduce the
semantic distances between the connected nodes but the hard targets do not
reflect this structural information. In contrast, with our SALS, the losses on
the nodes of different $r_{y_{i}}(i)$ are better balanced thanks to the
structure-aware soft targets. In this way, the nodes with lower $r_{y_{i}}(i)$
no longer dominate the weight updating of GNNs’ training, so that GNNs can be
trained to form more discriminative representations for the nodes of higher
$r_{y_{i}}(i)$.

Fig. 6 presents the learned representations obtained by a 3-layer GCN trained
with hard targets and one with SALS, where we visualize the representations of
the half of nodes in the Pubmed’s training set of higher $r_{y_{i}}(i)$. It is
shown that the hidden layers supported by SALS learn more discriminative
features, thanks to the regularization given by SALS. This meets our analysis
in the Section 3.3 and observations from Fig. 5. The structure-aware soft
targets produced by our SALS supervised GNNs to have more balanced gradient
magnitudes on the nodes of different $r_{y_{i}}(i)$, which enables GNNs to
take more efforts to learn more discriminative node representations than the
original hard targets. These highly discriminative features potentially help
produce better class predictions than less discriminative features.

Figure 7: Reliability diagrams of a 3-layer GCN, and it with LS, SALS on the
Cora dataset.

(a) Pubmed

(b) CoraFull

Figure 8: The test accuracy (z-axis) of GCN with SALS in (a) and ResGCN with
SALS in (b) under different values of the hyper-parameters $\epsilon$ and
$\gamma$.

Fig. 7 shows the 10-bin reliability diagram of a 3-layer GCN trained on Cora
Guo et al. (2017). The grey dashed line represents perfect calibration where
the output likelihood (confidence) perfectly predicts the accuracy. Without
label smoothing, the model trained with hard targets is clearly over-
confident, since in expectation the accuracy is always below the confidence.
With our SALS, we observe that the reliability diagram slope is now much
closer to a slope of 1 and the model is better calibrated than the original
GCN and it with LS. This meets our analysis in Sec. 3.3, that the structure-
aware soft targets correspond to suitable semantic positions that match the
GNNs’ inductive bias. Our SALS smoothens the targets to make the labels of
connected nodes correlated, which aligns better with the characteristics of
graph data than the hard targets, LS, and OLS.

Last but not least, we evaluate how sensitive our SALS is to the hyper-
parameters: $\epsilon$ that controls effects of label smoothing, and $\gamma$
that balances the neighborhood label and the pre-defined uniform distribution.
We visualize the results in Fig. 8. The performance of GNNs with SALS is
generally smooth when parameters are in certain ranges. However, too small
values of $\epsilon$ and $\gamma$ result in low performances, which should be
avoided in practice. Moreover, increasing $\epsilon$ from 0.1 to 0.3 and
$\gamma$ from 0.5 to 0.7 improves the effectiveness of SALS on all datasets,
demonstrating that the label smoothing guided by graph structures plays an
important role in improving the performance of GNNs.

##  5 Conclusion

In this work, we propose Structure-aware Label Smoothing (SALS) as an
enhancement component to popular node classification models. SALS can capture
the prior knowledge about the graph structures for supervising the label
distributions. It produces structure-aware soft targets for every node in an
adaptive manner. Experiments on seven benchmark datasets proved our SALS’s
enhancement on various models for both transductive and inductive node
classification. Specifically, SALS reduces the over-fitting and calibration
errors of GCN models, which uses the graph structural knowledge to enrich the
label distribution An interesting future direction is to extend our methods to
other modalities such as text and images, where we can construct graph
topology in the semantic space and extend our SALS to learn better label
distributions.

## References

  * Bojchevski and Günnemann [2018] Aleksandar Bojchevski and Stephan Günnemann.  Deep gaussian embedding of graphs: Unsupervised inductive learning via ranking, 2018.
  * Bruna et al. [2013] Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun.  Spectral networks and locally connected networks on graphs.  arXiv preprint arXiv:1312.6203, 2013.
  * Chiang et al. [2019] Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh.  Cluster-gcn: An efficient algorithm for training deep and large graph convolutional networks.  In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 257–266, 2019.
  * Chorowski and Jaitly [2016] Jan Chorowski and Navdeep Jaitly.  Towards better decoding and language model integration in sequence to sequence models, 2016.
  * Frasca et al. [2020] Fabrizio Frasca, Emanuele Rossi, Davide Eynard, Ben Chamberlain, Michael Bronstein, and Federico Monti.  Sign: Scalable inception graph neural networks.  arXiv preprint arXiv:2004.11198, 2020.
  * Gao et al. [2018] Hongyang Gao, Zhengyang Wang, and Shuiwang Ji.  Large-scale learnable graph convolutional networks.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1416–1424, 2018.
  * Guo et al. [2017] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger.  On calibration of modern neural networks.  In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1321–1330. JMLR. org, 2017.
  * Hamilton et al. [2017] Will Hamilton, Zhitao Ying, and Jure Leskovec.  Inductive representation learning on large graphs.  In Advances in neural information processing systems, pages 1024–1034, 2017.
  * Haonan et al. [2019] Lu Haonan, Seth H Huang, Tian Ye, and Guo Xiuyan.  Graph star net for generalized multi-task learning.  arXiv preprint arXiv:1906.12330, 2019.
  * Hartuv and Shamir [2000] Erez Hartuv and Ron Shamir.  A clustering algorithm based on graph connectivity.  Information processing letters, 76(4-6):175–181, 2000.
  * Kearnes et al. [2016] Steven Kearnes, Kevin McCloskey, Marc Berndl, Vijay Pande, and Patrick Riley.  Molecular graph convolutions: moving beyond fingerprints.  Journal of computer-aided molecular design, 30(8):595–608, 2016\.
  * Kipf and Welling [2016] Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  arXiv preprint arXiv:1609.02907, 2016.
  * Li et al. [2019] Guohao Li, Matthias Müller, Ali Thabet, and Bernard Ghanem.  Deepgcns: Can gcns go as deep as cnns?  arXiv preprint arXiv:1904.03751, 2019.
  * London and Getoor [2014] Ben London and Lise Getoor.  Collective classification of network data.  Data Classification: Algorithms and Applications, 399, 2014.
  * Mathur et al. [1997] Lynette Knowles Mathur, Ike Mathur, and Nanda Rangan.  The wealth effects associated with a celebrity endorser: The michael jordan phenomenon.  Journal of Advertising research, 37(3):67–74, 1997.
  * McAuley and Leskovec [2012] Julian McAuley and Jure Leskovec.  Image labeling on a network: using social-network metadata for image classification.  In European conference on computer vision, pages 828–841. Springer, 2012.
  * Müller et al. [2019] Rafael Müller, Simon Kornblith, and Geoffrey E Hinton.  When does label smoothing help?  In Advances in Neural Information Processing Systems, pages 4696–4705, 2019.
  * Qu et al. [2019] Meng Qu, Yoshua Bengio, and Jian Tang.  Gmnn: Graph markov neural networks.  arXiv preprint arXiv:1905.06214, 2019.
  * Rong et al. [2019] Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang.  Dropedge: Towards deep graph convolutional networks on node classification.  In International Conference on Learning Representations, 2019.
  * Schlichtkrull et al. [2018] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling.  Modeling relational data with graph convolutional networks.  In European Semantic Web Conference, pages 593–607. Springer, 2018\.
  * Shchur et al. [2018] Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann.  Pitfalls of graph neural network evaluation.  arXiv preprint arXiv:1811.05868, 2018.
  * Szegedy et al. [2016] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, Red Hook, NY, USA, 2017. Curran Associates Inc.
  * Veličković et al. [2017] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio.  Graph attention networks.  arXiv preprint arXiv:1710.10903, 2017.
  * Wang and Leskovec [2020] Hongwei Wang and Jure Leskovec.  Unifying graph convolutional neural networks and label propagation.  arXiv preprint arXiv:2002.06755, 2020.
  * Wu et al. [2019] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S Yu.  A comprehensive survey on graph neural networks.  arXiv preprint arXiv:1901.00596, 2019.
  * Xu et al. [2018] Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka.  Representation learning on graphs with jumping knowledge networks.  arXiv preprint arXiv:1806.03536, 2018.
  * Zeng et al. [2019] Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor Prasanna.  Graphsaint: Graph sampling based inductive learning method.  arXiv preprint arXiv:1907.04931, 2019.
  * Zhang et al. [2018] Jiani Zhang, Xingjian Shi, Junyuan Xie, Hao Ma, Irwin King, and Dit-Yan Yeung.  Gaan: Gated attention networks for learning on large and spatiotemporal graphs.  arXiv preprint arXiv:1803.07294, 2018.
  * Zhang et al. [2021] Chang-Bin Zhang, Peng-Tao Jiang, Qibin Hou, Yunchao Wei, Qi Han, Zhen Li, and Ming-Ming Cheng.  Delving deep into label smoothing.  IEEE Transactions on Image Processing, 30:5984–5996, 2021.
  * Zhou et al. [2018] Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun.  Graph neural networks: A review of methods and applications.  arXiv preprint arXiv:1812.08434, 2018.
  * Zhuang and Ma [2018] Chenyi Zhuang and Qiang Ma.  Dual graph convolutional networks for graph-based semi-supervised classification.  In Proceedings of the 2018 World Wide Web Conference, pages 499–508, 2018.
  * Zoph et al. [2018] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le.  Learning transferable architectures for scalable image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.


# Photon Classification with Gradient Boosted Trees at CLAS12

G. Matousek11footnotetext: Corresponding author. , A. Vossen

###### Abstract

Dihadron semi-inclusive deep inelastic scattering (SIDIS) of 10.6 GeV
longitudinally polarized electrons off the proton has been measured using the
CLAS12 detector at Jefferson Lab. Two separate channels, $\pi^{+}\pi^{0}$ and
$\pi^{-}\pi^{0}$, were analyzed, requiring the reconstruction of diphoton
pairs. In this analysis, we addressed the problem of false neutral particles
being reconstructed by CLAS12’s event builder, polluting the otherwise
physical combinatorial background underneath the $\pi^{0}$ peak. A photon
classifier using a Gradient Boosted Trees (GBTs) architecture was trained with
Monte Carlo simulations to reduce the amount of background $\pi^{0}$’s. We
show that the nearest-neighbor features learned by the model lead to a
substantial increase in signal vs. background discrimination compared to
previous CLAS12 $\pi^{0}$ analyses. The machine learning approach recovers
several times more dihadron statistics for the dataset.

##  1 Introduction

Our understanding of the spin structure of nucleons has seen several
advancements with the help of recent deep inelastic scattering (DIS)
experiments. In semi-inclusive DIS (SIDIS), a single hadron is identified in
the final state along with the reconstructed scattered electron. Measured
asymmetries in the azimuthal angle, $\phi_{h}$, of these hadrons, with respect
to either or some combination of the beam and target spin reveal the presence
of spin-dependent parton distribution functions (PDFs) within nucleons.
Extracting these PDFs in single hadron SIDIS involves modeling the intrinsic
transverse momentum $k_{T}$ dependence of the partons. By studying the
azimuthal modulations of dihadron SIDIS, these model-dependent approaches
towards accessing PDFs are avoided. This is because the two hadron state has
an additional degree of azimuthal freedom, $\phi_{R}$, which allows certain
PDFs to be accessed without convolutions over transverse momentum space as
$k_{T}$ is integrated out. In particular, the elusive twist-3 colinear PDF
$e(x)$ can be extracted in such a way by measuring the $(\phi_{h},\phi_{R})$
asymmetries in dihadron SIDIS.
In addition to offering a more targeted approach for measuring spin-sensitive
PDFs, dihadron SIDIS allows us to study more complex spin-related effects in
hadronization. At leading twist, dihadron fragmentation functions (DiFFs)
encode the probability for a quark to fragment into a hadron pair $h_{1}h_{2}$
[1]. The DiFF $G_{1}^{\perp}$, which describes the quark helicity dependence
on dihadron production, has had little experimental investigation up to this
point.
Previous beam spin asymmetries of $ep\rightarrow e\pi^{+}\pi^{-}X$ SIDIS
dihadrons were measured using the CLAS12 detector system [2]. For our study,
we extend this analysis using the same dataset to $\pi^{+}\pi^{0}$ and
$\pi^{-}\pi^{0}$ dihadrons. Exploring the same asymmetries with different
final states allow us to explore the isospin dependence of our fragmentation
functions (see [3, 4]), as well as PDFs such as $e(x)$ with different
systematics.

##  2 Experiment

Dihadron $\pi^{\pm}\pi^{0}$ SIDIS was measured at Jefferson National Lab using
the CLAS12 (CEBAF Large Acceptance Spectrometer 12-GeV) detector system [5].
The data set corresponds to the Run Group A (RG-A) configuration which
operated between Spring 2018 and Spring 2019. The experiment scattered a
longitudinally polarized electron beam in the energy range of $10.2-10.6$ GeV
with a run-by-run polarization of $86-89$% on a fixed, unpolarized liquid
hydrogen target.
At CLAS12, a sector-based electron trigger is used to select inclusive
scattering events, followed by a skimming procedure to extract candidate DIS
events (ex: $Q^{2}>1$ GeV2). Forward detectors ($5<\theta<35^{\circ}$)
including drift chambers and electromagnetic calorimeters provide
identification and four-momentum reconstruction of our electrons, charged
pions, and photons.

##  3 Machine Learning

The main challenge encountered during the $\pi^{\pm}\pi^{0}$ analysis was the
abundance of non-physical $\gamma\gamma$ background beneath the $\pi^{0}$
peak. This background can be seen in Fig. 1, where are large fraction of
photons beneath the peak do not have $\pi^{0}$ parent (i.e. are not
combinatorial backgrounds). At CLAS12, the absence of an ionized track in the
drift chambers pointing towards a non-negligible energy deposition in the ECal
is indicative of a neutral particle candidate ($\gamma$ or $n$, which are
separated using timing). The particle reconstruction algorithm implemented
across many CLAS12 analyses, called the Event Builder, incorrectly assigns
background calorimeter clusters to physical photons from the event. The
calorimeter clustering also tends to mistakenly "split" one local hotspot into
several particles, as opposed to merging them into one single particle. The
nature of these false neutrals is shown in a sample reconstructed Monte Carlo
SIDIS event in Fig. 2. Despite only 2 true event photons, 11 are
reconstructed, with a majority of them grouped near other neutrals, or near
other energetic hadrons.

Figure 1: Origin of diphoton signal and background distributions from
reconstructed CLAS12 Monte Carlo. SIDIS cuts correspond to $Q^{2}>1$ GeV2,
$y<0.8$, and $W>2$ GeV.

Figure 2: A sample Monte Carlo event drawn in ($\theta$,$\phi$) space.
$\theta$ is the scattering angle of the particle off the beamline, and $\phi$
is the azimuthal angle. Marker size roughly scales with particle energy. On
the left shows only the Monte Carlo particles, whereas the right overlays
particle reconstructed by the Event Builder

In this work, a machine learning model was developed to solve the non-physical
diphoton background problem. The architecture we decided upon was Gradient
Boosted Decision Trees (GBTs) classifier which was implemented using the open-
source python package CatBoost [6]. For each reconstructed photon in an event,
the model was trained to classify if the photon was physical (i.e. originated
from the SIDIS event). The training sample was collected by taking photons
from reconstructed Monte Carlo and flagging those that had a generated Monte
Carlo counterpart as physical (signal). The 16 input features of the model are
split into 3 categories: event-wide (such as the number of photons in the
event), those intrinsic to the photon (such as its $E_{\mathrm{tot}}$,
$\theta$, and $E_{\mathrm{PCAL}}$), and, the most critical to the model’s
success, those intrinsic to its nearest neighbors. In particular, it was
predicted and later confirmed through testing that the angular distance
between the photon and its $k$-nearest charged hadrons, neutrals, and event
electron were essential for accurate discrimination of signal vs. background.
The discriminating power of several of these features are shown in Fig. 3,
shown alongside a figure of merit scan for determining the optimal
$p$-threshold for the GBT model. Since we do not use the relative energy
between neighboring photons, we prevent the model from learning the resonant
structure of the $\pi^{0}$ peak. A final model was developed using 1,000 trees
with a maximum depth of 10, adhering to a symmetric growth policy. A learning
rate of 0.1 was applied to moderate gradient descent, and log loss was
evaluated on a validation set split 75/25 with the training data. Log loss was
recorded for each of the 1,000 generations, and the model with the lowest
validation loss was selected to minimize overfitting.

Figure 3: Left: Feature distributions between true Monte Carlo (solid) and ML
predictions. The four shown are the nearest photon neighbor relative angle
(top left), nearest hadron neighbor relative angle (top right), photon
scattering angle (bottom left) and nearest hadron neighbor relative energy
(top right). Right: Figure of merit extraction for optimal $p$-threshold cut
of the GBT model.

##  4 Results

In Fig. 4 we overlay the GBT model output when applied to data and Monte Carlo
photons. The agreement of these distributions indicates the similarity of the
data and Monte Carlo input feature space, supporting the claim that the model
can be effectively used to classify signal data photons. In Fig. 5 we show the
diphoton spectra of SIDIS $\pi^{+}\pi^{0}$ dihadrons. For each subfigure, we
plot the distribution before applying any photon purity cuts, after using
CLAS12 standard photon purity cuts (in this case, $E_{\gamma}>0.6$ GeV), and
after using our machine learning approach. On the right figure, we apply an
additional exclusivity cut on the final state missing mass to select exclusive
$ep\rightarrow e\rho^{+}(n)$ events. We see that our machine learning model
reduces the background fraction of $\gamma\gamma$ pairs underneath the
$\pi^{0}$ peak while maintaining a significantly large amount signal
$\pi^{0}$’s. The figure of merit for the GBT approach is 360, compared to the
much smaller 182 figure of merit for the traditional cuts. In the exclusive
case, where we expect no combinatorial diphoton background, the machine
learning model’s background diphoton distribution is flat near zero. Overall,
we’ve shown that a GBT model trained on nearest-neighbor features tackles the
false neutral backgrounds at CLAS12, providing a more statistically
significant data sample for our dihadron study.

Figure 4: Output histograms of the trained GBTs when applying them on data and
Monte Carlo. Left: $\pi^{+}\pi^{0}$ dihadrons. Right: $\pi^{-}\pi^{0}$
dihadrons.

Figure 5: Diphoton spectra of SIDIS $\pi^{+}\pi^{0}$ dihadrons. Left: No
exclusivity cut applied. Right: Exclusivity cut applied.

## References

  * [1] S. Gliske, A. Bacchetta and M. Radici, _Production of two hadrons in semi-inclusive deep inelatic scattering_ , _Physical Review D_ 90 (2014) 114027.
  * [2] T. Hayward, C. Dilks, A. Vossen, H. Avakian, S. Adhikari, G. Angelini et al., _Observation of beam spin asymmetries in the process ep → e’ $\pi^{+}\pi^{-}$ x with clas12_, _Physical Review Letters_ 126 (2021) 152501.
  * [3] H.H. Matevosyan, A. Kotzinian and A.W. Thomas, _Dihadron fragmentation functions in the quark-jet model: Longitudinally polarized quarks_ , _Physical Review D_ 96 (2017) 074010.
  * [4] H.H. Matevosyan, A. Kotzinian and A.W. Thomas, _Dihadron fragmentation functions in the quark-jet model: Longitudinally polarized quarks_ , _Physical Review D_ 96 (2017) 074010.
  * [5] V.D. Burkert, L. Elouadrhiri, K.P. Adhikari et al., _The CLAS12 Spectrometer at Jefferson Laboratory_ , _Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment_ 959 (2020) 163419.
  * [6] “CatBoost - state-of-the-art open-source gradient boosting library with categorical features support.”