# Universal Graph Continual Learning

Thanh Duc Hoang1, Do Viet Tung1, Duy-Hung Nguyen1,
Bao-Sinh Nguyen1, Huy Hoang Nguyen2, and Hung Le3
1Cinnamon AI, Vietnam
{kris, ace, hector<EMAIL_ADDRESS>
2University of Oulu, Finland
<EMAIL_ADDRESS>
3Deakin University, Australia.
<EMAIL_ADDRESS>
###### Abstract

We address catastrophic forgetting issues in graph learning as incoming data
transits from one to another graph distribution. Whereas prior studies
primarily tackle one setting of graph continual learning such as incremental
node classification, we focus on a universal approach wherein each data point
in a task can be a node or a graph, and the task varies from node to graph
classification. We propose a novel method that enables graph neural networks
to excel in this universal setting. Our approach perseveres knowledge about
past tasks through a rehearsal mechanism that maintains local and global
structure consistency across the graphs. We benchmark our method against
various continual learning baselines in real-world graph datasets and achieve
significant improvement in average performance and forgetting across tasks.

##  1 Introduction

The recent literature has shown the high potential of graph neural networks
(GNNs) in individual tasks from various domains, such as social and molecule
networks McCallum et al. (2000); Hamilton et al. (2017). However, when graph
data come in sequence, GNNs suffer from catastrophic forgetting French (1999).
In particular, if we treat a subgraph as a data point, a stream of data
arriving from different task distributions may cause graph models to merely
learn the current task using current data but forget the older ones. Thus,
GNNs underperform on data from previously trained tasks. The forgetting can
occur locally (node-unit) or globally (graph-unit), depending on the task,
which can be node or graph classification Galke et al. (2020); Carta et al.
(2021); Wang et al. (2022).

There exist many continual learning algorithms to cope with catastrophic
forgetting. Unfortunately, they are mainly designed for primitive neural
networks such as multiple layer perceptron (MLP) and convolutional neural
network (CNN), where the task is image classification Kirkpatrick et al.
(2017); Rusu et al. (2016). Prior works have developed methods for GNN,
primarily for node incrementation Wang et al. (2022) developed a method to
cast vertex classification as a graph classification task by transforming each
node of the graph into a feature graph, which might not consider relations in
the original graph. Furthermore, it is unclear whether the approach can be
extended to other settings involving graph incrementation. Recently, ZHANG et
al. (2022) have introduced a benchmark on continual learning for both node and
graph classification, partly covering incremental graph learning. However, the
benchmark lacks the scenario where each data point is a graph, and the task is
node classification, which can be seen in document Park et al. (2019) and
scene graphs Tang et al. (2020).

In this work, we address continual learning in a more general context where
the incoming data point can be either a node or graph and the task can be
either node or graph classification. Hereinafter, we name the setting
Universal Graph Continual Learning (UGCL). First, we provide a taxonomy of
scenarios in UGCL consisting of node-unit node classification (NUNC), graph-
unit node classification (GUNC), and graph-unit graph classification (GUGC).
Then, we propose a new method to alleviate catastrophic forgetting in all the
scenarios. Our approach adapts rehearsal and distillation techniques to graph-
based models to enhance knowledge retention. We maintain a replay memory of
nodes coupled with their neighbors and continuously draw samples from the
memory to remind the model of previous graphs through distillation mechanisms.
We argue that persevering local and global structure representations are
critical for the effectiveness of graph modeling in UGCL. Hence, we enforce
these representations produced by the model in a task to be close to those of
the previous tasks. As such, we store models corresponding to all tasks and
minimize the difference between the representations.

We demonstrate the performance of our method and other continual learning
baselines in the three scenarios. For NUNC and GUGC, we employ 5 established
datasets to verify our method. For GUNC, we create a new benchmark consisting
of 3 public document image understanding datasets, in which a document is
represented as a graph of text boxes, and the task is to extract the content
and the type of these text boxes (i.e. node classification). In any case, our
proposed approach outperforms the other continual learning methods
significantly and shows competitive performance against the theoretical upper-
bound baselines.

The contributions of our study are as follows:

  * •

We formalize a comprehensive taxonomy of graph continual learning and propose
a new method to handle all cases in the taxonomy. Our method leverages
rehearsal to preserve graph data’s local and global representations.

  * •

We create a new continual learning benchmark for graph-unit node
classification, following the Class-Incremental setting with the task as node
classification.

  * •

We conduct extensive experiments for node and graph classification to validate
our method using a total of 8 datasets (see Table 2).

##  2 Related Work

Recent studies on catastrophic forgetting (CF) mainly consist of experience
replay, regularization, and parameter isolation. Experience replay methods
store old data via experience replay buffers and replay them while learning a
new task Isele and Cosgun (2018). Regularization-based methods leverage
additional loss terms Febrinanto et al. (2022) to consolidate knowledge in the
learning process for new tasks and keep previous knowledge. For instance, EWC
Kirkpatrick et al. (2017) keeps the network parameters close to the optimal
parameters of previous tasks using L2 regularization while parameter Isolation
methods Mallya and Lazebnik (2017); Le and Venkatesh (2022) keep an
independent model for each task to prevent any possible CF among tasks. As
another example, progressive networks Rusu et al. (2016) grow new branches for
new tasks while freezing previous task parameters to avoid modification of
prior knowledge. These methods are designed for simple neural networks such as
MLP or CNN and are validated only in image classification. It is a non-trivial
task to apply them to graph data, which are commonly found in real-world
applications in a streaming fashion, such as social networks, road routes, and
citation graphs.

Feature graph networks (FGNs) Wang et al. (2022) is an early approach that
addresses CF in GNNs. FGN takes the features as nodes and turns nodes into
graphs so that new nodes in the regular graph turn into individual training
samples. As a result, the node classification task becomes a graph
classification task. Research in rehearsal methods is also applied to graph
data such as ER-GNN Zhou and Cao (2021). This work introduced three novel
approaches to selecting prior nodes to be stored in the experience buffer that
aims to prevent the forgetting phenomenon in continual node classification.
ContinualGNN Wang et al. (2020) proposed to combine both rehearsal-based
methods and regularization-based methods to tackle the CF issue when the graph
evolves, and new patterns appear over time. During the new pattern detection
process, they store a small amount of high-influence nodes of existing
patterns. Then, a regularization-based method similar to EWC is employed to
keep the current model parameters do not go too far from previous model
parameters so that both prior knowledge and current knowledge are maintained.

However, all the aforementioned approaches focus on one evolving graph over
time. As far as our knowledge, the problem with multiple separate graphs from
a task has not been studied. The closest to this task is ContinualGNN Wang et
al. (2020), formulating nodes and neighbors belonging to old patterns as one
single graph. In the prior study, each data point is a single graph, but these
graphs can have common nodes, whereas we aim to deal with multi-separated
graphs. It is crucial to derive a method that can work in any setting,
including a single big graph and multi-separated graphs. Our method can be
cast as a hybrid of regularization-based and rehearsal-based methods, as it
attempts to preserve knowledge inferred in previous tasks to overcome
catastrophic forgetting through representation constraints and replaying.
Unlike previous replay methods, however, we do not focus on which samples to
replay; instead, we integrate task-agnostic local and global information from
graph structures into continual learning.

##  3 Problem Formulation

In the following, we first summarize all scenarios in graph learning and their
characteristics regarding continual learning. Then, we detail the problem of
graph-unit continual learning, especially GUNC, which is currently
underexplored in the literature.

Task | $N_{\mathrm{graphs}}$ | Unit | Classification
---|---|---|---
NUNC | Single | Node | Node
GUGC | Multiple | Graph | Graph
GUNC | Multiple | Graph | Node

Table 1: The distinction of graph-unit node classification (GUNC) tacked in
this work compared to node-unit node classification (NUNC) and graph-unit
graph classification (GUGC).

###  3.1 Graph Learning Taxonomy

A graph is defined as $G=(V,E)$ where $V$ is the set of nodes and $E\subset
V^{2}$ is the set of edges. Each node $v\in V$ is coupled with a feature
vector $x\in X\subset\mathbb{R}^{F}$. Each edge $e\in E$ is associated with a
weight vector $w\in\mathbb{R}^{W}$.

In Table 1, we categorize regular graph learning into 3 distinct tasks based
on the number of graphs $N_{g}$ and the definition of a sample (or unit):
node-unit node classification (NUNC), graph-unit graph classification (GUGC),
and graph-unit node-classification (GUNC). First, in NUNC and GUNC, we learn a
predictor $f$ to assign a label $y\in Y$ for every node $v\in V$, given a
graph $G$ with its node features $X$ and edge weights $W$. Unlike the node-
unit version working on a single big graph, our task usually involves multiple
individual graphs. Big graph datasets like CORA McCallum et al. (2000) often
have a big node feature vector of size $1000$ constructed from a long text
paragraph which sometimes makes it less dependent on the graph topology.
Meanwhile, graph-unit data extracted from datasets like CORD Park et al.
(2019) have only a smaller feature vector of size smaller than $100$ made from
a few words and numbers. Consequently, graph-unit learning is rather topology-
dependent in this case. In a single big graph with a million nodes, we can use
message-passing to propagate information from far away nodes. On the other
hand, multiple graphs in graph-unit learning often have less number of nodes
(about $200$ nodes), thus a shallower topology. Hence, with those limitations,
graph learning and continual learning with GUNC are prone to be harder than
their node-unit counterparts. NUNC has been formulated and studied extensively
under continual learning context Zhou and Cao (2021); Wang et al. (2022). The
following section will focus more on GUNC and GUGC.

###  3.2 Graph-Unit Continual Learning

In the continual learning setting for GUNC and GUGC, we have the same
objective as regular graph-unit learning, but the data comes in the form of a
sequence
$G^{t}={(v^{t}_{i},e^{t}_{i},x^{t}_{i},w^{t}_{i},y^{t}_{i})_{i=1}^{M^{t}}}$
where $t$ is the task index and $M^{t}\ll M$ is the number of samples in the
sequence. Assume that all items in $G^{t}$ belong to a distribution $P^{t}$ or
$(v^{t}_{i},e^{t}_{i},x^{t}_{i},w^{t}_{i},y^{t}_{i})\sim P^{t}$, our task is
to maximize the likelihood of a label $y^{t}\sim P^{t}(y|v,e,x,w)$ . The test
sample might belong to one of the past tasks or the current task $t$.
Therefore, our graph-unit continua learning objective is not only to perform
current task training $t$ but also not to forget past tasks ${t^{\prime}}$ to
overcome catastrophic forgetting Li and Hoiem (2017); Kirkpatrick et al.
(2017); Rusu et al. (2016).

In particular, we focus our experiments on a specific continual learning
setting called class-incremental learning (CIL). In CIL, each task contains an
exclusive subset of classes $Y^{t}\subset Y$. According to Hsu et al. (2018),
CIL is the most practical yet most difficult scenario where we have disjoint
output spaces $Y^{t}\cap Y^{t^{\prime}}=\emptyset$ between tasks which
naturally leads to disjoint label distribution $P^{t}(Y)\cap
P^{t^{\prime}}(Y)=\emptyset$ and thus $P^{t}(G)\neq P^{t^{\prime}}(G)$. We
note that it differs from task-incremental learning (TIL), where the latter
has a big relaxation with the known task identity or task descriptor.
Therefore, one could build a multi-head architecture to treat each task
separately, which totally eliminates the risk of cross-task prediction.

##  4 Method

In this work, we aim to develop a system to universally overcome catastrophic
forgetting for NUNC, GUGC, and GUNC. The workflow of the proposed system is
shown in Figure 1. Given an input graph and embedded feature of nodes, our
experience replay (ER) backbone selects replay samples and stores them in a
Replay Buffer Rolnick et al. (2018). Then, we utilize Local Structure
Distillation and Global Structure Distillation to remind the model of the
global-local structure of graph samples in previous tasks. The backbone of our
method is based on the rehearsal approach, which often dominates in continual
learning because the replayed samples provide direct information about past
tasks, giving huge advantages against regularization-based approaches. Whereas
most prior replay mechanisms focus on which samples to replay, we investigate
another aspect. As such, we argue that aside from task-related information,
additional knowledge can be derived from replayed samples in the form of a
task-agnostic distillation loss. Although we discuss only one replay mechanism
as the backbone, our distillation mechanism is not limited to any specific
one.

###  4.1 Graph Experience Replay

Experience Replay (ER) is well-known in the continual image classification
setting with its simple implementation yet effectiveness. Regarding graph
learning, the ER adaptation requires more or fewer changes depending on the
specific learning task. Similar to the image domain, ER for graphs Zhou and
Cao (2021) stores a portion of data belonging to classes from past tasks in
the replay buffer $G_{b}$. Then, in each training step of the current task, we
query $k$ samples to optimize together with the original training data to
prevent catastrophic forgetting. For GUGC, we store graphs belonging to
classes from past tasks with all of their nodes in the replay buffer. For
GUNC, although we only need nodes belonging to classes from past tasks, we
also store their corresponding graphs to use the connectivity information for
later computation. Hence one item (either node or graph) in our replay buffer
always comes with an associated graph. For NUNC, we apply a similar procedure
except that all nodes are from a single graph. Therefore, unlike NUNC having
all nodes connected to each other, GUNC has only some connections from nodes
in the same graph, thus making it harder to learn (see more in Sec 3). In the
following, we describe the implementation of the replay buffer.

First, our replay buffer $G_{b}$ is implemented as a fixed-size list whose
each element is a graph/ node with its label $y\in Y^{t}$. To expand $G_{b}$,
we collect all training data in $G^{t}$ after finishing the training on
$G^{t}$. At this point, our replay buffer $G_{b}$ might become bigger than the
allowed size $\delta$, which triggers a replacement procedure.

The replacement strategy iterates between 2 steps: (1) finding the class that
has the biggest number of samples in the replay buffer, and (2) removing one
sample belonging to that class to ensure class balance Prabhu et al. (2020).
To choose which node to remove, we can use a random selection procedure or a
ranking selection like Mean Feature and Coverage Maximization Zhou and Cao
(2021). Finally, with the replay buffer $G_{b}$, we can optimize our model
continually, minimizing the Experience Replay Loss

$\displaystyle\mathcal{L}^{ER}=\alpha
L(f(v,e,x,w),y)+(1-\alpha)L(f(v_{b},e_{b},x_{b},w_{b}),y_{b})$ (1)

Here, $(v,e,x,w,y)\sim G^{t}$, $(v_{b},e_{b},x_{b},w_{b},y_{b})\sim~{}G_{b}$,
$\alpha$ is the coefficient balancing the learning of the current and past
tasks, $f$ is the predictor, and $L$ is the conventional cross-entropy loss.
We use a uniform sampling strategy to sample examples from the replay buffer
$G_{b}$ for simplicity.

Figure 1: Overview of the proposed method. For simplicity, we show $N$ tasks,
where the model will be continuously learned and updated. The left side gives
the building and updating of the Replay Buffer (RB) across tasks. While the
right side describes the replay and knowledge distillation process.
Nodes/Graphs sampled from RB, along with those in the current task, will be
used to gain structured knowledge from the previous task model by using Local
Structure Distillation and Global Structure Distillation processes. The RB
will be expanded with current task graphs by replacement strategies.

###  4.2 Local-Global Knowledge Distillation

As graph learning is mainly about iterative feature aggregation or message
passing across local neighbors, transferring learned graph structure
representations is critical to retain a graph model performance. Since the
experiment replay strategy reminds the model of previous task samples within
the task objective, it may not help explicitly transfer graph topology
structure representation learned by trained models. Inspired by distillation
approaches Hinton et al. (2014); Yim et al. (2017), we propose a way to
effectively transfer the acquired structure knowledge from models trained in
old tasks. Prior distillation strategies extract the output Hinton et al.
(2014), intermediate activation Zagoruyko and Komodakis (2016); Huang and Wang
(2017), or label Hou et al. (2019); Douillard et al. (2020) from CNN models
with image inputs. Unlike them, our method focuses on graph neural networks,
which handle a more general input from mixed domains (visual, text,
molecules). While there are some studies Deng and Zhang (2021); Yang et al.
(2020) try to distill knowledge from previous graph models to a new model,
they merely focus on learning the structure of the input graph or only
learning local structure. Our work aims to transfer global and local
characteristics of graph structure representations embedded in trained models
to a model in the new task, thereby preventing the model from forgetting key
characteristics of past tasks.

####  4.2.1 Local Structure Distillation

Aggregating neighbor information plays an important role in graph learning,
which depends much on typical local context to make the prediction. Therefore
keeping the model from forgetting previously learned local context or
structure between a node and its neighbors is necessary. We hypothesize that
the model trained on previous tasks has already encoded the local structure
representations of past tasks. Hence, using past samples from the replay
buffer can reconstruct the representations in the past models. These
representations will be used to enforce the current model to output similar
local structure representations as the past models’.

Let $g$ denote the graph associated with the item sampled from the replay
buffer. We use this graph to extract the neighbors of the considering node to
build the local structure representations. We construct a representation of
the local structure through 2 steps: (1) extracting node representations and
(2) forming local structure representations.

Given a set of node features $X=\\{x_{1},x_{2},\dots,x_{N}\\}$ from the graph
$g$ containing $N$ nodes and an adjacency matrix $A$ that represents the edges
among nodes. The hidden representation of node $v_{i}$ at the $l$-th layer,
denoted as $\theta_{i}^{(l)}$, is formulated as

$\displaystyle\theta_{i}^{(l)}=\sigma(\sum_{j\in
K_{(i)}}A_{ij}\theta_{j}^{l-1}W^{l})$ (2)

where $K{(i)}$ denotes the neighbors of node $v_{i}$ , $\sigma(\cdot)$ is an
activation function, and $W^{l}$ is the transformation matrix of the $l$-th
layer. $\theta_{i}^{(0)}$ represents the input feature of node $v_{i}$.
$W^{l}$ is a weight matrix that defines the aggregation strategy from
neighbors. Inspired by Mcpherson et al. (2001), we represent the local
structure representation by measuring the difference between the features of a
node and the features of neighbor nodes. More specifically, the local
structure difference would be formulated as follows

$\displaystyle S_{(g,i)}^{c}$
$\displaystyle=\theta_{i}^{c}-\frac{1}{N_{i}}\sum_{n=1}^{N_{i}}\theta_{n}^{c}$
(3) $\displaystyle S_{(g,i)}^{o}$
$\displaystyle=\theta_{i}^{o}-\frac{1}{M_{i}}\sum_{m=1}^{M_{i}}\theta_{m}^{o}$
(4)

Eqs. (3) and (4) build the local structure embedding vectors as the difference
between a node feature and its neighbor mean feature, where $\theta_{i}^{c}$
and $\theta_{i}^{o}$ are node features from the current model and the previous
model, respectively. $\theta_{n}^{c}$ and $\theta_{m}^{o}$ are neighbor node
features from the current and old models, respectively. $N_{i}$ and $M_{i}$
are the neighbor size where neighbors were selected based on their connection
with the node $v_{i}$ which is defined in the adjacency matrix $A$. We then
try to minimize the distance between two embeddings by minimizing the
following Local Structure Loss

$\mathcal{L}^{LS}=\frac{1}{K_{g}\cdot
K_{n}}\sum_{j=1}^{K_{g}}\sum_{i=1}^{K_{n}}1-cos(S_{(j,i)}^{c},S_{(j,i)}^{o})$
(5)

where $cos$ is the cosine similarity, $K_{g}$ is the number of associated
graphs sampled from the replay buffer, and $K_{n}$ is the number of nodes
sampled from those graphs. This mechanism prevents the current model from
forgetting previously learned local structure representation in previous tasks
while not interfering directly with the learning of the current task since the
regularization occurs only in feature space, not in the final prediction
layer, as in Experience Replay. The choice of $K_{n}$ and its impact will be
discussed in the Experiment section.

####  4.2.2 Global Structure Distillation

The latent interactions among non-neighbor nodes also play an essential role
in learning tasks that require global interactions, such as graph
classification. Preserving the global structure of how the previous model
embeds would keep the current model from forgetting such information learned
in previous tasks. We explicitly train the current model to preserve the
global structure to distil representational knowledge from the previous model
better. To achieve this, we introduce the Global Structure Loss to maximize
similarities among all node features.

Specifically, we distil the representations learned from the previous task
model by computing similarities between graph embeddings from the current and
the previous task’s model as follows

$\mathcal{L}^{GS}=\frac{1}{K_{g}}\sum_{j=1}^{K_{g}}1-cos(f^{P}_{j},f^{C}_{j})$
(6)

where $f^{P}_{j}$ and $f^{C}_{j}$ are the graph embeddings from the previous
and the current task’s model, respectively, where they are computed by
applying weighted sum and max pooling to the node features generated by the
models followed by a linear layer. $K_{g}$ is the number of graphs retrieved
from the replay buffer, similar to that in the local structure distillation.
Here, replaying graph samples of previous tasks prevents the current model
from forgetting learned global structure information in previous tasks, which
assists the representation learning of the entire graph distribution.

Finally, the final objective function $\mathcal{L}^{Total}$ is a linear
combination of the 3 losses and the current task loss:

$\displaystyle\mathcal{L}^{Total}=\mathcal{L}^{Task}+\alpha\mathcal{L}^{ER}+\beta\mathcal{L}^{LS}+\gamma\mathcal{L}^{GS}$
(7)

where ${\mathcal{L}^{Task}}$ is the objective loss of the current tasks,
$\alpha$, $\beta$ and $\gamma$ are hyper-parameters to be tuned while
training.

##  5 Experiments

###  5.1 Implementation Details

Dataset | $N_{\mathrm{graphs}}$ | $N_{\mathrm{classes}}$ | $N_{\mathrm{tasks}}$ | $N_{\mathrm{classes}}/N_{\mathrm{tasks}}$
---|---|---|---|---
Graph-unit node classification
SROIE | $973$ | $4$ | $4$ | $1^{*}$
CORD | $1000$ | $30$ | $5$ | $6$
WILDRECEIPT | $1739$ | $12$ | $4$ | $3$
Graph-unit graph classification
ENZYMES | $600$ | $6$ | $3$ | $2$
Aromaticity | $3868$ | $30$ | $15$ | $2$
Node-unit node classification
CoraFull | $1$ | $70$ | $35$ | $2$
Reddit | $1$ | $40$ | $20$ | $2$
Arxiv | $1$ | $40$ | $20$ | $2$

Table 2: Description of benchmark datasets. *SROIE has #classes/task = 1 but
is still valid because we provide an additional class named None/other in
negative sampling.

####  5.1.1 Model Architecture

Both node classification (GUNC, NUNC) and graph classification (GUGC) share
the same graph model architecture as a simple graph convolutional network
(GCN) with 3 GCN layers Kipf and Welling (2016). Before being fed to the first
GCN layer, the input features go through a linear layer to learn a more
compact representation. Except for the first layer, each GCN layer has a
residual connection from its input to its output. For GUGC benchmarks, we add
a WeightedSumAndMax ZHANG et al. (2022) pooling layer to calculate the graph
embedding. Finally, we pass the final GCN output through a linear layer with a
softmax activation to output the multiclass probability for each node/graph.

####  5.1.2 Training Protocol

We follow the traditional setting of Class Incremental learning. Specifically,
we first train the GNN with either node classification or graph classification
objective on the first task consisting of $N^{1}_{\mathrm{classes}}$ classes.
Next, from the second task to the last task, we keep expanding the model
output layer with additional $N^{t}_{\mathrm{classes}}$ units. Strictly, we
only provide the current task labels, even though our model has past task
output units. As a result, the model performance vastly degrades if no
mechanism is implemented to prevent catastrophic forgetting. Additionally, we
keep all the model checkpoints after every task to run the evaluation later.
Besides, keeping past checkpoints is necessary for our distillation
strategies.

####  5.1.3 Evaluation Protocol

After training our model on each task, we evaluate its performance on the
testing data of the current task and all previous tasks. To be in line with
Chaudhry et al. (2018); Lopez-Paz and Ranzato (2017); ZHANG et al. (2022), we
also utilize the metrics Average Performance (AP) and Average Forgetting (AF).

####  5.1.4 Baselines

We extensively compare our method with non-continual learning baselines and
continual learning baselines, listed below.

Non-CL baselines:

  * •

Finetune Girshick et al. (2013): This baseline follows the above training and
evaluating protocols without any further mechanisms to prevent catastrophic
forgetting. This baseline is a basic starting point for further integration of
advanced continual learning techniques.

  * •

Feature extraction Donahue et al. (2013); Razavian et al. (2014): The baseline
trains the model on the first task dataset and then freezes all previous
layers while adding new output units to the classification layer (see Training
Protocol). With this baseline, we want to validate whether a model can still
learn a new task without forgetting past tasks by keeping the same
representation.

  * •

Independent Paz and Ranzato (2017): Instead of expanding the model, we train a
whole new GNN with $N_{\mathrm{classes}}/N_{\mathrm{tasks}}$ output units for
every encountering task. As a result, the Forgetting Measure for this baseline
is always zero. At the same time, the Average Accuracy is calculated by
averaging the current task accuracies but not the final ones like the other
baselines:

$A_{N_{\mathrm{tasks}}}=\frac{1}{N_{\mathrm{tasks}}}\sum_{j=1}^{N_{\mathrm{tasks}}}a_{j,j}$

Often, the negative impact of forgetting is bigger than the positive impact of
the forward transfer. Thus, the result of this baseline is considered an upper
bound for other methods.

  * •

Joint training Caruana (1998): This baseline is very similar to Finetune, but
we keep the past datasets alongside the current task dataset to train the
model. Therefore, the result of this baseline is also another upper bound for
other methods.

CL baselines. By adding different mechanisms to prevent catastrophic
forgetting over the Finetune baseline, we obtain several advanced baselines:
LwF Li and Hoiem (2017), DK Hinton et al. (2015), EWC Kirkpatrick et al.
(2017), MAS Aljundi et al. (2018), TWP Liu et al. (2021), GEM Lopez-Paz and
Ranzato (2017), and ER Rolnick et al. (2018). Since some of the methods are
originally for non-graph data, we slightly adapt them to our tasks as follows:

  * •

In LwF, we strictly follow the original implementation with the distillation
loss calculated based on the output of the current model and the model in the
previous task for an input graph in our current training data. Specifically,
given an input graph, we calculate the distillation loss for all node
features.

  * •

DK is implemented similarly, except for the loss function changes to KL
Divergence Loss Joyce (2011).

  * •

For EWC, we also follow the original implementation that employs the Fisher
information matrix to regularize the loss function so that network parameters
are close to the learned parameters of previous tasks.

  * •

MAS can be implemented similarly to EWC, except we use the gradient of the
output with respect to a parameter to evaluate its importance.

  * •

On the other hand, TWP is designed for graphs with a loss to preserve the
topological information of the graphs between tasks.

  * •

For GEM, we store training samples (graphs/ nodes) in the replay buffer to
then modify the current task gradients with the gradients calculated with the
replay buffer data.

  * •

ER is the base method for our work that was already described in Sec 4.1. For
the replacement strategy, we use random selection. We note that since GUGC and
NUNC are common settings in the literature ZHANG et al. (2022), we can
implement or reuse baselines on those settings than on the less well-known
setting GUNC.

Experience Replay with Structure Distillation (Ours). Our method integrates
task-agnostic knowledge into ER. Like other ER-related methods, the replay
buffer size acts as one hyperparameter. Moreover, to apply Structure
Distillation to NUNC, GUNC, and GUGC, we loop through a set of $T_{g}$ graph
samples from both the replay buffer and input graphs. In our experiments, we
sampled graphs from the replay buffer equal to the batch size of the input
graph for distilling structure knowledge. Similar to other baselines like LwF
and DK, the distillation loss in our method transfers naturally from node
classification to graph classification. In total, we examine 3 variants, which
are ER with global structure distillation (ER-GS), ER with local structure
distillation (ER-LS), and ER with both global-local structure distillation
(ER-GS-LS).

Method | CORD | SROIE | Wild Receipt
---|---|---|---
AP (%) $\uparrow$ | AF (%) $\uparrow$ | AP (%) $\uparrow$ | AF (%) $\uparrow$ | AP (%) $\uparrow$ | AF (%) $\uparrow$
Finetune | $15.3\pm 0.3$ | $-76.7\pm 0.2$ | $12.4\pm 0.4$ | $-80.1\pm 0.1$ | $23.6\pm 0.1$ | $-78.8\pm 0.1$
Feature Extraction | $19.6\pm 1.3$ | $-8.3\pm 1.7$ | $22.9\pm 0.2$ | $-11.9\pm 0.8$ | $23.6\pm 0.1$ | $-10.5\pm 0.6$
Independent | $79.9\pm 1.5$ | - | $84.9\pm 0.6$ | - | $84.6\pm 0.1$ | -
Joint | $90.9\pm 0.8$ | - | $80.9\pm 1.4$ | - | $83.1\pm 0.3$ | -
EWC | $14.5\pm 2.0$ | $-33.0\pm 1.6$ | $12.6\pm 0.2$ | $-35.4\pm 0.1$ | $19.3\pm 0.4$ | $-34.1\pm 1.3$
LwF | $15.5\pm 1.0$ | $-16.0\pm 0.9$ | $13.8\pm 1.0$ | $-32.9\pm 1.7$ | $20.2\pm 0.5$ | $-45.8\pm 8.7$
DK | $15.9\pm 0.3$ | $-59.9\pm 2.2$ | $11.1\pm 0.2$ | $-67.8\pm 5.6$ | $22.6\pm 0.4$ | $-76.8\pm 0.1$
ER | $79.4\pm 0.4$ | $-2.3\pm 0.3$ | $81.0\pm 1.4$ | $-1.5\pm 0.2$ | $81.6\pm 0.3$ | $-2.4\pm 0.4$
ER-GS (Ours) | $82.7\pm 1.1$ | $-2.9\pm 0.5$ | $84.0\pm 0.7$ | $-1.2\pm 0.2$ | $82.4\pm 0.1$ | $-2.0\pm 0.2$
ER-LS (Ours) | $80.4\pm 1.1$ | $-3.6\pm 0.3$ | $84.2\pm 0.2$ | $-1.3\pm 0.3$ | $82.3\pm 0.3$ | $-2.3\pm 0.4$
ER-GS-LS (Ours) | $\mathbf{84.0\pm 0.4}$ | $\mathbf{-1.9\pm 0.3}$ | $\mathbf{84.4\pm 0.3}$ | $\mathbf{-1.2\pm 0.2}$ | $\mathbf{83.0\pm 0.1}$ | $\mathbf{-1.2\pm 0.6}$

Table 3: Performance comparisons on the task of Document Image Understanding.
Bold/underlined denote the best/second best-performing CL technique for each
column. We note that non-CL baselines (first rows) serve as theoretical bounds
for reference purposes. $-$ denotes ignored AF calculations.

###  5.2 GUNC: Document Image Understanding

####  5.2.1 Datasets

We consider 3 document image understanding datasets: SROIE Huang et al.
(2019), CORD Park et al. (2019), and WILDRECEIPT Sun et al. (2021) (Table 2).
We can set up graph-unit node classification on document datasets with a few
heuristic steps (refer to Task Description). As mentioned, document image
understanding datasets often have less informative feature vectors and shallow
topology, which causes difficulty for graph-unit node classification. For
example, in CORD, the maximum number of nodes is $84$, and the input feature
vector size is $51$. For SROIE and WILDRECEIPT, the quantities are $(240,43)$
and $(217,81)$ respectively. Thus, doing continual learning for GUNC for those
document datasets is rather challenging.

In the graph-unit node classification task, given the input nodes with
associated features and also the graph, the task is to generate the embedded
feature for each node such that nodes with different classes can be separated.
We adopt CORD, SROIE, and WILDRECEIPT datasets that contain, 30, 4, and 12
classes, respectively. For CORD, we keep the original train-validation-test
split while we further split the training data of SROIE and WILDRECEIPT with a
ratio 80-20 for the training and validation set.

####  5.2.2 Task Description

For document image understanding using graph-unit node classification (GUNC),
we first build a graph where each node is equivalent to a text box.
Subsequently, the label for each node is the category of the corresponding
text line. For example, in CORD Park et al. (2019), we have categories like
$\mathrm{store\\_name}$, $\mathrm{price}$. Next, to form the edge information
and build an adjacency matrix, we adopt a heuristic rule based on the
horizontal/vertical alignments between nodes as well as their distances Qian
et al. (2018). Particularly, since different documents have different sizes,
we use a relative distance compared with the document image size to justify an
edge.

After that, with the set of distinct classes as $C$, we construct
$N_{\mathrm{tasks}}$ disjoint same-length subsets $(G^{t})$ as
$N_{\mathrm{tasks}}$ continual learning tasks. In each subset, we ignore nodes
with labels $y\notin C^{t}$ in the training loss for that set. A detailed task
setting is shown in Table 2 for GUNC, including SROIE, CORD, and WILDRECEIPT.
Often, in GUNC, we leverage non-target nodes as negative samples to boost the
overall accuracy. Therefore, in GUNC, tasks can have only 1 class like in the
case of SROIE without losing generality.

####  5.2.3 Hyperparameter Tuning

We use grid-search to tune common hyperparameters. Particularly, the number of
epochs is sampled in the range of $[50,100]$ with a discretization step of 10,
and the learning rate of Adam optimizer is sampled from the set $\\{10^{-2}$,
$10^{-3}$, $10^{-4}$, $10^{-5}\\}$. For EWC, LwF, and DK, the old-new task
weight is sampled in the range of $[0.1,0.9]$. For ER, we set the replay
buffer size to $1000$ nodes from all classes. For our methods, we optimize the
loss weights $\alpha$, $\beta$ and $\gamma$ in the range of $[0.1,0.9]$ using
a faster hyperparameter search: Bayesian Optimization Wu et al. (2019); Nguyen
et al. (2021). For ER-LS, and ER-GS-LS, we set $K_{n}$ equal to batch size
which is one as graphs in the document are not large and dense compared to
other established graph datasets.

####  5.2.4 Results

The results on the 3 document image understanding datasets CORD, SROIE, and
WILDRECEIPT are shown in Table 3. Through reusing past samples from previous
tasks, ER can largely improve the performance of the fine-tuning strategy and
is significantly better than baselines, including EWC, LwF, DK, and Feature
Extraction. Particularly, the model can remember a past task by utilizing a
set of past representative samples in the replay buffer. This direct strategy
turns out to be more effective with graph data since graph-unit continual
learning often contains distinct tasks, which is unlike image classification,
where different tasks share similar structures like digits (MNIST LeCun et al.
(1998)). For instance, a task in CORD can contain $\mathrm{address}$
information while another task can contain $\mathrm{price}$. As a result,
regularization-based methods like EWC, LwF, and DK tend to perform poorly on
our GUNC benchmark. Our method further improves the vanilla ER with the
ability to transfer the global and local structure information, resulting in
the best performance among all the comparison methods.

Regarding evaluation metrics, the average improvements of our method over
Finetune on CORD, SROIE, and WILDRECEIPT are $68.7\%$, $72.0\%$, and $59.4\%$.
As stated in the baseline subsection, joint training can be considered an ER-
based method whose replay buffer contains all past data. Thus, it is
reasonable that our proposed method, in general, has lower performance than
this upper bound baseline. Surprisingly, SROIE benefits most from our
framework, which performs even better than joint training. However, without
the local-structure (LS) and global-structure (GS) distillation, ER baseline
alone only has a similar performance as the joint training on SROIE. That
indicates the importance of introducing additional topology information in the
form of LS and GS. Besides, our proposed method requires fewer samples than
the Joint baseline, which results in a faster training speed. Formally, a
joint training uses $\frac{N_{\mathrm{tasks}}(N_{\mathrm{tasks}}+1)}{2}$ task
data while our method uses only $N_{\mathrm{tasks}}$ task data. Thus, our
method is $\frac{(N_{\mathrm{tasks}}+1)}{2}$ times the speed of the joint
training baseline. For example, in CORD, $N_{\mathrm{tasks}}=5$, we have a $3$
times faster training speed. Note that we ignore the cost for the replay
buffer because its fixed size does not scale with the number of tasks.

Method | ENZYMES | Aromaticity
---|---|---
AP (%) $\uparrow$ | AF (%) $\uparrow$ | AP (%) $\uparrow$ | AF (%) $\uparrow$
Finetune | $23.0\pm 0.5$ | $-47.5\pm 7.5$ | $5.4\pm 0.2$ | $-66.3\pm 2.0$
Joint | $93.2\pm 6.2$ | - | $77.2\pm 1.3$ | -
EWC | $17.2\pm 0.9$ | $-51.1\pm 6.1$ | $7.5\pm 1.6$ | $-69.9\pm 2.5$
MAS | $21.6\pm 1.6$ | $-20.0\pm 2.5$ | $7.8\pm 1.0$ | $-80.5\pm 2.1$
GEM | $20.0\pm 0.0$ | $-57.5\pm 1.0$ | $9.3\pm 2.4$ | $-75.6\pm 1.1$
TWP | $22.5\pm 0.9$ | $-56.5\pm 6.5$ | $6.5\pm 1.6$ | $-45.4\pm 7.7$
LwF | $21.0\pm 1.1$ | $-55.1\pm 5.2$ | $5.7\pm 3.3$ | $-16.1\pm 4.7$
ER | $22.5\pm 0.9$ | $-35.0\pm 2.5$ | $37.2\pm 1.5$ | $14.0\pm 1.3$
ER-GS (Ours) | $21.7\pm 2.1$ | $-35.0\pm 7.2$ | $35.4\pm 1.6$ | $6.2\pm 0.7$
ER-LS (Ours) | $23.3\pm 1.7$ | $-25.0\pm 7.5$ | $40.0\pm 1.7$ | $14.5\pm 0.6$
ER-GS-LS (Ours) | $\mathbf{25.5\pm 1.7}$ | $\mathbf{-20.0\pm 5.1}$ | $\mathbf{45.4\pm 1.5}$ | $\mathbf{17.6\pm 1.2}$

Table 4: Performance comparisons under task Graph Classification on different
datasets. Bold/underlined denote the best/second best-performing CL technique
for each column. We note that non-CL baselines (first rows) serve as
theoretical bounds for reference purposes. $-$ indicates ignored AF
calculation.

The improvements mainly come from incorporating information from both the data
and graph learned structure representation, which provides additional
supervision that can fully utilize the graph structures and limited samples.
Specifically, our methods are remarkably less forgetting compared to other
baselines with AF of $-1.9\%$, $-1.2\%$, and $-1.2\%$ in CORD, SROIE, and Wild
Receipt, respectively. The results are also nearly approach results of
Independent and Joint approaches that require all training samples across
tasks and more computing resources.

###  5.3 GUGC: Molecule Classification

####  5.3.1 Datasets

We consider 2 molecule graph classification datasets: ENZYMES, and Aromaticity
Xiong et al. (2019) (see details in Table 2). ENZYMES is a dataset of protein
tertiary structures which contains 600 enzymes. The task is to classify each
enzyme into one of the 6 Enzyme Commission numbers (EC number). Aromaticity
Xiong et al. (2019) is a molecule dataset with the task of predicting the
number of aromatic atoms in molecules. The node features and adjacency
matrices are available in the dataset. We split the data into
training/validation/test sets with a ratio of 80/10/10.

####  5.3.2 Task Description

For graph classification using graph-unit graph classification (GUGC), we are
given the processed graphs with all the necessary information. Besides, unlike
GUNC, we do not have to filter out non-target nodes of each task since the
labels of each graph group each subset. As there is no None/other class in
graph labels, we have to ensure at least 2 classes are presented in each task,
so we just group 2 classes into one task incrementally, resulting in 3 and 15
binary classification tasks for ENZYMES and Aromaticity, respectively.

####  5.3.3 Hyperparameter Tuning

To tune the hyperparameters, we follow exactly the procedure in ZHANG et al.
(2022) for EWC, MAS, GEM, TWP, and LwF. For ER, we use 10 graphs as the buffer
size for each class in small datasets like ENZYMES, while big datasets like
Aromaticity use 100 graphs. For our methods, based on our experience with the
previous task, we set the loss weights $\alpha$, $\beta$ and $\gamma$ as
$0.5$, $0.5$, and $0.5$, respectively. In ER-LS and ER-GS-LS, we set
$K_{n}=15$ as the graph size in this task is much bigger compared to the graph
in the GUNC task.

####  5.3.4 Results

We report the results on 2 datasets ENZYMES and Aromaticity in Table 4. We
strictly follow the class-incremental setting, training procedure, and data
processing introduced in a GUGC benchmark ZHANG et al. (2022). Overall, the
performance of Finetune method is among the worst both in terms of AP and AF.
While our ER variants still demonstrate outstanding performance. Specifically,
on the ENZYMES dataset, we observe a big gap of 27.5% AF and a slight
improvement of 2.5% AP compared to Finetune. On Aromaticity, big improvements
happen in AP (40.0%) and AF (83.9%). Furthermore, the positive AF shows that
our method can overcome not only catastrophic forgetting but also bring
positive forward transfer. Finally, ER-GS-LS achieves the best performance
compared to other ER-based methods. This proves the usefulness of our Local-
Global Knowledge Distillation, even in the graph-level classification task.

###  5.4 NUNC: Social Graph Learning

####  5.4.1 Datasets

We consider 3 public datasets: CoraFull McCallum et al. (2000), Reddit
Hamilton et al. (2017), and
Arxiv111https://ogb.stanford.edu/docs/nodeprop/#ogbn-arxiv. We strictly follow
the class-incremental setting, training procedure, and data processing
introduced in a NUNC benchmark ZHANG et al. (2022).

####  5.4.2 Hyperparameter Tuning

To tune the hyperparameters, we follow strictly the procedure in ZHANG et al.
(2022) for EWC, MAS, GEM, TWP, and LwF. For ER, we use 10 nodes as the buffer
size for each class in small datasets like CoraFull, while big datasets like
Reddit use 100 nodes. For our methods, based on our experience with the
previous tasks, we set the loss weights $\alpha$, $\beta$, and $\gamma$ as
$0.5$, $0.5$, and $0.5$, respectively. In LS related method, we set $K_{n}=15$
for CoraFull and Arxiv dataset, while on Reddit we set $K_{n}=5$. We provide
details of performance for some choices of $K_{n}$ in Table 6.

####  5.4.3 Results

As shown in Table 5, we compare our methods with other baselines in the
benchmark. Overall, on both Arxiv and Reddit datasets, the performance of our
proposed methods including ER-GS, ER-LS, and ER-GS-LS surpasses other advanced
baselines like EWC, MAS, or ER by a large margin. On Corafull, our ER-GS-LS
ranks highest with a substantial gap among CL baselines followed by TWP, we
create a significant gap of 36.6% AP and 39.6% AF to the Finetune baseline. On
top of that, our method also gain the highest scores on the rest two datasets,
Arxiv and Reddit. Specifically, on Arxiv, the gap is 29.9% AP and 63.1% AF to
the Finetune baseline, and our method can come even close to the upper bound
baseline Joint with a gap of 5.9% AP. For Reddit, our Local Structure
Distillation (ER-LS) is suboptimal due to the densely connected nodes in a
graph, which makes global information to be more favored in this case. This
has been proven to be truly effective in the Global Structure Distillation
setting. Then combining local structure information can be redundant or even
harm global information. Therefore, the ER-GS-LS baseline performed worse than
the Global Structure Distillation (ER-GS). Nevertheless, our method still has
a gap of 70.6% AP and 70.3% AF with the Finetune baseline. With this result,
we can confirm the usefulness of the Local-Global Knowledge Distillation in
the node-level node classification task.

Method | CoraFull | Arxiv | Reddit
---|---|---|---
AP (%) $\uparrow$ | AF (%) $\uparrow$ | AP (%) $\uparrow$ | AF (%) $\uparrow$ | AP (%) $\uparrow$ | AF (%) $\uparrow$
Finetune | $2.9\pm 0.0$ | $-94.7\pm 0.1$ | $4.9\pm 0.0$ | $-87.0\pm 1.5$ | $5.1\pm 0.3$ | $-94.5\pm 2.5$
Joint | $80.6\pm 1.5$ | - | $40.7\pm 2.3$ | - | $84.3\pm 1.7$ | -
EWC | $3.7\pm 0.6$ | $-93.0\pm 1.2$ | $4.9\pm 0.0$ | $-88.9\pm 0.3$ | $10.6\pm 1.5$ | $-92.9\pm 1.6$
MAS | $2.9\pm 1.2$ | $-88.3\pm 2.1$ | $4.8\pm 1.1$ | $-82.8\pm 2.4$ | $9.7\pm 1.3$ | $-93.7\pm 2.2$
GEM | $6.4\pm 1.5$ | $-90.0\pm 2.1$ | $4.9\pm 1.2$ | $-86.4\pm 2.4$ | $28.4\pm 3.5$ | $-71.9\pm 4.2$
TWP | $\mathbf{13.1\pm 2.1}$ | $\mathbf{-80.7\pm 2.3}$ | $4.8\pm 0.9$ | $-88.3\pm 1.5$ | $9.3\pm 1.2$ | $-91.3\pm 2.3$
LwF | $2.7\pm 0.7$ | $-93.7\pm 1.6$ | $4.9\pm 1.1$ | $-87.0\pm 2.3$ | $5.2\pm 1.5$ | $-81.0\pm 1.8$
ER | $3.3\pm 1.8$ | $-90.9\pm 2.1$ | $29.3\pm 1.6$ | $-49.6\pm 2.0$ | $71.7\pm 7.0$ | $-28.4\pm 5.8$
ER-GS (Ours) | $2.9\pm 1.4$ | $-90.6\pm 1.1$ | $29.9\pm 1.8$ | $-46.5\pm 1.9$ | $\mathbf{79.7\pm 4.7}$ | $\mathbf{-19.9\pm 5.0}$
ER-LS (Ours) | $4.5\pm 1.2$ | $-86.7\pm 1.5$ | $33.4\pm 1.0$ | $\mathbf{-36.0\pm 1.3}$ | $72.7\pm 4.6$ | $-27.3\pm 4.3$
ER-GS-LS (Ours) | $\mathbf{39.5\pm 3.1}$ | $\mathbf{-55.1\pm 2.9}$ | $\mathbf{34.8\pm 0.9}$ | $\mathbf{-23.9\pm 1.1}$ | $75.7\pm 4.4$ | $-24.2\pm 5.1$

Table 5: Performance comparisons on the task of Node-Unit Node Classification.
Bold/underlined denote the best/second best-performing CL technique for each
column. We note that non-CL baselines (first rows) serve as theoretical bounds
for reference purposes. $-$ indicates ignored AF calculation.

$K_{n}$ | CoraFull | Arxiv | Reddit | Aromaticity
---|---|---|---|---
AP (%) $\uparrow$ | AF (%) $\uparrow$ | AP (%) $\uparrow$ | AF (%) $\uparrow$ | AP (%) $\uparrow$ | AF (%) $\uparrow$ | AP (%) $\uparrow$ | AF (%) $\uparrow$
$0$ | $2.9\pm 1.4$ | $-90.6\pm 1.1$ | $29.9\pm 1.8$ | $-46.5\pm 1.9$ | $79.7\pm 4.7$ | $-19.9\pm 5.0$ | $35.4\pm 1.6$ | $6.2\pm 0.7$
$1$ | $3.6\pm 1.2$ | $-92.9\pm 1.1$ | $31.5\pm 1.2$ | $-37.6\pm 1.3$ | $71.4\pm 2.7$ | $-28.7\pm 3.5$ | $33.5\pm 1.2$ | $10.3\pm 0.9$
$3$ | $4.8\pm 1.1$ | $-91.8\pm 0.9$ | $32.1\pm 0.8$ | $-44.2\pm 1.2$ | $73.8\pm 2.6$ | $-26.1\pm 2.9$ | $37.4\pm 1.4$ | $13.5\pm 0.8$
$5$ | $6.2\pm 1.3$ | $-90.5\pm 1.4$ | $33.9\pm 0.5$ | $-38.2\pm 0.4$ | $\mathbf{75.7\pm 4.4}$ | $\mathbf{-24.2\pm 5.1}$ | $42.8\pm 1.2$ | $14.7\pm 0.9$
$10$ | $26.6\pm 2.7$ | $-67.8\pm 2.3$ | $34.0\pm 0.9$ | $-33.4\pm 0.9$ | $72.1\pm 3.6$ | $-27.9\pm 4.1$ | $41.9\pm 1.3$ | $14.6\pm 1.5$
$15$ | $\mathbf{39.5\pm 3.1}$ | $\mathbf{-55.1\pm 2.9}$ | $\mathbf{34.8\pm 0.9}$ | $-23.9\pm 1.1$ | $73.7\pm 3.5$ | $-26.3\pm 3.9$ | $\mathbf{45.4\pm 1.5}$ | $17.6\pm 1.2$
$20$ | $38.2\pm 3.2$ | $-56.28\pm 2.8$ | $34.2\pm 1.2$ | $\mathbf{-20.35\pm 1.4}$ | $72.1\pm 3.8$ | $-27.1\pm 4.2$ | $44.8\pm 1.2$ | $\mathbf{10.7\pm 0.9}$

Table 6: Performance comparisons on the selection of $K_{n}$ values for our
ER-GS-LS. Bold denote the best-performing CL technique for each column.

###  5.5 Model Analysis

In this section, we study the impact of the replay buffer on our distillation
strategy. The hyperparameter $K_{n}$ determines the contribution of the
samples from the replay buffer to the local and global distillation process.
$K_{n}=0$ is equivalent to ER-GS variant of our method. We examine different
values of $K_{n}$ and report the performance of our method in Table 6.

As the results demonstrate, sampling more nodes, i.e. larger $K_{n}$, in the
graphs from the replay buffer brings significant improvement in both $AP$ and
$AF$, especially in CoraFull, we have a substantial gap of 35% in $AP$ with
$K_{n}=15$. This result confirms the effectiveness of our method when the
model was given samples in previous tasks to gain the learned local and global
structure representation. In general, with a higher $K_{n}$ value, our model
can be consistently improved; however, at $K_{n}=20$, the performance stops
improving, which might indicate we have reasonable local information to distil
and increasing to a higher value might not help.

##  6 Discussion

In this paper, we systematically formulated Universal Graph Continual Learning
(UGCL) and proposed a novel approach to prevent catastrophic forgetting issues
under both node classification and graph classification settings. We
extensively evaluated several common continual learning approaches on UGCL.
The experiments show that standard regularization-based methods cannot
effectively maintain prior knowledge in graphs, while rehearsal-based and
parameter-isolation-based methods can perform better. The results also
highlight that our proposed solution using memory replay and subgraph
representation constraints often achieves the best results. The method almost
reaches the upper-bound result on both average accuracy and forgetting measure
metrics. Our work is the first step toward the UGCL setting. In the following
work, we will explore our method for large-scale graph problems such as scene
graphs and transport networks.

## References

  * Aljundi et al. (2018) Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars.  Memory aware synapses: Learning what (not) to forget.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 139–154, 2018.
  * Carta et al. (2021) Antonio Carta, Andrea Cossu, Federico Errica, and Davide Bacciu.  Catastrophic forgetting in deep graph networks: an introductory benchmark for graph classification.  _ArXiv_ , abs/2103.11750, 2021.
  * Caruana (1998) Rich Caruana.  Multitask learning.  In Sebastian Thrun and Lorien Y. Pratt, editors, _Learning to Learn_ , pages 95–133. Springer, 1998.  doi: 10.1007/978-1-4615-5529-2_5.  URL https://doi.org/10.1007/978-1-4615-5529-2_5.
  * Chaudhry et al. (2018) Arslan Chaudhry, Puneet Kumar Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr.  Riemannian walk for incremental learning: Understanding forgetting and intransigence.  _CoRR_ , abs/1801.10112, 2018.  URL http://arxiv.org/abs/1801.10112.
  * Deng and Zhang (2021) Xiang Deng and Zhongfei Zhang.  Graph-free knowledge distillation for graph neural networks.  In _IJCAI_ , 2021.
  * Donahue et al. (2013) Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell.  Decaf: A deep convolutional activation feature for generic visual recognition.  _CoRR_ , abs/1310.1531, 2013.  URL http://arxiv.org/abs/1310.1531.
  * Douillard et al. (2020) Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle.  _PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning_ , pages 86–102.  11 2020.  ISBN 978-3-030-58564-8.  doi: 10.1007/978-3-030-58565-5_6.
  * Febrinanto et al. (2022) Falih Febrinanto, Feng Xia, Kristen Moore, Chandra Thapa, and Charu C. Aggarwal.  Graph lifelong learning: A survey.  _IEEE Computational Intelligence Magazine_ , 18:32–51, 2022\.
  * French (1999) Robert M French.  Catastrophic forgetting in connectionist networks.  _Trends in cognitive sciences_ , 3(4):128–135, 1999.
  * Galke et al. (2020) Lukas Galke, Benedikt Franke, Tobias Zielke, and Ansgar Scherp.  Lifelong learning of graph neural networks for open-world node classification.  _2021 International Joint Conference on Neural Networks (IJCNN)_ , pages 1–8, 2020.
  * Girshick et al. (2013) Ross B. Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  _CoRR_ , abs/1311.2524, 2013.  URL http://arxiv.org/abs/1311.2524.
  * Hamilton et al. (2017) Will Hamilton, Zhitao Ying, and Jure Leskovec.  Inductive representation learning on large graphs.  _Advances in neural information processing systems_ , 30, 2017.
  * Hinton et al. (2014) Geoffrey Hinton, Jeff Dean, and Oriol Vinyals.  Distilling the knowledge in a neural network.  pages 1–9, 03 2014.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2(7), 2015.
  * Hou et al. (2019) Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin.  Learning a unified classifier incrementally via rebalancing.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 831–839, 2019.  doi: 10.1109/CVPR.2019.00092.
  * Hsu et al. (2018) Yen Chang Hsu, Yen Cheng Liu, and Zsolt Kira.  Re-evaluating continual learning scenarios: A categorization and case for strong baselines.  _CoRR_ , abs/1810.12488, 2018.  URL http://arxiv.org/abs/1810.12488.
  * Huang and Wang (2017) Zehao Huang and Naiyan Wang.  Like what you like: Knowledge distill via neuron selectivity transfer.  07 2017.
  * Huang et al. (2019) Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and C. V. Jawahar.  Icdar2019 competition on scanned receipt ocr and information extraction.  In _2019 International Conference on Document Analysis and Recognition (ICDAR)_ , pages 1516–1520, 2019.  doi: 10.1109/ICDAR.2019.00244.
  * Isele and Cosgun (2018) David Isele and Akansel Cosgun.  Selective experience replay for lifelong learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Joyce (2011) James M Joyce.  Kullback-leibler divergence.  In _International encyclopedia of statistical science_ , pages 720–722. Springer, 2011.
  * Kipf and Welling (2016) Thomas N. Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  _CoRR_ , abs/1609.02907, 2016.  URL http://arxiv.org/abs/1609.02907.
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ , 114(13):3521–3526, 2017.
  * Le and Venkatesh (2022) Hung Le and Svetha Venkatesh.  Neurocoder: General-purpose computation using stored neural programs.  In _International Conference on Machine Learning_ , pages 12204–12221. PMLR, 2022.
  * LeCun et al. (1998) Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11):2278–2324, 1998.
  * Li and Hoiem (2017) Zhizhong Li and Derek Hoiem.  Learning without forgetting.  _IEEE transactions on pattern analysis and machine intelligence_ , 40(12):2935–2947, 2017.
  * Liu et al. (2021) Huihui Liu, Yiding Yang, and Xinchao Wang.  Overcoming catastrophic forgetting in graph neural networks.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 35, pages 8653–8661, 2021.
  * Lopez-Paz and Ranzato (2017) David Lopez-Paz and Marc’Aurelio Ranzato.  Gradient episodic memory for continual learning.  In _NIPS_ , 2017.
  * Mallya and Lazebnik (2017) Arun Mallya and Svetlana Lazebnik.  Packnet: Adding multiple tasks to a single network by iterative pruning.  _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7765–7773, 2017.
  * McCallum et al. (2000) Andrew Kachites McCallum, Kamal Nigam, Jason Rennie, and Kristie Seymore.  Automating the construction of internet portals with machine learning.  _Information Retrieval_ , 3(2):127–163, 2000\.
  * Mcpherson et al. (2001) Miller Mcpherson, Lynn Smith-Lovin, and James Cook.  Birds of a feather: Homophily in social networks.  _Annual Review of Sociology_ , 27:415–, 01 2001.  doi: 10.3410/f.725356294.793504070.
  * Nguyen et al. (2021) Duy-Hung Nguyen, Bao-Sinh Nguyen, Nguyen Viet Dung Nghiem, Dung Tien Le, Mim Amina Khatun, Minh-Tien Nguyen, and Hung Le.  Robust deep reinforcement learning for extractive legal summarization.  In _Neural Information Processing: 28th International Conference, ICONIP 2021, Sanur, Bali, Indonesia, December 8–12, 2021, Proceedings, Part VI 28_ , pages 597–604. Springer, 2021.
  * Park et al. (2019) Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, and Hwalsuk Lee.  {CORD}: A consolidated receipt dataset for post-{ocr} parsing.  In _Workshop on Document Intelligence at NeurIPS 2019_ , 2019.  URL https://openreview.net/forum?id=SJl3z659UH.
  * Paz and Ranzato (2017) David Lopez Paz and Marc Aurelio Ranzato.  Gradient episodic memory for continual learning.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 6467–6476, 2017.
  * Prabhu et al. (2020) Ameya Prabhu, Philip HS Torr, and Puneet K Dokania.  Gdumb: A simple approach that questions our progress in continual learning.  In _European conference on computer vision_ , pages 524–540. Springer, 2020.
  * Qian et al. (2018) Yujie Qian, Enrico Santus, Zhijing Jin, Jiang Guo, and Regina Barzilay.  Graphie: A graph-based framework for information extraction.  _ArXiv_ , abs/1810.13083, 2018.
  * Razavian et al. (2014) Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson.  CNN features off-the-shelf: an astounding baseline for recognition.  _CoRR_ , abs/1403.6382, 2014.  URL http://arxiv.org/abs/1403.6382.
  * Rolnick et al. (2018) David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, and Greg Wayne.  Experience replay for continual learning.  In _Neural Information Processing Systems_ , 2018.
  * Rusu et al. (2016) Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  _arXiv preprint arXiv:1606.04671_ , 2016.
  * Sun et al. (2021) Hongbin Sun, Zhanghui Kuang, Xiaoyu Yue, Chenhao Lin, and Wayne Zhang.  Spatial dual-modality graph reasoning for key information extraction.  _CoRR_ , abs/2103.14470, 2021.  URL https://arxiv.org/abs/2103.14470.
  * Tang et al. (2020) Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang.  Unbiased scene graph generation from biased training.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 3716–3725, 2020.
  * Wang et al. (2022) Chen Wang, Yuheng Qiu, Dasong Gao, and Sebastian Scherer.  Lifelong graph learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13719–13728, 2022.
  * Wang et al. (2020) Junshan Wang, Guojie Song, Yi Wu, and Liang Wang.  Streaming graph neural networks via continual learning.  In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_, pages 1515–1524, 2020.
  * Wu et al. (2019) Jia Wu, Xiu-Yun Chen, Hao Zhang, Li-Dong Xiong, Hang Lei, and Si-Hao Deng.  Hyperparameter optimization for machine learning models based on bayesian optimization.  _Journal of Electronic Science and Technology_ , 17(1):26–40, 2019.
  * Xiong et al. (2019) Zhaoping Xiong, Dingyan Wang, Xiaohong Liu, Feisheng Zhong, Xiaozhe Wan, Xutong Li, Zhaojun Li, Xiaomin Luo, Kaixian Chen, Hualiang Jiang, et al.  Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism.  _Journal of medicinal chemistry_ , 63(16):8749–8760, 2019.
  * Yang et al. (2020) Yiding Yang, Jiayan Qiu, Mingli Song, Dacheng Tao, and Xinchao Wang.  Distilling knowledge from graph convolutional networks.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 7072–7081, 2020.
  * Yim et al. (2017) Junho Yim, Donggyu Joo, Ji-Hoon Bae, and Junmo Kim.  A gift from knowledge distillation: Fast optimization, network minimization and transfer learning.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 7130–7138, 2017.
  * Zagoruyko and Komodakis (2016) Sergey Zagoruyko and Nikos Komodakis.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  12 2016.
  * ZHANG et al. (2022) Xikun ZHANG, Dongjin Song, and Dacheng Tao.  CGLB: Benchmark tasks for continual graph learning.  In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2022.  URL https://openreview.net/forum?id=5wNiiIDynDF.
  * Zhou and Cao (2021) Fan Zhou and Chengtai Cao.  Overcoming catastrophic forgetting in graph neural networks with experience replay.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 4714–4722, 2021.


# Mitigating Black-Box Adversarial Examples with Bayesian Neural Modeling for
Enhanced End-to-End Speech Recognition

###### Abstract

In this work, we aim to enhance the system robustness of end-to-end automatic
speech recognition (ASR) against adversarially-noisy speech examples. We focus
on a rigorous and empirical “black-box adversarial robustness” setting (e.g.,
on-device or cloud applications) that the adversarial noise is only generated
by black-box optimization (e.g., evolutionary and zeroth-order estimation)
without accessing gradient information of a targeted ASR model directly. We
propose a new Bayesian neural network (BNN) based adversarial detector, which
could model latent distributions against adaptive adversarial perturbation
with divergence measurement. We further simulate deployment scenarios of RNN
Transducer, Conformer, and wav2vec-2.0 based ASR systems with proposed
adversarial detection system. Leveraging the proposed BNN based detection
system, we improve detection rate by +2.77 to +5.42% (relative +3.03 to
+6.26%) and reduce the word error rate by 5.02 to 7.47 % on LibriSpeech
datasets compared to current model enhancement methods against adversarial
speech examples.

Index Terms—  Adversarial Robustness, Robust Speech Recognition, Speech
Recognition Safety, and Sequence Modeling

##  1 Introduction

End-to-end automatic speech recognition [lee1990acoustic,
graves2006connectionist] (ASR) has many applications in human society,
empowering voice-based intelligent control, spoken language understanding
[tur2008calo], on-device services [yang2020multi], and web-based speech
interactions [mendels2015improving]. These high-performance speech
applications benefit from neural network-based ASR systems that have highly
accurate on-device performance with fixed model parameters. Even when the
model parameters of a simulated ASR system are secured by reliable data
protection [yang2021pate], encryption [zhang2019encrypted], and security
measures, recent concerns [yang2020characterizing, zelasko2021adversarial]
about query-free robustness evaluation highlight the need for designing an ASR
system to be robust against noise samples generated by adversarial
optimization. As an example shown in Fig. 1(a), query-free optimization is
applied on random signals to synthesize high-confidence false examples to
maliciously manipulate ASR output (e.g., “open the door”). The aforementioned
noise evaluation on a simulated ASR is called black-box adversarial robustness
and could be categorized in a regime of ASR robustness against environmental
noise (e.g., acoustic conditions) as a crucial challenge for ASR designs.
Table 1 provides an overview of the access constraints for black-box
robustness, which is our focus in this work.

To improve robustness against adversarial examples, adversarial training-based
approaches [yang2020characterizing, sun2018training] have been widely studied,
mainly by applying augmented noisy examples with correct labels to re-train a
targeted neural network model to improve its generalization. However,
incorporating adversarial training [shafahi2020universal] into a large-scale
neural network is costly and unstable due to its sensitivity to high-
dimensional decision boundaries. Furthermore, adversarial training is often
considered to be not economical and realistic for online ASR systems, where
the model parameters have to be updated frequently, and the augmented noises
need to be re-generated for each time.

Recently, adversarial detection mechanisms [yang2018characterizing,
li2021detecting] have emerged as a new alternative to improve system
robustness against adversarial examples. Adversarial detection provides low
training-cost solutions that are easily incorporated into an existing end-to-
end system with an option to control the safety risk by filtering out
suspicious inputs. In this work, we take advantage of randomness properties
from Bayesian neural networks to design a high-performance adversarial
detector empowered by statistical estimation on test samples. As shown in Fig.
1(b), the proposed method is based on an additional variational layer (Flipout
[wen2018flipout]) for distribution estimation that could be easily integrated
into end-to-end ASR architectures. Next, we will review existing methods for
mitigating adversarial speech examples and demonstrate the novelty of our
system design.

Fig. 1: Illustration of (a) black-box adversarial robustness for automatic
speech recognition (ASR) from query-based optimization (e.g., on-device or
cloud services) and (b) proposed Bayesian neural network (BNN) enhanced model
robustness by adversarial detection.

##  2 Related Work

Table 1: Type between black-box ($\mathcal{A}_{E}$ and $\mathcal{A}_{Z}$) and
white-box ($\hat{\mathcal{A}}$) adversarial robustness evaluation. In this
work, we focuses on black-box settings with access constraints during noise
generation.

Noise Types Para. Access Gradient Info. Output Access Ref. $\mathcal{A}_{E}$:
Evolutionary No No Yes [khare2019adversarial, alzantot2019genattack]
$\mathcal{A}_{Z}$: Zeroth-order No No (estimated) Yes [chen2017zoo,
tu2019autozoom] $\hat{\mathcal{A}}$: Fast Gradient Yes Yes Yes
[carlini2018audio]

###  2.1 Adversarial Robustness for Speech Processing

To study adversarial robustness for ASR, additive noise $\delta$ is applied to
the original waveform under an environmental noise ratio = $10$dB reported in
the previous evaluation [yakura2018robust, yang2020characterizing]. The noise
is trainable with an objective to degrade model performance or create
malicious outputs (e.g., “open the door”). The adversarial robustness for
speech modeling is first studied with a simple open-box (so-called “white-
box”) setting, where the $\delta$ is simply generated by gradient information,
such as the fast-gradient-sign method (FGSM) [goodfellow2014explaining] and
projected gradient descent (PGD) [madry2018towards] attacks against the
model’s decision boundaries. However, white-box adversarial evaluations
heavily rely on the hypothesis that the targeted model could be accessed for
its gradient information during the noise generation process. This hypothesis
is generally not true for real-world speech-based applications in recent
discussions and evaluations [tramer2020adaptive, zelasko2021adversarial,
yang2020characterizing]. For example, the end-point device and cloud users
cannot extract gradient information from a simulated ASR model directly, where
the model is prohibited from external visits and protected by different layers
of information security frameworks.

By contrast, closed-model adversarial (so-called “black-box”) evaluations aim
to study strict and empirical settings, where the additive environmental noise
must be generated without knowing model parameters. Black-box optimization
techniques for the neural model have been used in these studies, such as
evolutionary learning [alzantot2019genattack] and zeroth-order optimization
[tu2019autozoom, chen2017zoo] from sample queries. Multiple-objective
evolutionary optimization [khare2019adversarial] has highlighted black-box
adversarial robustness challenges against popular ASR back-ends, including
DeepSpeech [hannun2014deep] and Kaldi ASR [povey2011kaldi]. By using only
limited query information from input/output pairs, a high acoustic similarity
(0.97 to 0.98) is preserved between black-box generated audio and the original
audio, while resulting in large relative word error rate (WER) degradation (up
to 980%). Meanwhile, for improving adversarial robustness, adversarial
training is often augmented with correctly labeled noisy samples for training.
However, adversarial training requires a long training times and high model
complexity. Moreover, adversarial training is costly for on-device simulation
once the ASR model needs updating. Ideally, parallel model enhancement
solutions, including adversarial detection and noise filtering frameworks,
would provide tangible, low-complexity, and energy-efficient solutions toward
safe and reliable ASR. Next, we will review the existing adversarial detection
benchmarks to mitigate the black-box perturbation for ASR.

###  2.2 Distribution Modeling against Black-Box Perturbation

To concretely address empirical adversarial robustness, adaptive noise
evaluation has to been studied or even required to include from most recent
benchmarks, where the adaptive noise is jointly optimized with both targeted
and enhancement modules during the adversarial evaluation. For example,
temporal dependency [yang2018characterizing] (TD) based detection methods have
shown the best performance compared to other defensive methods. TD detector
randomly divides the inputs sequences into multiple uniform segments and
computes the output distribution of whole audio and segmented audio segments
for abnormality detection on the adversarial input. More recently, the self-
attention mechanism [yang2020characterizing] has been incorporated into the TD
properties and further denoises the adversarial features with multi-scale
representation learning. Nevertheless, there is less discussion on how to
design an efficient framework to improve the distributional modeling (e.g.,
abnormal) of the latent representations with the TD properties for the neural
ASR model. In this work, we propose a novel design building upon RNN-T,
conformer, and wav2vec-2.0 with Bayesian neural networks [wen2018flipout] to
improve distributional modeling. Specifically, incorporating randomness in
neural networks have been recently proven improved smoothness
[li2021detecting, bekasovs2018bayesian] of neural predictors, thus providing
stronger robustness guarantees.

##  3 Black-Box Adversarial-Robust Neural Speech Recognition

###  3.1 Evaluating Black-Box Adversarial Robustness

We first consider an end-to-end classification model $M$, where the classifier
is trained with model parameters $\theta$ to produce output prediction (e.g.,
words or phonemes), $M(\mathbf{x};\theta)=\mathbf{y}$, from an input
$\mathbf{x}$ (e.g., acoustic features). The loss function
$L_{\theta}(\mathbf{y},\hat{\mathbf{y}})$ is optimized using gradient descent
to minimize the prediction error between $\mathbf{y}$ and $\hat{\mathbf{y}}$.
In black-box adversarial robustness, we assume that there is no information
about model parameter $\theta$ available to the adversarial noise generator.
As shown in the Eq. 1, untargeted black-box adversarial noise can only be
generated by queries $\mathbf{x},\mathbf{y}$ to maximize the prediction loss:

$\mathbf{x}_{adv}=\mathbf{x}+\delta;\leavevmode\nobreak\
\operatorname*{argmin}_{\delta}\mathbb{E}_{(\mathbf{x},\mathbf{y})\sim\mathcal{D}}\left[L(\mathbf{x}+\delta,\mathbf{y})\right].$
(1)

We have fixed a maximum [alzantot2019genattack] querying number 10k regarding
$\\{\mathbf{x},\mathbf{y}\\}$ input-output pairs in our studies refer to the
established black-box robustness benchmarks. We consider adaptive adversarial
noise setting in this work, where the noise is jointly optimized against a
simulated “ASR associated with an adversarial detector” as a rigorous and firm
setting [tramer2020adaptive, carlini2019evaluating] for empirical adversarial
robustness evaluation with a $\delta_{\max}$=0.01 under $l_{\infty}$-norm.

$\mathcal{A}_{E}$: Multi-objective evolutionary adversarial perturbation. The
method initializes a population of examples around the given input example $x$
picking random examples from a uniform distribution defined over the sphere of
radius $\delta_{\max}$ centered on the original example. The algorithm
computes an adversarial noisy input such that
$\left\|\mathbf{x}-\mathbf{x}_{adv}\right\|_{\infty}\leq\delta_{\max}$. This
is achieved by adding random noise in the range
$\left(-\delta_{\max},\delta_{\max}\right)$ to each dimension of the input
vector $\mathbf{x}$. We follow the multi-objective optimization method
proposed in [khare2019adversarial] to ensure acoustic similarity, computed as
the cosine similarly ($\geq$ 0.95) of MFCCs between the noisy and original
speech inputs, for the ASR robustness evaluation. Since the work of
[khare2019adversarial] does not provide a query number for study, we use the
benchmark black-box algorithm from [alzantot2019genattack] to improve the
sample efficiency to generate adversarial examples.

$\mathcal{A}_{Z}$: Estimated gradient noises with zeroth-order optimization,
where random additive noises are synthesized with estimated gradient
information by zeroth-order optimization on open source ASR models. We follow
an established zeroth order optimization benchmark proposed by AutoZoom
[tu2019autozoom] to generated distortion with an adaptive random gradient
estimation strategy on the ASR system without accessing the target model
parameters. AutoZoom leverages upon jointly optimizing latent embeddings (from
an offline encoder) of a target input to generate query-efficient distortion
by coordinate-wise gradient estimation based on random vector. We use
wav2vec2.0 [baevski2020wav2vec] as the offline encoder to extract latent
embeddings from inputs speech. The unknown gradient information
$\frac{\partial f(\mathbf{x})}{\partial\mathbf{x}_{i}}$ of the $i$-th input
speech sample is estimated by:
$b\cdot\frac{f(\mathbf{x}+\beta\mathbf{u})-f(\mathbf{x})}{\beta}\cdot\mathbf{u}\approx\frac{\partial
f(\mathbf{x})}{\partial\mathbf{x}_{i}}$, where $\beta>0$ represents as a
smoothing parameter (e.g., coordinate-wise estimated gradient), $\mathbf{u}$
is a vector with unit-length. We use scaling parameter ($b$) as $1$ followed a
optimal rate formula in [duchi2015optimal]. The gradient information has used
to computing the standard querying-based black-box loss function defined in
[chen2017zoo]. We refer to [chen2017zoo, tu2019autozoom] for more details.

###  3.2 Bayesian Neural Network for Adversarial Detection

In previous works, BNN and other randomization methods have been used to
improve robust classification accuracy, but they were not simulated for
improve adversarial detection performance on ASR.

Blundell _et al._ [blundell2015weight] introduced an efficient algorithm to
learn BNN parameters. BNN is subject to model the distribution of the hidden
parameters $w$ with given the random variables ($x,y$), instead of estimating
the maximum likelihood values $w_{\text{MLE}}$ for the weights. From a
Bayesian perspective, each stochastic parameter is now a random variable
sampling from a distribution instead of being a fixed (deterministic)
parameter. Given the input $x$ and label $y$, a BNN aims to estimate the
posterior over the weights $p(\bm{w}\mid x,y)$ given the prior $p(\bm{w})$.
The real posterior can be inferenced by a parametric distribution
$q_{\bm{\theta}}(\bm{w})$, where the unknown trainable parameter $\bm{\theta}$
is estimated by minimizing the KL divergence:
$\mathcal{D}_{\mathrm{KL}}\left(q_{\bm{\theta}}(\bm{w})\|p(\bm{w}\mid
x,y)\right)$ over $\bm{\theta}$. $q_{\theta}$ is factorized Gaussian
distribution with neural parameters, where
$q_{\theta_{i}}\left(\bm{w_{i}}\right)=\mathcal{N}\left(\bm{w}_{i};\mu,\sigma^{2}\right)$.

The objective function of the evidence lower bound for training BNNs is
reformulated from expression of KD divergence as shown in (2), which is a sum
of a data-dependent part and a regularization part, for each training data
pair $\left(x_{i},y_{i}\right)\in\bm{D}$ and
$\left(\bm{w}_{i}\right)\in\bm{W}$:

$\underset{\bm{\theta}}{\operatorname*{argmax}}\left\\{\small\underset{\bm{W}\sim
q_{\theta}}{\mathbb{E}}[\log
p(\bm{D}\mid\bm{W})]-\mathcal{D}_{\mathrm{KL}}\left(q_{\bm{\theta}}\|p\right)\right\\}$
(2)

where $\bm{D}$ represents the data distribution; $\bm{W}$ represents random
weights distribution. In the first term of objective (2), the probability of
$y_{i}$ given $x_{i}$ and weights is the output of the model. This part
represents the classification loss. The second term of objective (2) is trying
to minimize the divergence between the prior and the parametric distribution,
as a form of regularization.

Based on the theoretical justification for BNN modeling [liu2018adv,
tran2019bayesian, li2021detecting], we consider $f(\bm{x},\bm{w})$ as a model
with $\bm{x}\sim\mathcal{C}_{\bm{x}}$ and $\bm{w}\sim\mathcal{C}_{\bm{w}}$,
where $\mathcal{C}_{\bm{w}}$ is any distribution that is symmetric about
$\bm{w}_{0}=\mathbb{E}[\bm{w}]$, such as
$\mathcal{N}\left(\bm{w}_{0},\bm{I}\right).$ If
$\nabla_{\bm{x}}f(\bm{x},\bm{w})$ can be approximated by the first-order
Taylor expansion at $\bm{w}_{0}$, we have:

$\mathcal{W}(f(\bm{x}+\bm{\delta},\bm{w}),f(\bm{x},\bm{w}))\geq\mathcal{W}\left(f\left(\bm{x}+\bm{\delta},\bm{w}_{0}\right),f\left(\bm{x},\bm{w}_{0}\right)\right)$
(3)

where $\bm{\delta}$ represents an adversarial perturbation and $\mathcal{W}$
represents a translation-invariant sliced-Wasserstein distance
[kolouri2016sliced] measuring distribution dispersion with one-dimensional
closed form convergence [kolouri2016sliced]. The inequality performs that
parameters associated with randomness will enlarge the distributional
differences between normal and adversarial outputs. Thus, we can utilize the
distributional differences of the BNN layer as shown in Fig. 1(b) to detect
adversarial examples. We estimate the 1-Wasserstein distance between the
distributions to detect adversarial examples. We show that BNNs can enlarge
distributional differences with this distance metric. For a model
$f(\bm{x},\bm{w})$ with $\bm{x}\sim\mathcal{C}_{\bm{x}}$ and
$\bm{w}\sim\mathcal{C}_{\bm{w}}$, where $\mathcal{C}_{\bm{w}}$ is any
distribution that satisfies $\bm{w}$ is symmetric about
$\bm{w}_{0}=\mathbb{E}[\bm{w}]$, such as
$\mathcal{N}\left(\bm{w}_{0},\bm{I}\right)$. We use a efficient stochastic
“Flipout” [wen2018flipout] method for BNN (as shown in Fig. 1(b)) to sample
pseudo-independent posteriors for each input audio, which utilizes a Monte
Carlo approximation of the distribution integrating with advantages of
significantly lower variance and complexity.

Distribution Estimation with BNN: To interpret the detection behavior of
predictions by BNNs, we visualize the distribution of different audio inputs,
including the training set (blue), the test set (green), and the adversarial
samples (red) during the untargeted noisy evaluation. As shown in Fig. 2, the
BNN is accurate and robust from the aforementioned stochastic modeling to
measure and identify the adversaries in LibriSpeech dataset.

Fig. 2: Standard deviation distributions of hidden layer output of the
proposed adversarial detection by Bayesian neural network.

##  4 Experiments

###  4.1 Training Datasets and ASR Architecture

LibriSpeech dataset: The LibriSpeech [panayotov2015librispeech] corpus is a
collection of large-scale audiobooks. We use 960 hours of LibriSpeech for our
evaluation. For the language modeling results, LibriSpeech-960 also provides
the n-gram language models and the corresponding texts taken from Project
Gutenberg books, comprising 803M tokens and 977K unique words. We randomly
select 2,000 samples from “dev-clean” for two adversarial robustness
evaluation tasks (“targeted words” or “non-targeted words” perturbation) in
this study.

ASR architectures for adversarial robustness evaluation:

(1) RNN Transformer: We use RNN-T ASR model with five LSTM layers as an
encoder. Each LSTM layer has 1024 hidden units. The prediction network in
RNN-T comes with two LSTM layers of 1024 units and an embedding layer of 512
units. The one-best sequence is produced by combining the scores of RNN-T with
the language model scores times with a coefficient We take length-normalized
probabilities from the language model (LM) network for its LM scores and
determine the coefficient $\lambda$ using grid search on a development data
set; $\lambda$ was fixed at $0.006$ for all reported experiments.

(2) Conformer ASR: ConformerNet [gulati2020conformer] has been used in
streaming ASR applications. It contains two feed-forward modules sandwiching
the multi-headed self-attention module and the convolution module. The first
feed-forward module is built up by half-step residual weights. The second part
of feed-forward module is connected to a final layer-norm module.

(3) ASR based on wav2vec-2.0 [baevski2020wav2vec], a pretraining method used
to provide waveform-level representations for end-to-end speech recognition.
We follow the setup of “large from scratch” presented in Baevski _et al._
[baevski2020wav2vec] to provide a downstream task for wav2vec-2.0 and evaluate
its black-box adversarial robustness.

###  4.2 Black-Box Adversarial Robustness Baseline

Task 1 ($\mathcal{T}$1) for untargeted adversarial robustness evaluation: We
first evaluate all the ASR models with the (1) multi-objective evolutionary
($\mathcal{A}_{E}$) and (2) zeroth-order ($\mathcal{A}_{Z}$) estimated
gradient information for untargeted noise evaluation. As shown in the Eq. (1),
the optimization process is based on finding a minimum noise under
$l_{p}$-norm constraints (e.g., environmental noise level in our case) to let
the final prediction diverge as far as possible from its original output
prediction. The $l_{\infty}$-norm untargeted adversarial robustness evaluation
is important, owing to a recent theoretical justification [weng2018evaluating]
of its relationship to an upper bound of model robustness toward other noise
(e.g., $l_{2}$ as Gaussian).

Fig. 3: Task 1 baseline under black-box adversarial noise.

As shown in Fig. 3, RNN-T, Conformer, and wav2vec-2.0 models show relatively
small variances between clean test set and additive Gaussian noise (denoted as
$\mathcal{A}_{G}$) under an SNR of 10dB. However, all the evaluated ASR
systems show major performance degradation (increased WER) by evaluating with
evolutionary adversarial noise (denoted as $\mathcal{A}_{E}$) and zeroth-order
optimized based adversarial noise (denoted as $\mathcal{A}_{Z}$) shared the
same SNR ratio (10dB) with the additive Gaussian noise setting. While
adversarial noises ($\mathcal{A}_{E}$ and $\mathcal{A}_{Z}$) causes a severe
23.11% average WER degradation, Gaussian-based perturbation only leads to a
0.38% to 0.42% WER increase. Interestingly, the ConformerNet ASR system shows
slightly better robustness in terms of WER compared to RNN-T and wav2vec-2.0,
which degrades by 25.27% absolute under $\mathcal{A}_{E}$ noise and 21.27%
under $\mathcal{A}_{Z}$ noise. The findings could be due to its scale-free
feature learning during the patch-wise processes based on its enhanced
convolution architecture, which echos to some findings in
[yang2020characterizing].

Table 2: Task 1 with detection methods: WER (%) reduction by filtering out adversarial examples and detection performance. ASR | RNN-T | Conformer | wav2vec-2.0
---|---|---|---
Method | TD | BNN | TD | BNN | TD | BNN
WER | -8.41 | -13.43 | -9.64 | -15.09 | -7.45 | 14.92
AUC | 82.31 | 90.12 | 83.42 | 92.34 | 80.09 | 91.09
FP | 12.23 | 6.23 | 10.29 | 5.23 | 14.32 | 5.78
FN | 6.43 | 2.34 | 7.12 | 2.34 | 8.23 | 3.23

Next, we integrate an additional adversarial detector into the ASR to detect
and filter the a mixed of 6k noisy samples (2k for $\mathcal{A}_{G}$; 2k for
$\mathcal{A}_{E}$; 2k for $\mathcal{A}_{Z}$). We compare the WER deduction by
using a TD detector and a BNN distance-estimated detector in $\mathcal{T}$1
and report the value of WER deduction, area under the curve (AUC), false
positive (FP), and false negative (FN) ratios in Tab. 2. We observe that
Conformer-based ASR is already improved on the four evaluation metrics. The
BNN-based detection and filtering system demonstrates superior results
compared to the TD detection baseline [yang2018characterizing].

Task 2 ($\mathcal{T}$2) for targeted adversarial robustness evaluation: Unlike
untargeted perturbation, targeted sentence adversarial robustness evaluation
aims to prevent the ASR from making misleading predictions. For example, ”open
the door” is one classical malicious target used in adversarial noise
evaluation. To generate noise with a targeted output, the optimization process
tries to find a minimal noise signal $\delta$ that minimize prediction loss
between noisy prediction $\textbf{y}_{\text{adv}}$ and target
$\textbf{y}_{\text{target}}$. The second row of Table 3 shows that the output
predictions of evaluated ASR models are not robust a mix of noisy Librispeech
data. For $\mathcal{T}$2 experiments, we select (1) local smoothing
[yang2018characterizing] (LS), which uses a sliding window of fixed length for
local smoothing to reduce the adversarial perturbation; (2) downsampling
[yang2018characterizing] (DS), where we downsample the original 16 kHz audio
to 8 kHz, resulting in a audio file with a band-limit, that avoids sacrificing
the quality of the recovered audio while reducing the adversarial noise in the
reconstruction phase; and (3) TD methods from [yang2018characterizing]. (4)
For model defense based on speech enhancement (SE), we select the state-of-
the-art self-attention U-Net gated speech enhancement [yang2020characterizing]
against black-box adversarial noise. As shown in Tab. 3, the BNN-based
detection and filtering outperforms all the existing methods and renders
91.94% of $\mathcal{A}_{E}$ and 94.13% of $\mathcal{A}_{T}$ adversarial
perturbations unsuccessful—a new model robustness benchmark when evaluating
with a mix of noisy Librispeech data.

Table 3: The unsuccessful rate (UR%) indicating how often adversarial noise fails to manipulate an ASR model with malicious output of “open the door”. A higher UR value indicates a robust system performance with selected defense algorithms (lines 3 to 5) discussed in Section 4.3. $\mathcal{A}_{E}$ represents evolutionary noise and $\mathcal{A}_{Z}$ represents zeroth-order optimized noise as summarized in Tab. 1. Evaluation | RNN-T | Conformer | wav2vec-2.0 | Avg.
---|---|---|---|---
$\mathcal{A}_{E}$ noise [khare2019adversarial] | 6.76 | 8.79 | 6.39 | 7.31
$\mathcal{A}_{E}$ \+ LS | 10.39 | 10.27 | 10.92 | 10.52
$\mathcal{A}_{E}$ \+ DS | 9.39 | 11.23 | 8.92 | 9.84
$\mathcal{A}_{E}$ \+ TD [yang2018characterizing] | 88.73 | 86.34 | 84.50 | 86.52
$\mathcal{A}_{E}$ \+ SE [yang2020characterizing] | 80.99 | 81.42 | 81.34 | 81.25
$\mathcal{A}_{E}$ \+ BNN | 91.23 | 93.34 | 91.25 | 91.94
$\mathcal{A}_{Z}$ noise [tu2019autozoom] | 24.76 | 25.76 | 23.39 | 24.63
$\mathcal{A}_{Z}$ \+ LS | 24.32 | 25.83 | 23.92 | 24.69
$\mathcal{A}_{Z}$ \+ DS | 23.93 | 25.92 | 23.95 | 24.60
$\mathcal{A}_{Z}$ \+ TD [yang2018characterizing] | 90.87 | 92.34 | 89.12 | 90.77
$\mathcal{A}_{Z}$ \+ SE [yang2020characterizing] | 91.32 | 93.42 | 89.34 | 91.36
$\mathcal{A}_{Z}$ \+ BNN | 94.53 | 95.34 | 92.52 | 94.13

##  5 Conclusions

We have demonstrated a new framework leveraging Bayesian neural networks for
latent distribution modeling to improve adversarial robustness for end-to-end
speech recognition. The newly proposed BNN method attains the best results
compared to existing enhancement methods under black-box adversarial
evaluation. The experimental results suggest that recent Conformer and
wav2vec-2.0 methods also suffer from the adversarial evaluation challenges for
both untargeted and targeted adversarial evaluation. The proposed low-
complexity detection method gives us promising new ways to design robust and
reliable ASR systems.

#### Acknowledgment

The authors thank Ariya Rastrow, Mat Hans and Björn Hoffmeister from Alexa AI
for their valuable comments and discussion.