# Content Augmented Graph Neural Networks

Fatemeh Gholamzadeh Nasrabadi<EMAIL_ADDRESS>Department of Computer
Engineering, Amirkabir University of Technology (Tehran Polytechnic), Tehran,
Iran AmirHossein Kashani<EMAIL_ADDRESS>Department of Computer
Engineering, Amirkabir University of Technology (Tehran Polytechnic), Tehran,
Iran Pegah Zahedi<EMAIL_ADDRESS>Department of Computer Engineering,
Amirkabir University of Technology (Tehran Polytechnic), Tehran, Iran Mostafa
Haghir Chehreghani<EMAIL_ADDRESS>(corresponding author)
Department of Computer Engineering, Amirkabir University of Technology (Tehran
Polytechnic), Tehran, Iran

###### Abstract

In recent years, graph neural networks (GNNs) have become a popular tool for
solving various problems over graphs. In these models, the link structure of
the graph is typically exploited and nodes’ embeddings are iteratively updated
based on adjacent nodes. Nodes’ contents are used solely in the form of
feature vectors, served as nodes’ first-layer embeddings. However, the filters
or convolutions, applied during iterations/layers to these initial embeddings
lead to their impact diminish and contribute insignificantly to the final
embeddings. In order to address this issue, in this paper we propose
augmenting nodes’ embeddings by embeddings generating from their content, at
higher GNN layers. More precisely, we propose models wherein a structural
embedding using a GNN and a content embedding are computed for each node.
These two are combined using a combination layer to form the embedding of a
node at a given layer. We suggest methods such as using an auto-encoder or
building a content graph, to generate content embeddings. In the end, by
conducting experiments over several real-world datasets, we demonstrate the
high accuracy and performance of our models.

Keywords— Graphs (networks), graph neural networks, node embedding
(representation), content embedding, structural embedding

##  1 Introduction

Graphs are an important tool to model data and their relationships. They are
used in many domains and applications such as social networks, scientific
networks, and protein protein interaction networks. Several data analysis and
machine learning tasks and problems, including classification, regression,
clustering and link prediction, have been studied for nodes of a graph or for
a collection of graphs. The state of the art recent algorithms for solving
these problems rely on computing embeddings (representations) for nodes or
graphs. In the embedding (representation) learning task, each node or each
graph is mapped to a low-dimensional vector space, so that nodes or graphs
that are similar to each other in the graph space, should find similar
embeddings in the vector space. Graph neural networks (GNNs) provide a
powerful and popular tool to compute embeddings.

A variety of graph neural networks have been proposed in the literature to
generate high-quality embeddings that can enhance the performance of different
tasks. They mostly rely on the link structure of the graph and iteratively
update the embedding of a node according to its embedding and the embeddings
of its neighbors in the previous iteration (layer). As a result, the local
neighborhood of a node plays a critical role in its computed final embedding.
However, nodes of a graph usually contain a rich content that can be used to
improve several tasks such as node classification and clustering. Existing
GNNs mostly use this content only in the form of feature vectors that are fed
into them, as nodes’ first-layer embeddings. However, consecutive filters
(convolutions) that are applied during various iterations/layers, reduce the
impact of nodes’ feature vectors, so that they find little influence on nodes’
final embeddings. As a result, the discriminative power of nodes’ content
information, which can be useful in many applications, is mostly ignored.

Motivated by this observation and in order to preserve the impact of nodes’
contents at higher GNN layers, in this paper we propose novel methods that
augment nodes’ embeddings with nodes’ content information, at higher GNN
layers. More precisely, we propose a model wherein a structural embedding
using a GNN and a content embedding are computed for each node. These two are
combined using a combination layer to form the final embedding of a node at a
given layer. We propose two methods to generate content embeddings of nodes.
In the first method, we build it using an auto-encoder applied to nodes’
initial feature vectors to improve them. In the second method, we construct it
by forming a content-similarity graph and applying a GNN to this content graph
to obtain nodes’ content embeddings. Our content augmentation methods are
independent of the used GNN model and can be aligned with any of them. In this
paper, we apply them to three well-known graph neural networks: GCN [15], GAT
[22] and GATv2 [3]. By conducting experiments over several real-world
datasets, we show that our content augmentation techniques considerably
improve the performance of GNNs.

The rest of this paper is organized as follows. In Section 2, we provide an
overview on related work. In Section 3, we present preliminaries and
definitions used in the paper. In Section 4, we present our methods for
augmenting graph neural networks with nodes’ content information. We
empirically evaluate the performance of our proposed methods in Section 5.
Finally, the paper is concluded in Section 6.

##  2 related work

Kipf and Welling [15] introduced the concept of graph convolutional networks
(GCNs), with a primary emphasis on node classification. They illuminated that
graph signal processing convolutions essentially amass feature insights from
neighboring nodes encompassing a designated node. Their method involves
generating node embeddings rooted in localized neighborhoods crafted around
individual nodes. In GraphSAGE [10], Hamilton et al. proposed the utilization
of a versatile aggregation function Agg, as follows:

$\bm{z}_{u}^{(k)}=\textsf{sig}\left(\bm{W}^{(k)}\cdot\textsf{Agg}\left(\left\\{\bm{z}_{u}^{(k-1)},\forall
u\in N(v)\right\\}\right),\bm{B}^{(k)}\bm{z}_{v}^{(k-1)}\right),$

where matrices $\bm{W}^{(k)}$ and $\bm{B}^{(k)}$ are trainable weight matrices
and sig is the sigmoid function.

Velickovic et al. [22] introduced Graph Attention Networks (GATs), where they
considered non-equal importance weights for the neighbors of a node. They
suggested using an attention mechanism $\alpha$. Then the attention
coefficient between a pair of nodes $v$ and $u$ is defined as follows:

$e_{vu}=\alpha\left(\bm{W}^{(k)}\bm{z}_{u}^{(k-1)},\bm{W}^{(k)}\bm{z}_{v}^{(k-1)}\right),$

where matrices $\bm{W}^{(k)}$ are trainable weight matrices. Jin et al. [13]
introduced SimP-GCN, a graph convolutional network composed of the subsequent
steps. Initially, it employs an adaptive technique to fuse the structural and
node features. Then, it employs a learning approach to forecast pairwise
feature similarity derived from the hidden embeddings of designated node
pairs.

Alon and Yahav [1] studied the challenge of over-squashing in deep graph
neural networks, a problem that intensified with the growing number of GNN
layers. They suggested maintaining the essential information flow from the
initial layers to a central bottleneck layer. They also proposed a mechanism
that creates new edges, connecting crucial nodes to the target node. In the
GIANT model [6], the authors addressed feature scarcity in graph-agnostic
contexts by leveraging a pre-trained BERT model to predict neighborhoods of
raw textual data. They extracted feature vectors from the BERT model and
seamlessly integrated them into the GNN model.

Li et al. [17] proposed a general class of structure related features, called
Distance Encoding (DE), that captures the distance between the node set whose
representation is to be learned and each node in the graph. It can apply
different graph-distance measures such as shortest path distance or PageRank
scores. You et al. [26] presented the class of Identity-aware Graph Neural
Networks (ID-GNNs). ID-GNN extends GNNs by inductively considering nodes’
identities during message passing. It first extracts the ego network centered
at the node, then performs rounds of heterogeneous message passing, where
different sets of parameters are applied to the center node than to other
surrounding nodes.

Huang et al. [12] proposed an approach to mitigate the computational
complexity of GNN models. In their model, GNN serves as a post-processing
layer, while the initial layer uses the power of a multi-layer perceptron
(MLP). Thost and Chen [21] introduced the DAGNN model to enhance the
performance of multi-layer GNN models for acyclic graphs. DAGNN processes
nodes based on the partial order stipulated by the Directed Acyclic Graph
(DAG), utilizing current-layer information to compute the present-layer
representation of the target node. Chehreghani [5] argued that the
explainability of GNN models is on par with that of decision trees and,
consequently, surpasses the explainability of conventional neural networks.
Brody et al. [3] proposed a straightforward modification to GAT’s attention
function, by reordering its internal operations. They introduced the GATv2
method, which exhibits enhanced expressiveness compared to GAT.

For an in-depth exploration of GNNs, interested readers can refer to [23].
Another aspect of GNNs is their ability to handle dynamic graphs, as
demonstrated by works such as [14, 4]. These methods address scenarios that
require effective updates to computed embeddings after modifications of the
graph, such as: node insertion, node deletion, edge addition, edge deletion,
and weight adjustments. The literature also includes shallow approaches [18,
8, 7], where the embeddings of nodes are considered as parameters that should
be learned. However, these shallow models often exhibit inferior performance
than the GNNs, especially in learning tasks such as node classification.

##  3 Preliminaries

Throughout the paper, we use the following standard for notations and symbols:
lowercase letters for scalars, uppercase letters for sets, multisets and
graphs, bold lowercase letters for vectors and bold uppercase letters for
matrices. We assume the reader is familiar with basic concept in graph theory.
By $G=(V,E)$, we refer to a graph whose node set is $V$ and edge set is $E$.
By $n$ we denote the number of nodes of the graph. We assume that each node
has a feature vector of size $d$. By $X\in\mathbb{R}^{(n\times d)}$, we denote
the matrix of nodes’ feature vectors. By finding vector embeddings for nodes
of a graph, we want to encode them as low-dimensional vectors that summarize
the structure of the graph. A node embedding function g is a mapping that maps
each node in the graph into a low-dimensional vector of real values [9]. The
most popular methods to generate vector embeddings are graph neural networks.

At a high level, GNNs consist of the following steps:

  1. i)

for each node $v$, a neighborhood $N(v)$ is constructed,

  2. ii)

at the first layer, the embedding of node $v$ consists of its features (or
e.g., the zero vector, if attributes are absent), and

  3. iii)

at each layer $k+1$, the embedding of node $v$ is computed using a function f
that takes as input the layer-$k$ embedding of $v$ and the layer-$k$
embeddings of the nodes in the neighborhood of $v$. Function f consists of the
following components: i) a linear transformation defined by (at least) one
trainable weight matrix $W$ that converts a lower layer embedding into a
message, ii) an activation function $\sigma$, which is element-wise applied to
the each generated message to induce non-linearity to the model, and iii) an
aggregation function agg which takes a number of vectors as input, aggregates
them, and generates a vector as the higher layer embedding of the node.

As a result, function f is defined as follows:

$\bm{z}_{v}^{(k+1)}=\textsf{f}(v,N(v))=\mathsf{agg}\left(\sigma\left(\bm{W}\cdot\bm{z}_{u}^{(k)},\forall
u\in N(v)\cup\\{v\\}\right)\right),$ (1)

where $\bm{z}_{v}^{(k+1)}$ and $\bm{z}_{u}^{(k)}$ are respectively the vector
embeddings of $v$ at layers $k+1$ and $k$. A widely used activation function
is ReLU, defined as follows: $\textsf{ReLU}(x)=\textsf{max}(0,x)$.

##  4 Content augmented graph neural networks

As already mentioned, our primary concern in this paper lies in the vanishing
of essential first-layer information, specifically content information, as it
propagates through multi-layer convolutional GNNs. Consequently, the models
tend to over-rely on graph structural information during the final decision-
making process and pay less attention to nodes’ content. In this section, we
present two effective models that are designed to augment GNNs with content
information. The first model is proper for the supervised setting, wherein
enough labeled examples should be accessible during training. This is because
of the auto-encoder, utilized within this model: although its encoder/decoder
components are inherently unsupervised, in our model, they should be trained
with enough number of examples to learn dimension reduction patterns,
effectively. The second model works fine in the semi-supervised setting,
wherein a limited amount of labeled data is accessible during training. It
constructs a content graph, alongside the input graph, and applies GNN models
to both of them. GNNs are known for their superior performance in the semi-
supervised setting. As a result, the second model demonstrates outstanding
performance in this setting.

In the rest of this section, first, we briefly discuss how the input graph is
preprocessed, before feeding it into GNN models. Then we describe our
supervised and semi-supervised content augmentation methods.

###  4.1 Preprocessing

In our data preprocessing phase, we adopt a bag-of-words approach [16] for
content processing, creating the initial vectors that will be employed by the
GNN layers. The bag-of-words technique represents the content information of
the input data, serving as a foundation for subsequent graph-based operations.
It should be noted that due to variations in the datasets, this approach
results in different first-layer vector sizes. We will elaborate on this in
Section 5, providing insights into the impact of diverse dataset
characteristics on the GNNs’ performance.

Furthermore, to enhance information flow and preserve local content within the
graph, it is a common technique to augment the graph structure by adding self-
loops to each node. Our experiments show that the addition of self-loops
improves the performance in cases where the dataset structure is denser than
usual. For example, self-loops boost both the base model and our proposed
model in the DBLP and BlogCatalog datasets. However, in the case of Cora and
CiteSeer, we observe a decrease in performance. We consider this self-loop
addition as a hyperparameter and report the best results obtained when
considering or ignoring this feature. We also apply a dropout layer to GNNs to
prevent overfitting.

###  4.2 Supervised content augmentation of graph neural networks (AugS)

Figure 1 presents the high level structure of our proposed model for content
augmentation of graph neural networks, in the supervised setting. This model
is compatible with any graph neural network and does not depend on the
specific model used. As mentioned earlier, this model is primarily effective
in supervised scenarios, as deep auto-encoders tend to have limited
functionality in cases of data scarcity. We refer to this supervised content
augmentation model as AugS-GNN. In the following, we describe each component
of the model in details. 111Our implementation of AugS is publicly available
at: https://github.com/amkkashani/AugS-GNN

Figure 1: The high-level architecture of AugS-GNN. Red units are computational
and blue units are data.

####  4.2.1 Structural and content embeddings

As mentioned earlier, we create two distinct embeddings for each node. First,
we utilize the GNN model, which incorporates both the graph structure and
features generated from the bag-of-words approach. In this way, a structural
embedding is built for each node. Then, in order to add a stronger dimension
of content information, at higher GNN layers and for each node, we generate a
content embedding. This is done by feeding the first-layer embedding (the
initial feature vector) of each node into an auto-encoder. The output of the
encoder component of the auto-encoder serves as the content embedding.

For auto-encoder, we use the model based on multiple MLP encoder layers and
multiple MLP decoder layers, introduced by Hinton [11]. In the encoder part,
each layer’s size is reduced by half compared to the previous layer, while the
decoder part follows an opposite pattern, with each layer’s size being doubled
relative to the preceding one. The unsupervised loss function used during
training is defined by Equation 2, which helps train the the model’s
parameters:

$J(\theta)=\sum_{i=0}^{n}||f_{dec}\circ f_{enc}(x^{(i)})-x^{(i)}||_{2}^{2},$
(2)

where $\Theta$ is the set of trainable parameters of the encoders and
decoders, operator $\circ$ is function composition, $f_{\text{enc}}$ and
$f_{\text{dec}}$ respectively denote the encoder and decoder functions (each
consists of several MLP layers) and $x^{(i)}$ represents the initial feature
vector of node $i$. The loss function computes the squared L2 norm of the
distance vector between the decoder’s composition of the encoder’s output and
the original input. Using it, we try to train the auto-encoder parameters in a
way that each vector, after encoding and decoding, becomes close to its
original form as much as possible. The summation over $n$ considers feature
vectors of all nodes of the training dataset. After completing the training
phase, we use the output of the last encoder (the input of the first decoder),
as the content embedding. This layer is called the bottleneck layer. It is
worth highlighting that the parameters of the auto-encoder are not jointly
learned with the parameters of the whole model, as a separate loss function is
used to train the model. We set the input dimension based on the number of
node features and the bottleneck dimension to $64$.

####  4.2.2 Combination layer

In this layer, we combine the structural and content embeddings obtained for
each node, to form a unique embedding for it. Our combination layer consists
of two phases: the fusion phase, wherein the two structural and content
embeddings are fused to form a single vector; and the dimension reduction
phase, wherein the dimentionality of the vector obtained from the first phase
is reduced. For the first phase, we can explore several fusion functions,
including: concatenation where the two vectors are concatenated, sum where an
element-wise sum is applied to the vectors and max where an element-wise max
is applied to the vectors. Our Experiments demonstrate that concatenation
consistently outperforms the other fusion methods across all datasets.
Therefore in this paper, we specifically highlight the results achieved using
the concatenation function.

For the second phase, we incorporate an MLP, whose parameters are trained
jointly with the other parameters of the model (unlike the parameters of the
auto-encoder used to generate the content embedding).

####  4.2.3 Prediction head

The embeddings generated by AugS can be used as the input for several
downstream tasks and problems such as classification, regression, and sequence
labeling. The method used to address these tasks is called the prediction
head. Various machine learning techniques, such as SVD, decision trees, and
linear regression, can be used in the prediction process. In the experiments
of this paper, our focus is to assess the model’s performance in classifying
nodes. To do so, we use an MLP, consisting of two dense hidden layers each one
with 16 units, as the prediction head. The parameters of the prediction head
are jointly trained with the other parameters of the whole model.

We train our model using the cross entropy loss function [20]. For a single
training example, it is defined as follows:

$-\sum_{k=1}^{c}T_{k}\log(S_{k}),$ (3)

where $c$ is the number of classes and $T_{k}$ and $S_{k}$ respectively
represent the true probability and the estimated probability of belonging the
example to class $k$. The total cross entropy is defined as the sum of cross
entropies of all training examples.

We use the following setting to train the model. We set the number of epochs
to $200$, batch size to $32$, dropout ratio to $0.05$ (in the MLP of the
prediction head, we set it to $0.2$), and the number of GNN layers to $2$. We
use the Adam algorithm as the optimizer and ReLU as the activation function in
the hidden layers and softmax in the output layer.

###  4.3 Semi-supervised content augmentation of graph neural network (AugSS)

In this section, we present our second augmentation method, specifically
tailored to work in a semi-supervised setting. This approach involves the
construction of an auxiliary graph based on nodes’ content attributes, which
is subsequently integrated with the input graph during GNN processing. By
adopting this strategy, we infuse the GNN with the influence of nodes’
contents, effectively mitigating the issue of content information degradation
within the GNNs. We refer to this semi-supervised content augmentation model
as AugSS-GNN. Similar to AugS, this approach is compatible with any graph
neural network and does not rely on the particular model employed. 222Our
implementation of AugSS is publicly available at:
https://github.com/FatemehGholamzadeh/AugSS-GNN

####  4.3.1 Content graph construction

Our second approach for content augmentation in GNNs involves the creation of
a ”content graph”, which is fed into the GNN framework. Henceforth, in this
paper, we refer to the input graph as the ”structural graph”, to distinguish
it from the content graph. The construction of the content graph is done
through the utilization of inherent features within the graph nodes. Here are
two steps for building the content graph:

  1. 1.

We compute pairwise similarities between initial feature vectors of nodes. We
can employ diverse metrics suitable for assessing the likeness of two vectors,
such as Euclidean distance, dot product, or cosine similarity. In this paper,
we specifically employ cosine similarity.

  2. 2.

If the similarity value of the feature vectors of two nodes exceeds a certain
threshold $\epsilon$, we create an edge between the two nodes. It is important
to note that the threshold value varies across different datasets. In this
study, we employ a grid search methodology to determine the optimal threshold
for each dataset.

####  4.3.2 Model’s architecture

Our semi-supervised strategy for integrating the content graph into the
training procedure of graph neural networks is illustrated in Figure 2. Apart
from the structural graph, we introduce the content graph as an input to the
graph neural network. GNNs take a feature matrix of graph nodes as input.
Therefore, both the structural and content graphs require an initial feature
matrix to be provided as input to the GNN. In our approach, we employ the same
feature matrix $X$ as the initial feature matrix, for both the structural
graph and the content graph. We examined alternative techniques, such as
vectors generated by the DeepWalk algorithm [19], or initializing the feature
matrix for the content graph with a set of structural graph features. However,
these methods did not yield satisfactory results. Let’s denote the structural
graph by $G$ and the content graph by $G^{\prime}$. All graph neural networks
used in this section have two graph convolution layers. First, a graph
convolution layer, called Conv1, is applied to each of the graphs, considering
the adjacency matrix specific to that graph. It is important to note that each
of these graph convolutions has its own weight parameters, and they do not
share trainable weight parameters.

Suppose that Conv1 maps input vectors with dimension $d$ to output vectors
with dimension $d^{\prime}$. After applying this layer to $G$ and
$G^{\prime}$, we will have two output matrices with the same size $n\times
d^{\prime}$. In fact, for each node we will have two vectors of dimension
$d^{\prime}$. At this stage, we combine these two vectors using an aggregation
method, which could be, for example, element-wise averaging or summing, or
their concatenation, to obtain a vector of dimension $2d^{\prime}$. Then, we
reduce the dimensionality of this vector to obtain a vector of dimensions
$d^{\prime}$, which is used as input to the next layer.

In our experiments, we also employed an alternative aggregation method that
led to improved results. We introduced two scalar trainable parameters,
$w_{1}$ and $w_{2}$, serving as the weights for the structural graph and
content graph outputs, respectively. As a result, if we denote the structural
graph output with $h_{1}$ and the content graph output with $h_{2}$, the
combined output can be computed as follows:

$h=w_{1}\times h_{1}+w_{2}\times h_{2}.$ (4)

We train these two weights throughout the network’s training process, along
with the other parameters. This dynamic training approach allows us to
adaptively determine the significance of each embedding component.

The second layer of the graph neural network (Conv2), which is also the final
layer, takes input vectors of dimension $d^{\prime}$ and maps them to vectors
of dimension $c$, where $c$ is the number of classes in node classification
tasks. Therefore, by applying the softmax function, the class of each node is
determined. Similar to the AugS method, we use cross entropy as the loss
function. Figure 2 illustrates our proposed model’s architecture.

Figure 2: The high-level architecture of AugSS-GNN.

##  5 Experiments

In this section, by conducting experiments over several real-world datasets,
we show that our content augmenting techniques considerably improve the
performance of GNNs. We choose three well-known GNN models to improve by our
augmentation techniques: GCN [15], GAT [22] and GATv2 [3]. The experiments are
conducted on a Colab T4 GPU with 16 GB VRAM available for computation.

###  5.1 Datasets

To evaluate our models, we utilize five widely used datasets: Cora [25],
CiteSeer [25], DBLP [2], BlogCatalog [24] and Wiki [24]. The specifications of
these real-world datasets are summarized in Table 1. We select these datasets
for evaluation due to their inclusion of both graph-based and textual content.
We partition each dataset into training, validation, and test sets.

Table 1: Summary of our real-world datasets. Dataset | #nodes | #edges | #attributes | #classes
---|---|---|---|---
CiteSeer | 3312 | 4732 | 3703 | 6
Cora | 2708 | 5278 | 1433 | 7
Wiki | 2405 | 17981 | 4973 | 17
DBLP | 17716 | 105734 | 1639 | 4
BlogCatalog | 5196 | 343486 | 8189 | 6

###  5.2 Evaluation measures

We adopt accuracy and macro-F1 as the evaluation criteria. Accuracy measures
the proportion of correctly classified instances. Macro-F1 accounts for both
precision and recall, considering class imbalances and providing a balanced
assessment of the model’s performance. They are formally defined as follows.

$\text{accuracy}=\frac{\text{number of correctly classified test
examples}}{\text{total number of test examples}}\times 100.$ (5)

$precision_{class(k)}=\frac{TP_{class(k)}}{TP_{class(k)}+FP_{class(k)}},$ (6)

where $TP_{\text{class}(k)}$ represents the number of correctly predicted
examples in class $k$ and $FP_{\text{class}(k)}$ indicates the number of
examples predicted as class $k$ but belong to other classes.

$recall_{class(k)}=\frac{TP_{class(k)}}{TP_{class(k)}+FN_{class(k)}},$ (7)

where $FN_{\text{class}(k)}$ is defined as the number of examples in class $k$
that are predicted as belonging to other classes.

$F1_{class(k)}=\frac{2\times precision_{class(k)}\times
recall_{class(k)}}{precision_{class(k)}+recall_{class(k)}},$ (8)

and

$macro\text{-}F1=\frac{1}{c}\times\sum_{k=1}^{c}F1_{class(k)},$ (9)

where $c$ is the number of classes.

###  5.3 Results

In this section, we present the results of our experiments, in both supervised
and semi-supervised settings.

####  5.3.1 Supervised content augmentation

In the supervised setting, we split each dataset in half for training and
testing. We report accuracy and macro-F1 scores for both the GNN models and
the AugS-GNN models. We run each model for $10$ times and report the average
results as well as the standard deviations. The accuracy results reported in
Table 2 depict improvements over most of the cases. In a number of cases (DBLP
and BLOGCAT), the improvements of AugS are significant. Additionally, the
macro-F1 scores reported in Table 3 demonstrate noticeable improvements of
AugS, indicating the effectiveness of our suggested model over imbalance
datasets that have classes with few instances.

Table 2: Comparing accuracy scores of the basic GNN models against their
augmented counterparts, using the AugS method.

| CORA | CITESEER | WIKI | DBLP | BLOGCAT
---|---|---|---|---|---
GCN |  86.92±0.26 | 73.32±0.08 |  74.62±0.15 |  80.21±0.02 |  71.88±2.97
AugS-GCN | 87.41±0.55 |  72.44±0.38 | 75.58±0.94 | 83.09±0.13 | 86.04±1.33
GAT |  85.11±0.00 | 75.83±0.13 |  73.47±0.74 |  72.83±0.019 |  59.05±0.41
AugS-GAT | 87.65±0.46 |  74.76±0.52 | 73.64±0.60 | 83.00±0.19 | 72.52±0.96
GATv2 |  87.10±0.11 | 75.90±0.14 |  72.96±0.58 |  67.08±0.62 |  62.29±0.33
AugS-GATv2 | 87.52±0.37 |  74.43±0.73 | 73.40±1.10 | 83.69±0.25 | 70.94±0.81

Table 3: Comparing macro F1-scores of the basic GNN models against their
augmented counterparts, using the AugS method.

| CORA | CITESEER | WIKI | DBLP | BLOGCAT
---|---|---|---|---|---
GCN |  84.87$\pm$0.12 |  69.33$\pm$0.07 | 64.00$\pm$0.53 |  64.60$\pm$0.06 |  69.62$\pm$4.68
AugS-GCN | 86.48$\pm$0.50 | 87.10$\pm$0.11 |  59.89$\pm$3.01 |  78.73$\pm$0.22 | 85.637$\pm$1.34
GAT |  82.47$\pm$0.03 | 71.59$\pm$0.18 |  52.03$\pm$2.20 |  50.20$\pm$0.054 |  58.19$\pm$0.40
AugS-GAT | 86.22$\pm$0.65 |  71.48$\pm$0.79 | 58.79$\pm$2.14 | 79.08$\pm$0.36 | 71.88$\pm$0.95
GATv2 |  84.87$\pm$0.12 | 71.50$\pm$0.17 |  51.47$\pm$2.30 |  39.47$\pm$0.39 |  61.615$\pm$0.37
AugS-GATv2 | 85.80$\pm$0.62 |  71.21$\pm$0.11 | 58.77$\pm$2.21 | 80.12$\pm$0.39 | 70.57$\pm$0.78

####  5.3.2 Semi-supervised content augmentation

In this section, we assess the performance of AugSS-GNN, within a semi-
supervised framework. To provide a limited amount of training data for the
models, we select $20$ labeled samples per class from our datasets and
randomly sample $500$ data points for validation. Additionally, we allocate
$1000$ data points for evaluating the models. In the case of the wiki dataset,
some classes have fewer than $20$ samples, prompting us to select just $5$
training samples from these classes. We present accuracy (Table 4) and
macro-F1 scores (Table 5) for both the base models (GCN, GAT, and GATv2) and
their AugSS models. These reported scores are based on the results of $10$
runs of the models, where the average accuracy and macro-F1 scores as well as
the standard deviations are provided in the tables.

Our experiments illustrates the impact of incorporating a content graph into
the training process of graph neural networks. By using AugSS, across various
datasets, we observe noticeable improvements in accuracy and F1-score, ranging
from approximately $2\%$ to $14\%$. Therefore, AugSS considerably enhances the
performance of GNNs, when a limited amount of labeled data is available during
the training phase. This improvement in the performance of GNN models is due
to the role of nodes’ content information in their discrimination, which is
better modeled and reflected to the final classifier, using our content
augmentation method.

Table 4: Comparing accuracy scores of the basic GNN models against their
augmented counterparts, using the AugSS method.

| CORA | CITESEER | WIKI | DBLP | BLOGCAT
---|---|---|---|---|---
GCN |  80.81 $\pm$ 0.60 |  69.32 $\pm$ 0.69 |  68.65 $\pm$ 1.26 |  74.92 $\pm$ 1.03 | 75.38 $\pm$ 0.83
AugSS-GCN | 82.48 $\pm$ 0.93 | 72.859 $\pm$ 0.55 | 72.61 $\pm$ 0.84 | 77.31 $\pm$ 0.65 |  72.87 $\pm$ 0.92
GAT |  81.17 $\pm$ 1.21 |  68.14 $\pm$ 1.51 |  52.89 $\pm$ 3.92 |  75.4 $\pm$ 0.96 |  63.54 $\pm$ 2.59
AugSS-GAT | 82.14 $\pm$ 0.71 | 72.463 $\pm$ 0.53 | 59.5 $\pm$ 0.74 | 76.33 $\pm$ 0.71 | 67.6 $\pm$ 2.83
GATv2 |  80.64 $\pm$ 0.62 |  68.2 $\pm$ 1.11 |  50.61 $\pm$ 3.31 |  74.85 $\pm$ 1.36 |  67.57 $\pm$ 2.10
AugSS-GATv2 | 81.62 $\pm$ 0.42 | 72.50 $\pm$ 0.69 | 59.7 $\pm$ 1.55 | 76.42 $\pm$ 0.82 | 81.28 $\pm$ 1.33

Table 5: Comparing macro F1-scores of the basic GNN models against their
augmented counterparts, using the AugSS method.

| CORA | CITESEER | WIKI | DBLP | BLOGCAT
---|---|---|---|---|---
GCN | 79.917 ± 0.6 | 65.627 ± 0.9 | 58.775 ± 1.6 | 70.943 ± 0.9 | 74.702 ± 0.9
AugSS-GCN | 81.355 ± 0.9 | 69.119 ± 0.6 | 62.781 ± 1.1 | 74.03 ± 0.8 | 71.994 ± 1.0
GAT | 80.362 ± 1.0 | 68.459 ± 0.0 | 43.291 ± 2.9 | 72.109 ± 0.8 | 62.991 ± 2.2
AugSS-GAT | 81.035 ± 0.9 | 70.457 ± 2.1 | 48.855 ± 1.5 | 72.813 ± 0.8 | 66.79 ± 2.6
GATv2 | 79.745 ± 0.6 | 64.628 ± 0.9 | 41.602 ± 2.1 | 71.879 ± 1.3 | 66.414 ± 2.3
AugSS-GATv2 | 80.659 ± 0.6 | 67.785 ± 0.5 | 47.623 ± 1.7 | 72.948 ± 1.2 | 80.742 ± 1.4

##  6 Conclusion

In this paper, we proposed novel methods that augment nodes’ embeddings with
their content information at higher GNN layers. In our methods, for each node
a structural embedding and a content embedding are computed and combined using
a combination layer, to form the embedding of the node at a given layer. We
presented techniques such as using an auto-encoder or building a content graph
to generate nodes’ content embeddings. By conducting experiments over several
real-world datasets, we showed that our models considerably improve the
accuracy of GNN models.

## References

  * [1] Uri Alon and Eran Yahav.  On the bottleneck of graph neural networks and its practical implications.  arXiv preprint arXiv:2006.05205, 2020.
  * [2] Aleksandar Bojchevski and Stephan Günnemann.  Deep gaussian embedding of graphs: Unsupervised inductive learning via ranking.  arXiv preprint arXiv:1707.03815, 2017.
  * [3] Shaked Brody, Uri Alon, and Eran Yahav.  How attentive are graph attention networks?  In International Conference on Learning Representations, 2022.
  * [4] Mostafa Haghir Chehreghani.  Dynamical algorithms for data mining and machine learning over dynamic graphs.  Wiley Interdiscip. Rev. Data Min. Knowl. Discov., 11(2), 2021.
  * [5] Mostafa Haghir Chehreghani.  Half a decade of graph convolutional networks.  Nature Machine Intelligence, 4(3):192–193, 2022.
  * [6] Eli Chien, Wei-Cheng Chang, Cho-Jui Hsieh, Hsiang-Fu Yu, Jiong Zhang, Olgica Milenkovic, and Inderjit S Dhillon.  Node feature extraction by self-supervised multi-scale neighborhood prediction.  arXiv preprint arXiv:2111.00064, 2021.
  * [7] Yuxiao Dong, Nitesh V. Chawla, and Ananthram Swami.  metapath2vec: Scalable representation learning for heterogeneous networks.  In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, August 13 - 17, 2017, pages 135–144. ACM, 2017.
  * [8] Aditya Grover and Jure Leskovec.  node2vec: Scalable feature learning for networks.  In Balaji Krishnapuram, Mohak Shah, Alexander J. Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi, editors, Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pages 855–864. ACM, 2016\.
  * [9] Aditya Grover and Jure Leskovec.  Node2vec: Scalable feature learning for networks.  InProceedings of KDD 2016, 2016.
  * [10] William L. Hamilton, Zhitao Ying, and Jure Leskovec.  Inductive representation learning on large graphs.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 1024–1034, 2017.
  * [11] Geoffrey E Hinton and Ruslan R Salakhutdinov.  Reducing the dimensionality of data with neural networks.  science, 313(5786):504–507, 2006.
  * [12] Qian Huang, Horace He, Abhay Singh, Ser-Nam Lim, and Austin R Benson.  Combining label propagation and simple models out-performs graph neural networks.  arXiv preprint arXiv:2010.13993, 2020.
  * [13] Wei Jin, Tyler Derr, Yiqi Wang, Yao Ma, Zitao Liu, and Jiliang Tang.  Node similarity preserving graph convolutional networks.  In 14th ACM International Conference on Web Search and Data Mining (WSDM) 2021, Jerusalem, Israel, March 8-12, 2021. ACM, 2021.
  * [14] Seyed Mehran Kazemi, Rishab Goel, Kshitij Jain, Ivan Kobyzev, Akshay Sethi, Peter Forsyth, and Pascal Poupart.  Representation learning for dynamic graphs: A survey.  J. Mach. Learn. Res., 21:70:1–70:73, 2020.
  * [15] Thomas N. Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
  * [16] Quoc Le and Tomas Mikolov.  Distributed representations of sentences and documents.  In International conference on machine learning, pages 1188–1196. PMLR, 2014.
  * [17] Pan Li, Yanbang Wang, Hongwei Wang, and Jure Leskovec.  Distance encoding: Design provably more powerful neural networks for graph representation learning.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  * [18] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena.  Deepwalk: online learning of social representations.  In Sofus A. Macskassy, Claudia Perlich, Jure Leskovec, Wei Wang, and Rayid Ghani, editors, The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’14, New York, NY, USA - August 24 - 27, 2014, pages 701–710. ACM, 2014.
  * [19] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena.  Deepwalk: Online learning of social representations.  ACM KDD, 2014.
  * [20] Claude Elwood Shannon.  A mathematical theory of communication.  The Bell system technical journal, 27(3):379–423, 1948.
  * [21] Veronika Thost and Jie Chen.  Directed acyclic graph neural networks.  arXiv preprint arXiv:2101.07965, 2021.
  * [22] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio.  Graph attention networks.  In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
  * [23] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and Philip S. Yu.  A comprehensive survey on graph neural networks.  IEEE Trans. Neural Networks Learn. Syst., 32(1):4–24, 2021.
  * [24] Renchi Yang, Jieming Shi, Xiaokui Xiao, Yin Yang, Juncheng Liu, and Sourav S. Bhowmick.  Scaling attributed network embedding to massive graphs.  CoRR, abs/2009.00826, 2020.
  * [25] Zhilin Yang, William Cohen, and Ruslan Salakhudinov.  Revisiting semi-supervised learning with graph embeddings.  In International conference on machine learning, pages 40–48. PMLR, 2016.
  * [26] Jiaxuan You, Jonathan M. Gomes-Selman, Rex Ying, and Jure Leskovec.  Identity-aware graph neural networks.  In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 10737–10745. AAAI Press, 2021.


# The (In)Effectiveness of Intermediate Task Training
For Domain Adaptation and Cross-Lingual Transfer Learning

Sovesh Mohapatra1,$\dagger$, Somesh Mohapatra 2,$\dagger$

# The (In)Effectiveness of Intermediate Task Training
For Domain Adaptation and Cross-Lingual Transfer Learning

Sovesh Mohapatra1,$\dagger$, Somesh Mohapatra 2,$\dagger$

###### Abstract

Transfer learning from large language models (LLMs) has emerged as a powerful
technique to enable knowledge-based fine-tuning for a number of tasks,
adaptation of models for different domains and even languages. However, it
remains an open question, if and when transfer learning will work, i.e.
leading to positive or negative transfer. In this paper, we analyze the
knowledge transfer across three natural language processing (NLP) tasks - text
classification, sentimental analysis, and sentence similarity, using three
LLMs - BERT, RoBERTa, and XLNet - and analyzing their performance, by fine-
tuning on target datasets for domain and cross-lingual adaptation tasks, with
and without an intermediate task training on a larger dataset. Our experiments
showed that fine-tuning without an intermediate task training can lead to a
better performance for most tasks, while more generalized tasks might
necessitate a preceding intermediate task training step. We hope that this
work will act as a guide on transfer learning to NLP practitioners.

## Introduction

Knowledge-based transfer learning leverages zero or few-shot learning from a
pre-trained model to predict for a range of similar tasks (You et al. 2020;
Raffel et al. 2020; Houlsby et al. 2019). The ability to use a pre-trained
model, as-is or with very limited training, has proposed a very lucrative
opportunity, as compared to training from scratch for every single task (Pan
2020; Day and Khoshgoftaar 2017) The applications of transfer learning have
ranged from NLP to image, and even video tasks (Kim et al. 2020; Salza et al.
2022; Bengio 2012).

In recent works, people have applied transfer learning to a range of NLP
tasks, observing mixed results, both positive and negative transfer (Zhang et
al. 2022). Pruksachatkun et al. (2020) showed how transfer learning with
intermediate task training could affect a number of target and probing
English-language NLP tasks. In most cases, positive transfer from LLMs, such
as BERT, has been noted for similar language NLP tasks, like hate speech
classification (Mozafari, Farahbakhsh, and Crespi 2019), propaganda detection
(Vlad et al. 2019), and biomedical NLP tasks (Peng, Yan, and Lu 2019).
Negative transfer has been shown in attempts to transfer an English Part-of-
Speech (POS) tagger to a Hindi corpus (Dell’Orletta 2009; Rayson et al. 2007),
and other NLP tasks (Wang et al. 2019).

Transfer learning for domain adaptation has been widely studied and applied
across language and medical fields (Xu, He, and Shu 2020; Ghafoorian et al.
2017; Kouw and Loog 2018). Savini and Caragea (2022) showed how intermediate
task training on sarcasm helped in transfer learning, similar to Felbo et al.
(2017) and Baroiu and Traușan-Matu (2022). However, in another domain
adaptation task, Meftah et al. (2021) showed that knowledge transfer between
related seemingly similar domains like news and tweets resulted in negative
transfer, probing the results using both quantitative and qualitative methods.

Cross-lingual tasks are another area where transfer learning strategies have
shown a lot of potential (Ahmad et al. 2020; Luo et al. 2021). Chen et al.
(2018) have shown that when language-invariant and language-specific features
are coupled at the instance level.

In this work, we analyze the effect of intermediate task training on a larger
dataset for three different NLP tasks - text classification, sentiment
analysis, and sentence similarity - and evaluate three language models - BERT,
RoBERTa, and XLNet. For each NLP task, we have one domain adaptation and
another cross-lingual task. In total, we have eighteen experiments on a range
of NLP tasks.

## Methodology

Here, we present an overview of our methodology, including information on
transfer learning for intermediate task training, and domain adaptation, and
cross-lingual fine-tuning and evaluation for the NLP tasks, and the datasets.

In each of the following tasks, both intermediate task training and fine-
tuning were performed by training over 70% of the dataset, and evaluated on
the remaining 30%. For the intermediate task training, each pre-trained LLM
was trained for 100 epochs using the large dataset. For fine-tuning after and
without intermediate task training, transfer learning to the target dataset
was performed by training for 10 epochs. In both cases of transfer learning,
all the model weights were updated, or none of the layers were frozen.

Model instances for LLMs, BERT, RoBERTa, and XLNet, were obtained from the
respective GitHub repositories (Qiu et al. 2020).

In each of the NLP tasks, the dataset used for the intermediate task training
from the LLM is at least an order of magnitude larger than the dataset used
for fine-tuning the model to the target task. The target task for domain
adaptation, and the respective datasets, have been chosen to be in a similar
field, as of the intermediate task dataset. For cross-lingual target tasks, we
have tried to ensure that the task is in the same domain, and the language has
semantic and syntactic similarity.

### Text classification

For text classification, we performed intermediate task training using the
IMDB movie reviews dataset (Maas et al. 2011). The IMDB movie reviews dataset
has 50,000 examples classifying the movie reviews into two classes: positive
and negative.

To evaluate domain adaptation, we used a randomly sampled subset of SMS spam
collection dataset, with 5,600 examples (Delany, Buckley, and Greene 2012),
annotated as spam or not. For cross-lingual task evaluation, we created a
dataset of French and Spanish movie reviews, with 100 examples of each
language, obtained by web scraping. Each example was translated into English
using Google Translate and annotated as positive or negative.

### Sentiment analysis

IMDB genre classification dataset was used for intermediate task training for
the sentiment analysis task (Kumar et al. 2022). This dataset consisted of
49,000 examples, classified into 27 different movie genres.

We sampled the GoEmotions dataset for 5,200 examples across all 28 different
emotions, including neutral (Demszky et al. 2020) for the domain adaptation
task. Similar to text classification, we manually created a dataset of French
and German sentences with 1,700 examples, and used Google Translate to
translate them into English, and then manually annotate them into 13 different
emotions.

### Sentence similarity

We used the Paraphrase Adversaries from Word Scrambling (PAWS) dataset,
containing human-labeled sentence similarity for more than 49,000 pair-wise
examples, for the intermediate task training (Zhang, Baldridge, and He 2019).

For domain adaptation, we collected about 2,100 stock names, listed on the New
York Stock Exchange and NASDAQ-traded stocks. We sampled 4,000 exampled from
the PAWS-X dataset to evaluate cross-lingual sentence similarity (Yang et al.
2019).

Figure 1: Accuracy on the test dataset against the fine-tuning epoch of the
intermediate task transfer learnt LLMs has been noted for different tasks. The
rows represent the NLP tasks, and the columns represent the transfer learning
tasks. Specific datasets used for the intermediate task training have been
mentioned to the right of the NLP task, and the transfer learning datasets
used for training and testing have been noted on the graphs.

## Results and Discussion

To understand the effects of fine-tuning with and without intermediate task
training, we analyzed the accuracy of the three NLP tasks for both domain
adaptation and cross-lingual predictions.

### RoBERTa and BERT with intermediate task training are the best models,
depending on the task

In our observations, we observed that fine-tuning a post-intermediate task
training transfer learnt RoBERTa LLM outperformed in three out of six tasks,
across three NLP tasks (text classification, sentiment analysis, sentence
similarity) and two transfer learning experiments (domain adaptation, cross-
lingual performance), while BERT outperformed others in the rest three (Figure
1). In both cases of text classification - domain adaptation and cross-lingual
prediction, and sentiment analysis - domain adaptation, BERT outperformed
RoBERTa. XLNet was consistently the worst performing model in all of our
experiments.

Table 1: Accuracy of the target-task fine-tuned large language models with and
without intermediate task training has been noted for the different NLP tasks.
The specific LLMs are BERT, RoBERTa, and XLNet, with the appended note
mentioning if the specific model had an intermediate task training or not.
Models with intermediate task training (I) on the larger dataset followed by
fine-tuning (F) to the target task have been noted as ‘ModelName-IF’, such as
BERT-IF, and if the model has only been fine-tuned, then it has been noted as
‘ModelName-F’, such as BERT-F. In target tasks per NLP task, the first task is
for domain adaptation, and the next one is for cross-lingual adaptation. The
best performing model accuracy for each NLP target-task has been noted in
bold.

NLP Task | Target Task | BERT-IF | BERT-F | RoBERTa-IF | RoBERTa-F | XLNet-IF | XLNet-F
---|---|---|---|---|---|---|---
Text Classification | SMS Spam | 0.62 | 0.71 | 0.55 | 0.73 | 0.48 | 0.58
| French/Spanish Movie Reviews | 0.67 | 0.63 | 0.51 | 0.58 | 0.46 | 0.44
Sentimental Analysis | GoEmotions | 0.89 | 0.83 | 0.76 | 0.71 | 0.62 | 0.59
| French/German Emotions | 0.61 | 0.68 | 0.66 | 0.72 | 0.53 | 0.57
Sentence Similarity | Stock Ticker | 0.52 | 0.67 | 0.67 | 0.72 | 0.47 | 0.55
| PAWS-X | 0.61 | 0.67 | 0.71 | 0.74 | 0.62 | 0.58

Similar trends for transfer learning using LLMs, where RoBERTa and BERT have
similar performance, and both outperform XLNet, have been recently reported
for detection of clickbaits (Rajapaksha, Farahbakhsh, and Crespi 2021), fake
reviews (Gupta, Gandhi, and Chakravarthi 2021), and mental illness prediction
(Ameer et al. 2022).

Irrespective of the tasks, we noted that the change in accuracy against the
fine-tuning epochs for the LLMs the followed a similar trend. RoBERTa and
XLNet improved on the accuracy metric for about 5 to 7 epochs, and then the
accuracy started to go down. BERT had almost similar accuracy with minor
improvements, or even worsening, as the fine-tuning epochs increased. We
attribute such behavior to an observation of the linguistic information
storage noted by pruning of layers in BERT, RoBERTa, and XLNet in a recent
work (Durrani, Sajjad, and Dalvi 2021), stating that BERT stored the
information deep in the network, while RoBERTa and XLNet localized it in the
lower layers. Their observations qualitatively indicate higher rates of fine-
tuning in the early epochs for RoBERTa and XLNet, aligning with our
observations. In the later epochs, such behavior aligns with the bias-variance
trade-off that comes with fine-tuning a generalized model to the target task,
where too much training erodes the generalizability of the intermediate task
training and starts overfitting to the target dataset.

### Fine-tuned RoBERTa, without intermediate task training, outperforms other
transfer learnt LLMs in most tasks

Across all models, both with and without intermediate task training, RoBERTa,
only fine-tuned on the target dataset, has the highest accuracy in four out of
six tasks, while BERT with intermediate task training followed by fine-tuning
leads in the other two - text classification - cross-lingual prediction and
sentiment analysis - domain adaptation (Table 1). Comparison of accuracy with
and without intermediate task training for the LLMs provides us the
opportunity to discuss the effects of positive and negative transfer, and thus
the (in)effectiveness of transfer learning for domain adaptation and cross-
lingual predictions.

The negative transfer noted across most of the tasks indicated that
intermediate task training led to an unnecessary and ineffective
generalization for the domain adaptation and cross-lingual tasks. For
instance, in the case of text classification - domain adaptation, the
intermediate task training on IMDB movie reviews, followed by fine-tuning, led
to the best model as BERT with an accuracy of 0.62, whereas direct fine-tuning
led to an accuracy of 0.71, 0.73, and 0.58 for BERT, RoBERTa, and XLNet,
respectively. The similar performance of BERT and RoBERTa, in this specific
task, show how fine-tuning is effective. The performance of XLNet, with an
accuracy of 0.58, which is closer to the best intermediate task trained -
fine-tuned model’s accuracy of 0.62, further provides convincing argument for
the case. We observed similar trends for the negative transfer for other
tasks, except in the cases where BERT with both intermediate task training and
fine-tuning on the target dataset outperformed the only fine-tuned models. We
further do a case-by-case analysis and try to provide a qualitative argument
for the positive and negative transfer.

For text classification, we noted that the SMS spam collection, though
seemingly similar to the movie reviews, is fundamentally different with
distinct spamming patterns, thereby leading to better performance on direct
fine-tuning. However, cross-lingual predictions for French/Spanish movie
reviews follow a linguistically, specifically the semantics and syntax,
similar notation to the English reviews, thereby leading to a positive
transfer with the intermediate task training.

In the context of sentiment analysis, sentiments in GoEmotions dataset are
similar to the movie genres, for instance, the emotion love corresponding to
the genre romance, which might be leading to an internal covariate shift with
the fine-tuning, preceded by the intermediate task training, thus being aided
by a positive transfer. However, the French-German emotions classification
have only thirteen emotions, instead of the twenty-seven genres and twenty-
eight emotions in the GoEmotions domain adaptation task, which might lead to a
harder covariate shift in the confusion matrix for classification.
Additionally, it shows that the syntactic knowledge of the English language
might not be the only factor in understanding the emotion behind the
French/German text.

Ultimately, for sentence similarity, we attribute the directly fine-tuned
model performing better on the financial domain adaptation to the lack of
inherent pattern for the full names of tickers, which have been adapted from a
very specialized domain. In this case, the linguistic knowledge from the
general PAWS dataset does not transfer any finance-specific knowledge to the
model. We believe that a similar argument can be made for the confusing nature
of syntax in any particular language used when paraphrasing, thereby leading
to a negative transfer for a cross-lingual sentence similarity task.

## Limitations and Future Work

In the case of domain adaptation, the focus on SMS spam similarity to movie
reviews, genre to emotions, is a limited field of view. We would like to
expand on these tasks and extend the study to multiple cases to see if there
are design principles based on similarity of the domain that could predict
positive or negative transfer. We believe that future work including multiple
domains, with a quantitative similarity on the domains themselves, would be
helpful in arriving at conclusions about the effectiveness of transfer
learning in domain adaptation.

For transfer learning for cross-lingual tasks, we are limited by the small
lexicons considering closely related languages to English. Furthermore, the
the model performance on the cross-lingual tasks is dependent on the task and
the diversity in the source data. Considering unrelated languages, such as
Chinese, Korean, Hindi and more would shed light on how transfer learning
strategies could be developed to provide better predictions, without the
necessity of training models from scratch.

## Conclusion

In this work, we analyzed the effectiveness of transfer learning by fine-
tuning to a target task, with and without an intermediate task training step.
We observed that in almost all cases, the intermediate task training leads to
a negative transfer for both domain adaptation and cross-lingual NLP tasks. On
the performance of different LLMs, we noted that BERT and RoBERTa performed
similarly, and outperformed XLNet on the tasks when an intermediate task
training step was involved. However, when we compared all models both with and
without the intermediate task training step, only fine-tuned RoBERTa emerged
as a clear winner. We hope that these results, and those that arise as a part
of this work in the near future, inform the community on the positive and
negative effects of transfer learning with an intermediate task training step.

## References

  * Ahmad et al. (2020) Ahmad, Z.; Jindal, R.; Ekbal, A.; and Bhattachharyya, P. 2020.  Borrow from rich cousin: transfer learning for emotion detection using cross lingual embedding.  _Expert Systems with Applications_ , 139: 112851.
  * Ameer et al. (2022) Ameer, I.; Arif, M.; Sidorov, G.; Gomez-Adorno, H.; and Gelbukh, A. 2022.  Mental illness classification on social media texts using deep learning and transfer learning.  _arXiv preprint arXiv:2207.01012_.
  * Baroiu and Traușan-Matu (2022) Baroiu, A.-C.; and Traușan-Matu, S. 2022.  Automatic Sarcasm Detection: Systematic Literature Review.  _Information_ , 13(8): 399.
  * Bengio (2012) Bengio, Y. 2012.  Deep learning of representations for unsupervised and transfer learning.  In _Proceedings of ICML workshop on unsupervised and transfer learning_ , 17–36. JMLR Workshop and Conference Proceedings.
  * Chen et al. (2018) Chen, X.; Awadallah, A. H.; Hassan, H.; Wang, W.; and Cardie, C. 2018.  Multi-source cross-lingual model transfer: Learning what to share.  _arXiv preprint arXiv:1810.03552_.
  * Day and Khoshgoftaar (2017) Day, O.; and Khoshgoftaar, T. M. 2017.  A survey on heterogeneous transfer learning.  _Journal of Big Data_ , 4(1): 1–42.
  * Delany, Buckley, and Greene (2012) Delany, S. J.; Buckley, M.; and Greene, D. 2012.  SMS spam filtering: Methods and data.  _Expert Systems with Applications_ , 39(10): 9899–9908.
  * Dell’Orletta (2009) Dell’Orletta, F. 2009.  Ensemble system for Part-of-Speech tagging.  _Proceedings of EVALITA_ , 9: 1–8.
  * Demszky et al. (2020) Demszky, D.; Movshovitz-Attias, D.; Ko, J.; Cowen, A.; Nemade, G.; and Ravi, S. 2020\.  GoEmotions: A dataset of fine-grained emotions.  _arXiv preprint arXiv:2005.00547_.
  * Durrani, Sajjad, and Dalvi (2021) Durrani, N.; Sajjad, H.; and Dalvi, F. 2021.  How transfer learning impacts linguistic knowledge in deep NLP models?  _arXiv preprint arXiv:2105.15179_.
  * Felbo et al. (2017) Felbo, B.; Mislove, A.; Søgaard, A.; Rahwan, I.; and Lehmann, S. 2017.  Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm.  _arXiv preprint arXiv:1708.00524_.
  * Ghafoorian et al. (2017) Ghafoorian, M.; Mehrtash, A.; Kapur, T.; Karssemeijer, N.; Marchiori, E.; Pesteie, M.; Guttmann, C. R.; Leeuw, F.-E. d.; Tempany, C. M.; Ginneken, B. v.; et al. 2017.  Transfer learning for domain adaptation in MRI: Application in brain lesion segmentation.  In _International conference on medical image computing and computer-assisted intervention_ , 516–524. Springer.
  * Gupta, Gandhi, and Chakravarthi (2021) Gupta, P.; Gandhi, S.; and Chakravarthi, B. R. 2021.  Leveraging transfer learning techniques-bert, roberta, albert and distilbert for fake review detection.  In _Forum for Information Retrieval Evaluation_ , 75–82.
  * Houlsby et al. (2019) Houlsby, N.; Giurgiu, A.; Jastrzebski, S.; Morrone, B.; De Laroussilhe, Q.; Gesmundo, A.; Attariyan, M.; and Gelly, S. 2019.  Parameter-efficient transfer learning for NLP.  In _International Conference on Machine Learning_ , 2790–2799. PMLR.
  * Kim et al. (2020) Kim, Y.-G.; Kim, S.; Cho, C. E.; Song, I. H.; Lee, H. J.; Ahn, S.; Park, S. Y.; Gong, G.; and Kim, N. 2020.  Effectiveness of transfer learning for enhancing tumor classification with a convolutional neural network on frozen sections.  _Scientific Reports_ , 10(1): 1–9.
  * Kouw and Loog (2018) Kouw, W. M.; and Loog, M. 2018.  An introduction to domain adaptation and transfer learning.  _arXiv preprint arXiv:1812.11806_.
  * Kumar et al. (2022) Kumar, S.; Kumar, N.; Dev, A.; and Naorem, S. 2022.  Movie genre classification using binary relevance, label powerset, and machine learning classifiers.  _Multimedia Tools and Applications_ , 1–24.
  * Luo et al. (2021) Luo, J.; Wang, J.; Cheng, N.; Xiao, E.; Xiao, J.; Kucsko, G.; O’Neill, P.; Balam, J.; Deng, S.; Flores, A.; et al. 2021.  Cross-language transfer learning and domain adaptation for end-to-end automatic speech recognition.  In _2021 IEEE International Conference on Multimedia and Expo (ICME)_ , 1–6. IEEE.
  * Maas et al. (2011) Maas, A. L.; Daly, R. E.; Pham, P. T.; Huang, D.; Ng, A. Y.; and Potts, C. 2011\.  Learning Word Vectors for Sentiment Analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , 142–150. Portland, Oregon, USA: Association for Computational Linguistics.
  * Meftah et al. (2021) Meftah, S.; Semmar, N.; Tamaazousti, Y.; Essafi, H.; and Sadat, F. 2021.  On the hidden negative transfer in sequential transfer learning for domain adaptation from news to tweets.  In _Proceedings of the Second Workshop on Domain Adaptation for NLP_ , 140–145.
  * Mozafari, Farahbakhsh, and Crespi (2019) Mozafari, M.; Farahbakhsh, R.; and Crespi, N. 2019.  A BERT-based transfer learning approach for hate speech detection in online social media.  In _International Conference on Complex Networks and Their Applications_ , 928–940. Springer.
  * Pan (2020) Pan, S. J. 2020.  Transfer learning.  _Learning_ , 21: 1–2.
  * Peng, Yan, and Lu (2019) Peng, Y.; Yan, S.; and Lu, Z. 2019.  Transfer learning in biomedical natural language processing: an evaluation of BERT and ELMo on ten benchmarking datasets.  _arXiv preprint arXiv:1906.05474_.
  * Pruksachatkun et al. (2020) Pruksachatkun, Y.; Phang, J.; Liu, H.; Htut, P. M.; Zhang, X.; Pang, R. Y.; Vania, C.; Kann, K.; and Bowman, S. R. 2020.  Intermediate-task transfer learning with pretrained models for natural language understanding: When and why does it work?  _arXiv preprint arXiv:2005.00628_.
  * Qiu et al. (2020) Qiu, X.; Sun, T.; Xu, Y.; Shao, Y.; Dai, N.; and Huang, X. 2020.  Pre-trained models for natural language processing: A survey.  _Science China Technological Sciences_ , 63(10): 1872–1897.
  * Raffel et al. (2020) Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; Liu, P. J.; et al. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(140): 1–67.
  * Rajapaksha, Farahbakhsh, and Crespi (2021) Rajapaksha, P.; Farahbakhsh, R.; and Crespi, N. 2021.  BERT, XLNet or RoBERTa: The Best Transfer Learning Model to Detect Clickbaits.  _IEEE Access_ , 9: 154704–154716.
  * Rayson et al. (2007) Rayson, P.; Archer, D. E.; Baron, A.; Culpeper, J.; and Smith, N. 2007.  Tagging the Bard: Evaluating the accuracy of a modern POS tagger on Early Modern English corpora.  In _Proceedings of the Corpus Linguistics conference: CL2007_.
  * Salza et al. (2022) Salza, P.; Schwizer, C.; Gu, J.; and Gall, H. C. 2022.  On the effectiveness of transfer learning for code search.  _IEEE Transactions on Software Engineering_.
  * Savini and Caragea (2022) Savini, E.; and Caragea, C. 2022.  Intermediate-task transfer learning with BERT for sarcasm detection.  _Mathematics_ , 10(5): 844.
  * Vlad et al. (2019) Vlad, G.-A.; Tanase, M.-A.; Onose, C.; and Cercel, D.-C. 2019.  Sentence-level propaganda detection in news articles with transfer learning and BERT-BiLSTM-capsule model.  In _Proceedings of the second workshop on natural language processing for internet freedom: Censorship, Disinformation, and Propaganda_ , 148–154.
  * Wang et al. (2019) Wang, X.; Li, L.; Ye, W.; Long, M.; and Wang, J. 2019.  Transferable attention for domain adaptation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, 5345–5352.
  * Xu, He, and Shu (2020) Xu, W.; He, J.; and Shu, Y. 2020.  Transfer learning and deep domain adaptation.  _Advances and Applications in Deep Learning_ , 45.
  * Yang et al. (2019) Yang, Y.; Zhang, Y.; Tar, C.; and Baldridge, J. 2019.  PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification.  In _Proc. of EMNLP_.
  * You et al. (2020) You, K.; Kou, Z.; Long, M.; and Wang, J. 2020.  Co-tuning for transfer learning.  _Advances in Neural Information Processing Systems_ , 33: 17236–17246.
  * Zhang et al. (2022) Zhang, W.; Deng, L.; Zhang, L.; and Wu, D. 2022.  A survey on negative transfer.  _IEEE/CAA Journal of Automatica Sinica_ , 9: 1–25.
  * Zhang, Baldridge, and He (2019) Zhang, Y.; Baldridge, J.; and He, L. 2019.  PAWS: Paraphrase Adversaries from Word Scrambling.  In _Proc. of NAACL_.