# Pre-training Graph Neural Network for Cross Domain Recommendation

Chen Wang1, Yueqing Liang2, Zhiwei Liu1, Tao Zhang1, and Philip S. Yu1
1Department of Computer Science, University of Illinois at Chicago, USA
2Business School, University of Sydney, Sydney, AU  {cwang266, zliu213,
tzhang90<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

A recommender system predicts users’ potential interests in items, where the
core is to learn user/item embeddings. Nevertheless, it suffers from the data-
sparsity issue, which the cross-domain recommendation can alleviate. However,
most prior works either jointly learn the source domain and target domain
models, or require side-features. However, jointly training and side features
would affect the prediction on the target domain as the learned embedding is
dominated by the source domain containing bias information. Inspired by the
contemporary arts in pre-training from graph representation learning, we
propose a pre-training and fine-tuning diagram for cross-domain
recommendation. We devise a novel Pre-training Graph Neural Network for Cross-
Domain Recommendation (PCRec), which adopts the contrastive self-supervised
pre-training of a graph encoder. Then, we transfer the pre-trained graph
encoder to initialize the node embeddings on the target domain, which benefits
the fine-tuning of the single domain recommender system on the target domain.
The experimental results demonstrate the superiority of PCRec. Detailed
analyses verify the superiority of PCRec in transferring information while
avoiding biases from source domains.

###### Index Terms:

Recommender system; Cross-domain; Pre-training; Contrastive learning

Figure 1: An illustration of PCRec model architecture in specific example. The
model is mainly composed of three setps. In the pre-training stage, a model
with the structural learning of nodes’ embedding is used to learn the source
domain’s structure. The target domain is put into the pre-trained model to
initialize nodes’ embedding and fine-tune by a bipartite recommendation system
in the fine-tuning stage. In the last, applied fine-tuned embedding to
predict.

##  I Introduction

A recommender system predicts the potential interests of users to items, where
the core is to learn user/item embeddings. Matrix factorization is an early
method for collaborative filtering (CF), where it learns user/item embeddings
and reconstructs their interactions with inner product [1, 2]. The general
idea of CF approach is based on observation of the preferences of other users
that are similar to the historical preferences of the target user. Later on,
deep recommender systems [3, 4, 5, 6] have shown that capturing deep features
in a supervised or unsupervised manner is more appealing than shallow models
such as CF to capture similarity and implicit relationship between items.
However, due to the cold-start and data-sparsity issues [7, 8], models learned
from a single domain are unable to achieve satisfactory performance.
Therefore, Cross-Domain Recommendation (CDR) [9, 10, 11, 12, 13] has been
proposed. It transfers the information from source domains to a target domain,
such that the recommendation performance on the target domain can be improved.

The CDR has been verified as an effective way to alleviate the aforementioned
issues [10, 11, 12, 13]. The essential idea in CDR is to use the common users
on both domains to transfer relevant information, which can be achieved from
two perspectives: 1) modeling user information in source domain as auxiliary
information in target domain [10, 12]; or 2) jointly training shareable
parameters on both domains [11, 14]. Meanwhile, some recent works [13, 15] try
to combine both. However, existing works assume that the information from a
source domain is relevant to the prediction objectives in the target domain,
which is not necessarily true. If a source domain contains dominant bias
against the target domain, the prediction on the target domain would be
misled, which thus spoils the recommendation performance.

To this end, we should devise a new CDR paradigm, which can not only transfer
the information but also protect the prediction on the target domain from
being dominated by the bias from source domains. Inspired by recent
developments of pre-training frameworks in computer vision [16, 17], natural
language processing [18], graph representation learning [19], and sequential
recommendation [20], we intent to design a novel pre-training framework for
the CDR problem. The advantages are twofold: Firstly, the pre-trained model on
source domains transfer auxiliary information to the target domain. Secondly,
the fine-tuning step on target domain ensures the prediction is dominated by
the information in the target domain, thus overcoming the bias from source
domains.

Though existing works have discussed how to pre-train a model on source
domains and fine-tune it on the target domain [21], it is still challenging to
propose a suitable framework for the CDR problem. Firstly, previous works [19,
22, 18] pre-train a model on a large corpus with side information available,
which is not the case for CDR. The only available data in CDR is the user-item
interactions, which is very sparse. Additionally, it is unclear how to exert
the pre-trained model while avoiding its bias impeding the prediction on the
target domain.

Therefore, we propose to Pre-train a graph neural network for Cross-domain
Recommendation (PCRec) by transferring the graph structural information from
source domain graphs to the target domain. During the pre-training stage, we
adopt the Self-Supervised Learning (SSL) [23, 24, 25] scheme to train a graph
encoder. SSL reduces the prediction bias from source domains as it pre-trains
the model without prediction objectives. To be more specific, we sample and
embed two sub-graphs of a node as positive pairs and employ contrastive
learning to maximize the agreements between them.

Then, we transfer the pre-trained graph encoder to the target domain. However,
since the encoder is also dominated by the structural bias in the source
domain, we thus exert the pre-trained encoder to initialize node embeddings
for another single domain model. According to our empirical study, we
recognize that adopting a simple Matrix Factorization (MF) model on the target
domain during the fine-tuning stage [26] significantly outperforms other
complex models, such as LightGCN (LGCN) [27]. We hypothesis that a simple MF
retains a balance between source domain and target domain information, while
other complex models over-emphasize the data on the target domain and thus
diminish the source domain information. The contributions are as follows:

  * •

To the best of our knowledge, we are the first work investigating pre-training
GNN to tackle the CDR problem.

  * •

We design a novel framework to handle the CDR problem, which pre-trains a
graph encoder on source domain to initialize the user/item embeddings in the
target domain.

  * •

Extensive experiments are conducted to complete CDR and analyze our proposed
framework.

##  II Related Work

###  II-A Cross-domain recommendation

The cross-domain recommendation system mainly alleviates two major CF-based
model bottlenecks: user/item data sparsity and the cold-start problem. The
trained model may be over-fitting when user/item data is too sparsity to cover
the universal data distribution. The cold start problem often exists when new
users have no historical shopping records and the lack of features of new
products. Both issues lead to a decline in the effectiveness of the
recommendation system.

Conventional CDR has two approaches [28] to address the above problems: (1)
content-based transfer, and (2) embedding-based transfer. Content-based
transfer mainly links different domains by identifying similar content
information while no common users/items in this situation [29, 30]. In
contrast, Embedding-based transfer focuses on user/item relevance, such as
multiple domains have common users or common items. This approach first trains
the CF-based model (bayesian personalized ranking [26], neural collaborative
filtering [5], etc.)to extract user/item embeddings and then transfer these
embeddings through common or similar users/items across domains. In this work,
we focus on embedding-based transfer. Unlike learning common users/items
attributes, we adopt self-supervised contrastive learning and graph neural
networks to learn users’ structural embedding from the source domain and then
transfer it to the targer domain.

###  II-B Contrastive learning

GNNs models couple with contrastive learning to learn graph or node level
representations without relying on supervisory data [31]. Then the trained
model can transfer the learned representations to a priori unknown downstream
tasks. In general, the contrastive learning method needs to create multiple
views for each instance in the dataset through various data augmentations
[32]. Two views are positive pairs, one is the original instance, and another
is the augmented instance. We also need two negative views generated from the
different instances. The ultimate goal of contrastive learning is to shorten
the distance of the positive pair while pushing the negative sample away.
Mutual information (MI) is often the measurement in contrastive learning.

##  III Proposed Model

###  III-A Preliminary

Following existing works [33, 19], we model the user-item interactions as a
bipartite graph. We denote the graph as
$\mathcal{G}=(\mathcal{U},\mathcal{I},\mathcal{E})$, where $\mathcal{U}$,
$\mathcal{I}$ and $\mathcal{E}$ denote the set of users, items, and edges,
respectively. Regarding the CDR problem, we denote a source domain graph as
$\mathcal{G}^{(s)}$ and a target domain graph as $\mathcal{G}^{(t)}$. For each
node, we extract its context information from the $r$-ego network which is
defined as:

######  Definition 1 (An $r$-ego network.)

The $r$-hop neighbors for a node $u$ are defined as
$\mathcal{S}_{u}=\\{i:d(u,i)\leq r\\}$ where $d(u,i)$ is the shortest path
distance between $u$ and $i$ in the graph $\mathcal{G}$. The $r$-ego network
of vertex $u$, denoted as $\mathcal{G}_{u}$, is a sub-graph composed by
$\mathcal{S}_{u}$ and the corresponding edges between $\mathcal{S}_{u}$.

Next, we present how to pre-train a graph encoder
$f:\mathcal{G}\rightarrow\mathbb{R}^{d}$ upon the source domain graph
$\mathcal{G}^{(s)}$ by adopting the self-supervised learning scheme [19, 24].

###  III-B Pre-training on Source Domain

We adopt the SSL scheme during the pre-training phase, which employs
contrastive learning to optimize the graph encoder. Specifically, the SSL has
three components: 1) the data augmentation, which constructs positive and
negative sub-graph pairs of a node, 2) the graph encoder to embed the sub-
graphs, and 3) the contrastive loss to optimize the encoder.

####  III-B1 Data Augmentation

Contrastive learning requires the construction of positive pairs and negative
samples of a node. As the only available data in the source domain graph is
the interactions, we construct positive pairs as two sub-graphs of one node.
The sub-graphs should share similar structure information to warrant two sub-
graphs to be positive pairs. Therefore, we sample them from the $r$-ego
network of a node.

We first conduct two random walks on node $u$’s $r$-ego network
$\mathcal{G}_{u}$ (the superscript is ignored for simplicity), to generate two
sub-graphs $g^{q}$ and $g^{k}$, which are regarded as a positive pair. After
constructing positive sub-graph pairs for nodes, we treat those sub-graphs
generated from different $r$-ego networks as negative samples. We demonstrate
the sub-graph construction process in Figure 1(a), where $g_{u}^{q}$ and
$g_{u}^{k}$ are a positive pair since they are sampled from the same node. We
use $g^{k^{-}}_{1}$ and $g^{k^{-}}_{2}$ to denote the negative samples, which
are sub-graphs sampled from the $r$-ego network of another node.

####  III-B2 GNN Encoder

After retrieving those sub-graphs, we feed them into two graph encoders
$f^{q}$ and $f^{k}$, which is illustrated in Figure 1(a). We encode the sub-
graph $g^{q}$ with graph encoder $f^{q}$, while encoding other sub-graphs with
$f^{k}$. Correspondingly, we generate low-dimensional representative vectors
$\mathbf{e}^{q}$ and $\mathbf{e}^{k}$ for the positive pair $g^{q}$ and
$g^{k}$, respectively. In this work, we choose the Graph Isomorphism Network
(GIN) [34] to be the graph encoder because GIN exhibits powerful ability in
distinguishing a broad class of graphs [35]. In general, other GNN models can
also be used as the encoder. We leave this study as future work.

####  III-B3 Contrastive Loss Function

We adopt the contrastive loss InfoNCE [36] to self-supervisedly optimize the
graph encoder, which maximizes the agreements between positive pairs. The
InfoNCE loss is formulated as follows:

$\mathcal{L}_{\text{InfoNCE}}=-\log\frac{\text{exp}(\mathbf{e}_{q}^{\intercal}\mathbf{e}_{k}/\boldsymbol{\tau})}{\sum_{i=1}^{n}\text{exp}(\mathbf{e}_{q}^{\intercal}\mathbf{e}_{i}/\boldsymbol{\tau})},$
(1)

where $\boldsymbol{\tau}$ is the temperature hyper-parameter. Minimizing this
objective is equivalent to maximizing the similarity between positive pairs,
i.e. $\mathbf{e}_{q}$ and $\mathbf{e}_{k}$, while minimizing the similarity
between negative pairs, i.e. $\mathbf{e}_{q}$ and $\mathbf{e}_{i}$ where
$i\neq k$. In practice, we view those instances as a query embedding
$\mathbf{e}_{q}$ and a set of key embeddings
$\\{\mathbf{e}_{i}\\}|_{i=1}^{n}$. The contrastive loss looks up a single key
(denoted by $\mathbf{e}_{k}$) that $\mathbf{e}_{q}$ matches in the key set.

In contrastive learning, maintaining the K-size look-up key set is essential.
Intuitively, as the denominator in Eq. (1) expresses, larger key set size
leads to better sampling of the underlying data space. Due to the
computational constraint, we adopt the MoCo [37] training scheme, which
maintains a dynamic set of keys with a queue and a moving-averaged encoder.
MoCo is able to increase the key set size without additional backpropagation
costs. Formally, if denoting the parameters of $f_{k}$ as $\theta_{k}$ and
those of $f_{q}$ as $\theta_{q}$, MoCo updates $\theta_{k}$ as
$\theta_{k}\leftarrow m\theta_{k}+(1-m)\theta_{q}$, where $m\in[0,1)$ is a
momentum hyper-parameter.

###  III-C Fine-tuning on Target Domain

After obtaining the pre-trained GIN model from the source domain, we should
transfer the model to the target domain. However, due to the pre-trained
encoder is dominated by the structural bias of the source domain graph,
directly fine-tuning the encoder in the target domain cannot avoid the
interference of the bias from the source domain. Instead, we employ the pre-
trained GIN model to initialize the node embeddings in the target domain.
Then, we fine-tune a simple Matrix Factorization (MF) [26] model with
initialized embeddings to infer the final embeddings of nodes. We illustrate
this process as in Figure 1(b). The fine-tuning on the target domain is
optimized with the Bayesian Personalized Ranking (BPR) loss function, which is
formulated as follows [26]:

$\mathcal{L}_{\text{BPR}}=-\sum_{u=1}^{M}\sum_{i\in\mathcal{N}_{u}}\sum_{j\notin\mathcal{N}_{u}}\log\sigma(\hat{y}_{ui}-\hat{y}_{uj})+\lambda\|\Theta\|^{2}$
(2)

where $\mathcal{N}_{u}$ denotes the set of items which are the neighbors of
node $u$, $\hat{y}_{ui}$, and $\hat{y}_{uj}$ denote the rating of user $u$ on
item $i$, and the rating of user $u$ on item $j$ individually.

Alternatively, the MF-based recommeder system can be substituted by any other
single domain recommendation models, e.g., LightGCN. However, the empirical
results indicate that a simple MF-based model outperforms a complex signal
domain recommender system.

###  III-D Recommendation

We will use the embedding optimized during the fine-tuning stage to make a
further recommendation. We compute the score between a user and an item as:
$\hat{r}_{ui}=\boldsymbol{e}_{u}^{\intercal}\boldsymbol{e}_{i}$, where
$\mathbf{e}_{u}\in\mathbb{R}^{d}$ and $\mathbf{e}_{i}\in\mathbb{R}^{d}$ are
the user and item embeddings, respectively. The score $\hat{r}_{ui}$ is used
to rank those items for users.

TABLE I: Statistics of the datasets. Data |  `#` User |  `#` Item | Sparsity | Domain
---|---|---|---|---
Amazon-GGF | 31,230 | 40,648 | 99.96`%` | Source
Amazon-PP | 14,180 | 4,970 | 99.80`%` | Target

##  IV Experiments

TABLE II: The overall comparison. The performance is measured in the target dataset Amazon-PP.  Model | Recall@20 | Recall@40 | MAP@20 | MAP@40
---|---|---|---|---
LGCN | 0.1935 | 0.3107 | 0.0203 | 0.0248
CMF | 0.0712 | 0.1329 | 0.0080 | 0.0109
CoNet | 0.1153 | 0.2314 | 0.0160 | 0.0161
JSCN | 0.1329 | 0.2520 | 0.0263 | 0.0307
PCRec | 0.2756 | 0.4445 | 0.0264 | 0.0329
Impro. | 42.43% | 43.06% | 0.38% | 7.17%

We conduct experiments to respond the following research questions (RQs):

  * •

RQ1: Can PCRec outperform existing single domain and cross-domain
recommendation methods?

  * •

RQ2: In the case of cross-domain, how far away neighbor information
aggregation is helpful to represent the node embedding?

  * •

RQ3: In CDR problem, how can we effectively transfer the information from
source domain to target domain?

###  IV-A Data

We conduct experiments on two datasets from the Amazon Review Data
(2018)111Available at https://nijianmo.github.io/amazon/index.html.: Grocery
and Gourmet Food (Amazon-GGF) and Prime Pantry (Amazon-PP). Their statistics
are shown in Table I. For cross-domain models, we set Amazon-GGF as the source
domain and Amazon-PP as the target domain, while for single domain model, we
solely use the target domain data. Due to distinct sparsity on both datasets,
we adopt the 5-core222Each of the remaining users has at least k ratings.
setting for Amazon-PP, and 10-core setting for Amazon-GGF. The number of
common users between Amazon-PP and Amazon-GGF is 4,275.

###  IV-B Experimental Settings

####  IV-B1 Baselines

We compare our proposed PCRec with three cross-domain and one single-domain
methods. CMF [38] is a matrix factorization-based cross-domain rating
prediction model. CoNet [13] and JSCN [33] both are joint learning model with
differernt ways to transfer one domian knowledge to the other. One single
domain method is LightGCN, which devises a light graph convolution for
training efficiency and generation ability.

####  IV-B2 Evaluation Protocol

We randomly split 80% and 20% of the interactions in the target domain as
training and testing set, respectively. We randomly choose 10% of the training
data for validation during training. We use Recall@${K}$ and MAP@${K}$ to
evaluate the top-${K}$ recommendation performance where ${K}=[20,40]$.

####  IV-B3 Hyper-parameter Settings

In the pre-training, we apply Adam optimizer with a learning rate of 0.005,
$\tau$ is 0.07, and MoCo (K=512) momentum $m$ is 0.999. We change the learning
rate to 0.001 in adam optimizer, and the early stopping strategy is the same
as LightGCN. Furthermore, PCRec method inherits the optimal values of other
shared hyper-parameters.

###  IV-C Overall Comparison (RQ1)

We compare PCRec with various baselines regarding the performance on the
target domain Amazon-PP. The overall comparison is reported in Table II and we
have the following findings:

  * •

PCRec can significantly outperforms other methods on recall, e.g., achieving
42.43% on Recall@20. against the second-best one. This is because PCRec can
effectively transfer the information from source domains to target domain
while protecting the recommendation on the target domain from being dominated
by the source domain. The performance variant between MAP and Recall is
because the number of positive samples in the data is small.

  * •

Among those baselines, LGCN performs the best with respect to Recall@{20,40},
even better than CDR methods, which indicates the effectiveness of using GNN
to learn node embeddings. Nevertheless, it is still much worse than PCRec
since it is cannot transfer the information from the source domain.

  * •

JSCN outperforms other baseline CDR methods, which shows the benefits of using
spectral graph convolution to encode user-item interaction. However, it
performs worse than PCRec, because it jointly learns the source and target
domain embeddings, which leads to the interference of the noise in the source
domain.

TABLE III: The influence of distance on neighbors aggregation.  Variants | Recall@20 | Recall@40 | MAP@20 | MAP@40
---|---|---|---|---
PCRec-2hop | 0.0329 | 0.0663 | 0.0044 | 0.0064
PCRec-3hop | 0.0297 | 0.0613 | 0.0037 | 0.0039
PCRec-L3 | 0.1938 | 0.3267 | 0.0200 | 0.0259
PCRec-L1 | 0.1996 | 0.3264 | 0.0201 | 0.0258
PCRec | 0.2756 | 0.4445 | 0.0264 | 0.0329

###  IV-D Variants Analysis (RQ2)

In this section, we aim to analyse the specific designs of our proposed PCRec
framework regarding the neighbors aggregation.

At the pre-training stage, how to generate the subgraphs is critical.
Regarding this point, two designs can be modified:

  * •

The first one is the choice of $r$ concerning the $r$-hop neighbors. We change
the $r$ from 2 to 3, and results are generated by applying the pre-trained GIN
without fine-tuning to reflect the intrinsic effect. Intuitively, PCRec should
perform better at 3-hop rather than 2-hop due to more neighbor information
being included. However, as presented in Table III, PCRec-2hop outperforms
PCRec-3hop. We argue that this is because, with the distance increasing, the
similarity between positive samples will decrease, thus hindering node
representation learning.

  * •

The second design that can be changed is the data augmentation method.
Usually, there are four ways to augment graph data, node dropping, edge
dropping, random walk, and attribute masking [39]. The first three mechanisms
do not require side information. Thus they can be adopted for the CDR problem.
In PCRec, we generate the subgraphs through random walk, which can be
supported by the graph structure assumption [40]. The other two methods can be
explored in the future.

For the fine-tuning stage, we study how the complexity of fine-tuning model
will impact the performance. We change PCRec’s MF to 1-layer LGCN (PCRec-L1)
and 3-layer LGCN (PCRec-L3). There is no significant difference between
PCRec-L1 and PCRec-L3, and they are both much worse than PCRec. We hypothesis
that a complex model will over-emphasize the target domain, thus hindering
knowledge transfer. Therefore, a simple model with elaborate initialization
may be able to retain the balance.

TABLE IV: Exploration of various ways to transfer the information from source domain to target domain of CDR. | Recall@20 | Recall@40 | MAP@20 | MAP@40
---|---|---|---|---
LGCN | 0.1935 | 0.3107 | 0.0203 | 0.0248
Pre-Only | 0.0329 | 0.0663 | 0.0044 | 0.0064
CU-PE | 0.1911 | 0.3198 | 0.0199 | 0.0258
CU-PM | 0.2277 | 0.3880 | 0.0227 | 0.0293
PCRec | 0.2756 | 0.4445 | 0.0264 | 0.0329

###  IV-E Transfer in CDR (RQ3)

To explore the process of implementing knowledge transfer in CDR, we present
how do we finally get to the PCRec framework by altering three key components
step by step:

  * •

We first examine the necessity of fine-tuning in CDR. We transfer the source
knowledge by the pre-trained GIN model without fine-tuning, which is named
Pre-Only. As shown in Table IV, the performance of Pre-Only is the worst,
which implies that naively adopting the pre-trained model would bring severe
bias to the target domain. Therefore, we add a fine-tuning model to PCRec.

  * •

Secondly, we consider two plans to transfer the knowledge to common users: 1)
straightly using the pre-trained embeddings as the initialization of fine-
tuning (named CU-PE), 2) applying the pre-trained model to generate the
initialization of fine-tuning (named CU-PM). As presented in Table IV, CU-PM
not only performs better than CU-PE but also surpasses the LGCN, while CU-PE
falls behind it. In the CDR problem, the result indicates that transferring
the inherited structure information from the model has better performance than
copying the common users’ embedding from the source domain to the target
domain directly.

  * •

Finally, we explore the role of nodes other than common users, which leads to
our final PCRec. In CDR problem, other nodes should be carefully processed.
But as they can also receive the information transferred by the pre-trained
encoder, we suggest that they should be included. The experiment result
demonstrates that the target performance would not be dominated by the source
domain in this way of transferring.

##  V CONCLUSION

In this work, we investigate the possibility of pre-training a GNN to transfer
structural representations in the source domain to the target domain to
address the cold-start problem in the CDR task. We propose a novel framework,
PCRec, which pre-trains a graph encoder to learn node embeddings from the
source domain and apply it to the target domain to initiate embedding. A
simple MF method during the fine-tuning stage can significant outperform other
complicated methods on all metrics.

## Acknowledgments

This work is supported in part by NSF under grants III-1763325, III-1909323,
III-2106758, and SaTC-1930941.

## References

  * [1] Y. Koren, R. M. Bell, and C. Volinsky, “Matrix factorization techniques for recommender systems,” _Computer_ , vol. 42, no. 8, pp. 30–37, 2009.
  * [2] R. Salakhutdinov and A. Mnih, “Probabilistic matrix factorization,” in _Advances in Neural Information Processing Systems 20_ , J. C. Platt, D. Koller, Y. Singer, and S. T. Roweis, Eds. Curran Associates, Inc., 2007, pp. 1257–1264.
  * [3] H. Wang, N. Wang, and D. Yeung, “Collaborative deep learning for recommender systems,” in _Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Sydney, NSW, Australia, August 10-13, 2015_ , L. Cao, C. Zhang, T. Joachims, G. I. Webb, D. D. Margineantu, and G. Williams, Eds. ACM, 2015, pp. 1235–1244.
  * [4] Z. Liu, M. Wan, S. Guo, K. Achan, and P. S. Yu, “Basconv: Aggregating heterogeneous interactions for basket recommendation with graph convolutional neural network,” in _Proceedings of the 2020 SIAM International Conference on Data Mining_. SIAM, 2020, pp. 64–72.
  * [5] X. He, L. Liao, H. Zhang, L. Nie, X. Hu, and T. Chua, “Neural collaborative filtering,” in _Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017_ , R. Barrett, R. Cummings, E. Agichtein, and E. Gabrilovich, Eds. ACM, pp. 173–182.
  * [6] Z. Liu, X. Li, Z. Fan, S. Guo, K. Achan, and P. S. Yu, “Basket recommendation with multi-intent translation graph neural network,” in _2020 IEEE International Conference on Big Data (Big Data)_ , 2020, pp. 728–737.
  * [7] G. Adomavicius and A. Tuzhilin, “Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions,” _IEEE Trans. Knowl. Data Eng._ , vol. 17, no. 6, pp. 734–749, 2005.
  * [8] A. I. Schein, A. Popescul, L. H. Ungar, and D. M. Pennock, “Methods and metrics for cold-start recommendations,” in _SIGIR 2002: Proceedings of the 25th Annual International ACM SIGIR_ , K. Järvelin, M. Beaulieu, R. A. Baeza-Yates, and S. Myaeng, Eds.
  * [9] M. M. Khan, R. Ibrahim, and I. Ghani, “Cross domain recommender systems: A systematic literature review,” _ACM Comput. Surv._ , vol. 50, no. 3, pp. 36:1–36:34, 2017.
  * [10] T. Man, H. Shen, X. Jin, and X. Cheng, “Cross-domain recommendation: An embedding and mapping approach,” in _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017_ , C. Sierra, Ed. ijcai.org, 2017, pp. 2464–2470.
  * [11] A. Farseev, I. Samborskii, A. Filchenkov, and T. Chua, “Cross-domain recommendation via clustering on multi-layer graphs,” in _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017_ , N. Kando, T. Sakai, H. Joho, H. Li, A. P. de Vries, and R. W. White, Eds. ACM, 2017, pp. 195–204.
  * [12] L. Hu, J. Cao, G. Xu, L. Cao, Z. Gu, and C. Zhu, “Personalized recommendation via cross-domain triadic factorization,” in _22nd International World Wide Web Conference, WWW ’13, Rio de Janeiro, Brazil, May 13-17, 2013_ , D. Schwabe, V. A. F. Almeida, H. Glaser, R. Baeza-Yates, and S. B. Moon, Eds. International World Wide Web Conferences Steering Committee / ACM, 2013, pp. 595–606.
  * [13] G. Hu, Y. Zhang, and Q. Yang, “Conet: Collaborative cross networks for cross-domain recommendation,” in _Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM 2018, Torino, Italy, October 22-26, 2018_ , A. Cuzzocrea, J. Allan, N. W. Paton, D. Srivastava, R. Agrawal, A. Z. Broder, M. J. Zaki, K. S. Candan, A. Labrinidis, A. Schuster, and H. Wang, Eds. ACM, 2018, pp. 667–676.
  * [14] X. Wang, Z. Peng, S. Wang, P. S. Yu, W. Fu, and X. Hong, “Cross-domain recommendation for cold-start users via neighborhood based feature mapping,” in _Database Systems for Advanced Applications - 23rd International Conference, DASFAA 2018, Gold Coast, QLD, Australia, May 21-24, 2018, Proceedings, Part I_ , ser. Lecture Notes in Computer Science, J. Pei, Y. Manolopoulos, S. W. Sadiq, and J. Li, Eds., vol. 10827. Springer, 2018, pp. 158–165.
  * [15] P. Li and A. Tuzhilin, “DDTCDR: deep dual transfer cross domain recommendation,” in _WSDM_. ACM, 2020, pp. 331–339.
  * [16] Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, “Unsupervised feature learning via non-parametric instance discrimination,” in _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_. IEEE Computer Society, 2018, pp. 3733–3742.
  * [17] K. He, H. Fan, Y. Wu, S. Xie, and R. B. Girshick, “Momentum contrast for unsupervised visual representation learning,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_. IEEE, 2020, pp. 9726–9735.
  * [18] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [19] J. Qiu, Q. Chen, Y. Dong, J. Zhang, H. Yang, M. Ding, K. Wang, and J. Tang, “GCC: graph contrastive coding for graph neural network pre-training,” in _KDD ’20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020_ , R. Gupta, Y. Liu, J. Tang, and B. A. Prakash, Eds. ACM, 2020, pp. 1150–1160.
  * [20] Z. Liu, Y. Chen, J. Li, P. S. Yu, J. McAuley, and C. Xiong, “Contrastive self-supervised sequential recommendation with robust augmentation,” _arXiv preprint arXiv:2108.06479_ , 2021.
  * [21] W. Hu*, B. Liu*, J. Gomes, M. Zitnik, P. Liang, V. Pande, and J. Leskovec, “Strategies for pre-training graph neural networks,” in _International Conference on Learning Representations_ , 2020.
  * [22] W. Hu, B. Liu, J. Gomes, M. Zitnik, P. Liang, V. S. Pande, and J. Leskovec, “Pre-training graph neural networks,” _CoRR_ , vol. abs/1905.12265, 2019\.
  * [23] X. Li, Z. Liu, S. Guo, Z. Liu, H. Peng, P. S. Yu, and K. Achan, “Pre-training recommender systems via reinforced attentive multi-relational graph neural network,” in _IEEE BigData_. IEEE, 2021.
  * [24] J. Wu, X. Wang, F. Feng, X. He, L. Chen, J. Lian, and X. Xie, “Self-supervised graph learning for recommendation,” _arXiv preprint arXiv:2010.10783_ , 2020\.
  * [25] Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen, “Graph contrastive learning with augmentations,” _Advances in Neural Information Processing Systems_ , vol. 33, 2020.
  * [26] S. Rendle, C. Freudenthaler, Z. Gantner, and L. Schmidt-Thieme, “BPR: bayesian personalized ranking from implicit feedback,” in _UAI 2009, Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, QC, Canada, June 18-21, 2009_ , J. A. Bilmes and A. Y. Ng, Eds. AUAI Press, 2009, pp. 452–461.
  * [27] X. He, K. Deng, X. Wang, Y. Li, Y. Zhang, and M. Wang, “Lightgcn: Simplifying and powering graph convolution network for recommendation,” in _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020_ , J. Huang, Y. Chang, X. Cheng, J. Kamps, V. Murdock, J. Wen, and Y. Liu, Eds. ACM, 2020, pp. 639–648.
  * [28] F. Zhu, Y. Wang, C. Chen, J. Zhou, L. Li, and G. Liu, “Cross-domain recommendation: Challenges, progress, and prospects,” in _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021_ , Z. Zhou, Ed. ijcai.org, 2021, pp. 4721–4728.
  * [29] P. Winoto and T. Y. Tang, “If you like the devil wears prada the book, will you also enjoy the devil wears prada the movie? A study of cross-domain recommendations,” _New Gener. Comput._ , vol. 26, no. 3, pp. 209–225, 2008\.
  * [30] S. Tan, J. Bu, X. Qin, C. Chen, and D. Cai, “Cross domain recommendation based on multi-type media fusion,” _Neurocomputing_ , vol. 127, pp. 124–134, 2014\.
  * [31] L. Wu, H. Lin, Z. Gao, C. Tan, and S. Z. Li, “Self-supervised on graphs: Contrastive, generative, or predictive,” _CoRR_ , vol. abs/2105.07342, 2021\.
  * [32] K. Hassani and A. H. K. Ahmadi, “Contrastive multi-view representation learning on graphs,” in _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , ser. Proceedings of Machine Learning Research, vol. 119. PMLR, 2020, pp. 4116–4126.
  * [33] Z. Liu, L. Zheng, J. Zhang, J. Han, and P. S. Yu, “JSCN: joint spectral convolutional network for cross domain recommendation,” in _IEEE BigData_. IEEE, 2019, pp. 850–859.
  * [34] K. Xu, W. Hu, J. Leskovec, and S. Jegelka, “How powerful are graph neural networks?” in _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net, 2019.
  * [35] L. Babai and L. Kucera, “Canonical labelling of graphs in linear average time,” in _20th Annual Symposium on Foundations of Computer Science, San Juan, Puerto Rico, 29-31 October 1979_. IEEE Computer Society, 1979, pp. 39–46.
  * [36] A. van den Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” _CoRR_ , vol. abs/1807.03748, 2018.
  * [37] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 9729–9738.
  * [38] A. P. Singh and G. J. Gordon, “Relational learning via collective matrix factorization,” in _Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Las Vegas, Nevada, USA, August 24-27, 2008_ , Y. Li, B. Liu, and S. Sarawagi, Eds. ACM, 2008, pp. 650–658.
  * [39] Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen, “Graph contrastive learning with augmentations,” in _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., 2020.
  * [40] J. Leskovec, J. M. Kleinberg, and C. Faloutsos, “Graphs over time: densification laws, shrinking diameters and possible explanations,” in _Proceedings of the Eleventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Chicago, Illinois, USA, August 21-24, 2005_ , R. Grossman, R. J. Bayardo, and K. P. Bennett, Eds. ACM, 2005, pp. 177–187.


# Generative Data Augmentation using LLMs improves Distributional Robustness
in Question Answering

Arijit Ghosh Chowdhury
University of Illinois Urbana Champaign
<EMAIL_ADDRESS>
Aman Chadha
Stanford University, Amazon Alexa AI
<EMAIL_ADDRESS>
Work does not relate to position at Amazon.

###### Abstract

Robustness in Natural Language Processing continues to be a pertinent issue,
where state of the art models under-perform under naturally shifted
distributions. In the context of Question Answering, work on domain adaptation
methods continues to be a growing body of research. However, very little
attention has been given to the notion of domain generalization under natural
distribution shifts, where the target domain is unknown. With drastic
improvements in the quality and access to generative models, we answer the
question: How do generated datasets influence the performance of QA models
under natural distribution shifts? We perform experiments on 4 different
datasets under varying amounts of distribution shift, and analyze how "in-the-
wild" generation can help achieve domain generalization. We take a two-step
generation approach, generating both contexts and QA pairs to augment existing
datasets. Through our experiments, we demonstrate how augmenting reading
comprehension datasets with generated data leads to better robustness towards
natural distribution shifts.

##  1 Introduction

Natural language processing has seen substantial progress over the last few
years owing to the emerging abilities of pre-trained language models. In many
benchmarks, large pre-trained models adapted to a target dataset reach or even
surpass human performance. However, very often these models fail to generalize
to changing test distributions.

Through this work, we perform a systematic study of how "in-the-wild"
generation can affect the distributional robustness of question-answering
models trained on the popular Stanford Question Answering Dataset (SQUAD)
Rajpurkar et al. (2016). Synthetic data generation is a widely adopted method
for domain adaptation in QA systems Shakeri et al. (2020) Yue et al. (2021)
Yue et al. (2022). However, domain adaptation methods have access to
unlabelled/labelled data belonging to the target domain, and do not account
for unseen natural distribution shifts. Our work studies the effect of
generated data on distribution shifts where the target domain is unseen.

The conception of a dataset has undergone significant evolution in recent
times. This transformation has been catalyzed by the advent of generative
models trained ’in-the-wild’, such as those described in Brown et al. (2020),
Bubeck et al. (2023), and Touvron et al. (2023). These models, which utilize
vast and diverse datasets across a range of domains, have facilitated the
infusion of the web with synthesized data of high calibre, applicable to an
extensive array of conceptual topics. Interestingly, these models are not
merely confined to generation based on a pre-established distribution; they
possess the capacity for repeated prompting, resulting in the creation of
markedly diverse data. In the context of this emerging model paradigm, our
research investigates the following query: How do generated datasets affect
the distributional robustness of Question Answering models? Specifically, we
focus on Natural Distribution Shifts, where the test distribution is naturally
occurring Wang et al. (2021), where robustness can be defined around model’s
performance under distribution shift.

We present an overview of our generation setup in Figure 1. For generating
data, utilize GPT-3.5 Brown et al. (2020), and create a question-answering
dataset using questions provided in the SQUAD Rajpurkar et al. (2016) dataset.
We use a dual generation approach, by first prompting the language model to
generate a context for a question given in the SQUAD dataset, and then
generating question-answer pairs for the newly generated context.

As the prevalence of generative AI continues to grow, it is anticipated that
vast quantities of synthesized data will be utilized directly for machine
learning model training. It is crucial to acknowledge, however, that there
currently exists a scarcity of standardized evaluation and critique methods
for such datasets. The generation of data unveils an array of unique
opportunities and obstacles, thus warranting a more comprehensive evaluation
process. Therefore, as a key part of our research, we present SQUAD-G, a
question-answering dataset derived from generated data. This dataset will be
made accessible to the public, facilitating straightforward and replicable
benchmarking, alongside the assessment of the utility and critique of
generated datasets, thus ensuring reliable and trustworthy natural language
processing.

Our Contributions ➠ We propose a framework to improve the distributional
robustness of reading comprehension models in the presence of natural
distribution shifts. ➠ Through a thorough quantitative evaluation, we evaluate
the capabilites of LLMs to generate high quality synthetic data for question
answering tasks.  ➠ We release a new dataset to benchmark further work on
evaluating models trained on generated data.

##  2 Related Work

Figure 1: Overview of the generation system. Our method creates a generated
dataset which is then augmented with the real dataset to train a question
answering model.

###  2.1 Distribution Shift in NLP

Ramponi and Plank (2020) provide a comprehensive survey of domain adaptation
and generalization problems in NLP, dividing methods into data centric and
model centric. In our work, we adopt a data centric approach towards domain
generalization. Wang et al. (2022) note that there is often a significant
overlap in test-train data for QA models, and SOTA models often fail to
perform well on questions that cannot be memorized from training data. This
motivates our work to create diverse datasets for training and evaluation.
This work also highlights the scarcity of research done for generalization in
QA models, especially for natural distribution shifts. Arora et al. (2021)
outline the several different kinds of out-of-distribution data in NLP, and
highlight the need to better benchmark performance on these samples.

###  2.2 Robustness in Question Answering

Longpre et al. (2019) performed one of the first experiments towards domain
agnostic question answering using data augmentation techniques that utilize
pretrained models.Miller et al. (2020) introduce three new datasets for QA
models, constructed from New York Times articles, Reddit Posts, and Amazon
Product Reviews. This is the first comprehensive study analyzing the effect of
natural distribution shift on QA models. Our work builds on this by utilizing
these datasets to evaluate our data augmentation method. Furthermore,Miller et
al. (2020) conduct an extensive evaluation of the robustness of different
model and adaptation methods under 15 distribution shifts in question
answering. However, they do not explore new methods that incorporate better
generalization in QA models. Liu et al. (2021) highlight similar challenges in
open domain question answering, breaking down the root causes towards failure.
Our work is grounded in extractive question answering, and aims to be the
first study that studies the effects of creating datasets using large language
models for downstream QA tasks.

###  2.3 Generalization using Generated Data

Gowal et al. (2021) demonstrate how generated data conditioned on the training
set can lead to adversarial robust models, while Bartolo et al. (2021) create
synthetic data to improve adversarial robustness in QA models. Our work
confirms and extends this hypothesis for QA models specifically for natural
distribution shifts. More recently, context generation has been used as a data
augmentation for text classification Mekala et al. (2022) and information
retrieval Su et al. (2022). Our work "instructs" Wei et al. (2022) a GPT-3.5
model to generate a context for a given question. Along similar lines, Bansal
and Grover (2023) demonstrate how Stable Diffusion
111https://github.com/CompVis/stable-diffusion can be used to create diverse
datasets for image classification. We leverage the generation capabilities of
large language models to generate diverse datasets for downstream tasks, that
can be inferenced using smaller models.

##  3 Methodology

###  3.1 Context Generation

We first generate contexts by conditioning it on a question present in the
SQUAD dataset. This allows the language model to generate a paragraph that can
be used to generate question-answer pairs. Since the paragraph is generated
using an existing question, the generated context is consistent with the
informative trivia format of SQUAD-like datasets. To maintain further
consistency, the generated context is clipped to be within 250 words, based on
the average context length present in the SQUAD dataset. We prompt GPT 3.5
(gpt-3.5-turbo) 222https://platform.openai.com/docs/models in the following
manner: Generate a paragraph which answers the following question: (question)
. Here the question is sampled from the SQUAD dataset. Figure 1 demonstrates
the generation process. Additionally, the Appendix A contains examples from
the generation process.

###  3.2 Question Answer Generation

After the context is created, the generated paragraph is used to create
question-answer pairs. This is done by using a T5 based question generation
model Lopez et al. (2020) that is trained on the SQUAD dataset, which takes a
paragraph has an input and returns a question-answer pair. We use the open
source333https://github.com/patil-suraj/question-generation implementation for
this model. Additionally we also filter out QA pairs based on round-trip
consistency Alberti et al. (2019).

##  4 Experiments

Dataset | SQUAD | NewWiki | NYT | Amazon | Reddit
---|---|---|---|---|---
Metrics | F1 | EM | F1 | EM | F1 | EM | F1 | EM | F1 | EM
Real data | 90.4 | 83.0 | 89.4 | 79.2 | 86.4 | 76.1 | 79.9 | 66.4 | 80.1 | 67.1
Generated data | 79.5 | 64.6 | 80.1 | 65.3 | 76.5 | 63.2 | 72.4 | 59.5 | 72.7 | 60.2
Real + Generated data | 92.7 | 84.7 | 91.1 | 80.4 | 88.9 | 79.3 | 80.3 | 67.1 | 81.7 | 68.7

Table 1: Performance on natural distribution shift datasets.

Dataset | SQUAD | NewWiki | NYT | Amazon | Reddit
---|---|---|---|---|---
Metrics | F1 | EM | F1 | EM | F1 | EM | F1 | EM | F1 | EM
Real + 50% Generated data | 91.4 | 81.1 | 90.4 | 82.2 | 87.4 | 77.1 | 79.7 | 65.4 | 80.3 | 67.4
Real + 100% Generated data | 92.7 | 84.7 | 91.1 | 80.4 | 88.9 | 79.3 | 80.3 | 67.1 | 81.7 | 68.7
Real + 200% Generated data | 92.9 | 84.8 | 91.3 | 80.7 | 88.5 | 79.1 | 80.9 | 67.3 | 80.8 | 68.1

Table 2: Performance on varying amounts of generated data.

###  4.1 Setup

We train an extractive reading comprehension model using SQUAD V1.1, using the
RoBERTA-Base model across all our experiments. We use a learning rate of
$3e-5$, a batch size of $16$ and run our experiments for $3$ epochs each. We
use the implementation provided by HuggingFace, and run our models on a stand-
alone Nvidia A100 GPU provided by Google Colab. We do not use GPT-3.5 as a
baseline since the purpose of this study is to specifically measure the
performance by smaller models.

For all our experiments, we measure F1 and Exact Match scores to quantify
performance on Natural Distribution Shift (NDS) datasets.

###  4.2 Datasets

We use the following datasets created by Miller et al. (2020) to set up our
test bed:

The New Wikipedia dataset contains newer QA pairs from wikipedia articles used
by the SQUAD V1.1 dataset. Contains 7938 test samples from 48 contexts. The
New York Times dataset contains articles from New York times which are then
used to annotate QA pairs in the same format as SQUAD. It is ensured that the
passage length statistics stay the same. Contains 10,065 test samples from 46
articles. Reddit dataset contains articles from Reddit where the authors
concatenated each post’s title with its body. This dataset contains 9,803 test
samples from 1969 posts. The Amazon Product Reviews dataset contains user
generated product reviews from the "Home and Kitchen" category on Amazon. This
data contains 9,885 test samples from 1909 reviews.

##  5 Results

###  5.1 Does generated data help with distributional robustness?

We evaluate the F1 and Exact Match scores of models trained with different
datasets on natural distribution shifts (NDS) benchmarks. We note the average
EM and F1 numbers across three random seeds in Table 1. The models are trained
on an equal amount of real and generated data.

We find that the model, when trained on SQUAD, when subjected to natural
distribution shift datasets, the model’s performance significantly
deteriorates. A noteworthy observation was that exclusive training on the
generated data resulted in substandard performance on both the SQUAD and its
Natural Distribution Shift (NDS) datasets. The inferior absolute performance
could be potentially attributed to the distribution disparity between the
source and the generated training datasets. Interestingly, we observe that for
the model trained on the generated data, the performance gaps on the real
validation dataset and its NDS datasets are low, which might be attributed to
the benefits of training on diverse generated data.

Across all experiments, we note that the performance decrease is slightly less
pronounced across the New York Times test set, which resembles the Wikipedia
based training set in SQUAD most closely. Similarly, it is not surprising that
the performance drop is substantial on user generated noisy datasets like
Amazon Reviews and Reddit posts.

Finally, we expose our model to an evenly-distributed blend of real and
generated datasets, with the goal of investigating the impact of generative
augmentations. Our results reveal that the absolute performance of the model,
when trained with a combination of real and generated data, either parallels
or exceeds the performance of models trained exclusively on either real or
generated datasets, across all naturally distributed datasets. This
observation suggests that the incorporation of real data into the training
process is indeed essential for attaining superior absolute performance.

To summarize, while using solely generated data improves robustness at the
expense of absolute performance, a blend of real and artificially generated
data presents the ideal balance for robust and precise training.

###  5.2 How much generated data is needed?

Here, we investigate how different combinations of the generated dataset can
help the classifiers take advantage of the complementary strengths of the two
data sources (Table 2).

To do so, we assessed the average performance of models trained with three
different input mixing combinations created by using 50%, 100%, and 200% of
the generated dataset. We observed an increase in performance on shifted
datasets as the size of the generated data increases while keeping the amount
of real data fixed. However, when the proportion of the generated data
increases twofold while keeping the proportion of the real data fixed, we
observe that the performance gains are only marginal. Additionally, we note
that using only half of the generated data does not provide enough meaningful
signal in terms of diversity and does not lead to major performance
improvements compared to training on real data.

Overall, we found that the ideal split between real and generated data is a
50-50 split where the two datasets are able to compliment each other, in terms
of providing both diversity and in-domain samples at the same time.

##  6 Conclusion and Future Avenues

We created a framework that enhances the performance of reading comprehension
models by supplementing real datasets with a diverse dataset generated by
contemporary, real-world generative models. Our findings indicate that this
training method yields superior results on test datasets and those with
natural distribution shifts, due to the added robustness from training on the
generated data as opposed to traditional methods. We studied how varying
amounts of generated data impact these patterns. We found that data generated
by large, current language models prove more effective for developing sturdy
models in Natural Language Processing (NLP). Our work also suggests a positive
trend towards using high-quality synthetic data to train smaller, specialized
models. In the future, we want to explore a more extensive comparison against
question generation methods and how this paradigm fits into fine-tuning larger
language models.

##  7 Limitations

The development of robust question answering models using synthetic data
derived from modern large language models (LLMs) demonstrates a notable
progression in the field of generative AI for trustworthy machine learning
(ML). However, this study does not delve into other vital facets such as
fairness and privacy. The primary focus of our methodology is to generate data
from a large language model in a zero-shot fashion for topics that are
adequately encapsulated within its range. Nonetheless, it becomes essential to
fine-tune the base model for distributions that aren’t sufficiently
represented in its training data. In spite of these limitations, the central
contributions of this paper continue to hold substantial value and offer key
insights for advancing the positive influence in the realm of trustworthy ML.

## References

  * Alberti et al. (2019) Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019\.  Synthetic qa corpora generation with roundtrip consistency.
  * Arora et al. (2021) Udit Arora, William Huang, and He He. 2021.  Types of out-of-distribution texts and how to detect them.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10687–10701, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Bansal and Grover (2023) Hritik Bansal and Aditya Grover. 2023.  Leaving reality to imagination: Robust classification via generated datasets.
  * Bartolo et al. (2021) Max Bartolo, Tristan Thrush, Robin Jia, Sebastian Riedel, Pontus Stenetorp, and Douwe Kiela. 2021.  Improving question answering model robustness with synthetic adversarial data generation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 8830–8848, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023.  Sparks of artificial general intelligence: Early experiments with gpt-4.
  * Gowal et al. (2021) Sven Gowal, Sylvestre-Alvise Rebuffi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy Mann. 2021.  Improving robustness using generated data.
  * Liu et al. (2021) Linqing Liu, Patrick S. H. Lewis, Sebastian Riedel, and Pontus Stenetorp. 2021.  Challenges in generalization in open domain question answering.  _CoRR_ , abs/2109.01156.
  * Longpre et al. (2019) Shayne Longpre, Yi Lu, Zhucheng Tu, and Chris DuBois. 2019.  An exploration of data augmentation and sampling techniques for domain-agnostic question answering.  In _Proceedings of the 2nd Workshop on Machine Reading for Question Answering_ , pages 220–227, Hong Kong, China. Association for Computational Linguistics.
  * Lopez et al. (2020) Luis Enrico Lopez, Diane Kathryn Cruz, Jan Christian Blaise Cruz, and Charibeth Ko Cheng. 2020.  Transformer-based end-to-end question generation.  _ArXiv_ , abs/2005.01107.
  * Mekala et al. (2022) Dheeraj Mekala, Tu Vu, Timo Schick, and Jingbo Shang. 2022.  Leveraging QA datasets to improve generative data augmentation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9737–9750, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Miller et al. (2020) John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. 2020.  The effect of natural distribution shift on question answering models.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Ramponi and Plank (2020) Alan Ramponi and Barbara Plank. 2020.  Neural unsupervised domain adaptation in NLP—A survey.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6838–6855, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Shakeri et al. (2020) Siamak Shakeri, Cicero Nogueira dos Santos, Henry Zhu, Patrick Ng, Feng Nan, Zhiguo Wang, Ramesh Nallapati, and Bing Xiang. 2020.  End-to-end synthetic data generation for domain adaptation of question answering systems.  _arXiv preprint arXiv:2010.06028_.
  * Su et al. (2022) Dan Su, Mostofa Patwary, Shrimai Prabhumoye, Peng Xu, Ryan Prenger, Mohammad Shoeybi, Pascale Fung, Anima Anandkumar, and Bryan Catanzaro. 2022.  Context generation improves open domain question answering.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.
  * Wang et al. (2021) Xuezhi Wang, Haohan Wang, and Diyi Yang. 2021.  Measure and improve robustness in nlp models: A survey.  _arXiv preprint arXiv:2112.08313_.
  * Wang et al. (2022) Xuezhi Wang, Haohan Wang, and Diyi Yang. 2022.  Measure and improve robustness in NLP models: A survey.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4569–4586, Seattle, United States. Association for Computational Linguistics.
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022.  Finetuned language models are zero-shot learners.
  * Yue et al. (2021) Zhenrui Yue, Bernhard Kratzwald, and Stefan Feuerriegel. 2021.  Contrastive domain adaptation for question answering using limited text corpora.  _arXiv preprint arXiv:2108.13854_.
  * Yue et al. (2022) Zhenrui Yue, Huimin Zeng, Ziyi Kou, Lanyu Shang, and Dong Wang. 2022.  Domain adaptation for question answering via question classification.  _arXiv preprint arXiv:2209.04998_.

## Frequently Asked Questions (FAQs)

  * ✽

How are we sampling questions to generate paragraphs?

➠

One question is sampled per context in the original SQUAD dataset to condition
the paragraph generation.

  * ✽

Why don’t we generate new contexts from scratch?

➠

This is done to create topically consistent datasets, and run a controlled
experiment where only determine whether LLM generated contexts provide
linguistic and stylistic diversity.

  * ✽

Why don’t we prompt GPT-3/3.5/4 to generate QA pairs as well?

➠

In our attempts to do so, we realised that GPT prompts do not generate outputs
that stay faithful to the extractive nature of the SQUAD dataset. Furthermore,
we wanted to limit our API usage.

##  Appendix A Appendix: Example Generations

This section provides supplementary material in the form of additional
examples to bolster the reader’s understanding of the concepts presented in
this work.

Original Context |  The American bison is a large bovid which inhabited much of western North America prior to the 1800s, living on the prairies in large herds. However, the vast herds of bison attracted market hunters, who killed dozens of bison for their hides only, leaving the rest to rot. Thousands of these hunters quickly eliminated the bison herds, bringing the population from several million in the early 1800s to a few hundred by the 1880s. Conservation efforts have allowed the population to increase, but the bison remains near-threatened.
---|---
Sampled Question |  "What type of creature is the American bison?"
Prompt |  Generate a paragraph that answers the following question: "What type of creature is the American bison?"
Generated Context |  The American Bison, often colloquially referred to as buffalo, is a North American species of bison that once roamed across the vast expanses of the continent in large herds. These colossal creatures are mammals, belonging to the family Bovidae, which includes other large, hoofed mammals such as domestic cattle and goats. Adult bison are imposing creatures; they are known to stand up to 6.5 feet tall at the shoulder and can weigh over a ton, making them the heaviest land animals in North America.They have a robust body structure with a large, muscular hump over the shoulders and a shaggy, dark brown winter coat which gets lighter in the summer. Their large heads sport a pair of short, sharp horns, used both for defense and in fights for status within the herd. Despite their bulk, bison are surprisingly agile and fast, capable of reaching speeds up to 35 mph.The American Bison plays a crucial role in maintaining the health and biodiversity of the prairie ecosystem.
Generated Questions |  Q: To which species does the term "buffalo" colloquially refer in North America? A: American Bison, Q: How does the American Bison use its horns? A: for defense and in fights for status within the herd

Table 3: Example #1 – Context, Question, and Answer Generation.

Original Context |  Exhibitions and annual horse shows in all districts and a national horse and cattle show at Lahore are held with the official patronage. The national horse and cattle show at Lahore is the biggest festival where sports, exhibitions, and livestock competitions are held. It not only encourages and patronises agricultural products and livestock through the exhibitions of agricultural products and cattle but is also a colourful documentary on the rich cultural heritage of the province with its strong rural roots.
---|---
Sampled Question |  "What is Punjab’s biggest festival?"
Prompt |  Generate a paragraph that answers the following question: "What is Punjab’s biggest festival?"
Generated Context |  Punjab’s biggest festival is Baisakhi, also known as Vaisakhi, celebrated with great fervor and enthusiasm in the region. It’s a significant festival for the Punjabi people and is observed annually on the 13th or 14th of April. Baisakhi is not only a harvest festival, marking the time when farmers reap their rabi (winter) crops, but it also has deep religious significance, particularly for Sikhs. It commemorates the formation of the Khalsa Panth, a Sikh political and military group, by Guru Gobind Singh Ji in 1699. The day is marked by visiting gurdwaras, participating in processions, dancing the traditional Bhangra and Gidda, and sharing meals in the community. The joyous festival is a celebration of culture, heritage, and communal unity, encapsulating the vibrant spirit of Punjab.
Generated Questions |  Q: When is Baisakhi observed? A: annually on the 13th or 14th of April, Q: Why is Baisakhi significant for farmers? A: a harvest festival, marking the time when farmers reap their rabi (winter) crops

Table 4: Example #2 – Context, Question, and Answer Generation.

Original Context |  In Canada, the Royal Newfoundland Constabulary was founded in 1729, making it the first police force in present-day Canada. It was followed in 1834 by the Toronto Police, and in 1838 by police forces in Montreal and Quebec City. A national force, the Dominion Police, was founded in 1868. Initially the Dominion Police provided security for parliament, but its responsibilities quickly grew. The famous Royal Northwest Mounted Police was founded in 1873. The merger of these two police forces in 1920 formed the world-famous Royal Canadian Mounted Police.
---|---
Sampled Question |  "When did Canada get its first police?"
Prompt |  Generate a paragraph that answers the following question: "When did Canada get its first police?"
Generated Context |  The first policing institution in Canada was established in the early 19th century. The Royal Newfoundland Constabulary, founded in St. John’s, Newfoundland in 1729, is considered the oldest civil law enforcement body in North America. However, it primarily served as a paramilitary unit, dealing with the city’s disorderly sailors and soldiers. Later, the North-West Mounted Police (NWMP), which is often seen as the direct predecessor to the current Royal Canadian Mounted Police (RCMP), was formed on May 23, 1873. The establishment of the NWMP marked a significant shift in the approach to law enforcement in Canada, transitioning from a military-style model to a community-focused approach. The NWMP, and later the RCMP, played a crucial role in establishing the rule of law across Canada, particularly in the west and north during the periods of rapid expansion and development in the late 19th and early 20th centuries.
Generated Questions |  Q: What is the oldest civil law enforcement body in North America? A: The Royal Newfoundland Constabulary, Q: Which organization is seen as the direct predecessor to the Royal Canadian Mounted Police? A: the North-West Mounted Police

Table 5: Example #3 – Context, Question, and Answer Generation.

Original Context |  The Earth of the early Archean (4,000 to 2,500 million years ago) may have had a different tectonic style. During this time, the Earth’s crust cooled enough that rocks and continental plates began to form. Some scientists think because the Earth was hotter, that plate tectonic activity was more vigorous than it is today, resulting in a much greater rate of recycling of crustal material. This may have prevented cratonisation and continent formation until the mantle cooled and convection slowed down. Others argue that the subcontinental lithospheric mantle is too buoyant to subduct and that the lack of Archean rocks is a function of erosion and subsequent tectonic events.
---|---
Sampled Question |  "During what time period was the Archean era?"
Prompt |  Generate a paragraph that answers the following question: "During what time period was the Archean era?"
Generated Context |  The Archean Eon, one of the four principal eons of Earth’s geologic time scale, occurred approximately 4.0 to 2.5 billion years ago. It follows the Hadean Eon and precedes the Proterozoic Eon. The Archean era is divided into four geologic time periods, known as eras: the Eoarchean, Paleoarchean, Mesoarchean, and Neoarchean. This time period was crucial in Earth’s history as it saw the formation of the planet’s crust and the development of the earliest forms of life. Despite the extreme conditions, with high temperatures and a lack of oxygen, early life forms such as cyanobacteria began to emerge, which eventually played a role in shaping the Earth’s atmosphere by producing oxygen through photosynthesis.
Generated Questions |  Q: What eon follows the Hadean Eon and precedes the Proterozoic Eon?A: The Archean Eon, Q: Despite what conditions did early life forms begin to emerge during the Archean Eon? A: the extreme conditions, with high temperatures and a lack of oxygen

Table 6: Example #4 – Context, Question, and Answer Generation.