# Interpreting mechanism of Synergism of drug combinations using attention
based hierarchical graph pooling

1st Zehao Dong  Washington University in St. Louis
<EMAIL_ADDRESS>2rd Yixin Chen  Washington University in St. Louis
<EMAIL_ADDRESS>3st Philip Payne  4nd Fuhai Li  Washington University in
St. Louis
<EMAIL_ADDRESS>Washington University in St. Louis
<EMAIL_ADDRESS>
###### Abstract

The synergistic drug combinations provide huge potentials to enhance
therapeutic efficacy and to reduce adverse reactions. However, effective and
synergistic drug combination prediction remains an open question because of
the unknown causal disease signaling pathways. Though various deep learning
(AI) models have been proposed to quantitatively predict the synergism of drug
combinations. The major limitation of existing deep learning methods is that
they are inherently not interpretable, which makes the conclusion of AI models
un-transparent to human experts, henceforth limiting the robustness of the
model conclusion and the implementation ability of these models in the real-
world human-AI healthcare. In this paper, we develop an interpretable graph
neural network (GNN) that reveals the underlying essential therapeutic targets
and mechanism of the synergy (MoS) by mining the sub-molecular network of
great importance. The key point of the interpretable GNN prediction model is a
novel graph pooling layer, Self-Attention based Node and Edge pool (henceforth
SANEpool), that can compute the attention score (importance) of nodes and
edges based on the node features and the graph topology. As such, the proposed
GNN model provides a systematic way to predict and interpret the drug
combination synergism based on the detected crucial sub-molecular network. We
evaluate SANEpool on molecular networks formulated by genes from 46 core
cancer signaling pathways and drug combinations from NCI ALMANAC drug
combination screening data. The experimental results indicate that 1) SANEpool
can achieve the current state-of-art performance among other popular graph
neural networks; and 2) the sub-molecular network detected by SANEpool are
self-explainable and salient for identifying synergistic drug combinations.

##  I Introduction

The combinatorial drug therapy has been of crucial importance in the modern
clinical disease treatment and drug discovery [Hopkins, 2008]. Drug
combination that can produce more beneficial combinatorial effects than each
constituent is said to be synergistic, and the synergic behavior always allows
the lower doses of drugs in the combination than their individual potencies,
thus reducing the induction of the drug resistance [Podolsky and Greene, 2011]
and overcoming the side effects [Chandrasekaran et al., 2016, Radic-Sarikas et
al., 2017, O’Neil et al., 2016] associated with high doses of the single drug
usage. Hence, drug combination therapy provides greatly promising avenue
towards the treatment of the most dreadful multi-factorial diseases, such as
cancer [Devita Jr et al., 1975, Crino et al., 1997, Carew et al., 2008,
Shuhendler et al., 2010], diabetes and bacterial infections. In contrast to
the synergism, the therapeutic efficacy of some drug combinations can be
simply additive, or even sub-additive. As such, there has been growing
interests in investigating the synergy mechanism of drug combinations to
distinguish the synergistic combinations from non-synergistic ones.

Frequently, the synergy of drug combinations is tested in pre-clinical model
environments, such as high-throughput screening (HTS) instruments [Mott et
al., 2015, Griner et al., 2014], where thousands of combinatorial experiments
are simultaneously implemented under actionable hypothesis and conditions to
profile the synergism. However, the testing space can be extremely massive due
to the large amount of drug combinations, cell lines, dose choices, and
patient samples, hence it can be impractical to traverse the whole testing
space [Holbeck et al., 2017]. Furthermore, the transition from some pre-
clinical environments to the clinical practice sometimes can also cause
failure [Yang et al., 2018]. As such, various computational (AI) models [Zhang
et al., 2021, Janizek et al., 2018, Chen and Guestrin, 2016, Preuer et al.,
2018] are developed to assist the synergy analysis of drug combinations.

Basically, the computational AI models takes massive omics data and chemical
structures data as input, and then adopt deep learning algorithms to predict
the synergy score. Then the AI models can be trained by experimental data to
quantify the presence or absence of the synergism. For instance, several
proposed machine learning models build ensembles tree to predict the synergy
scores (TreeCombo [Chen and Guestrin, 2016], Random Forest [Sidorov et al.,
2019]). While various deep neural networks are developed to learn the node
embeddings based on chemical and gene expression profiles in specific cell
lines, and then the learnt drug embeddings are used to predict the drug pair
synergy (DeepSynergy [Preuer et al., 2018], MatchMaker [Kuru et al., 2021]).
However, a major limitation is that current applied AI models are inherently
not interpretable. The interpretability of AI models is usually desirable by
the pharmacy industry and the real-world AI-assisted healthcare, as it allows
to incorporate human expertise in the decision making to provide more robust
conclusion.

In recent years, there has been a growing interest in the machine learning
communities to develop graph neural networks for learning from graph-
structured data, such as social networks [Monti et al., 2017, Ying et al.,
2018a], protein networks [Fout, 2017, Zitnik et al., 2018], etc. To learn from
these graphs, various differentiable graph neural network (GNN) frameworks
have been proposed to capture graph structures. As some of the earliest works,
spectral-based GNNs [Bruna et al., 2013, Defferrard et al., 2016] define graph
convolution operations from the perspective of graph signaling processing.
However, spectral GNNs rely on the decomposition of graph Laplacian and
operate on the graph spectra. Thus, they are dependent on the structures of
the whole graphs and are not localized in space. As a result, it is difficult
to apply them to graphs with varying characteristics or large-scale graphs. On
the other hand, spatial-based GNNs define each node’s receptive field based on
the node’s spatial relations and implement localized graph convolution within
this field. Therefore, they can overcome the aforementioned limitations and
have achieved impressive results on various graph learning tasks, ranging from
node classification [Hamilton et al., 2017], link prediction [Schütt et al.,
2017, Zhang and Chen, 2018], to graph classification [Dai et al., 2016].

While the neighborhood aggregation effectively learns node representations,
another challenge in graph-level prediction tasks is to learn meaningful
graph-level representation from node representations, and various graph
pooling modules are proposed for this purpose. In general, pooling modules
fall into three categories: summing-based approaches, sorting-based
approaches, and hierarchical approaches. Summing-based pooling modules [Henaff
et al., 2015] generate graph representations by summing nodes’ representations
extracted through graph convolution operations. Sorting-based pooling modules
convert non-Euclidean graph data into grid data by sequentially reading nodes
in a meaningful and consistent order. For instance, PATCHY-SAN [Niepert et
al., 2016] generates node order via a node labeling procedure and a
canonicalization process to select fixed-sized patches with a fixed number of
ordered neighbors for each node. DGCNN [Zhang et al., 2018] proposes to sort
nodes for pooling according to their structural roles within the graph.
However, since the summing-based graph pooling and the sorting-based graph
pooling stack multiple graph convolution layers to propagate information and
globally implement the graph down-sampling via a pooling module, the generated
graph representation is inherently flat. In order to extract hierarchical
graph representations, DiffPool [Ying et al., 2018b] uses different GNNs to
separately implement neighborhood aggregation and graph pooling, and it
provides a framework to hierarchically pool nodes across a broad set of
graphs.

To address aforementioned challenges, we introduce SANEpool (Self-Attention
based Node and Edge pool), an interpretable graph neural network to mine the
drug combination synergism by systematically extracting the target gene sub-
network that intrigues the synergic behavior. Various medical chemistry
researchs have shown that cancer is driven by genetic and epigenetic
alterations, many of which can be mapped into signaling pathways that controls
the survival and migration/invasion of cancer cells. As one previous signaling
pathway analysis suggests [Sanchez-Vega et al., 2018], 89$\%$ of tumor samples
had at least one driver alteration in one of 10 cancer-related signaling
pathways that is responsible for the tumor development, while 57$\%$ and
30$\%$ had one and multiple potentially druggable targets respectively.
Another example [Pan et al., 2017] is that the dug combination of Venetoclax
and Idasanutlin can generate antileukemic efficacy in the treatment of acute
myeloid leukemia by inhibiting antiapoptotic Bcl-2 family proteins and
activating the p53 pathway at same time. Thus, inhibited signaling targets
analysis shows great potential of facilitating the drug combination synergism
discovery. Following this intuition, each SANEpool layer implements the
standard graph convolution layer (GCN) to generate attention features that
encodes the gene (and drug) information as well as the topology information of
the molecular network. Based on these attention features and the pre-defined
order of genes, the probability that a gene or an interaction (gene-gene
interaction, gene-drug interaction) will cause the synergy performance is
calculated, and then genes and interactions that are unlikely to influence the
synergism of drug combination will be filtered out. The proposed model is
composed of multiple SANEpool layers, and will output the target gene sub-
network for the interpretable and robust synergy prediction.

We validate the proposed model on drug combinations from NCI ALMANAC [Sidorov
et al., 2019], and the proposed architecture achieves the current state-of-art
performance in the synergy score prediction and scoring task. Furthermore,
through visualizations of the detected target gene sub-network of different
cancer cell lines, we observe that the proposed model (SANEpool) can detect
salient target gene patterns that causes the synergic drug combinations, which
reveals the synergism mechanism in drug combination discovery.

##  II Related Works

#### Graph neural networks:

Graph neural networks (GNNs) have revolutionized the field of learning with
graph structured data, and empirically achieved the current state-of-art
performance in various graph learning tasks, ranging from node classification,
link prediction to graph classification. Broadly, GNNs [Schlichtkrull et al.,
2018, Zhang et al., 2018, Ying et al., 2018b, Verma and Zhang, 2018, Dai et
al., 2016, Niepert et al., 2016] follow a recursive neighborhood aggregation
scheme where the node features from the neighborhood of each node are
aggregated to update the node’s feature. Such framework allows GNNs to capture
the graph topology as well as node features, hence unleashing the
representation learning ability among graphs.

#### Pan-cancer biomarkers:

The genotype-oriented therapies for pan-caner biomarkers have been approved by
the US Food and Drug Administration. These biomarkers amplify our knowledge of
genomic profiling across various malignancies by revealing the prevalence of
certain oncogenic alternations, hence playing important roles in the drug
combination discovery. According to the previous study [Yao et al., 2021],
30$\%$ of recurrent alternations across tumor types from 10000 patients with
metastatic cancers are targetable, and various genotype-oriented therapies are
detected based on the genomic profiling. For instance, the neurotrophic
receptor kinases (NTRK) family genes 1-3 were identified in various pediatric
cancers. Then a clinical trial of the Trk inhibitor larotrectinib demonstrated
the antitumor activity and hence led to the usage of arotrectinib as treatment
of cancers harboring NTRK fusions.

#### Machine Learning in drug synergy prediction:

The drug synergy analysis is beneficial as it provides a useful resource for
novel predicted drug combinations. However, manually discovering the synergism
in practice is still challenging due to the high cost and the limited number
of synergistic drug combinations approved by the Food and Drug Administration.
Hence, the computational model shows huge potential to find the mechanism of
synergy (MoS) in a biological meaningful manner. Currently, various
computational models, ranging from unsupervised learning models [Jin et al.,
2011, Wu et al., 2010, Chen et al., 2016] to supervised learning models [Li et
al., 2015, Xu et al., 2011], have been proposed for the purpose of predicting
the synergy of drug combinations, and have achieved the expressive
performance. Broadly, these computational models, such as DeepSynergy [Preuer
et al., 2018] and Matchmaker [Kuru et al., 2021], take as input massive
chemical descriptors of tested drug pair and cell line gene expression
profile, and then use multi-layer perceptrons (MLPs) based deep learning
models to predict the synergy score of drug combinations. Although these
models effectively predict the synergy score of drug combinations, they are
inherently not interpretable, while the interpretability is crucial for the
real-world application. As the drug synergy has been reported to be largely
determined by the biomolecular network topology [Yin et al., 2014], many deep
learning models, such as , DeepSignalingSynergy [Zhang et al., 2021] and IDSP
[Dong et al., 2021], incorporate the gene-gene interactions and gene-drug
interaction into model design to allow the model make interpretable
predictions that explains the underlying MoS.

##  III Methodology

In this section, we introduce the proposed interpretable graph pooling layer,
Self-Attention based Node and Edge Pool (SANEpool). Based on this graph
pooling layer, we develop an interpretable graph neural network to detect the
biological meaningful gene sub-network for synergic and interpretable drug
combination prediction. The key point of SANEpool is to contain the attention
score (importance) of nodes and edges though the node features, graph topology
and the domain knowledge (order of genes in a gene network), then the
attention scores make it possible to filter out less important (less relevant)
nodes and edges for prediction. In Section 3.1, we introduce the problem
formulation of the interpretable drug prediction task. In Section 3.2, we
develop the mechanism of SANEpool, and the overall interpretable model
architecture is described in Section 3.3. The problem formulation and the
architecture of SANEpool are illustrated in Figure 1 and Figure 2,
respectively.

Figure 1: Overview of the problem formulation. The objective is to
systematically detect molecular sub-networks to predict the synergy score of
the input drug pairs and to explain the drug combinatorial synergies across a
large group of patients and cell lines

###  III-A Problem Configuration

In this work, we study the molecular networks of cancer drug combination
therapies in an inductive manner, where the tested drug pairs are unseen
during the training phase. The molecular networks contain genes from $46$
famous signaling pathways in KEGG dataset and drug pairs from NCI Almanac
dataset, while the interactions between genes and drugs can be obtained from
DrugBank database (version 5.1.5, released 2020-01-03) [Wishart et al., 2018].
The objective is to predict the synergic score of each drug pair based on the
gen-drug interaction network. In order to make the prediction interpretable,
SANEpool proposed to detect the sub-gene network (i.e. the red gene nodes in
Figure 1) that are most relevant to the drug pair effects. Then the detected
sub-graph provides insight into the molecular mechanism of resistant or
sensitive response of cancer drug combinations.

Let $G=(V,E)$ be the molecular network (graph), where $V$ is the node sets
that contains gene nodes and drug nodes, $E$ is the edges set that
characterizes the interactions between nodes. For the notation convenience, we
use $A$ to denote the adjacency matrix of a graph. Since the molecular graph
are inherently undirected and has no self-loop, adjacency matrix A is a
symmetric matrix. We use $X\in R^{n\times h}$ to denote the input node
feature, where $n$ is the number of nodes, and $h$ is the dimension of input
features. Hence, the graph can also be represented as the pair of the node
feature and adjacency matrix such that $G=(X,A)$.Furthermore, we use $Z^{t}\in
R^{n\times h_{t}}$ to denote the node representation in layer $t$, where
$h_{t}$ is the dimension of node representation.

###  III-B The proposed SANEpool model

Figure 2: The proposed SANEpool layer and the overall architecture. SANEpool
layer incorporates node features and graph topologies through a GNN layer,
then the output is used to compute the attention score of nodes and edges,
based on which we can detect important nodes and edges through top-K sorting
or threshold filtering. The overall model takes a hierarchical graph pooling
architecture.

The (self-)attention mechanism plays important roles in various machine
learning models including natural language processing architectures [Parikh et
al., 2016, Devlin et al., 2018, Yan et al., 2020], graph classification
architectures [Velickovic et al., 2018, Dong et al., 2022], sequential
prediction algorithms [Cheng et al., 2016], adversarial learning models [Zhang
et al., 2019], and etc. The attention mechanism allows input features to be
the criteria for the attention itself [Vaswani et al., 2017], thus can
distinguish the relative importance between features during the information
aggregation process. In order to incorporate the node features and graph
topology in the attention scores for the nodes and edges pooling, we follow
the idea of SAGpool [Lee et al., 2019] and utilize a graph convolution layer
to aggregate such information and to compute the attention feature matrix
$H^{t}$.

$\displaystyle H^{t}=f(\tilde{D}^{-1}\tilde{A}Z^{t}\Theta^{t})$ (1)

where $\tilde{A}=A+I$ is the adjacency matrix with added self-loops,
$\tilde{D}$ is the corresponding diagonal degree matrix of $\tilde{A}$ such
that $\tilde{D}_{ii}=\sum_{j=1}^{n}\tilde{A}_{ij}$, and $f$ is the activation
function. The matrix $\Theta^{t}\in\mathbb{R}^{h_{t}\times F_{t}}$ is the
trainable parameter to coordinate the attention score of nodes and edges.
Various graph convolution layers [Defferrard et al., 2016, Niepert et al.,
2016, Kipf and Welling, 2016] have been proposed, these GNN formulas can be
used as substitution of Equation 1. All GNN layers follows the same
information aggregation framework, hence, the extracted attention features
$H^{t}$ contain information of node features as well as the graph topology.

Next, we discuss how to compute the attention scores of nodes and edges based
on extracted attention features $H^{t}$ and the domain knowledge of gene
networks. The node attention score is determined as the cosine of the angle
between the attention feature vector and a trainable projection vector $p$
(Equation 2). The attention score Att measures the probability that gene nodes
will cause the synergism of drug combinations. Hence, we can sort node
attention scores and adopt the top-$k$ selection technique to hierarchically
select the gene sub-network for the purpose of synergism prediction, and such
process is formulated as:

$\displaystyle\textit{Att}_{i}$
$\displaystyle=\frac{p^{t}H^{t}_{i}}{||p^{t}||^{\frac{1}{2}}||H^{t}_{i}||^{\frac{1}{2}}}$
(2) idx $\displaystyle=\textit{top}(\textit{Att},k)$ (3)

In the top node selection process (i.e. Equation 3), when $k\in N$, we adopt
the tok-$k$ node selection by keeping fixed number of nodes in each layer as
DGCNN [Zhang et al., 2018]; On the other hand, we can also implement node
selection method proposed by [Cangea et al., 2018] to retain a proportion of
nodes when $k\in(0,1]$. Based on selected node index idx, we can constructed
the graph downsampling as $Z^{t+1}=Z^{t}(idx,:)$ and $A^{t+1}=A^{t}(idx,idx)$.

Let $G^{t+1}=(Z^{t+1},A^{t+1})$. Since the proportion of retained information
(attention score) of nodes in $G^{t+1}$ are different, the connectivity
strength between nodes can be different. Hence, we should also provide a
consistent mechanism to characterize such bias. One intuitive approach is to
apply the graph attention mechanism based on the extracted attention features
$H^{t}$:

$e_{i,j}=\textit{relu}(\textit{MLP}(H^{t}_{i}\ \lVert\ H^{t}_{j}))$ (4)

where the symbol $\lVert$ indicates the concatenation operation. Due to the
universal approximation theorem [Hornik et al., 1989, Hornik, 1991], such
formulation can approximate any continues function that measures the
connectivity strength. However, a major limitation of this framework is the
memory cost. For large-scale graphs like gene networks, the memory usage to
compute the attention score of edge may limit the practical ability of the
proposed model. Luckily, the molecular network (graph) takes advantage that
each gene node in the graph has a corresponding predefined order (i.e. the
gene name), which serves as a canonical order of nodes in the gene networks.
Hence, we directly model the interaction strength between nodes (genes) based
on the pre-defined order of genes in each layer through a trainable parameter
matrix $W$, where $W_{i,j}$ reflects the connection strength between nodes
with the canonical label $i$ and $j$. Then the the edge weight in the subgraph
$G^{t+1}$ is trainable through the equation $A^{t+1}=A^{t+1}W^{t}(idx,idx)$.
Further discussions are provided in Appendix LABEL:sec:append1.

###  III-C The Overall Architecture

Hierarchical Graph Pooling. The overall architecture of the proposed
interpretable model takes the hierarchical graph pooling structure [Cangea et
al., 2018, Lee et al., 2019], and Figure 2 illustrates the overall
architecture. The model stacks multiple SANEpool layers followed by a graph
convolution layer, to hierarchically extract a key sub-graph from the input
graph. In other words, the proposed SANEpool layer is used to downsample the
important sub-network (sub-graph). After the downsampling process, another GNN
(graph convolution) layer is used to aggregate information based on sub-graph
$G^{t+1}$ (Equation 5) to update the node representation.

$Z^{t+1}=\textit{GNN}(Z^{t}[\textit{idx}^{t},:],A^{t+1})$ (5)

where $Z^{t}[\textit{idx}^{t},:]$ is the node representation matrix of the
downsampled sub-graph in the $t$-th SANEpool layer. Then, the output of the
last graph convolution layer is used for the prediction task.

Readout mechanism. Inspired by IGMC [Zhang and Chen, 2019], the proposed model
takes the node representation of two drug nodes to make prediction. The graph
convolution framework indicates that such node representations encode the
enclose rooted subtrees around the drug nodes in the pooled graph, hence
representing the relations and interactions between drugs and genes. Ideally,
we hope that the readout phase should be invariant to the drug node order, as
the same drug pairs always have the same clinical performance regardless their
order. Let $u_{1}$, $u_{2}$ be the output of two drug nodes, then the readout
layer follows the factorization decoder in decagon [Kuru et al., 2021]:

$\textit{score}=u^{T}_{1}D^{T}Du_{2}$ (6)

where $D\in\mathbb{R}^{h_{T},h_{T}}$ is the trainable parameter matrix in the
decoder, where $h_{T}$ is the dimension of node representations in the last
graph convolution layer $T$. The parameter matrix $D$ models the interaction
effects between every two dimensions in drug representations $u_{1}$ and
$u_{2}$.

###  III-D Comparison to Other Graph Pooling Methods

The proposed SANEpool directly addresses the interpretability challenge and
the graph pooling challenge in the real-world graph-level prediction tasks.
Here we give some comparison between SANEpool with other related graph pooling
algorithms.

#### Comparison with Sortpool.

Both SANEpool and Sortpool [Zhang et al., 2018] propose to sort nodes
according to the structural role (i.e. ’importance’) of nodes in the graph.
However, DAGNN is inherently flat, while SANEpool aggregate information in a
hierarchical way. Thus, SANEpool is capable of capturing more informative
graph representation for the downstream prediction task.

#### Comparison with SAGpool.

Both SANEpool and SAGpool [Lee et al., 2019] use GNNs to learn the projection
and self-attention scores to perform graph pooling which selects the top-
ranked nodes and filters out the rest. However, compared with SAGpool, the key
innovation of SANEpool is to directly implement the edge pooling through a
trainable parameter matrix $W$ based on the pre-defined node order (so as the
graph size) in the molecular networks. As gene nodes in molecular networks
always has extreme high centrality, edge pooling always help to deal with the
feature smoothing challenge, hence capturing more reasonable and salient sub-
networks for the prediction task.

#### Comparison with Diffpool.

Both SANEpool and Diffpool [Ying et al., 2018b] learn graph representation in
a hierarchical way. However, Diffpool focus on the relational analysis of node
clusters, while SANEpool detect the critical sub-network based on the
downstram tasks. Hence, SANEpool supports the pathway-based analysis in the
drug combination prediction, thus providing interpretable results for
healthcare.

##  IV Experiments

TABLE I: Performance evaluation Model | Pearson’r $\uparrow$ | MSE $\downarrow$ | Interpretability
---|---|---|---
DeepSignalingSynergy | 0.631$\pm$ 0.019 | 45.218 $\pm$ 1.889 | False
GIN | 0.565 $\pm$ 0.042 | 51.732 $\pm$ 5.636 | False
GCN | 0.494 $\pm$ 0.049 | 58.585 $\pm$ 5.618 | False
DAGNN | 0.509 $\pm$ 0.025 | 57.827 $\pm$ 3.174 | False
GAT | 0.571 $\pm$ 0.031 | 50.995 $\pm$ 3.021 | True
GG-NN | 0.509 $\pm$ 0.025 | 57.827 $\pm$ 3.174 | False
SAGpool | 0.537 $\pm$ 0.031 | 53.125 $\pm$ 4.116 | True
MinCUT pool | 0.581 $\pm$ 0.029 | 49.312 $\pm$ 3.917 | False
Diffpool | 0.577 $\pm$ 0.022 | 52.449 $\pm$ 3.155 | False
SANEpool (our model) |  0.656$\pm$ 0.016 |  44.352$\pm$ 2.241 | True
TABLE II: Cell line based K-S test results

Group1 | Group2 | Group3 | Group4 | Group5
---|---|---|---|---
Cell line | p value | Cell line | p value | Cell line | p value | Cell line | p value | Cell line | p value
UACC-62 | 2.421 $\mathrm{e}{-9}$ | NCI-H522 | 6.145 $\mathrm{e}{-10}$ | HT29 | 1.463 $\mathrm{e}{-13}$ | MDA-MB-435 | 1.028 $\mathrm{e}{-5}$ | A549/ATCC | 2.745 $\mathrm{e}{-12}$
OVCAR-8 | 2.821 $\mathrm{e}{-15}$ | HOP-62 | 2.745 $\mathrm{e}{-12}$ | HCT-15 | 3.341 $\mathrm{e}{-23}$ | RPMI-8226 | 6.145 $\mathrm{e}{-10}$ | MDA-MB-231/ATCC | 9.027 $\mathrm{e}{-3}$
OVCAR-3 | 1.227 $\mathrm{e}{-9}$ | HS 578T | 2.961 $\mathrm{e}{-3}$ | UO-31 | 2.495 $\mathrm{e}{-28}$ | BT-549 | 5.652 $\mathrm{e}{-13}$ | UACC-257 | 1.124 $\mathrm{e}{-23}$
LOX IMVI | 3.341 $\mathrm{e}{-23}$ | SW-620 | 2.852 $\mathrm{e}{-22}$ | MCF7 | 1.228 $\mathrm{e}{-17}$ | NCI-H460 | 1.573 $\mathrm{e}{-15}$ | EKVX | 2.495 $\mathrm{e}{-28}$
HOP-92 | 3.471 $\mathrm{e}{-18}$ | SF-268 | 3.721 $\mathrm{e}{-28}$ | K-562 | 7.747 $\mathrm{e}{-10}$ | T-47D | 2.403 $\mathrm{e}{-5}$ | MDA-MB-468 | 6.085 $\mathrm{e}{-6}$
MALME-3M | 1.014 $\mathrm{e}{-37}$ | SK-MEL-5 | 1.135 $\mathrm{e}{-7}$ | SF-295 | 4.636 $\mathrm{e}{-21}$ | NCI-H23 | 1.491 $\mathrm{e}{-10}$ | OVCAR-4 | 2.422 $\mathrm{e}{-9}$
SF-539 | 1.028 $\mathrm{e}{-5}$ | U251 | 1.135 $\mathrm{e}{-7}$ | PC-3 | 5.404 $\mathrm{e}{-24}$ | CAKI-1 | 7.256 $\mathrm{e}{-6}$ | HCT-116 | 1.173 $\mathrm{e}{-15}$
IGROV1 | 9.208 $\mathrm{e}{-11}$ | SK-OV-3 | 5.791 $\mathrm{e}{-11}$ | A498 | 2.821 $\mathrm{e}{-15}$ | NCI-H322M | 7.256 $\mathrm{e}{-6}$ | ACHN | 6.713 $\mathrm{e}{-15}$
HL-60(TB) | 5.034 $\mathrm{e}{-15}$ | KM12 | 1.579 $\mathrm{e}{-14}$ | NCI-H226 | 4.602 $\mathrm{e}{-12}$ | SK-MEL-28 | 9.620 $\mathrm{e}{-13}$ | DU-145 | 4.088 $\mathrm{e}{-26}$

We evaluate the effectiveness of the proposed SANEpool by comparing the
performance against powerful quantitative baselines in the synergy prediction
domain as well as various state-of-art graph neural networks (GNNs).
Furthermore, to illustrate the synergism detected by SANEpool, we also
implement visual analytics and statistical analysis to show the proposed
SANEpool can detect significantly different causal gene sub-networks for
synergic drug combinations.

###  IV-A Dataset Description

We formulate the interpretable synergic drug combination prediction problem as
a supervised learning problem. This problem takes as input biomolecular
networks (graphs) that consists of genes and drug pairs, then the objective is
to predict the synergy score of the drug pair in the input biomolecular
network based on the gene expressions and the network structure. In the data
collection process, we ensemble gene nodes from 46 well-known signaling
pathways (45 “signaling pathways” + cell cycle) [Feng et al., 2021] in KEGG
(Kyoto Encyclopedia of Genes and Genomes) database [Kanehisa and Goto, 2000],
where the gene-gene interactions are constructed based on the physical
signaling interactions from documented medical experiments. We also collect
FDA approved drug pairs whose target gene is included in aforementioned 46
signaling pathways from DrugBank database (version 5.1.5, released 2020-01-03)
[Wishart et al., 2018]. Below we list the details of the 46 signaling pathways
and 21 selected FDA approved drugs.

Signaling pathways used to formulate the input graphs: MAPK, FoxO, TGF-beta, T
cell receptor, Adipocytokine, ErbB, Sphingolipid, VEGF, B cell receptor,
Oxytocin, Ras, Phospholipase D, Apelin, Fc epsilon RI, Glucagon, Rap1, p53,
Hippo, TNF, Relaxin, Calcium, mTOR, Toll-like receptor, Neurotrophin, AGE-
RAGE, cGMP-PKG, PI3K-Akt, NOD-like receptor, Insulin, Cell cycle, cAMP, AMPK,
RIG-I-like receptor, GnRH, Chemokine, Wnt, C-type lectin receptor, Estrogen,
NF-kappa B, Notch, JAK-STAT, Prolactin, HIF-1, Hedgehog, IL-17, Thyroid
hormone.

FDA approved drugs used to formulate the input graphs: Celecoxib, Gefitinib,
Quinacrine hydrochloride, Tretinoin, Cladribine, Imatinib mesylate,
Romidepsin, Vinblastine sulfate (hydrate), Dasatinib, Lenalidomide, Sirolimus,
Vorinostat, Docetaxel, Mitotane, Sorafenib tosylate, Thalidomide, Everolimus,
Nilotinib, Tamoxifen citrate, Paclitaxel, Fulvestrant.

In the end, the synergy score is computed as the average combo-score [Pan et
al., 2017] with different doses on a given tumor cell line, and we collect the
combo-scores19 of drug combinations from the NCI Almanac dataset.

###  IV-B Baseline methods

#### Deep learning methods.

We compare SANEpool with the state-of-art deep learning methods for the
molecular-netwrok-based synergy prediction task, and it investigates the
discriminative importance of individual signaling pathways for drug
combination prediction.

#### Graph neural networks.

We also compare SANEpool with six widely adapted GNN baselines. These
baselines can be categorized into two types: (1) flat GNNs: Graph Attention
Network (GAT) [Veličković et al., 2018],Deep Graph CNN (DGCNN) [Zhang et al.,
2018], Graph Isomorphism Network (GIN) [Xu et al., 2018], and Graph
convolutional network (GCN) [Kipf and Welling, 2016]; (2) hierarchical GNNs:
Diffpool [Ying et al., 2018b] and SAGpool [Lee et al., 2019]. For GCN, GIN and
GAT, we stack 3 graph convolution layers with 64 output feature channels and
concatenate sum-pooled features from the layers to generate the graph
representation, which is then passed to an MLP to predict the graph label.

To provide robust model performance, we perform 10-fold cross-validation, and
report the accuracy averaged over 10 folds and the standard deviation of
validation accuracies across the 10 folds.

###  IV-C Experimental Results

Predictive Performance. In the experiment, we demonstrate the effectiveness of
SANEpool in predicting the synergy score of drug combinations over a
comprehensive list of baselines. The tested SANEpool model consists of a
SANEpool layer followed by two GNN layers. For GNN baselines (GCN, GAT, GIN,
DAGNN, SAGPool, Diffpool ) and the quantitative baseline,
DeepSignalingSynergy. We test their performance using open-source code. Table
I illustrates the experimental results. The experimental results indicate that
SANEpool can accurately predict the synergy score of drug combinations and
achieve the state-of-art predictive performance among the competitive
baselines.

Furthermore, (1) we find that SANEpool significantly improves the performance
over flat GNNs (i.e. GIN,GCN,GAT,DAGNN), which indicates that hierarchical
graph representation learning technique in molecular networks can provide
informative graph embedding with biological meaning. (2) SANEpool outperforms
other hierarchical graph pooling algorithms, and this observation indicates
that incorporating edge information in the graph pooling is a potential future
direction in the molecular network analysis, where graphs always have
thousands of high-centrality nodes.

Interpretability. The interpretability of deep learning models has been a
major limiting factor for the use of these models in real-world drug-
combination synergy analysis, since most usage cases require explanations of
the features used in the model. There is a nature tradeoff between the
interpretability and the accuracy of decision models in application, hence
it’s critical to find the balance between them. Currently, there are multiple
deep learning models which takes massive drug chemical structure information
and predict the synergy score in a fully untransparent manner, such as
DeepSynergy and MatchMarker. Although they can achieve expressive predictive
performance, the lack of interpretability somehow limits the power of these
models in real-world applications. Among the selected basslines,
DeepSignalingSynergy is constructed based on the standard multiple layer
perceptron model, hence it is inherently not interpretable. Similarly, GCN,
GIN, and GG-NN follow the basic neighborhood aggregation framework that
aggregates information form neighborhood of each node and then update the node
feature of the node based on the aggregated feature and node feature itself,
while such aggregation process are not interpretable, neither. In the
contrast, GAT and SAGPool can provide interpretable conclusions. For instance,
GAT provides interpretability by measuring the connection strength between
genes and graphs in the input biomolecular graphs through the attention
mechanism, and we can analyze the effect of drug nodes based the connection
strength. Though SAGPool doesn’t provide any information about how single
features impact the prediction, it allows user to analyze the effect of
combination of features through the hierarchical graph pooling. In analogous
to these interpretable models, in the next section, we will show that our
proposed SANEpool can detect the synergistic sub-network that distinguishs the
synergic drug combinations from non-synergic ones.

###  IV-D Statistical Analysis and Visual Analytics

In the experiment, we implement statistical analysis and visual analytics to
reveal the interpretability of the SANEpool model. For each cancer cell line,
there are multiple drug combinations targeting on the cell line, some of which
are synergic while others are not synergic. For each drug combination,
SANEpool can detect a corresponding gene subnetwork for the synergism
prediction. Hence, the proposed SANEpool assigns each gene a synergic
importance and a non-synergic importance, where the synergic importance is the
proportion that the gene is included in the sub-network of synergic drug
combinations, and the non-synergic importance is the proportion that the gene
is included in the subnetwork of non-synergic drug combinations. For instance,
if a gene node is never detected by any synergic drug combination that targets
on a specific cancer cell line, then it’s synergic importance is 0 and is
never used to predict the synergy score of synergic combinations. As Figure 3
shows, SANEpool can always detect similar gene patterns for synergic drug
combinations. In other world, for all cell lines, SANEpool can detect a core
sub-network for synergic drug combinations, thus providing insight to the
mechanism of the synergism performance of drug combinations.

Figure 3: This Figure describes the synergic importance of genes. The red
nodes represent genes have synergic importance large than 0.01, while the size
of nodes indicates the value of the synergic importance.

Next, we should decide whether the detected gene sub-networks for synergic
drug combinations are significantly different from that of non-synergistic
drug combinations. In order to do so, we can compare the distribution of the
synergic importance on genes and the distribution of the non-synergic
importance on genes. If these two distributions are significantly different,
we can infer that the detected gene sub-networks are also significantly
different. Hence, we implement the Kolmogorov-Smirnov test (K-S test) to
compare these distributions. In the K-S test, the null hypothesis assumes that
two distributions are same, and it computes a $D$ statistic as well as a
p-value corresponding to the $D$ statistic. Then we reject the null hypothesis
if the p-value is less than the significance level (0.05). We implement the
cell-line based test as well as the overall test that including all cell
lines. The p-value of the overall test is 3.9903e-27, hence the distribution
of synergic importance and non-synergic importance on nodes (which are
computed by SANEpool) are significantly different. The cell-line based K-S
test results are provided in Table II, and it shows that, for each cell line,
the detected gene sub-networks for synergic drug combinations and that for
non-synergic drug combinations are significantly different.

Figure 4: Heatmaps of cell line SF-295 and cell line K-562.

Besides the statistical analysis, we also use heat maps to show the difference
between detected sub-networks of synergic drug combinations and non-synergic
drug combinations. Here we provide examples on cell line SF-295 and cell line
K-562 in Figure 4, and more examples are provided in Appendix
LABEL:sec:append. In the heatmap, the values (synergy ¿ 0, synergy ¡ 0)
assigned to each gene are the cell-line-based synergic importance and non-
synergic importance, which indicates the proportion that the gene is included
in the gene sub-network (detected by SANEpool) of synergic drug combinations
(non-synergic drug combinations) targeting on the cell line. The synergy value
of a gene measures the possibility that the information of the gene, such as
the gene expression and gene copy number, contributes to the prediction
process of the drug synergy score. Hence, the difference of cell-line-based
synergy value (estimated probability that the gene is involved in the detected
sub-network of drug combinations with synergy ¿ 0) and cell-line-based non-
synergy value (estimated probability that the gene is involved in the detected
sub-network of drug combinations with synergy ¿ 0) can reflect the synergic
performance of the gene. That is, a larger difference indicates the gene
contributes more to the synergic drug combination than non-synergic drug
combinations. For instance, for cell line SK-562, the top 10 detected genes
are SIN3A, ETS2, WNT10B, SLC8A1, MTOR, KLF2, RGS2, SESN3, NRG1, TNFRSF11A.
Furthermore, these heatmaps also show that the difference between the detected
genes of synergic drug combinations and of non-synergic drug combinations are
significant.

Figure 5: Visualization of drug-gene interactions on selected cell lines. In
these graphs, red nodes represent genes in the subnetwork of all synergic drug
combinations on the cell line, purple nodes are drugs in randomly sampled
synergic drug combinations, blue nodes are drugs in randomly sampled non-
synergic drug combinations.

Furthermore, we also visualize the drug-gene interaction on the selected cell
lines in Figure 5 to show that synergic drug combinations are more likely to
target on the detected core signaling network thab non-synergic drug
combinations. For each cell line, we plot all genes (i.e. red nodes) in the
sub-networks (detected by SANEpool) of synergic drug combinations, and then
randomly sample a synergic drug combination (i.e. purple nodes) and some non-
synergic drug combinations (i.e. blue nodes). We find that the drugs in the
non-synergic combinations is very unlikely to target on subnetworks for
synergic drug combinations. Besides the cell-line-based synergy value and
cell-line-based non-synergy value of each gene, we also compute the synergy
value and non-synergy value of each gene based on all cell lines, and find a
core signaling network that consists of genes whose synergy value is larger
than 0.05. We visualize the core signaling network and all synergic drug
combinations in Figure 6, and we find that most synergic drug combinations
target on this core signaling network.

Figure 6: This Figure describes how synergic drugs targets on the detected
core signaling network. Synergic drug odes are visualized as purple nodes,
while red nodes represent genes in the detected core signaling network.

##  V Conclusion

In this paper, we have proposed an interpretable GNN architecture called
SANEpool (Self-Attention based Node and Edge pool) to predict the synergy
score of drug combinations and to investigate the underlying mechanism of the
synergy (MoS) by detecting salient molecular sub-networks. For each cell line
and each drug combination, SANEpool can detect a specific sub-network, and
SANEpool evaluates the contribution of each gene to synergic drug combinations
based on all detected (cell-line-based) sub-networks. Hence, cell-line
specific essential signaling gene targets are identified by SANEpool.
Furthermore, our observation also indicates that most synergistic drug
combinations inhibites the core signaling network detected by SANEpool. The
current work is limited by the number of drug combinations and cell lines. In
the future work, more drug combinations datasets with multi-omic data will be
integrated to uncover the mechanism of the synergy of effective drug
combinations.

##  VI Acknowledgement

This work is partially supported by the Children’s Discovery Institute (CDI)
M-II-2019-802 to Li.

## References

  * [Bruna et al., 2013] Bruna, J., Zaremba, W., Szlam, A., and LeCun, Y. (2013).  Spectral networks and locally connected networks on graphs.  arXiv preprint arXiv:1312.6203.
  * [Cangea et al., 2018] Cangea, C., Veličković, P., Jovanović, N., Kipf, T., and Liò, P. (2018).  Towards sparse hierarchical graph classifiers.  arXiv preprint arXiv:1811.01287.
  * [Carew et al., 2008] Carew, J. S., Giles, F. J., and Nawrocki, S. T. (2008).  Histone deacetylase inhibitors: mechanisms of cell death and promise in combination cancer therapy.  Cancer letters, 269(1):7–17.
  * [Chandrasekaran et al., 2016] Chandrasekaran, S., Cokol-Cakmak, M., Sahin, N., Yilancioglu, K., Kazan, H., Collins, J. J., and Cokol, M. (2016).  Chemogenomics and orthology-based design of antibiotic combination therapies.  Molecular systems biology, 12(5):872.
  * [Chen et al., 2016] Chen, D., Zhang, H., Lu, P., Liu, X., and Cao, H. (2016).  Synergy evaluation by a pathway–pathway interaction network: a new way to predict drug combination.  Molecular BioSystems, 12(2):614–623.
  * [Chen and Guestrin, 2016] Chen, T. and Guestrin, C. (2016).  Xgboost: A scalable tree boosting system.  In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785–794.
  * [Cheng et al., 2016] Cheng, J., Dong, L., and Lapata, M. (2016).  Long short-term memory-networks for machine reading.  arXiv preprint arXiv:1601.06733.
  * [Crino et al., 1997] Crino, L., Scagliotti, G., Marangolo, M., Figoli, F., Clerici, M., De Marinis, F., Salvati, F., Cruciani, G., Dogliotti, L., Pucci, F., et al. (1997).  Cisplatin-gemcitabine combination in advanced non-small-cell lung cancer: a phase ii study.  Journal of clinical oncology, 15(1):297–303.
  * [Dai et al., 2016] Dai, H., Dai, B., and Song, L. (2016).  Discriminative embeddings of latent variable models for structured data.  In International conference on machine learning, pages 2702–2711. PMLR.
  * [Defferrard et al., 2016] Defferrard, M., Bresson, X., and Vandergheynst, P. (2016).  Convolutional neural networks on graphs with fast localized spectral filtering.  arXiv preprint arXiv:1606.09375.
  * [Devita Jr et al., 1975] Devita Jr, V. T., Young, R. C., and Canellos, G. P. (1975).  Combination versus single agent chemotherapy: a review of the basis for selection of drug treatment of cancer.  Cancer, 35(1):98–110.
  * [Devlin et al., 2018] Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018).  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805.
  * [Dong et al., 2021] Dong, Z., Zhang, H., Chen, Y., and Li, F. (2021).  Interpretable drug synergy prediction with graph neural networks for human-ai collaboration in healthcare.  arXiv preprint arXiv:2105.07082.
  * [Dong et al., 2022] Dong, Z., Zhang, M., Li, F., and Chen, Y. (2022).  Pace: A parallelizable computation encoder for directed acyclic graphs.  arXiv preprint arXiv:2203.10304.
  * [Feng et al., 2021] Feng, J., Zhang, H., and Li, F. (2021).  Investigating the relevance of major signaling pathways in cancer survival using a biologically meaningful deep learning model.  BMC bioinformatics, 22(1):1–13.
  * [Fout, 2017] Fout, A. M. (2017).  Protein interface prediction using graph convolutional networks.  PhD thesis, Colorado State University.
  * [Griner et al., 2014] Griner, L. A. M., Guha, R., Shinn, P., Young, R. M., Keller, J. M., Liu, D., Goldlust, I. S., Yasgar, A., McKnight, C., Boxer, M. B., et al. (2014).  High-throughput combinatorial screening identifies drugs that cooperate with ibrutinib to kill activated b-cell–like diffuse large b-cell lymphoma cells.  Proceedings of the National Academy of Sciences, 111(6):2349–2354.
  * [Hamilton et al., 2017] Hamilton, W. L., Ying, R., and Leskovec, J. (2017).  Inductive representation learning on large graphs.  arXiv preprint arXiv:1706.02216.
  * [Henaff et al., 2015] Henaff, M., Bruna, J., and LeCun, Y. (2015).  Deep convolutional networks on graph-structured data.  arXiv preprint arXiv:1506.05163.
  * [Holbeck et al., 2017] Holbeck, S. L., Camalier, R., Crowell, J. A., Govindharajulu, J. P., Hollingshead, M., Anderson, L. W., Polley, E., Rubinstein, L., Srivastava, A., Wilsker, D., et al. (2017).  The national cancer institute almanac: a comprehensive screening resource for the detection of anticancer drug pairs with enhanced therapeutic activity.  Cancer research, 77(13):3564–3576.
  * [Hopkins, 2008] Hopkins, A. L. (2008).  Network pharmacology: the next paradigm in drug discovery.  Nature chemical biology, 4(11):682–690.
  * [Hornik, 1991] Hornik, K. (1991).  Approximation capabilities of multilayer feedforward networks.  Neural networks, 4(2):251–257.
  * [Hornik et al., 1989] Hornik, K., Stinchcombe, M., and White, H. (1989).  Multilayer feedforward networks are universal approximators.  Neural networks, 2(5):359–366.
  * [Janizek et al., 2018] Janizek, J. D., Celik, S., and Lee, S.-I. (2018).  Explainable machine learning prediction of synergistic drug combinations for precision cancer medicine.  BioRxiv, page 331769.
  * [Jin et al., 2011] Jin, G., Zhao, H., Zhou, X., and Wong, S. T. (2011).  An enhanced petri-net model to predict synergistic effects of pairwise drug combinations from gene microarray data.  Bioinformatics, 27(13):i310–i316.
  * [Kanehisa and Goto, 2000] Kanehisa, M. and Goto, S. (2000).  Kegg: kyoto encyclopedia of genes and genomes.  Nucleic acids research, 28(1):27–30.
  * [Kipf and Welling, 2016] Kipf, T. N. and Welling, M. (2016).  Semi-supervised classification with graph convolutional networks.  arXiv preprint arXiv:1609.02907.
  * [Kuru et al., 2021] Kuru, H. I., Tastan, O., and Cicek, E. (2021).  Matchmaker: a deep learning framework for drug synergy prediction.  IEEE/ACM Transactions on Computational Biology and Bioinformatics.
  * [Lee et al., 2019] Lee, J., Lee, I., and Kang, J. (2019).  Self-attention graph pooling.  In International conference on machine learning, pages 3734–3743. PMLR.
  * [Li et al., 2015] Li, P., Huang, C., Fu, Y., Wang, J., Wu, Z., Ru, J., Zheng, C., Guo, Z., Chen, X., Zhou, W., et al. (2015).  Large-scale exploration and analysis of drug combinations.  Bioinformatics, 31(12):2007–2016.
  * [Monti et al., 2017] Monti, F., Bronstein, M. M., and Bresson, X. (2017).  Geometric matrix completion with recurrent multi-graph neural networks.  arXiv preprint arXiv:1704.06803.
  * [Mott et al., 2015] Mott, B. T., Eastman, R. T., Guha, R., Sherlach, K. S., Siriwardana, A., Shinn, P., McKnight, C., Michael, S., Lacerda-Queiroz, N., Patel, P. R., et al. (2015).  High-throughput matrix screening identifies synergistic and antagonistic antimalarial drug combinations.  Scientific reports, 5(1):1–14.
  * [Niepert et al., 2016] Niepert, M., Ahmed, M., and Kutzkov, K. (2016).  Learning convolutional neural networks for graphs.  In International conference on machine learning, pages 2014–2023. PMLR.
  * [O’Neil et al., 2016] O’Neil, J., Benita, Y., Feldman, I., Chenard, M., Roberts, B., Liu, Y., Li, J., Kral, A., Lejnine, S., Loboda, A., et al. (2016).  An unbiased oncology compound screen to identify novel combination strategies.  Molecular cancer therapeutics, 15(6):1155–1162.
  * [Pan et al., 2017] Pan, R., Ruvolo, V., Mu, H., Leverson, J. D., Nichols, G., Reed, J. C., Konopleva, M., and Andreeff, M. (2017).  Synthetic lethality of combined bcl-2 inhibition and p53 activation in aml: mechanisms and superior antileukemic efficacy.  Cancer cell, 32(6):748–760.
  * [Parikh et al., 2016] Parikh, A. P., Täckström, O., Das, D., and Uszkoreit, J. (2016).  A decomposable attention model for natural language inference.  arXiv preprint arXiv:1606.01933.
  * [Podolsky and Greene, 2011] Podolsky, S. H. and Greene, J. A. (2011).  Combination drugs—hype, harm, and hope.  New England Journal of Medicine, 365(6):488–491.
  * [Preuer et al., 2018] Preuer, K., Lewis, R. P., Hochreiter, S., Bender, A., Bulusu, K. C., and Klambauer, G. (2018).  Deepsynergy: predicting anti-cancer drug synergy with deep learning.  Bioinformatics, 34(9):1538–1546.
  * [Radic-Sarikas et al., 2017] Radic-Sarikas, B., Tsafou, K. P., Emdal, K. B., Papamarkou, T., Huber, K. V., Mutz, C., Toretsky, J. A., Bennett, K. L., Olsen, J. V., Brunak, S., et al. (2017).  Combinatorial drug screening identifies ewing sarcoma–specific sensitivities.  Molecular Cancer Therapeutics, 16(1):88–101.
  * [Sanchez-Vega et al., 2018] Sanchez-Vega, F., Mina, M., Armenia, J., Chatila, W. K., Luna, A., La, K. C., Dimitriadoy, S., Liu, D. L., Kantheti, H. S., Saghafinia, S., et al. (2018).  Oncogenic signaling pathways in the cancer genome atlas.  Cell, 173(2):321–337.
  * [Schlichtkrull et al., 2018] Schlichtkrull, M., Kipf, T. N., Bloem, P., Berg, R. v. d., Titov, I., and Welling, M. (2018).  Modeling relational data with graph convolutional networks.  In European semantic web conference, pages 593–607. Springer.
  * [Schütt et al., 2017] Schütt, K. T., Kindermans, P.-J., Sauceda, H. E., Chmiela, S., Tkatchenko, A., and Müller, K.-R. (2017).  Schnet: A continuous-filter convolutional neural network for modeling quantum interactions.  arXiv preprint arXiv:1706.08566.
  * [Shuhendler et al., 2010] Shuhendler, A. J., Cheung, R. Y., Manias, J., Connor, A., Rauth, A. M., and Wu, X. Y. (2010).  A novel doxorubicin-mitomycin c co-encapsulated nanoparticle formulation exhibits anti-cancer synergy in multidrug resistant human breast cancer cells.  Breast cancer research and treatment, 119(2):255–269.
  * [Sidorov et al., 2019] Sidorov, P., Naulaerts, S., Ariey-Bonnet, J., Pasquier, E., and Ballester, P. J. (2019).  Predicting synergism of cancer drug combinations using nci-almanac data.  Frontiers in chemistry, page 509.
  * [Vaswani et al., 2017] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017).  Attention is all you need.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6000–6010.
  * [Velickovic et al., 2018] Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Lio’, P., and Bengio, Y. (2018).  Graph attention networks.  ArXiv, abs/1710.10903.
  * [Veličković et al., 2018] Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., and Bengio, Y. (2018).  Graph Attention Networks.  International Conference on Learning Representations.
  * [Verma and Zhang, 2018] Verma, S. and Zhang, Z.-L. (2018).  Graph capsule convolutional neural networks.  arXiv preprint arXiv:1805.08090.
  * [Wishart et al., 2018] Wishart, D. S., Feunang, Y. D., Guo, A. C., Lo, E. J., Marcu, A., Grant, J. R., Sajed, T., Johnson, D., Li, C., Sayeeda, Z., et al. (2018).  Drugbank 5.0: a major update to the drugbank database for 2018.  Nucleic acids research, 46(D1):D1074–D1082.
  * [Wu et al., 2010] Wu, Z., Zhao, X.-M., and Chen, L. (2010).  A systems biology approach to identify effective cocktail drugs.  In BMC systems biology, volume 4, pages 1–14. Springer.
  * [Xu et al., 2018] Xu, K., Hu, W., Leskovec, J., and Jegelka, S. (2018).  How powerful are graph neural networks?  arXiv preprint arXiv:1810.00826.
  * [Xu et al., 2011] Xu, K.-J., Hu, F.-Y., Song, J., and Zhao, X.-M. (2011).  Exploring drug combinations in a drug-cocktail network.  In 2011 IEEE International Conference on Systems Biology (ISB), pages 382–387. IEEE.
  * [Yan et al., 2020] Yan, S., Zheng, Y., Ao, W., Zeng, X., and Zhang, M. (2020).  Does unsupervised architecture representation learning help neural architecture search?  Advances in Neural Information Processing Systems, 33.
  * [Yang et al., 2018] Yang, J. C.-H., Mok, T., Han, B., Orlando, M., Puri, T., and Park, K. (2018).  A review of regimens combining pemetrexed with an epidermal growth factor receptor tyrosine kinase inhibitor in the treatment of advanced nonsquamous non–small-cell lung cancer.  Clinical lung cancer, 19(1):27–34.
  * [Yao et al., 2021] Yao, J., Arcila, M. E., Ladanyi, M., and Hechtman, J. F. (2021).  Pan-cancer biomarkers: Changing the landscape of molecular testing.  Archives of Pathology & Laboratory Medicine, 145(6):692–698.
  * [Yin et al., 2014] Yin, N., Ma, W., Pei, J., Ouyang, Q., Tang, C., and Lai, L. (2014).  Synergistic and antagonistic drug combinations depend on network topology.  PloS one, 9(4):e93960.
  * [Ying et al., 2018a] Ying, R., He, R., Chen, K., Eksombatchai, P., Hamilton, W. L., and Leskovec, J. (2018a).  Graph convolutional neural networks for web-scale recommender systems.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 974–983.
  * [Ying et al., 2018b] Ying, Z., You, J., Morris, C., Ren, X., Hamilton, W., and Leskovec, J. (2018b).  Hierarchical graph representation learning with differentiable pooling.  Advances in neural information processing systems, 31.
  * [Zhang et al., 2019] Zhang, H., Goodfellow, I., Metaxas, D., and Odena, A. (2019).  Self-attention generative adversarial networks.  In International conference on machine learning, pages 7354–7363. PMLR.
  * [Zhang and Chen, 2018] Zhang, M. and Chen, Y. (2018).  Link prediction based on graph neural networks.  In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 31, pages 5165–5175.
  * [Zhang and Chen, 2019] Zhang, M. and Chen, Y. (2019).  Inductive matrix completion based on graph neural networks.  arXiv preprint arXiv:1904.12058.
  * [Zhang et al., 2018] Zhang, M., Cui, Z., Neumann, M., and Chen, Y. (2018).  An end-to-end deep learning architecture for graph classification.  In Thirty-second AAAI conference on artificial intelligence.
  * [Zhang et al., 2021] Zhang, T., Zhang, L., Payne, P. R., and Li, F. (2021).  Synergistic drug combination prediction by integrating multiomics data in deep learning models.  In Translational Bioinformatics for Therapeutic Development, pages 223–238. Springer.
  * [Zitnik et al., 2018] Zitnik, M., Agrawal, M., and Leskovec, J. (2018).  Modeling polypharmacy side effects with graph convolutional networks.  Bioinformatics, 34(13):i457–i466.


# Transformers as Neural Augmentors: Class Conditional Sentence Generation via
Variational Bayes

M. Şafak Bilici  Department of Computer Engineering
Yildiz Technical University
Istanbul, Turkey
Email<EMAIL_ADDRESS>Mehmet Fatih Amasyali  Department of
Computer Engineering
Yildiz Technical University
Istanbul, Turkey
Email<EMAIL_ADDRESS>
###### Abstract

Data augmentation methods for Natural Language Processing tasks are explored
in recent years, however they are limited and it is hard to capture the
diversity on sentence level. Besides, it is not always possible to perform
data augmentation on supervised tasks. To address those problems, we propose a
neural data augmentation method, which is a combination of Conditional
Variational Autoencoder and encoder-decoder Transformer model. While encoding
and decoding the input sentence, our model captures the syntactic and semantic
representation of the input language with its class condition. Following the
developments in the past years on pre-trained language models, we train and
evaluate our models on several benchmarks to strengthen the downstream tasks.
We compare our method with 3 different augmentation techniques. The presented
results show that, our model increases the performance of current models
compared to other data augmentation techniques with a small amount of
computation power.

##  I Introduction

In recent years, remarkable performance of Transformer [1] based architectures
are shown on downstream tasks with or without pre-training. They are widely
used for downstream tasks as sentence classification, question answering,
coreference resolution. Besides that, using encoder-decoder Transformer model
allows process sequence-to-sequence tasks like machine translation,
summarization. In pre-training phase of these tasks, it is easy to collect a
corpus since pre-training is done in self-supervised fashion, generally.
However, at finetuning, it is required to collect significant number of
samples with labels. One way to increase number of training samples is data
augmentation. Nevertheless, data augmentation for NLP is not well-defined as
in computer vision. If we treat token values as just a numbers in input, we
reject the nature of the language. Hence, it is better to design more task-
oriented data augmentation methods in NLP with linguistic validity.

To address the sentence classification problem, we introduce a novel
generative data augmentation method for class conditional synthetic sentence
generation. Our model is a modified version of original Transformer model,
with a class conditional variational approximation between encoder and
decoder. After training class conditional variational Transformer, synthetic
sentences can be obtained with belonging desired class label. By doing this,
distribution of generated sentences is not too irrelevant from the original
sentences in desired class.

Figure 1: Model architecture of conditional variational Transformer.

Besides that, finetuning pre-trained models yet already computational
expensive; hence, a data augmentation method should require low computational
power. We show that our method works well with sub-sampling on used dataset.

We run our experiments on four datasets with different modifications and
approaches on our model.

##  II Related Work

###  II-A Data Augmentation for NLP

In recent years, data augmentation is studied for natural language tasks. A
traditional way to expand the existing data is using synonyms of words. This
paraphrasing method can be done by a large lexical databases of languages, or
by learning. As an example, in [2], authors experimented data augmentation by
using an English thesaurus. Also, there are several data augmentation methods
based on word level [3], [4]. Another way to use paraphrasing as data
augmentation is back-translation [5] which can be used to obtain more diverse
sentences as a neural data augmentation method. Following that, Transformers
are used as generative data augmentation technique [6]. In [7], authors
proposed two task specific data augmentation methods for Visual Question
Answering. Also, there are existing work on autoencoder based data
augmentation. Authors of [8] samples new sentences from a denoising
autoencoder with masked language corruption. A latent space based generative
approach is introduced in [9]. They use variational autoencoder [10] with
Gated Recurrent Unit [11].

###  II-B Text Based Variational Autoencoder

The idea of text based variational autoencoders is shown in [12] with a
combination of LSTM [13] and VAE. A more syntax informative text based VAE is
used in [14] with two distinct latent spaces. There are existing work on
controllable text generation from latent space [15], [16]. Prior to our work,
there are several Transformer based approaches. In [17], authors use
transformer based variational autoencoder with promising modifications. A
conditional design of variational Transformer is proposed in [18] for story
completion. Posterior collapse is common problem in text VAEs. This problem
occurs when the decoder ignores the latent vector and generation is done by
only decoder, like an auto-regressive language model. The study of preventing
posterior collapse is discussed in [19], [20] and [21]. We refer the reader to
[22], [19], for a detailed study on the properties of posterior collapse and
its relation to text based Variational Autoencoders.

##  III Proposed Method

###  III-A Background

An Autoencoder is a model for reconstructing the input. Reconstructing
includes encoding the input $\mathbf{x}$ into a smaller space, which is called
latent vector $z$, and decoding it to input space. The objective is to
minimize the metric loss $d(\cdot)$ between input $\mathbf{x}$ and
reconstructed variable $\hat{\mathbf{x}}$. When the objective is to represent
the latent variable $z$ in terms of a known distribution, Autoencoders are not
always working. If the distribution of $z$ is known, only the decoder of
Autoencoder is enough to generate new samples by passing $z$ as input.

As a solution for this problem, Variational Autoencoder is proposed in [10].
It is used for learning latent space representation via Variational Bayes.
Since we do not know the exact distribution of the input, our aim is to
approximate the unknown distribution to a known distribution. This procedure
is defined as Variational Bayes in literature. We thus choose a normal
distribution $p(z)\sim\mathcal{N}(0,\mathbf{I})$ as a known distribution, and
approximate distribution $p_{\text{encoder}}(\mathbf{z}\mid\mathbf{x}^{(i)})$
to this normal distribution at training time. Choosing simple and known
distribution helps us to generate samples from latent space $\mathbf{z}$ by
passing it to the probabilistic decoder $p_{\text{decoder}}$. Because, we
learn to encode the input to normal distribution, then decode it to same
space.

Thus, objective is to maximize the evidence lower bound (ELBO), which is
defined as

$\mathop{\mathbb{E}}[\log
p_{\text{decoder}}(\mathbf{x}^{(i)}\mid\mathbf{z})]-\mathbb{KL}(p_{\text{encoder}}(\mathbf{z}\mid\mathbf{x}^{(i)})\mid\mid
p(\mathbf{z}))$ (1)

where $KL$ refers to Kullback–Leibler divergence, $\mathbf{x}$ is the input
and $\mathbf{x}^{(i)}$ is the $i$th sample in minibatch $B$.
$p_{\text{encoder}}(\cdot)$ and $p_{\text{decoder}}(\cdot)$ are simply feed
forward layers in deep learning, for encoding and decoding.

###  III-B Modeling Class Conditional Variational Transformer

According to the Equation (1); we desing the Transformer model, introduced in
[1], by injecting conditional variational autoencoder between its encoder and
decoder. To formalize the idea, given a sequence of $N$ tokens,
$(t_{1},t_{2},...,t_{N})$, the encoder of the Transformer model is defined as
$Encoder:t_{i}\in\mathbb{R}^{N}\rightarrow\mathbf{c_{i}}\in\mathbb{R}^{N\times
D}$ where $D$ is the hidden dimension, and computes the contextual
representation $\mathbf{c}_{i}$ of each token $t_{i}$:

$\mathbf{c}_{1},\mathbf{c}_{2},...,\mathbf{c}_{N}=Encoder(t_{1},t_{2},...,t_{N})$
(2)

In the original Transformer model, contextual information from encoder flows
through to the decoder’s cross attention layer by mapping it to key and value.
However, recent state-of-the-art language models [23], [24], [25] show that
full sentence information can be obtained by only using contextual information
from first token $t_{1}$ (correspondingly the [CLS] token). In the same way,
we only use $\mathbf{c_{1}}$ token representation. The idea of conditional
variational autoencoder starts from vector $\mathbf{c}_{1}$. First, the
probabilistic encoder $p_{encoder}(\mathbf{z}\mid\mathbf{c}_{1})$ of CVAE
encodes sentence representation to a latent vector $\mathbf{z}$ with dimension
of $L$. Instead of conditioning only latent vector $\mathbf{z}$, decoder is
also conditioned to the class information of our sentence. This conditional
objective should be done by interacting latent vector with conditional
information. If we pass this class information to the decoder, the
reconstruction is conditioned both on the latent vector and class labels.
Prior work on conditional variational autoencoder sets an another condition
vector [26] and replace entries with condition features [27]. Instead of using
two vectors, we replace first entry of latent vector with class integer $C$ to
inject class information to decoding process. Denoting replaced vector
$\mathbf{z}$ as $\mathbf{z}^{\prime}$, decoder is reformulated as

$p_{decoder}(\mathbf{c}_{1}\mid\mathbf{z}^{\prime})$ (3)

Normally, the decoder of Transformer requires a whole sequence, however we use
only vector $\mathbf{c}_{1}$ as input to VAE’s encoder. After decoding in VAE,
the reconstructed variable is copied $N$ times to create a sequence like the
output of encoder of Transformer. This sequence of vectors is passed to the
decoder of Transformer as key and value.

Computationally, this formulation is more efficient than using whole
information from encoder’s output. It reduces the number of parameters of the
model. We experiment that if we use whole output of the encoder, the decoder
of Transformer tends to copy the sentences in training set. We thus believe
that this methods acts a role as regularizer. Besides that, authors of [28]
and [19] experimented that pooling the output of Transformer’s encoder
prevents posterior collapse. By following that, we use only $\mathbf{c}_{1}$
vector as a pooling strategy, to strengthen our model against posterior
collapse.

The decoder part of our Transformer has same architecture with original
Transformer. To make a formal definition between repeat vector
$\mathbf{c}_{i}^{\prime}$, the decoder computes logit vectors:

$\mathbf{l}_{1},\mathbf{l}_{2},...,\mathbf{l}_{N}=Decoder(\mathbf{x}^{\prime},\mathbf{c}_{i}^{\prime})$
(4)

where $\mathbf{x}^{\prime}$ is the shifted $\mathbf{x}$. These logits vectors
are simply our final predictions $p$ in Figure 1. Each logit corresponds to a
token from the vocabulary.

At inference time, conditional Transformer takes a random normal vector
$\mathbf{z}$ with a class label $z_{1}\leftarrow C$. This class label is
chosen by desired class of the sentence to be generated. Vector is passed to
decoder of VAE. At the same time, decoder of Transformer takes [START] token
to decode it autoregressively.

TABLE I: Results for IMDB and COLA on different models

Datasets | BERT | ALBERT | XLNet | DistilBERT
---|---|---|---|---
Baseline | Ours | Baseline | Ours | Baseline | Ours | Baseline | Ours
IMDB | 79.96 $\pm$ 0.686 | 80.82 $\pm$ 0.379 | 78.74 $\pm$ 0.17 | 79.28 $\pm$ 0.23 | - | - | 79.24 $\pm$ 0.11 | 80.05 $\pm$ 0.066
CoLA | 70.31 $\pm$ 0.103 | 70.73 $\pm$ 0.109 | - | - | 68.68 $\pm$ 0.438 | 68.9 $\pm$ 0.331 | 68.8 $\pm$ 0.273 | 69.61 $\pm$ 0.391

##  IV Experimental Setup

We evaluate our model on different benchmarks with different pre-trained
language models.

###  IV-A Benchmarks

TABLE II: different number of sentences based on different train samples with
Different Data Augmentation Methods

Dataset | # Train Samples | # Test Samples | # Generated Sentences
---|---|---|---
IMDB | 25000 | 25000 | 5000
CoLA | 8551 | 1043 | 5000
Rotten Tomatoes | 8530 | 1066 |  | 10000
---
(FWS, FWS + R, R)
TREC-10 | 5452 | 500 |  | 6000
---
(FWS, FWS + R, R)

####  IV-A1 CoLA

The Corpus of Linguistic Acceptability (CoLA) is one of the presented
benchmarks in General Language Understanding Evaluation (GLUE) [29]. The task
is to determine whether given sentence is a grammatically correct English
sentence.

####  IV-A2 TREC-10

TREC-10 [30] is a collection of questions that are annotated with its question
type. It is a multiclass classification task with class labels of
abbreviation, human, entity, location, numeric, description.

####  IV-A3 Rotten Tomatoes

Rotten Tomatoes [31] is a movie review dataset which is annotated for binary
classification on highly abstract sentiment representations.

####  IV-A4 IMDB

IMDB dataset [32] is a large collection of movie review dataset. The task is
to classify whether given sentence is positive review or not.

###  IV-B Sentence Generation Techniques

We follow different sentence generation techniques on different benchmarks for
data augmentation. For example, we generate 5000 different sentences for CoLA
dataset by using all training samples. On the other hand, since IMDB dataset
contains 25000 training samples, we perform random sub-sampling on this
dataset before training due to computation time characteristic of data
augmentation techniques.

This random generation process is done by passing [START] token to the
decoder, then decoding it in autoregressive fashion until seeing [END] token.
We see that this generated sentences without any rule tends to begin with the
most frequent starting words (for example ”A”, ”The”, ”I”) in trained dataset.

TABLE III: Results for Rotten Tomatoes and TREC-10 dataset on BERT model

Datasets | Baseline | Synonym Replacement | Contextual Replacement | Back-Translation | $\mathbf{R}$ | $\mathbf{FWS+R}$ | $\mathbf{FWS}$
---|---|---|---|---|---|---|---
Rotten Tomatoes | 74.27 $\pm$ 0.357 | 74.73 $\pm$ 0.622 | 75.77 $\pm$ 0.162 | 75.84 $\pm$ 0.649 | 75.58 $\pm$ 0.746 | 76.08 $\pm$ 0.425 | 76.3 $\pm$ 0.324
TREC-10 | 75.8 $\pm$ 0.113 | 78.1 $\pm$ 0.487 | 78.87 $\pm$ 0.523 | 79.2 $\pm$ 0.392 | 78.9 $\pm$ 0.408 | 79.73 $\pm$ 0.471 | 79.87 $\pm$ 0.57

To measure importance and contribution of first word, we choose a third
sentence generation technique: pre-sampling first word of a generated
sentence. To select less frequent words, we calculate frequency of each
starting word and normalize it to get their density function. Then, we raised
this distribution to the $3/4$rd power empirically. This allows us to generate
sentences that are relatively distinct from characteristics of current
sentences in dataset by passing {[START], $sample(p_{w_{1}},...,p_{w_{S}})$}
to the decoder at inference time.

However, since the aim is to generate a sentence that belongs to the class
$C$, third technique is not sufficient for such datasets. For example, if we
want to generate a sample belongs to class ”human” in TREC-10 dataset, it is
not quite right to start the sentence with ”where”. On account of this, for
TREC-10, we sample the first word from distinct distributions related to
desired class of sentence to be generated.

As shown in Table II, different number of sentences that are generated based
on different train samples. On the other hand, three set of sentences are
generated for Rotten Tomates and TREC-10. First Word Sampling ($FWS$) stands
for first word of all generated sentences are sampled from described
distribution. Random ($R$) stands for sentences are generated randomly,
without any pre-determined set of rules. At last, First Word Sampling + Random
($FWS+R$) means half of the generated sentences are $FWS$, other half is $R$.

All generation techniques include class conditional latent vector.

Algorithm 1 Class Conditional Sentence Generation

Input: model, $C$, use_fws
Output: a new sentence which belongs to class $C$

1:sample $z$ from $\mathcal{N}(0,\mathbf{I})$

2:$z[1]$ $\leftarrow C$

3:vae_output $\leftarrow$ model.vae_decoder($z$)

4:if use_fws == True then

5: $fw\leftarrow$ sample a first word from normalized frequencies

6:else

7: $fw\leftarrow$ None

8:end if

9:new_sentence $\leftarrow$ model.transformer_decoder(vae_output, $fw$)

10:return new_sentence

##  V Results

In this section, results are reported on 4 different benchmarks as stated in
Section IV. For IMDB, TREC-10 and Rotten Tomatoes; results are reported on
test set. For CoLA, we use dev set. To show the performance and contribution
of our model, we finetune several pre-trained models ([33], [34], [35], [36])
on original datasets and augmented datasets to compare. We run each experiment
three times and calculate the margin of error with 95% confidence level to
provide statistically significant evidence. We generate an equal number of
sentences for each class.

For IMDB and CoLA, we finetune BERT [23] and DistilBERT [25]. Additionaly, we
finetune ALBERT [37] for IMDB; XLNet [38] for CoLA.

Table I shows accuracy and margin of error values on IMDB and CoLA dataset
using different pre-trained models. For CoLA dataset, we train conditional
variational Transformer and generate 5000 samples to expand the dataset.
Although our method increases the performance compared to training with
original dataset, this increase is very small. Since linguistic acceptability
is affected negatively from very small changes, we believe that this is due to
characteristics of the dataset.

On IMDB dataset, we train conditional variational Transformer with randomly
chosen 7000 samples over 25000 samples. After that, 5000 samples are generated
to expand the dataset. As shown in Table I, this augmentation method has
considerable performance increase compared to CoLA experiments.

We choose three different augmentation techniques to compare with our method
on TREC-10 and Rotten Tomatoes. The first method is to replace a randomly
chosen work with its synonym on WordNet [39]. The second one is to replace a
randomly chosen word with most similar word using contextual word embeddings
[40]. The third method is back-translation. Sentences are translated into
German and then back into the English [41].

For each method, the same numbers of sentences are generated as in Table II.
Each sample is augmented in train dataset by iterating, if the size of
augmentation is larger than number of samples in train set, we choose random
samples to complete.

Table III shows accuracy and margin of error values on TREC-10 and Rotten
Tomatoes datasets using only BERT model. For these two dataset, we prepare 3
different augmented datasets based on different generation techniques ($R$,
$FWS+R$, $FWS$) as explained in Section IV. Experiments show that back-
translation outperforms synonym and contextual word replacement. For both
datasets, random generation ($R$) is below the other methods in terms of
accuracy.

There is considerable performance increase if we generate sequences with first
word sampling ($FWS+R$ and $FWS$) compared with baseline and other
augmentation methods.

##  VI Discussions

We propose a novel generative data augmentation method, which can generate new
sentences from desired class label. Our proposed model has several advantages
over other augmentation methods.

If a generative model for data augmentation generates sentences without any
class condition, there can be a corruption between class distributions.
Conditional variational Transformer keeps class information while encoding
semantic and syntactic properties of language on both binary and multiclass
classification datasets.

Generation of noised and corrupted sentences is inevitable outcome when it
comes to sentence generation. We believe that this sentences plays a role as
regularizer besides high quality generated sentences.

Conditional variational Transformer retains the advantages of generative
language models. We can generate a sentence from given sequence with class
conditionality. We calculate class based first word frequencies to generate
new sentences over less frequent words. Our experiments shows this approach
outperforms any random data generation procedure.

Experiments states that, our method performs well if the number of samples in
dataset is limited. We double the number of samples in Rotten Tomatoes and
TREC-10 datasets. This suggests that diversity can also be increased in small
datasets. Examples of sentences produced are given in Table IV in appendix.

##  VII Future Work

Since we apply our method to classification tasks, there are several tasks in
NLP. We believe that our method can be expanded to other tasks such as
sequence labeling, textual entailment 111Our models and promising future work
implementations are publicly available at
https://github.com/safakkbilici/Conditional-Variational-Transformer..

On the other hand, we don’t pre-train any models before training on presented
benchmarks. It is possible to encode more linguistic information via
unconditional latent space learning on big corpus. We plan to investigate the
pre-training objective and performance on downstream tasks in future work.

##  VIII Acknowledgment

This research is supported by TUBITAK, The Scientific and Technological
Research Council of Turkey with project number 120E100. Also, we would like to
thank to inzva community for providing us a valuable research environment and
GPU support.

## References

  * [1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems_ , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017\.
  * [2] X. Zhang, J. Zhao, and Y. LeCun, “Character-level convolutional networks for text classification,” 2016.
  * [3] Z. Xie, S. I. Wang, J. Li, D. Lévy, A. Nie, D. Jurafsky, and A. Y. Ng, “Data noising as smoothing in neural network language models,” 2017.
  * [4] J. Wei and K. Zou, “EDA: Easy data augmentation techniques for boosting performance on text classification tasks,” in _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 6382–6388. [Online]. Available: https://aclanthology.org/D19-1670
  * [5] R. Sennrich, B. Haddow, and A. Birch, “Improving neural machine translation models with monolingual data,” 2016.
  * [6] Y. Yang, C. Malaviya, J. Fernandez, S. Swayamdipta, R. Le Bras, J.-P. Wang, C. Bhagavatula, Y. Choi, and D. Downey, “Generative data augmentation for commonsense reasoning,” in _Findings of the Association for Computational Linguistics: EMNLP 2020_. Online: Association for Computational Linguistics, Nov. 2020, pp. 1008–1025. [Online]. Available: https://aclanthology.org/2020.findings-emnlp.90
  * [7] K. Kafle, M. Yousefhussien, and C. Kanan, “Data augmentation for visual question answering,” in _Proceedings of the 10th International Conference on Natural Language Generation_. Santiago de Compostela, Spain: Association for Computational Linguistics, Sep. 2017, pp. 198–202. [Online]. Available: https://aclanthology.org/W17-3529
  * [8] N. Ng, K. Cho, and M. Ghassemi, “SSMBA: Self-supervised manifold based data augmentation for improving out-of-domain robustness,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Online: Association for Computational Linguistics, Nov. 2020, pp. 1268–1283. [Online]. Available: https://aclanthology.org/2020.emnlp-main.97
  * [9] M. . Bilici and M. F. Amasyali, “Variational sentence augmentation for masked language modeling,” in _2021 Innovations in Intelligent Systems and Applications Conference (ASYU)_ , 2021, pp. 1–5.
  * [10] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” 2014.
  * [11] K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” 2014.
  * [12] S. R. Bowman, L. Vilnis, O. Vinyals, A. Dai, R. Jozefowicz, and S. Bengio, “Generating sentences from a continuous space,” in _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_. Berlin, Germany: Association for Computational Linguistics, Aug. 2016, pp. 10–21. [Online]. Available: https://aclanthology.org/K16-1002
  * [13] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [14] X. Zhang, Y. Yang, S. Yuan, D. Shen, and L. Carin, “Syntax-infused variational autoencoder for text generation,” in _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Florence, Italy: Association for Computational Linguistics, Jul. 2019, pp. 2069–2078. [Online]. Available: https://aclanthology.org/P19-1199
  * [15] Z. Hu, Z. Yang, X. Liang, R. Salakhutdinov, and E. P. Xing, “Toward controlled generation of text,” 2018.
  * [16] T. Zhao, R. Zhao, and M. Eskenazi, “Learning discourse-level diversity for neural dialog models using conditional variational autoencoders,” 2017.
  * [17] D. Liu and G. Liu, “A transformer-based variational autoencoder for sentence generation,” in _2019 International Joint Conference on Neural Networks (IJCNN)_ , 2019, pp. 1–7.
  * [18] T. Wang and X. Wan, “T-cvae: Transformer-based conditioned variational autoencoder for story completion,” in _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_. International Joint Conferences on Artificial Intelligence Organization, 7 2019, pp. 5233–5239. [Online]. Available: https://doi.org/10.24963/ijcai.2019/727
  * [19] S. Park and J. Lee, “Finetuning pretrained transformers into variational autoencoders,” in _Proceedings of the Second Workshop on Insights from Negative Results in NLP_. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 29–35. [Online]. Available: https://aclanthology.org/2021.insights-1.5
  * [20] T. Long, Y. Cao, and J. C. K. Cheung, “Preventing posterior collapse in sequence vaes with pooling,” _ArXiv_ , vol. abs/1911.03976, 2019.
  * [21] C. Li, X. Gao, Y. Li, B. Peng, X. Li, Y. Zhang, and J. Gao, “Optimus: Organizing sentences via pre-trained modeling of a latent space,” 2020.
  * [22] J. Lucas, G. Tucker, R. B. Grosse, and M. Norouzi, “Understanding posterior collapse in generative latent variable models,” in _DGS@ICLR_ , 2019.
  * [23] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186. [Online]. Available: https://aclanthology.org/N19-1423
  * [24] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” 2019.
  * [25] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter,” 2020.
  * [26] Z. Hu, Z. Yang, X. Liang, R. Salakhutdinov, and E. P. Xing, “Toward controlled generation of text,” 2017. [Online]. Available: https://arxiv.org/abs/1703.00955
  * [27] J. Lim, S. Ryu, J. W. Kim, and W. Y. Kim, “Molecular generative model based on conditional variational autoencoder for de novo molecular design,” _Journal of Cheminformatics_ , vol. 10, no. 1, Jul. 2018. [Online]. Available: https://doi.org/10.1186/s13321-018-0286-7
  * [28] T. Long, Y. Cao, and J. C. K. Cheung, “On posterior collapse and encoder feature dispersion in sequence vaes,” 2019. [Online]. Available: https://arxiv.org/abs/1911.03976
  * [29] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” in _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_. Brussels, Belgium: Association for Computational Linguistics, Nov. 2018, pp. 353–355. [Online]. Available: https://aclanthology.org/W18-5446
  * [30] X. Li and D. Roth, “Learning question classifiers,” in _COLING 2002: The 19th International Conference on Computational Linguistics_ , 2002. [Online]. Available: https://aclanthology.org/C02-1150
  * [31] B. Pang and L. Lee, “Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales,” in _Proceedings of the ACL_ , 2005.
  * [32] A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts, “Learning word vectors for sentiment analysis,” in _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_. Portland, Oregon, USA: Association for Computational Linguistics, June 2011, pp. 142–150. [Online]. Available: http://www.aclweb.org/anthology/P11-1015
  * [33] “bert-base-uncased,” https://huggingface.co/bert-base-uncased, accessed: 2021-12-23.
  * [34] “distilbert-base-uncased,” https://huggingface.co/distilbert-base-uncased, accessed: 2021-12-23.
  * [35] “xlnet-base-cased,” https://huggingface.co/xlnet-base-cased, accessed: 2021-12-23.
  * [36] “albert-base-v1,” https://huggingface.co/albert-base-v1, accessed: 2021-12-23.
  * [37] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “Albert: A lite bert for self-supervised learning of language representations,” 2020.
  * [38] Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” in _Advances in Neural Information Processing Systems_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, Inc., 2019.
  * [39] G. A. Miller, “Wordnet: A lexical database for english,” _Commun. ACM_ , vol. 38, no. 11, p. 39–41, nov 1995. [Online]. Available: https://doi.org/10.1145/219717.219748
  * [40] “Roberta base model,” https://huggingface.co/roberta-base, accessed: 2021-12-23.
  * [41] “facebook/wmt19-en-de,” https://huggingface.co/facebook/wmt19-en-de, accessed: 2021-12-23.
  * [42] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 2017.
  * [43] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, pp. 2818–2826.

TABLE IV: randomly chosen generated sentences for each dataset and class label

Datasets | $\mathbf{R}$ | $\mathbf{FWS}$
---|---|---
IMDB |  |  Negative (0):
---
This is a weak film I have rarely seen many of movies but
there was no plot to the same plot and as this film is simply awful.
The acting is poor the the camera work is just the nonexistent.
The acting is awful the cinematography is awful the only acceptable acting is
pathetic.
Positive (1):
This is a great movie I gave it a 10 out of 10 then that it is a little more
realistic movie
than most of the actors are great actors and actresses like they added a
shooting story.
The screenplay is very well done. The story in this movie is very well.
-
CoLA |  |  Unacceptable (0):
---
The boys guardians employer we elected president.
Acceptable (1):
The students demonstrated the technique this morning.
-
Rotten Tomatoes |  |  Negative (0):
---
The movie is a negligible work of manipulation but its so mechanical.
Positive (1): A shimmeringly lovely comingofage portrait of family and finally
roll
that could have viewers right out.
|  Negative (0):
---
Ordinary melodrama with weak dialogue and exactly as the direction of its lead
performances.
Positive (1):
Showtime is a film thats destined to win a documentary that is a certain
degree to raise.
TREC-10 |  |  Description (0):
---
What is the difference between pop music to drink?
Entity (1):
What is the correct way to use with James home with letter?
Abbreviation (2):
What is the abbreviation of the computer at General Motors?
Human (3):
Who is the first Taiwanese President to be popular sleep?
Numeric (4):
What is the temperature for having a typist to a beach?
Location (5):
What is the name of the Wilkes plantation in the Ewoks live on
|  Description (0):
---
How can I find out how much much do food if I know their questions?
Entity (1):
What is the best way to remove?
Abbreviation (2):
When reading classified ads what does EENTY other stand for?
Human (3):
What was the name of the American who was known for a senator in The Roman
Black swearing and 50s when?
Numeric (4):
When did the art live?
Location (5):
Where can I find a website about climbs of Mount Everest and sailors monument?

TABLE V: Variable Hyperparameters for Each Dataset. MSL stands for Maximum Sequence Length. Dataset | MSL | Batch Size | Epochs | Learning Rate
---|---|---|---|---
IMDB | 100 | 32 | 50 | 1E-4
CoLA | 70 | 64 | 95 | 2E-4
Rotten Tomatoes | 100 | 64 | 95 | 2E-4
TREC-10 | 50 | 64 | 70 | 2E-4

###  -A Generated Sentence Examples

In this section, we show one randomly chosen generated sentence for each
dataset and their class labels. For Rotten Tomatoes and TREC-10, $FWS$ samples
are shown as well. In Table IV, it is clear that our model can distinguish
negative and positive sentiment for IMDB and Rotten Tomatoes datasets.

If we examine the characteristics of movie review datasets, it has high
probability to start a sentence with ”this movie” phrase. With introducing
First Word Sampling approach $PWS$, diverse reviews can be generated.

On the other hand, our model captures the logical representation of question
types. Even though the TREC-10 dataset has limited number of train samples,
our model is capable to generate sentences which are not seen in train set.

###  -B Hyperparameters & Training Details

We use same hyperparameters for each dataset. Dimensions of keys, queries and
values are set to 64. Dimension of word vectors $d_{model}$ is set to 256. We
use 3 layers for encoder and decoder. For each multi-head attention layer,
number of heads is set to 8. We apply dropout with rate of 0.1. We don’t use
any learning rate scheduler. Adam optimizer [42] is used with $\beta_{1}=0.9$,
$\beta_{2}=0.999$. At training time, we apply label smoothing [43] with
$\epsilon_{ls}=0.1$. Other hyperparameters are given in Table V.

To prevent KL cost annealing, we follow the same technique in [12]. We
introduce a variable weight $w_{0}$ for KL term in the loss, then the weight
is increased gradually from 0 to 1 at training. We update value of the weight
with a simple logistic function

$w_{0}=\frac{1}{1+\exp(-k\cdot(t-x_{0}))}$ (5)

where $k=0.0025$, $x_{0}=2500$ and $t$ is the current step.