# Data-Free Adversarial Knowledge Distillation for Graph Neural Networks

Yuanxin Zhuang1 Lingjuan Lyu2 Chuan Shi1111Corresponding Author. Carl
Yang3&Lichao Sun4 1Beijing University of Posts and Telecommunications
2Sony AI 3Emory University 4Lehigh University {zhuangyuanxin,
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Graph neural networks (GNNs) have been widely used in modeling graph
structured data, owing to its impressive performance in a wide range of
practical applications. Recently, knowledge distillation (KD) for GNNs has
enabled remarkable progress in graph model compression and knowledge transfer.
However, most of the existing KD methods require a large volume of real data,
which are not readily available in practice, and may preclude their
applicability in scenarios where the teacher model is trained on rare or hard
to acquire datasets. To address this problem, we propose the first end-to-end
framework for data-free adversarial knowledge distillation on graph structured
data (DFAD-GNN). To be specific, our DFAD-GNN employs a generative adversarial
network, which mainly consists of three components: a pre-trained teacher
model and a student model are regarded as two discriminators, and a generator
is utilized for deriving training graphs to distill knowledge from the teacher
model into the student model. Extensive experiments on various benchmark
models and six representative datasets demonstrate that our DFAD-GNN
significantly surpasses state-of-the-art data-free baselines in the graph
classification task.

##  1 Introduction

An increasing number of machine learning tasks require dealing with a large
amount of graph data, which capture rich and complex relationships among
potentially billions of elements. Graph Neural Networks (GNNs) have become an
effective way to address the graph learning problem by converting the graph
data into a low dimensional space while keeping both the structural and
property information to the maximum extent. Recently, the rapid evolution of
GNNs has led to a growing number of new architectures as well as novel
applications Lu et al. (2021); Sun et al. (2018); Wu et al. (2021).

However, training a powerful GNN often requires heavy computation and storage.
Hence it is hard to deploy them into resource-constrained devices, such as
mobile phones. There has been a large literature Bahri et al. (2021) aiming to
compress and speed-up the cumbersome GNNs into lightweight ones. Among those
methods, knowledge distillation Hinton et al. (2015) is one of the most
popular paradigms for learning a portable student model from the pre-trained
complicated teacher by directly imitating its outputs.

Knowledge distillation (KD) is proposed by Hinton et al. Hinton et al. (2015)
for supervising the training of a compact yet efficient student model by
capturing and transferring the knowledge from a large complicated teacher
model. KD has received significant attention from the community in the recent
years Yang et al. (2021); Gou et al. (2021). Despite its successes, KD in its
classical form has a critical limitation. It assumes that the real training
data is still available in the distillation phase. However, in practice, the
original training data is often unavailable due to privacy concerns. Besides,
many large models are trained on millions or even billions of graphs Lu et al.
(2021). While the pre-trained models might be made available for the community
at large, making training data available also poses a lot of technical and
policy challenges.

An effective way to avert the above-mentioned issue is using the synthetic
graphs, i.e., data-free knowledge distillation Lopes et al. (2017); Liu et al.
(2021). Just as “data free” implies, there is no training data. Instead, the
data is reversely generated from the pre-trained models. Data-free
distillation has received a lot of attention in the field of computer vision
Fang et al. (2019); Lopes et al. (2017); Fang et al. (2021), which is however
rarely been explored in graph mining. Note that Deng et al. Deng and Zhang
(2021) have made some pilot studies on this problem and proposed graph-free
knowledge distillation (GFKD). Unfortunately, GFKD is not an end-to-end
approach. It only takes the fixed teacher model into account, ignoring the
information from the student model when generating graphs. Moreover, their
method is based on the assumption that an appropriate graph usually has a high
degree of confidence in the teacher model. In fact, the model maps the graphs
from the data space to a very small output space, thus losing a large amount
of information. Therefore, these generated graphs are not very useful for
distilling the teacher model efficiently which leads to an unsatisfactory
performance.

In this work, we propose a novel data-free adversarial knowledge distillation
framework for GNNs (DFAD-GNN). DFAD-GNN uses a knowledge distillation method
based on GAN Goodfellow et al. (2014). As illustrated in Figure 1, DFAD-GNN
contains one generator and two discriminators: one fixed discriminator is the
pre-trained teacher model, the other is the compact student model that we aim
to learn. The generator generates graphs to help transfer teachers’ knowledge
to students. Unlike previous work Deng and Zhang (2021), our generator can
fully utilize both the intrinsic statistics from the pre-trained teacher model
and the customizable information from the student model, which can help
generate high quality and diverse training data to improve the student model’s
generalizability. The contributions of our proposed framework can be
summarized as follows: 1) We study a valuable yet intractable problem: how to
distill a portable and efficient student model from a pre-trained teacher
model when the original training data is not available; 2) We propose a novel
data-free adversarial knowledge distillation framework for GNNs (DFAD-GNN) in
order to train a compact student model using the generated graphs and a fixed
teacher model. To the best of our knowledge, DFAD-GNN is the first end-to-end
framework for data-free knowledge distillation on graph structured data; and
3) Extensive experiments demonstrate that our proposed framework significantly
outperforms existing state-of-the-art data-free methods. DFAD-GNN can
successfully distill a student model with 81.8%-94.7% accuracy of the teacher
model on all six datasets.

##  2 Preliminary and Related Work

###  2.1 Graph Neural Networks

Graph Neural Networks (GNNs) have received considerable attention for a wide
variety of tasks Wu et al. (2020); Zhou et al. (2020). Generally, GNN models
can be unified by a neighborhood aggregation or message passing schema Gilmer
et al. (2017), where the representation of each node is learned by iteratively
aggregating the embeddings (“message”) of its neighbors.

As one of the most influential GNN models, Graph Convolutional Network (GCN)
Kipf and Welling (2016) performs a linear approximation to graph convolutions.
Graph Attention Network (GAT) Veličković et al. (2017) introduces an attention
mechanism that allows weighing nodes in the neighborhood differently during
the aggregation step. GraphSAGE Hamilton et al. (2017) is a comprehensive
improvement on the original GCN which replaced full graph Laplacian with
learnable aggregation functions. Graph Isomorphism Network (GIN) Xu et al.
(2018) uses a simple but expressive injective multiset function for neighbor
aggregation. These GNNs above will be employed as our teacher models and
student models in the experiments.

###  2.2 Knowledge Distillation

Knowledge distillation (KD) aims to transfer the knowledge of a (larger)
teacher model to a (smaller) student model Hinton et al. (2015); Wu et al.
(2022). It was originally introduced to reduce the size of models deployed on
devices with limited computational resources. Since then, this line of
research has attracted a lot of attention Gou et al. (2021). Recently, there
are a few attempts that try to combine knowledge distillation with graph
convolutional networks (GCNs). Yang et al. Yang et al. (2021) proposed a
knowledge distillation framework which can extract the knowledge of an
arbitrary teacher model and inject it into a well-designed student model. Jing
et al. Jing et al. (2021) proposed to learn a lightweight student GNN that
masters the complete set of expertise of multiple heterogeneous teachers.
These works aim to improve the performance of student models on semi-
supervised node classification task, rather than the graph classification task
we considered in this work. Furthermore, although the above-mentioned methods
obtained promising results, they cannot be effectively launched without the
original training dataset. In practice, the training dataset could be
unavailable for some reasons, e.g. transmission limitations, privacy, etc.
Therefore, it is necessary to consider a data-free approach for compressing
neural networks.

Techniques addressing data-free knowledge distillation have relied on training
a generative model to synthesize fake data. One recent work named graph-free
KD (GFKD) Deng and Zhang (2021) has proposed a data-free knowledge
distillation for graph neural network. GFKD learns graph topology structures
for knowledge distillation by modeling them with a multinomial distribution.
The training processes include two independent steps: (1) first, it uses a
pre-trained teacher model to generate fake graphs; (2) afterwards, it uses
these fake graphs to distill knowledge into the compact student model.
However, those fake graphs are optimized by an invariant teacher model without
considering the student model. Therefore, they are not very useful for
distilling the student model efficiently. In order to generate high quality
diverse training data to improve the student model’s generalizability, we
propose a data-free adversarial knowledge distillation framework for GNNs
(DFAD-GNN). Our generator is trained end-to-end which not only uses the pre-
trained teacher’s intrinsic statistics but also obtains the discrepancy
between teacher model and student model. We remark that the key differences
between GFKD and our model lies in the training process and the generator.

###  2.3 Graph Generation

Data-free knowledge distillation involves the generation of training data.
Motivated by the power of Generative Adversarial Networks (GANs) Goodfellow et
al. (2014), researchers have used them for generating graphs. Bojchevski et
al. proposed NetGAN Bojchevski et al. (2018), which uses the GAN framework to
generate random walks on graphs. De Cao and Kipf proposed MolGAN De Cao and
Kipf (2018), which generates molecular graphs using a simple multi-layer
perceptron (MLP).

In this work, we build on a min-max game between two adversaries who try to
optimize opposite loss functions. This approach is analogous to the
optimization performed in GANs to train the generator and discriminator. The
key difference is that GANs are generally trained to recover an underlying
fixed data distribution. However, our generator chases a moving target: the
distribution of data which is most indicative of the discrepancies of the
current student model and its teacher model.

Figure 1: The framework of DFAD-GNN.

##  3 DFAD-GNN Framework

As shown in Figure 1, DFAD-GNN mainly consists of three components: one
generator and two discriminators. One fixed discriminator is the pre-trained
teacher model $\mathcal{T}$, the other is the compact student model
$\mathcal{S}$ that we aim to learn. More specifically, the generator
$\mathcal{G}$ takes samples $z$ from a prior distribution and generates fake
graphs. Then the generated graphs are used to train a student model under the
supervision of the teacher model.

###  3.1 Generator

The generator $\mathcal{G}$ is used to synthesize fake graphs that maximize
the disagreement between the teacher $\mathcal{T}$ and the student
$\mathcal{S}$. $\mathcal{G}$ takes $D$-dimensional vectors
${z}\in\mathbb{R}^{D}$ sampled from a standard normal distribution
${z}\sim\mathcal{N}(0,I)$ and outputs graphs. For each $z$, $\mathcal{G}$
outputs an object: ${F}\in\mathbb{R}^{N\times T}$ that defines node features,
where $N$ is the node number and $T$ is node feature dimension. Then we
calculate adjacency matrix ${{A}}$ as follows:

${{A}}=\sigma\left({F}{F}^{\top}\right),$ (1)

where $\sigma(\cdot)$ is the logistic sigmoid function. We transform the range
of $A$ from $[0,1]$ to $\\{0,1\\}$ with a threshold $\tau$. If the element in
$A$ is larger than $\tau$, it is set as 1, otherwise 0.

The loss function used for $\mathcal{G}$ is the same as that used for
$\mathcal{S}$, except that the goal is to maximize it. In other words, the
student is trained to match the teacher’s predictions and the generator is
trained to generate difficult graphs for the student. The training process can
be formulated as:

$\max_{\mathcal{G}}\min_{\mathcal{S}}\mathbb{E}_{z\sim\mathcal{N}(0,1)}[\mathcal{D}(\mathcal{T}(\mathcal{G}(z)),\mathcal{S}(\mathcal{G}(z)))],$
(2)

where $\mathcal{D}\left(\cdot\right)$ indicates the discrepancy between the
teacher $\mathcal{T}$ and the student $\mathcal{S}$. If generator keeps
generating simple and duplicate graphs, student model will fit these graphs,
resulting in a very low model discrepancy between student model and teacher
model. In this case, the generator is forced to generate difficult and
different graphs to enlarge the discrepancy.

###  3.2 Adversarial Distillation

Overall, the adversarial training process consists of two stages: the
distillation stage that minimizes the discrepancy $\mathcal{D}$; and the
generation stage that maximizes the discrepancy $\mathcal{D}$, as shown in
Figure 1. We detail each stage as follows.

Algorithm 1 DFAD-GNN

Input: A pre-trained teacher model,$\mathcal{T}\left(X;\theta^{t}\right)$
Output: A comparable student model,$\mathcal{S}\left(X;\theta^{s}\right)$

1: Randomly initialize a student model $\mathcal{S}\left(X;\theta^{s}\right)$
and a generator $\mathcal{G}\left(z;\theta^{g}\right)$

2: for Epochs do

3: // Distillation Stage

4: for $k$ steps do

5: Generate graphs $X$ from $z$ with $\mathcal{G}\left(z;\theta^{g}\right)$

6: Calculate model discrepancy with $\mathcal{L}_{DIS}$

7: Update $\theta^{s}$ to minimize discrepancy with
$\nabla_{\theta^{s}}\mathcal{D}$

8: end for

9: //Generation Stage

10: Generate graphs $X$ from $z$ with $\mathcal{G}\left(z;\theta^{g}\right)$;

11: Calculate negative discrepancy with $\mathcal{L}_{GEN}$

12: Update $\theta^{g}$ to maximize discrepancy with
$\nabla_{\theta^{g}}-\mathcal{D}$

13: end for

####  3.2.1 Distillation Stage

In this stage, we fix the generator $\mathcal{G}$ and only update the student
$\mathcal{S}$ in the discriminator. We sample a batch of random noises ${z}$
and construct fake graphs with generator $\mathcal{G}$. Then each graph ${X}$
is fed to both the teacher and the student model to produce the output $q^{t}$
and $q^{s}$, where $q$ is a vector indicating the scores of different
categories.

In our approach, the choice of loss involves similar factors to those outlined
in GANs: multiple works have discussed the problem of vanishing gradients as
the discriminator becomes strong in case of GAN training Arjovsky and Bottou
(2017). Most prior work in model distillation optimized over the
KullbackLeibler Divergence (KLD) and Mean Square Error (MSE) between the
student and the teacher model. However, as the student model matches more
closely the teacher model, these two loss functions tend to suffer from
vanishing gradients. Specifically, back-propagating such vanishing gradients
through the generator can harm its learning. For our approach, we minimize the
Mean Absolute Error (MAE) between $q^{t}$ and $q^{s}$, which provides stable
gradients for the generator so that the vanishing gradients can be alleviated.
In our experiment, we empirically find that this significantly improves
student’s performance over other possible losses. Now we can define the loss
function for distillation stage as follows:

$\mathcal{L}_{DIS}=\mathbb{E}_{z\sim
p_{z}(z)}\left[\frac{1}{n}\left\|\mathcal{T}\left(\mathcal{G}(z)\right)-\mathcal{S}\left(\mathcal{G}(z)\right)\right\|_{1}\right].$
(3)

####  3.2.2 Generation Stage

The goal of the generation stage is to push the generation of new graphs. In
this stage, we fix the two discriminators and only update the generator. We
encourage the generator to produce more confusing training graphs. The loss
function used for generator is the same as for student except that the goal is
to maximize it:

$\displaystyle\mathcal{L}_{GEN}=-\mathcal{L}_{DIS}.$ (4)

With the generation loss, the error first back-propagates through the
discriminator (the teacher and the student model), then through the generator
to optimize it.

###  3.3 Optimization

The whole distillation process is summarized in Algorithm 1. DFAD-GNN trains
the student and the generator by iterating over the distillation stage and the
generation stage. Based on the learning progress of the student model, the
generator crafts new graphs to further estimate the model discrepancy. The
competition in this adversarial game drives the generator to discover more
knowledge. In the distillation stage, we update the student model for $k$
times so as to ensure its convergence. Note that compared with the
conventional method GFKD Deng and Zhang (2021), the time complexity of DFAD-
GNN mainly lies on the matrix multiplication of the generator, i.e.,
$O(TN^{2})$ where $N$ is the number of nodes and $T$ is the node feature
dimension. Although our method has higher time complexity, the performances
are much better. In addition, because there are not too many nodes in most
real world graph-level applications (usually less than 100), we remark that
our complexity is acceptable in practice.

##  4 Experiments

###  4.1 Datasets

We adopt six graph classification benchmark datasets including three
bioinformatics graph datasets, i.e., MUTAG, PTC_MR, and PROTEINS, and three
social network graph datasets, i.e., IMDB-BINARY, COLLAB, and REDDIT-BINARY.
The statistics of these datasets are summarized in Table 1. To remove the
unwanted bias towards the training data, for all experiments on these
datasets, we evaluate the model performance with a 10-fold cross validation
setting, where the dataset split is based on the conventionally used
training/test splits Niepert et al. (2016); Zhang et al. (2018); Xu et al.
(2018) with LIBSVM Chang and Lin (2011). We report the average and standard
deviation of validation accuracies across the 10 folds within the cross-
validation.

Dataset | #Graphs | #Classes | Avg#Graph Size
---|---|---|---
MUTAG | 188 | 2 | 17.93
PTC_MR | 344 | 2 | 14.29
PROTEINS | 1113 | 2 | 39.06
IMDB-BINARY | 1000 | 2 | 19.77
COLLAB | 5000 | 3 | 74.49
REDDIT-BINARY | 2000 | 2 | 427.62

Table 1: Summary of datasets.

Datasets | MUTAG | PTC_MR | PROTEINS | IMDB-BINARY | COLLAB | REDDIT-Binary
---|---|---|---|---|---|---
Teacher | GIN-5-128 | GIN-5-128 | GIN-5-128 | GIN-5-128 | GIN-5-128 | GIN-5-128
96.7±3.7 | 75.0±3.5 | 78.3±2.9 | 80.1±3.7 | 83.5±1.2 | 92.2±1.2
Student | GIN-5-32 | GIN-1-128 | GIN-5-32 | GIN-1-128 | GIN-5-32 | GIN-1-128 | GIN-5-32 | GIN-1-128 | GIN-5-32 | GIN-1-128 | GIN-5-32 | GIN-1-128
| (6.7%$\times m$) | (20.6%$\times m$) | (6.7%$\times m$) | (20.6%$\times m$) | (6.7%$\times m$) | (20.6%$\times m$) | (6.7%$\times m$) | (20.6%$\times m$) | (6.7%$\times m$) | (20.6%$\times m$) | (6.7%$\times m$) | (20.6%$\times m$)
KD | 96.7±5.1 | 95.3±4.6 | 76.6±5.9 | 77.0±8.1 | 76.0±5.1 | 78.8±3.2 | 80.4±3.4 | 82.0±3.5 | 82.8±1.6 | 83.6±1.9 | 90.4±2.2 | 91.7±1.9
RANDOM | 67.9±8.0 | 62.9±8.5 | 60.1±9.1 | 61.0±8.5 | 60.8±9.4 | 60.2±9.2 | 61.6±5.8 | 60.2±6.4 | 57.3±4.3 | 59.9±3.4 | 69.6±4.3 | 64.5±5.6
GFKD | 77.8±11.1 | 72.6±10.4 | 65.2±7.7 | 62.1±7.0 | 61.3±4.0 | 62.5±3.6 | 67.2±5.5 | 65.1±5.4 | 64.7±3.3 | 64.1±3.0 | 70.2±3.4 | 68.1±3.9
DFAD-GNN | 87.8±6.9 | 85.6±6.7 | 71.0±3.1 | 69.7±3.5 | 70.0±4.2 | 69.9±5.3 | 73.1±4.3 | 74.9±3.1 | 72.1±2.7 | 71.2±2.0 | 75.4±2.4 | 75.7±2.3
| (90.8%$\times t$) | (88.5%$\times t$) | (94.7%$\times t$) | (92.9%$\times t$) | (89.4%$\times t$) | (89.3%$\times t$) | (91.3%$\times t$) | (93.5%$\times t$) | (86.3%$\times t$) | (85.3%$\times t$) | (81.8%$\times t$) | (82.1%$\times t$)
Student | GCN-5-32 | GCN-1-128 | GCN-5-32 | GCN-1-128 | GCN-5-32 | GCN-1-128 | GCN-5-32 | GCN-1-128 | GCN-5-32 | GCN-1-128 | GCN-5-32 | GCN-1-128
| (3.3%$\times m$) | (10.6%$\times m$) | (3.3%$\times m$) | (10.6%$\times m$) | (3.3%$\times m$) | (10.6%$\times m$) | (3.3%$\times m$) | (10.6%$\times m$) | (3.3%$\times m$) | (10.6%$\times m$) | (3.3%$\times m$) | (10.6%$\times m$)
KD | 86.7±9.4 | 82.2±10.2 | 70.9±7.3 | 70.0±6.9 | 74.5±3.8 | 75.5±4.0 | 79.7±3.6 | 81.1±3.2 | 81.7±1.1 | 82.1±2.2 | 88.2±2.3 | 87.8±2.4
RANDOM | 58.9±19.3 | 55.6±21.1 | 59.4±10.1 | 55.6±8.1 | 59.2±8.4 | 57.9±8.0 | 52.3±2.4 | 55.1±2.8 | 55.6±4.4 | 55.3±5.5 | 59.3±3.7 | 57.7±4.2
GFKD | 70.0±11.2 | 69.1±10.3 | 65.0±8.2 | 61.9±8.5 | 62.9±7.7 | 61.4±8.8 | 63.5±5.3 | 65.2±5.7 | 65.7±2.6 | 64.2±1.9 | 65.3±2.6 | 65.1±2.7
DFAD-GNN | 74.1±9.3 | 76.4±8.8 | 67.7±2.9 | 67.9±3.5 | 67.2±5.0 | 65.7±3.7 | 69.5±4.8 | 68.6±5.4 | 70.3±1.2 | 69.8±1.8 | 69.9±1.3 | 70.4±1.9
| (76.2%$\times t$) | (79.0%$\times t$) | (90.3%$\times t$) | (90.5%$\times t$) | (85.8%$\times t$) | (83.9%$\times t$) | (86.8%$\times t$) | (85.6%$\times t$) | (84.2%$\times t$) | (83.6%$\times t$) | (75.8%$\times t$) | (76.4%$\times t$)
Student | GAT-5-32 | GAT-1-128 | GAT-5-32 | GAT-1-128 | GAT-5-32 | GAT-1-128 | GAT-5-32 | GAT-1-128 | GAT-5-32 | GAT-1-128 | GAT-5-32 | GAT-1-128
| (164.6%$\times m$) | (84.5%$\times m$) | (164.6%$\times m$) | (84.5%$\times m$) | (164.6%$\times m$) | (84.5%$\times m$) | (164.6%$\times m$) | (84.5%$\times m$) | (164.6%$\times m$) | (84.5%$\times m$) | (164.6%$\times m$) | (84.5%$\times m$)
KD | 87.8±8.9 | 82.2±10.2 | 73.2±5.5 | 69.7±6.8 | 76.6±3.4 | 74.5±4.6 | 80.7±3.0 | 79.9±2.8 | 80.3±1.5 | 81.5±1.2 | 90.9±1.9 | 90.6±2.0
RANDOM | 63.9±17.3 | 57.5±20.3 | 60.0±7.1 | 59.4±6.7 | 59.8±6.4 | 60.6±5.6 | 53.6±4.5 | 52.9±2.1 | 56.3±3.6 | 58.1±3.3 | 57.6±4.1 | 55.8±4.3
GFKD | 72.5±13.8 | 70.4±11.9 | 63.2±6.5 | 62.7±7.0 | 62.2±6.8 | 62.8±7.9 | 63.7±4.6 | 64.4±5.2 | 66.2±2.3 | 64.9±3.7 | 67.8±3.5 | 68.3±4.4
DFAD-GNN | 76.9±6.9 | 77.3±5.9 | 66.4±3.9 | 68.0±4.7 | 67.8±4.9 | 66.0±4.7 | 68.4±3.9 | 68.0±4.7 | 71.1±1.6 | 70.5±2.5 | 73.5±2.6 | 72.3±2.7
| (79.5%$\times t$) | (79.9%$\times t$) | (88.5%$\times t$) | (90.7%$\times t$) | (86.6%$\times t$) | (84.3%$\times t$) | (85.4%$\times t$) | (84.9%$\times t$) | (85.1%$\times t$) | (84.4%$\times t$) | (79.7%$\times t$) | (78.4%$\times t$)
Student |  | GraphSAGE
---
-5-32
| GraphSAGE
---
-1-128
| GraphSAGE
---
-5-32
| GraphSAGE
---
-1-128
| GraphSAGE
---
-5-32
| GraphSAGE
---
-1-128
| GraphSAGE
---
-5-32
| GraphSAGE
---
-1-128
| GraphSAGE
---
-5-32
| GraphSAGE
---
-1-128
| GraphSAGE
---
-5-32
| GraphSAGE
---
-1-128
| (5.9%$\times m$) | (11.1%$\times m$) | (5.9%$\times m$) | (11.1%$\times m$) | (5.9%$\times m$) | (11.1%$\times m$) | (5.9%$\times m$) | (11.1%$\times m$) | (5.9%$\times m$) | (11.1%$\times m$) | (5.9%$\times m$) | (11.1%$\times m$)
KD | 87.8±12.1 | 82.8±9.8 | 75.6±5.3 | 70.3±6.6 | 76.3±3.5 | 75.7±4.5 | 80.8±2.6 | 80.1±2.5 | 81.2±1.7 | 81.5±2.5 | 90.1±1.7 | 89.5±1.9
RANDOM | 62.2±17.4 | 57.8±22.7 | 61.1±7.0 | 59.9±6.9 | 57.4±8.5 | 55.7±6.3 | 52.6±2.8 | 53.2±2.9 | 54.6±3.6 | 55.5±2.7 | 54.6±4.5 | 54.4±4.0
GFKD | 67.7±12.9 | 68.1±12.1 | 62.5±5.9 | 63.0±6.6 | 63.3±7.7 | 61.8±7.9 | 62.3±5.2 | 63.1±6.0 | 63.3±2.3 | 64.7±3.2 | 63.6±3.8 | 64.0±3.7
DFAD-GNN | 76.5±7.3 | 75.9±6.5 | 66.9±3.7 | 67.5±3.9 | 69.0±6.1 | 67.8±5.4 | 67.5±4.9 | 69.0±3.4 | 68.9±1.1 | 69.6±2.1 | 71.3±3.1 | 69.1±2.9
| (79.1%$\times t$) | (78.5%$\times t$) | (89.2%$\times t$) | (90.0%$\times t$) | (88.1%$\times t$) | (86.6%$\times t$) | (84.3%$\times t$) | (86.1%$\times t$) | (82.5%$\times t$) | (83.4%$\times t$) | (77.3%$\times t$) | (74.9%$\times t$)

Table 2: Test accuracies (%) on six datasets. GIN-5-128 means 5 layers GIN
with 128 hidden units. (6.7%$\times m$) under student model means percentage
of student model parameters to teacher model parameters, $m$ is the number of
teacher model parameters. (90.8%$\times t$) under DFAD-GNN means percentage of
student model accuracy to teacher model accuracy, $t$ is the accuracy of the
corresponding teacher network.

###  4.2 Generator Architecture

We adopt a generator with fixed architecture for all experiments. The
generator takes a 32-dimensional vector sampled from a standard normal
distribution $\boldsymbol{z}\sim\mathcal{N}(\mathbf{0},\boldsymbol{I})$. We
process it with a 3-layer MLP of [64,128,256] hidden units respectively, tanh
is taken as the activation function. Finally, a linear layer is used to map
the 256-dimensional vectors to $N\times T$-dimensional vectors and reshape
them as node features $\mathbf{F}\in\mathbb{R}^{N\times T}$. Throughout our
experiments, we take the average number of nodes in the training data as $N$
and test the effect of $N$ in the ablation experiment.

###  4.3 Teacher/Student Architecture

To demonstrate the effectiveness of our proposed framework, we consider four
GNN models as teacher and student models for a thorough comparison, including:
GIN, GCN, GAT and GraphSAGE. Although the GNN model does not always require a
deep network to achieve good results, however, from Appendix B, there is no
fixed layer and hidden units that can make six datasets achieve the best
performance on four different models. For fair comparison, we use 5 layers
with 128 hidden units for teacher models. For the student model, we conduct
experiments to gradually reduce the number of layers
$l\in\left\\{5,3,2,1\right\\}$ and gradually reduce the number of hidden units
$h\in\left\\{128,64,32,16\right\\}$. We use a graph classifier layer which
first builds a graph representation by averaging all node features extracted
from the last GNN layer and then passing this graph representation to an MLP.

###  4.4 Implementation

For training, we use Adam optimizer with weight decay 5e-4 to update student
models. The generator is trained with Adam without weight decay. Both student
and generator are using a learning rate scheduler that multiplies the learning
rate by a factor 0.3 at 10%, 30%, and 50% of the training epochs. The number
of updates $k$ of the student model in Algorithm 1 is set to 5. The threshold
$\tau$ is empirically set to 0.5.

###  4.5 Baselines

We compare with the following baselines to demonstrate the effectiveness of
our proposed framework.

Teacher: The given pre-trained model which serves as the teacher in the
distillation process.

KD: The generator is removed, and the student model is trained on 100%
original training data in our framework.

RANDOM: The generator’s parameters are not updated and the student model is
trained on the noisy graphs generated by the randomly initialized generator.

GFKD: GFKD is a data-free KD for GNNs by modeling the topology of graph with a
multinomial distribution Deng and Zhang (2021).

(a) PROTEINS (b) COLLAB

Figure 2: Performance of different teachers to different students on PROTEINS
and COLLAB.

###  4.6 Experimental Results

We have pre-trained all datasets on GCN, GIN, GAT and GraphSAGE with 5 layers
and 128 hidden units (5-128 for short), and found that GIN performs best on
all datasets (Detailed experimental results can be found in the Appendix B).
Therefore, we adopt GIN as the teacher model in Table 2. We choose two
representative architecture 1-128 and 5-32 for four kinds of student models
(More experiments with other architectures can be found in the Appendix D).

(a) PROTEINS (b) COLLAB

Figure 3: Evaluation of different loss functions (teacher model is GIN-5-128).

(a) PTC_MR (b) IMDB-BINARY

Figure 4: Training with different percentages of real data. The intersection
of two lines indicates the needed percentage of the real training data to
achieve our data-free performance.

(a) PTC_MR (b) IMDB-B

Figure 5: Influence of the node number $N$ (Note that, the GFKD randomly
sample the number of nodes from $[10,22]$ for each graph).

From Table 2, it can be observed that KD’s performance is very close to even
outperforms the teacher model. That’s because KD is a data-driven method which
uses the same training data as the teacher model for knowledge distillation.
This also implies that the loss function of our DFAD-GNN is very effective in
distilling knowledge from the teacher model to the student model, as we apply
the same loss function in both KD and our DFAD-GNN.

We also observe that RANDOM delivers the worst performance as the generator is
not updated during the training process, thus the generator will not be able
to generate difficult graphs as the student model progresses. Consequently,
the student model fail to learn enough knowledge from the teacher, resulting
in poor results.

In terms of the efficacy of our DFAD-GNN, Table 2 shows that our DFAD-GNN
consistently outperforms the recent data-free method GFKD Deng and Zhang
(2021). We conjecture the potential reason that DFAD-GNN can significantly
outperform GFKD is the teacher encodes the distribution characteristics of the
original input graphs under its own feature space. Simply inverting graphs in
GFKD tends to overfit to the partial distribution information stored in this
teacher model. As a consequence, their generated fake graphs are lacking of
generalizability and diversity. In contrast, our generated graphs are more
conducive to transferring the knowledge of the teacher model to the student
model.

In terms of the stability, it can be seen from Table 2 that the standard
deviation of our DFAD-GNN is the smallest among all the data-free baselines
and across all datasets, indicating that our model can obtain relatively
stable prediction results.

Another interesting observation is that the performance of the compressed
model is not necessarily worse than the more complex model. As can be seen
from Table 2, that performance of a more compressed student model with 5-32 is
not necessarily worse than the student model with 1-128. Therefore, we
speculate that the performance of the student model may have no obvious
relationship with the degree of model compression, which requires further
investigation.

###  4.7 Model Analysis

##### Model Comparison.

Here, we select PROTEINS and COLLAB with the most training data on molecular
datasets and social datasets respectively for cross-training among the four
kinds of models. It can be seen from Figure 2, when GIN is used as a teacher
model, the overall performance of the student model is better, no matter which
type of the student model is adopted. When GIN is used as a student model, the
performance of the student model is significantly improved compared to other
student models. We speculate there may be two reasons: (1) Under the same
number of layers and hidden units, the GIN model has more parameters than GCN
and GraphSAGE, so GIN owns more powerful learning capability. Note that
although GAT has more parameters than GIN, it may be better at calculating
node attention weights and then performing node classification tasks; (2) GIN
is proposed to solve the problem of graph isomorphism. For our small molecule
graphs and social network graphs, GIN is more powerful than other models in
graph classification tasks.

##### Choice of Loss Function.

The choice of loss is of paramount importance to a successful distillation. We
explore five potential loss functions, including logit-MAE (calculate MAE with
pre-softmax activations, L-MAE for short), softmax-MAE (calculate MAE with
softmax outputs, S-MAE for short), MSE, KLD, and Cross Entropy (CE for short).
These losses are commonly used in the knowledge distillation literature Gou et
al. (2021). Figure 3 shows that using the MAE achieves significantly better
test accuracies compared to other loss functions. Generally speaking, it is
better to calculate MAE before softmax, because logits contain more
information. As mentioned earlier, when the student model matches more closely
to the teacher model, other loss functions tend to suffer from vanishing
gradients Fang et al. (2019). Specifically, backpropagating such vanishing
gradients through the generator can harm its learning. To prevent gradients
from vanishing, we use the MAE computed with the teacher and student logits.

##### Percentage of Training Data.

Although in the above reported results, we assume that the student cannot
obtain any training data. However, in practice, student may have partial
access to the training data. To reflect the practical scenario of knowledge
distillation, we conduct extra experiments on PTC_MR and IMDB-B, by varying
the percentage of training data from 0.5% to 100%, while keeping other hyper-
parameters the same as in the previous experiments. As illustrated in Figure
4, PTC_MR requires 17.1% of real data to achieve our results, while IMDB-B
requires only 5.6%. This is because PTC_ MR has less data, thus the percentage
of real data required is higher than IMDB-B.

##### Number of generated nodes.

To explore the influence of node number $N$ on the model performance, we
conduct experiments with varying sizes of $N$ on PTC_MR and IMDB-B. It can be
seen from Figure 5 that the model performs better when $N$ takes the value
near the average number of nodes in the training set. Far from the average
number, the performance will decrease correspondingly due to a large deviation
from real data.

##### Visualization of Generated Graphs.

The generated graphs and real graphs of IMDB-B and PTC_MR are shown in Figure
6. Although the generated graphs are not exactly the same as the real graphs,
they can be used to generate a student model with relatively good performance.

(a) IMDB-B real graph

(b) PTC_MR real graph

(c) IMDB-B fake graph

(d) PTC_MR fake graph

Figure 6: Graph visualization on IMDB-B and PTC_MR (The first row are real
graphs, the second row are fake graphs we generated).

##  5 Conclusion

This paper introduces a data-free adversarial knowledge distillation framework
on graph neural network for model compression. Without any access to real
data, we successfully reduce the discrepancy and obtain a student model with
relatively good performance. Our extensive experiments on graph classification
demonstrate that our framework can be effectively applied to different network
architectures. In the future, we will extend this work to multi-teacher
scenarios and continue to explore how to generate more complex graphs under
various generator structures.

## Acknowledgments

This work is supported in part by the National Natural Science Foundation of
China (No. 62192784, U20B2045, 62172052, 61772082, 62002029, U1936104). It is
also supported in part by The Fundamental Research Funds for the Central
Universities 2021RC28.

## References

  * Arjovsky and Bottou [2017] Martin Arjovsky and Léon Bottou.  Towards principled methods for training generative adversarial networks.  arXiv preprint arXiv:1701.04862, 2017.
  * Bahri et al. [2021] Mehdi Bahri, Gaétan Bahl, and Stefanos Zafeiriou.  Binary graph neural networks.  In CVPR, pages 9492–9501, 2021.
  * Bojchevski et al. [2018] Aleksandar Bojchevski, Oleksandr Shchur, Daniel Zügner, and Stephan Günnemann.  Netgan: Generating graphs via random walks.  In ICML, pages 610–619. PMLR, 2018.
  * Chang and Lin [2011] Chih-Chung Chang and Chih-Jen Lin.  Libsvm: a library for support vector machines.  ACM transactions on intelligent systems and technology (TIST), 2(3):1–27, 2011.
  * De Cao and Kipf [2018] Nicola De Cao and Thomas Kipf.  Molgan: An implicit generative model for small molecular graphs.  arXiv preprint arXiv:1805.11973, 2018.
  * Deng and Zhang [2021] Xiang Deng and Zhongfei Zhang.  Graph-free knowledge distillation for graph neural networks.  ArXiv, abs/2105.07519, 2021.
  * Fang et al. [2019] Gongfan Fang, Jie Song, Chengchao Shen, Xinchao Wang, Da Chen, and Mingli Song.  Data-free adversarial distillation.  arXiv preprint arXiv:1912.11006, 2019.
  * Fang et al. [2021] Gongfan Fang, Kanya Mo, Xinchao Wang, Jie Song, Shitao Bei, Haofei Zhang, and Mingli Song.  Up to 100x faster data-free knowledge distillation.  arXiv preprint arXiv:2112.06253, 2021.
  * Gilmer et al. [2017] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl.  Neural message passing for quantum chemistry.  In ICML, pages 1263–1272. PMLR, 2017.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  NeurIPS, 27, 2014.
  * Gou et al. [2021] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao.  Knowledge distillation: A survey.  International Journal of Computer Vision, 129(6):1789–1819, 2021\.
  * Hamilton et al. [2017] William L Hamilton, Rex Ying, and Jure Leskovec.  Inductive representation learning on large graphs.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 1025–1035, 2017.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * Jing et al. [2021] Yongcheng Jing, Yiding Yang, Xinchao Wang, Mingli Song, and Dacheng Tao.  Amalgamating knowledge from heterogeneous graph neural networks.  In CVPR, pages 15709–15718, 2021.
  * Kipf and Welling [2016] Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  arXiv preprint arXiv:1609.02907, 2016.
  * Liu et al. [2021] Yuang Liu, Wei Zhang, Jun Wang, and Jianyong Wang.  Data-free knowledge transfer: A survey.  arXiv preprint arXiv:2112.15278, 2021.
  * Lopes et al. [2017] Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner.  Data-free knowledge distillation for deep neural networks.  arXiv preprint arXiv:1710.07535, 2017.
  * Lu et al. [2021] Yuanfu Lu, Xunqiang Jiang, Yuan Fang, and Chuan Shi.  Learning to pre-train graph neural networks.  In AAAI, volume 35, pages 4276–4284, 2021.
  * Niepert et al. [2016] Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov.  Learning convolutional neural networks for graphs.  In International conference on machine learning, pages 2014–2023. PMLR, 2016.
  * Sun et al. [2018] Lichao Sun, Yingtong Dou, Carl Yang, Ji Wang, Philip S Yu, Lifang He, and Bo Li.  Adversarial attack and defense on graph data: A survey.  arXiv preprint arXiv:1812.10528, 2018.
  * Veličković et al. [2017] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio.  Graph attention networks.  arXiv preprint arXiv:1710.10903, 2017.
  * Wu et al. [2020] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip.  A comprehensive survey on graph neural networks.  IEEE transactions on neural networks and learning systems, 32(1):4–24, 2020.
  * Wu et al. [2021] Chuhan Wu, Fangzhao Wu, Yang Cao, Lingjuan Lyu, Yongfeng Huang, and Xing Xie.  Fedgnn: Federated graph neural network for privacy-preserving recommendation.  arXiv preprint arXiv:2102.04925, 2021.
  * Wu et al. [2022] Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Yongfeng Huang, and Xing Xie.  Communication-efficient federated learning via knowledge distillation.  Nature Communications, 13(1):1–8, 2022.
  * Xu et al. [2018] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka.  How powerful are graph neural networks?  arXiv preprint arXiv:1810.00826, 2018.
  * Yang et al. [2021] Cheng Yang, Jiawei Liu, and Chuan Shi.  Extract the knowledge of graph neural networks and go beyond it: An effective knowledge distillation framework.  In WWW, pages 1227–1237, 2021.
  * Zhang et al. [2018] Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen.  An end-to-end deep learning architecture for graph classification.  In Thirty-Second AAAI Conference on Artificial Intelligence, 2018\.
  * Zhou et al. [2020] Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun.  Graph neural networks: A review of methods and applications.  AI Open, 1:57–81, 2020.


# Human Age Estimation from Gene Expression Data using Artificial Neural
Networks

Salman Mohamadi  Computer Science and Electrical Engineering
West Virginia University
Morgantown, WV, USA
<EMAIL_ADDRESS>Gianfranco.Doretto  Computer Science and Electrical
Engineering
West Virginia University
Morgantown, WV, USA
<EMAIL_ADDRESS>Nasser M. Nasrabadi  Computer Science and
Electrical Engineering
West Virginia University
Morgantown, WV, USA
<EMAIL_ADDRESS>Donald A. Adjeroh  Computer Science and
Electrical Engineering
West Virginia University
Morgantown, WV, USA
<EMAIL_ADDRESS>
###### Abstract

The study of signatures of aging in terms of genomic biomarkers can be
uniquely helpful in understanding the mechanisms of aging and developing
models to accurately predict the age. Prior studies have employed gene
expression and DNA methylation data aiming at accurate prediction of age. In
this line, we propose a new framework for human age estimation using
information from human dermal fibroblast gene expression data. First, we
propose a new spatial representation as well as a data augmentation approach
for gene expression data. Next in order to predict the age, we design an
architecture of neural network and apply it to this new representation of the
original and augmented data, as an ensemble classification approach. Our
experimental results suggest the superiority of the proposed framework over
state-of-the-art age estimation methods using DNA methylation and gene
expression data.

##  I Introduction

Healthy aging requires accurate assessment and modification of healthcare
strategies and culture. However, this could not be done without understanding
the aging process in terms of transcriptional changes in cellular scale. In
fact, even though transcriptional changes of age-related genes differ among
different tissue types [1], modeling these changes of expression level across
the human lifespan, would extend the general understanding of aging and
heterogeneity in senescence of individuals with the same age. Along with this,
prior studies suggest that age prediction using trasncriptional changes such
as gene expression level will be feasible almost accurately and could be very
helpful in disease prevention and anti-aging therapeutics.

With an emphasis on the mentioned suggestion, this work relates to the general
problem of estimating human age from genomic data, a known genotype to
phenotype problem [2, 3]. The human genome has close to 30,000 genes. Each
gene responds to biological stress or defined experimental conditions, and
this response is captured by its level of expression under such condition. The
gene expression data records information about the observed levels of
expression for given set of genes in the genome, under specified experimental
conditions. Thus, our challenge is, given a data set of gene expression values
under specified experimental condition(s), in particular, gene expression data
from human dermal fibroblasts, the main cell type available in skin connective
tissue [4], from several individuals, how can we recover the age for a given
individual in the set? Addressing this question will have significant
implications in various fields, from health and precision medicine
(chronological age and biological age) to improved understanding of
experimental transcriptomic datasets. On one hand, this problem is made very
challenging because of the very limited data sets currently available. On the
other hand, the decreasing cost of data acquisition (and hence increasing data
availability) imply that this type of problem will become a dominant feature
in the health and in particular, personalized medicine of the future. This
work is also expected to have tremendous impact on forensic investigations
involving genomic data evidence, including human trafficking and child
exploitation.

Here, first we review the prior similar approaches to human age estimation,
including those that use information about gene expression, or other types
genomic data, and then we propose our framework based on a new approach to
gene expression data representation and inference. Our main contributions
include:

  * •

A new framework for age estimation using a dataset of gene expression data of
human dermal fibroblasts, which outperforms the state-of-the-art method on
this dataset and dataset of methylation data.

  * •

A novel data representation as well as data augmentation methods for gene
expression data, which allow us to apply well-known deep learning tools such
as artificial neural networks (ANNs).

The paper is organized as follows. The next section reviews related work.
Section 3 presents our proposed data representation scheme and data
augmentation method for gene expression data. Section 4 presents our neural
network-based framework for age prediction using gene expression data. Section
5 presents the experiments and the results. Section 6 concludes the paper.

##  II Background and Related Work

With increasing availability of different types of genomic data, accurate age
estimation has emerged as an important problem domain in different
applications. The connection between human age and genomic data has been the
focus of various studies, and has thus been investigated from many angles and
perspectives. Tsuji et al [5] found some association between age and
chromosome telomere lengths and used it for human age estimation. Richter et
al [6], tried to find a connection between age and mitochondria DNA while
working on probable cause of high mutation rate of mtDNA. Authors of reference
[7] extended the work on single tissue type to multi-tissue, hoping to
represent a better age prediction model for so many tissue types. Similarly, a
modeling on age based on epigenetic data is presented in [8]. With an emphasis
on general pattern analysis, Peters et al [9] characterized the age by
modeling variations within gene expression data inherently linked to the
aging. Authors of reference [10] modified their previous model on
discriminating relevant gene expression patterns related to young people and
old people. Moreover, a study on whole genome methylation data by reference
[11] strongly suggested general patterns of signature of aging. Xu et al [12]
proposed a method based on support vector regression to estimate the age from
DNA methylation data which achieved an average accuracy of 4.7 years. However,
there are only a handful of studies that try to connect human gene expression
data with age. Clearly, such a connection could have significant implications
for health and disease studies, for instance, in the area of human biological
age estimation [13, 14], or in personalized medicine. In other words, the
problem of age estimation can be studied using the increasingly available gene
expression data, with a potential for significantly improved accuracy at low
cost.

Holly et al [10] provided a statistical model based on gene expression data
for age-group classification which helped pave the way for this avenue of
research. Fleischer et al [15] developed a computational method to predict age
based on gene expression data from skin fibroblast cells using an ensemble of
machine learning classifiers. They generated an extensive RNA-seq dataset of
fibroblast cell lines derived from a total of 143 individuals whose ages range
from 1 to 94 years. Using this dataset, their method predicted chronological
age with a median error of 4 years, outperforming algorithms proposed by prior
studies that predicted age from DNA methylation [12, 17, 10], and from gene
expression data for fibroblasts [10].

This work is along this same line of estimating age for individuals using
information from their gene expression data. Our work is different in terms of
the new data representation scheme we introduce, and the proposed new data
augmentation method for this type of data. With these two innovations, it
becomes possible to build a parallel framework that can exploit the power of
deep learning techniques, such as artificial neural networks. While there are
also machine learning and deep learning tools for 1-D data representation that
could be used for these types of data, our 2-D data representation achieves
the highest accuracy compared to previous work. More importantly, one of the
key advantages of the spatial (2-D) representation is that, due to the fixed
spatial location of expression value associated with each gene, there is more
flexibility in exploiting potential long-range associations between genes [18,
19, 20], beyond just adjacent or co-located genes.

Figure 1: Spacial data representation for three individuals; left to right: 1
year old, 30 years old and 61 years old.

Figure 2: Data Augmentation Scheme.

##  III Data Representation and Augmentation

In this section we describe our novel data representation scheme and data
augmentation approach for gene expression data which enable us to use
artificial neural networks.

###  III-A Dataset and Normalization

Our database which is from [15], consists of a matrix of expression values
from 27,142 genes for a total of 143 individuals represented in two groups
(133+10). Age ranged from 1 year to 94 years. In fact, the dataset provides us
with expression value of each and every gene for all individuals. Certainly,
with only 143 samples, there is need for more data in order to train our
neural network. Thus, we need to perform data augmentation to generate
synthetic datasets. Each row contains the expression values from certain genes
across 143 individuals. Each column is the expression values of the 27,142
different genes for a given subject. Considering the rows, it seems that
despite the values in a column, values in a row are in a smaller range, i.e.,
expression values for a certain gene across all individuals are in a
relatively small dynamic range, but different genes expression values for a
certain individuals vary significantly. This has also been observed in other
general genomic datasets. [21, 22, 23].

Data normalization in known to accelerate the learning process of the network.
Thus, before doing any further steps such as data augmentation, we normalize
each row (specific gene) of the database individually, as a pre-processing
step. After applying standard normalization to each row, as shown below,
corresponding data samples of each row are centered at zero with unit
variance: ${z=\frac{(x-\mu)}{\sigma},}$ where ${\mu}$ and ${\sigma}$ are
respectively, the mean and standard deviation of the original data samples
$x$, and $z$ is the normalized data.

###  III-B Spatial Representation

Here, we propose a new data representation for the gene expression data from
an individual, and a methodology for data augmentation. In fact,data
representation and in general data visualization is very helpful in better
extracting the necessary information or modeling the data [38, 33]. To analyse
the data, we need to extract the columns of the data matrix, which captures
the expression values across all genes (27,142 genes), for each individual. As
we will explain in the next section, we design a deep neural network
classifier in order to perform the classification w.r.t the age for the
subjects. We extract each column and reshape it to a square two-dimensional
matrix, where each element of this matrix represents the expression values for
a specific gene. We reshape the vector by moving the elements in the vector to
a the matrix, filling from top left to the down right of the matrix
(165x165=27,225) which the last 83 elements of the matrix (27,225-27,142=83)
are padded with zeros. Therefore, we will have 143 squared matrices for the
143 subjects. Then we transform each matrix to a square image data, where each
pixel location of every image represents the gene expression value of a
specific gene. By so doing, we provide a spacial (2-D) representation for
expression values of all genes for each individual. Fig. 1 shows some examples
for three individuals in different age groups. More importantly, the figure
also shows the nature of the 2D representation for the gene expression data,
and the difficulty of analyzing such representations. It is noteworthy to
mention that while one could say that representing 1-D data in the form of 2-D
data may seem ad hoc (leading to possibly irrelevant spatial relationships),
this type of representation makes it simpler to explore and possibly exploit
potential long-range relationships between genes, for instance, using simple
2D convolution kernels. This may not be as simple, or as computationally
efficient, using 1-D representations. Further, since each pixel location in
this type of 2-D data (image) is associated with a unique gene, this could be
even helpful at the analysis stage (for instance, using artificial neural
networks) in terms of allowing us to easily compare a given gene across all
individuals (or subset of individuals) in the data set.

###  III-C Data Augmentation

Although the data representation is a 2D matrix of each column’s data samples,
the resulting images are not the same as traditional natural images, like
face, or natural real-world objects. First, based on our pre-evaluation
experiment, well-known data augmentation techniques for image data, such as
basic image manipulations [24], did not prove to be effective for this type of
image data. The reason may be the nature of the data, which is totally
different from normal image datasets. Further, in applying data augmentation
on this type of data, we need to recognize the meaning of the resulting
augmented data. For instance, simply flipping the image, or rotating the data
will lead to outcomes that may not be meaningful, given the information
represented in the images.

Thus, in order to augment this type of data, we should consider the basic
representation of the data, in other words, augment the data in the feature
space. In this line, besides machine learning tools, classical signal
processing methods and tools could be effective [28, 30, 37]. We need to
generate new features (expression values of a given gene, which is in a row)
for a certain subject (values in a given column), without significantly
shifting the statistical distribution of the sample data[25] to generate a new
subject with the same label (no change in the subject’s label),

Therefore, as shown in Fig. 2 and presented in corresponding equations, first
we add a random Gaussian variable with mean zero and standard deviation
${\sigma}$=${\beta}$*${\sigma_{d}}$, where ${\sigma_{d}}$ is the standard
deviation of the original data samples of a given row; then allocate the same
label of the original data (age of each column as an individual) to the
columns of the generated data. Here ${\beta}$ is an adjustable parameter
(based on our experiments, the best value for ${\beta}$ is 0.13). Here we have
${X\sim\mathcal{N}(\mu,\,\sigma^{2})\,}$ and
${P(x)=\frac{1}{{\sigma\sqrt{2\pi}}}e^{{{-\left({x-\mu}\right)^{2}}\mathord{\left/{\vphantom{{-\left({x-\mu}\right)^{2}}{2\sigma^{2}}}}\right.\kern-1.2pt}{2\sigma^{2}}}}}$.

In other words, we need to add some random variable to each row (feature)
considering the standard deviation of the row, without significantly changing
the data distribution. Otherwise we can not assign the label of the original
data to the generated data. Ideally, in order to be able to assign a label to
a generated data which is not the same as the label of the original data, we
need to comprehensively model the biological functionality and role of each
gene in aging. However, this is not practical with today’s technology.
Therefore, our approach to augment the data is to generate new data samples
which do not remarkably deviate from the original data samples distribution
individually (just a new copy of the data samples with a small deviation) so
that we can assign the same label to the generated data sample. While this may
be seen as a strong assumption from one viewpoint, this has already been used
by other data augmentation methods for different data types [26, 27]. Our
results show that this assumption is sound. Using this method we will
synthesize different versions of the dataset with the corresponding labels
[25, 27, 36].

We have tried several values for ${\beta}$, starting from 0.05 to 0.25. Using
the validation data, the best results is associated with ${\beta=0.13}$. Since
we have 27,142 different genes, we need to use 27,142 random random Gaussian
variables with mean zero and standard deviation associated with the standard
deviation of each row and add them to the data accordingly, in order to
generate a new set of the data with the same labels. For data augmentation,
since we added a random variable to each row for every set for data
augmentation, one may ask why just 5 sets of augmentations? In order to
address this question, we have also performed 5 more sets (totally 10 sets) of
data augmentation. However, the results did not improve remarkably. We suspect
that this may be caused by the fact that we may be generating similar copies
of the augmented data when we generate many sets.

As discussed later in Section 5, we need to design our experiments in such a
way that divides the dataset into several non-overlapping parts, where each
part should be independently used for training and testing in different rounds
of the experiment. We use a major proportion of the original data for training
and the remaining part as the test data. Data augmentation is performed only
on the training data. For data partitioning, the data is randomly divided into
four fixed partitions, then at every round of the experiment (four rounds in
total), we use three out of four parts for the data augmentation. Here, all
these three parts (${75\%}$ of the original data) plus augmented data are
designated for training, and the last part $(25\%)$ is used for testing.
Furthermore, we mention that, data augmentation as described above is applied
five times to the ${75\%}$ of the original data. Thus, at each round, the
training data will consist of 107 + (5 x 107) = 642 images, while the test
data consists of 36 images.

##  IV Age Estimation Framework

In this section we present our framework for age estimation from gene
expression data that exploits the outcome of our proposed data representation
and data augmentation schemes. After data representation and augmentation, we
feed the data to a network designed for age-group classification. Given our
2-D representation of the gene expression data, it become natural to analyze
such images using two dimensional convolutional neural networks (CNNs). We
emphasize that our data representation scheme (spatial 2-D rather than 1-D
representation) makes it natural to exploit the modeling capability of deep
learning architectures for the problem of age estimation using our type of
data. In fact, with our spatial representation, each pixel location represents
a specific gene, while the pixel intensity value indicates the expression
level for the gene. Thus, any specific gene is located in a fixed pixel
location in the image, possible relations between genes can be captured using
a simple convolution mask. This means that our neural network can more easily
learn these possible relationships, even between genes that may be widely
separated in the genome. This is confirmed in our experiments by changing the
order of genes in the spatial representation for some of individuals and then
training the network. In fact, one possible reason could be that even by
changing the order of the genes, almost for all individuals in a certain age
group, the expression values are quite similar.

Figure 3: Network architecture for age estimation via an ensemble of age-group
classifiers.

###  IV-A Network Architecture

The core objective in this work is to develop an effective framework for age
estimation from gene expression data. However, as mentioned earlier, in order
to accomplish it, we need a suitable data representation, and a proper method
for data augmentation, especially given the limited data sizes. Given our 2-D
spatial data representation, artificial neural networks (ANNs), for instance,
using convolutional neural networks (CNNs) become natural candidates for
learning in such a framework. Here, we elaborate on our network architecture
and the rationale behind its design. Initially, we performed pre-evaluation
experiment on some well-known deep neural networks, such as VGG19 and ResNet18
in order to emphasize the importance of compatibility between the network and
the data or task [39, 34]. Our preliminary experiments showed that, due to the
size of dataset, these deep neural networks did not produce good results.
Thus, for this work, we designed a shallower neural network as shown in Fig.
3. The first two layers are two identical convolutional blocks consisting of a
convolution layer followed by max-pooling (2 x 2) and ReLU activation
function. Then we incorporate three fully connected layers and finally a
softmax classifier as the last layer of the network.

###  IV-B Age Grouping

One major problem of age estimation from genetic datasets is the paucity of
data with age information. Our gene expression dataset has only 143
individuals, which is quite small for training a neural network. One way to
mitigate this problem is to use the data for classification of age groups,
rather than direct age estimation, which will be a regression problem.
Combining the classification results, one can then perform the required age
estimation. Previous work on this dataset [15], proposed several ensemble
classifiers using different machine learning tools. The best median absolute
error and mean absolute error reported were 4 and 7.7 years, respectively.
Inspired by this work, we developed our framework for ensemble classification
using neural networks on our proposed data representation and augmentation.
This resulted in an average median absolute error and mean absolute error of
4.1 and 3.69 years, respectively. Fleischer et al [15] incorporated twenty
sets of age groupings each with a fixed age bin widths of 20 years. Working
with so many age groups will obviously be computationally intensive, since we
have to perform classification for each grouping. In our work, we use just six
sets of age groupings, with possibly varying age bins, and bin sizes. Given
the small size of our dataset, the first sets of age-groupings used bin widths
of 20 years, but varying bin positions. This makes it easier for each age
group to have enough subjects for the system to learn during training. The
last two sets of groupings are defined based on biological rationale [29]. The
age ranges, size and structure of the six age-groupings (G1 to G6) are shown
below:

G1: [0-20], [21-40], [41-60], [61-80], [81-100]
G2: [0-5], [6-25], [26-45], [46-65], [66-85], [86-100]
G3: [0-10], [11-30], [31-50], [51-70], [71-90], [91-100]
G4: [0-15], [16-35], [36-55], [56-75], [76-95], [96-100]
G5: [0-2], [3-12], [13-24], [25-45], [46-64], [96-100]
G6: [0-11], [12-24], [25-49], [50-69], [70-100]

Figure 4: Pipeline of the proposed framework for age estimation from gene
expression data.

###  IV-C Combining Classifiers Results

Each classification round associated with one of the train/test partitions,
predicts an age group for the test subjects individually. Here, the label with
the highest probability in the output of the softmax is selected as the
predicted class or age-group. We need to combine the results to compute an
estimated age for each subject and then average the results over the four
rounds. Our approach for combining the results is similar to voting-based
regression.

Suppose that for a given test subject ${S_{x}}$, the six classifiers predict
the following intervals as their respective age bins: ${I_{1}}$, ${I_{2}}$,
${I_{3}}$, ${I_{4}}$, ${I_{5}}$ and ${I_{6}}$; Let these intervals be
represented by their corresponding start and end points: $<s_{i},e_{i}>$,
$i=1,2,...,6$. To compute a single number for the estimated age rather than an
age bin, first, we compute the mean of the intervals, next we compute the
average of all six means as the estimated age for ${S_{x}}$. That is,
$A_{x}=\frac{1}{6}{\sum_{i=1}^{6}(\frac{e_{i}+s_{i}}{2})}$, where $A_{x}$ is
the estimated age for subject $S_{x}$.

Besides the accuracy of the individual classifiers, as a metric of evaluation,
we also used the mean absolute error, and the median absolute error.

##  V Experiments

###  V-A Data Augmentation

As described in the section III-C, we add a random Gaussian variable with mean
zero and adaptive standard deviation to each row. We have
${X\sim\mathcal{N}(\mu,\,\sigma^{2})\,}$ where
${\sigma}$=${\beta}$*${\sigma_{d}}$ and ${\sigma_{d}}$ is the standard
deviation of the original data samples of a given row (each row consists of
the expression values of a certain gene across all individuals).

Next we assign the same label of the columns of the original data to the
columns of the generated data (each column contains the expression values of
all genes for a certain individual), where ${\beta}$ is a parameter adjusted
using validation data. Since we use the label of the original subject for the
generated sample subject, we need to adjust the ${\beta}$ so that the new
generated subject does not deviate notably from the original sample subject.
Therefore, we should try several values of ${\beta}$ and test the accuracy of
the classifiers (mean absolute error) over the validation data. We have tried
several ${\beta}$ values, starting from 0.05 to 0.25. We found that best
results were associated with ${\beta=0.13}$.

###  V-B Framework Implementation

We carried out a set of parallel experiments as depicted in Fig. 4.
Specifically, we randomly divided the original dataset (143 columns associated
with the gene expression values of 143 subjects) into four fixed parts. For
each experiment, one of the parts (${25\%}$) was selected as the test data and
the remaining three parts ($75\%$) were used for data augmentation and
training; which is divided to two parts, one for training (${65\%}$) and one
for validation (${10\%}$). The validation data is used for hyper-parameter
tuning and also adjusting the value of ${\beta}$.

As we have six sets of age-groupings, at each of the experiments we assigned
corresponding labels to each subject (image) based on the labeling protocol
presented in Section IV-B. Then we trained six identical classifiers with the
proposed architecture as shown in Fig. 3, each of them for one of the labeling
protocols. At the end of the experiments, we would have four sets of test
results, with each result associated with one of the four independent rounds
of the experiments (train/test partitions in Fig. 4), where each set of test
results provides us with six predicted labels (one label from each of the six
classifiers). The results of the six classifiers are combined to estimate of
the subject’s age. We used Adam algorithm [35] as the optimizer, using 100
epochs, and a learning rate of 0.001. Our loss function was the cross entropy
loss, and we wrote the code from scratch in Pytorch.

In binary classification, where the number of classes $N$ equals 2, the cross-
entropy loss can be calculated as:

$\begin{split}\mathcal{L}_{C}=-(y\log(p)+(1-y)\log(1-p)).\end{split}$ (1)

However, in this work we deal with a multiclass classification scheme where
the number of classes, $N$, can be 5, 6, or 7 depending on the corresponding
age grouping. Thus, we have:

$\begin{split}\mathcal{L}_{C}=-\sum_{c=1}^{N}(y_{o,c}\log(p_{o,c}))\end{split}$
(2)

where, $N$ is the number of classes, and $y$ is the binary indicator function
representing whether the output observation is the ground truth class.
Likewise, ${y_{o,c}}$ is defined for the multiclass scenario, and ${p_{o,c}}$
represents the probability of the output being ground truth class.

##  VI Results

###  VI-A Age-group classification results

Before proceeding with age estimation, we first investigated the performance
of the proposed data representation and network on age group classification,
using each of the six age groupings, since these will form the basis for our
age estimation. Table I shows the classification results for each of the six
sets of age-groupings. Table I shows the classification performance on each
age-grouping, using each of the four data partitions. The best accuracy is
associated with the classifier ${\\#1}$ and ${\\#6}$ respectively. Since the
label data is balanced, this may imply that these two age groupings are
possibly more effective for age prediction from a biological perspective. This
may be because these two groupings were derived based on some biological
considerations [29].

Classifier |  Accuracy ($P_{1}$) |  Accuracy ($P_{2}$) |  Accuracy ($P_{3}$) |  Accuracy ($P_{4}$)
---|---|---|---|---
C1 |  ${100\%}$ |  ${100\%}$ |  ${97\%}$ |  ${100\%}$
C2 |  ${69\%}$ |  ${69\%}$ |  ${72\%}$ |  ${72\%}$
C3 |  ${75\%}$ |  ${72\%}$ |  ${75\%}$ |  ${75\%}$
C4 |  ${78\%}$ |  ${81\%}$ |  ${78\%}$ |  ${81\%}$
C5 |  ${64\%}$ |  ${61\%}$ |  ${64\%}$ |  ${61\%}$
C6 |  ${89\%}$ |  ${89\%}$ |  ${92\%}$ |  ${92\%}$
TABLE I: Accuracy on different classifiers/age-groupings (see also Fig. 4).
$P_{1},P_{2},P_{3}$ and $P_{4}$ denote the results for experiments using
train/test data partitions $P_{1},P_{2},P_{3}$ and $P_{4}$, respectively. The
best accuracy is associated with the classifier ${\\#1}$ and ${\\#6}$
respectively.

###  VI-B Age estimation results and comparison

To ensure robustness, we expose our framework to more randomness, by
performing the whole experiment three times. Fig. 5 shows the plot of the true
age compared with the estimated age using our proposed method. The estimated
ages are generally close to the true ages. Table I summaries the performance
of the proposed framework. The table shows the impact of the 2-D
representation used in this work, and of the data augmentation. Using the
performance metric of mean absolute error (MAE), our proposed method
outperforms current state-of-the-art method on this dataset [15], and is
comparable to similar methods on other genomic datasets, such as epigenetic
data [8], or DNA methylation data [31]. Our method produced comparable results
in terms of the median absolute error (MdAE).

Figure 5: True vs predicted age. Ref. |  Algorithm |  Age bin |  MAE |  MdAE
---|---|---|---|---
[15] |  LDA ensemble |  20 |  7.7 |  4
[15] |  GNB ensemble |  20 |  16 |  8
[15] |  RF ensemble |  20 |  11.8 |  5
[15] |  SVM |  N/A |  11.9 |  10.2
Ours |  1-D ANN (no DA) |  varying |  11.8 |  9
Ours |  ANN (2D, no DA) |  varying |  7.1 |  7.7
Ours |  ANN (2D, DA) |  varying |  3.69 |  4.1
TABLE II: Accuracy of age estimation from fibroblast transcriptomes, for
various algorithms. Acronyms: MAE (Mean absolute error); MdAE (Median absolute
error); GNB (Gaussian naive Bayes); SVM (Support vector machines); RF (Random
forest); DA (data augmentation).

In fact, compared to the reference [15], it can be seen that our three key
contributions in this work, i.e., data representation, data augmentation, and
new framework for prediction, have helped to improve the prediction accuracy.
Without data augmentation it was not feasible to achieve better results using
the neural network. This is because almost all of the deep learning tools, and
in particular neural networks, need huge amounts of data to train[16, 32].
Moreover, our initial experiment without data augmentation did not lead to
desired training of the artificial neural networks, and consequently the
results were not satisfactory as it is reported in Table II. Our proposed
framework achieved a mean absolute error of 3.69 and median absolute error of
4.1 which shows the effectiveness of the method compared to the results of the
work on gene expression data such as [15, 9, 10] as well as work on
methylation data such as [7, 11, 12]

##  VII Conclusion

Human age estimation using genomic data has important implications for
research on multiple areas including understanding the individual aging,
relevant healthcare strategy and precision medicine regarding the aging.

In this paper, we proposed a new framework for age estimation using a dataset
of gene expression values, for subjects with age ranging from 1 year to 94
years. The core idea of the proposed framework is to use a new representation
of the information in the gene expression data, along with appropriate data
augmentation for this type of data to improve the accuracy. This is then
accompanied with a suitable neural network architecture to more accurately
estimate the age. Experiments using the proposed framework show the
effectiveness of the proposed approach. A comparative analysis with state-of-
the-art methods in age estimation using genomic data is also included.

## References

  * [1] Stegeman, R., and V. M. Weake. ”Transcriptional signatures of aging.” Journal of molecular biology 429, no. 16 (2017): 2427-2437.
  * [2] Dowell, Robin D., Owen Ryan, An Jansen, Doris Cheung, Sudeep Agarwala, Timothy Danford, Douglas A. Bernstein et al. ”Genotype to phenotype: a complex problem.” Science 328, no. 5977 (2010): 469-469.
  * [3] Mak, H. Craig, and Quincey Justman. ”Genotype-Phenotype Mapping Meets Single Cell Biology.” Cell systems 4, no. 1 (2017): 1-2.
  * [4] Kisiel, Marta A., and Agnes S. Klar. ”Isolation and culture of human dermal fibroblasts.” In Skin Tissue Engineering, pp. 71-78. Humana, New York, NY, 2019.
  * [5] Tsuji, Akiko, Atsushi Ishiko, Tomoya Takasaki, and Noriaki Ikeda. ”Estimating age of humans based on telomere shortening.” Forensic Science International 126, no. 3 (2002): 197-199.
  * [6] Richter, Christoph, Jeen-Woo Park, and Bruce N. Ames. ”Normal oxidative damage to mitochondrial and nuclear DNA is extensive.” Proceedings of the National Academy of Sciences 85, no. 17 (1988): 6465-6467.
  * [7] Horvath, Steve. ”DNA methylation age of human tissues and cell types.” Genome biology 14, no. 10 (2013): 1-20.
  * [8] Eipel, Monika, Felix Mayer, Tanja Arent, Marcelo RP Ferreira, Carina Birkhofer, Uwe Gerstenmaier, Ivan G. Costa, Stefanie Ritz-Timme, and Wolfgang Wagner. ”Epigenetic age predictions based on buccal swabs are more precise in combination with cell type-specific DNA methylation signatures.” Aging (Albany NY) 8, no. 5 (2016): 1034..
  * [9] Peters, Marjolein J., Roby Joehanes, Luke C. Pilling, Claudia Schurmann, Karen N. Conneely, Joseph Powell, Eva Reinmaa et al. ”The transcriptional landscape of age in human peripheral blood.” Nature communications 6, no. 1 (2015): 1-14.
  * [10] Holly, Alice C., David Melzer, Luke C. Pilling, William Henley, Dena G. Hernandez, Andrew B. Singleton, Stefania Bandinelli, Jack M. Guralnik, Luigi Ferrucci, and Lorna W. Harries. ”Towards a gene expression biomarker set for human biological age.” Aging cell 12, no. 2 (2013): 324-326.
  * [11] Hannum, Gregory, Justin Guinney, Ling Zhao, L. I. Zhang, Guy Hughes, SriniVas Sadda, Brandy Klotzle et al. ”Genome-wide methylation profiles reveal quantitative views of human aging rates.” Molecular cell 49, no. 2 (2013): 359-367.
  * [12] Xu, Cheng, Hongzhu Qu, Guangyu Wang, Bingbing Xie, Yi Shi, Yaran Yang, Zhao Zhao et al. ”A novel strategy for forensic age prediction by DNA methylation and support vector regression model.” Scientific reports 5, no. 1 (2015): 1-10.
  * [13] Belsky, Daniel W., Terrie E. Moffitt, Alan A. Cohen, David L. Corcoran, Morgan E. Levine, Joseph A. Prinz, Jonathan Schaefer et al. ”Eleven telomere, epigenetic clock, and biomarker-composite quantifications of biological aging: do they measure the same thing?.” American Journal of Epidemiology 187, no. 6 (2018): 1220-1230.
  * [14] Ashiqur Rahman, Syed, Peter Giacobbi, Lee Pyles, Charles Mullett, Gianfranco Doretto, and Donald A. Adjeroh. ”Deep learning for biological age estimation.” Briefings in bioinformatics 22, no. 2 (2021): 1767-1781.
  * [15] Fleischer, Jason G., Roberta Schulte, Hsiao H. Tsai, Swati Tyagi, Arkaitz Ibarra, Maxim N. Shokhirev, Ling Huang, Martin W. Hetzer, and Saket Navlakha. ”Predicting age from the transcriptome of human dermal fibroblasts.” Genome biology 19, no. 1 (2018): 1-8.
  * [16] Sun, Chen, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. ”Revisiting unreasonable effectiveness of data in deep learning era.” In Proceedings of the IEEE international conference on computer vision, pp. 843-852. 2017.
  * [17] Zhang, Yan, Rory Wilson, Jonathan Heiss, Lutz P. Breitling, Kai-Uwe Saum, Ben Schöttker, Bernd Holleczek, Melanie Waldenberger, Annette Peters, and Hermann Brenner. ”DNA methylation signatures in peripheral blood strongly predict all-cause mortality.” Nature communications 8, no. 1 (2017): 1-11.
  * [18] Dekker, Job, and Tom Misteli. ”Long-range chromatin interactions.” Cold Spring Harbor perspectives in biology 7, no. 10 (2015): a019356.
  * [19] Egecioglu, Defne, and Jason H. Brickner. ”Gene positioning and expression.” Current opinion in cell biology 23, no. 3 (2011): 338-345.
  * [20] Sanyal, Amartya, Bryan R. Lajoie, Gaurav Jain, and Job Dekker. ”The long-range interaction landscape of gene promoters.” Nature 489, no. 7414 (2012): 109-113.
  * [21] Liu, Yufei, Yuan Zhou, Xin Liu, Fang Dong, Chang Wang, and Zihong Wang. ”Wasserstein GAN-based small-sample augmentation for new-generation artificial intelligence: a case study of cancer-staging data in biology.” Engineering 5, no. 1 (2019): 156-163.
  * [22] Rodrigue, Nicolas, and Stéphane Aris-Brosou. ”Fast bayesian choice of phylogenetic models: Prospecting data augmentation–based thermodynamic integration.” Systematic biology 60, no. 6 (2011): 881-887.
  * [23] Royle, J. Andrew. ”Analysis of capture–recapture models with individual covariates using data augmentation.” Biometrics 65, no. 1 (2009): 267-274.
  * [24] Shorten, Connor, and Taghi M. Khoshgoftaar. ”A survey on image data augmentation for deep learning.” Journal of Big Data 6, no. 1 (2019): 1-48.
  * [25] DeVries, Terrance, and Graham W. Taylor. ”Dataset augmentation in feature space.” arXiv preprint arXiv:1702.05538 (2017).
  * [26] Wen, Qingsong, Liang Sun, Fan Yang, Xiaomin Song, Jingkun Gao, Xue Wang, and Huan Xu. ”Time series data augmentation for deep learning: A survey.” arXiv preprint arXiv:2002.12478 (2020).
  * [27] Le Guennec, Arthur, Simon Malinowski, and Romain Tavenard. ”Data augmentation for time series classification using convolutional neural networks.” In ECML/PKDD workshop on advanced analytics and learning on temporal data. 2016.
  * [28] Mohamadi, Salman, Hamidreza Amindavar, and SM Ali Tayaranian Hosseini. ”Arima-garch modeling for epileptic seizure prediction.” In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 994-998. IEEE, 2017.
  * [29] Geifman, Nophar, Raphael Cohen, and Eitan Rubin. ”Redefining meaningful age groups in the context of disease.” Age 35, no. 6 (2013): 2357-2366.
  * [30] Mohamadi, Salman, Donald A. Adjeroh, Behnoush Behi, and Hamidreza Amindavar. ”A new framework for spatial modeling and synthesis of genomic sequences.” In 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 2221-2226. IEEE, 2020.
  * [31] Zhang, Qian, Costanza L. Vallerga, Rosie M. Walker, Tian Lin, Anjali K. Henders, Grant W. Montgomery, Ji He et al. ”Improved prediction of chronological age from DNA methylation limits it as a biomarker of ageing.” bioRxiv (2018): 327890.
  * [32] Cho, Junghwan, Kyewook Lee, Ellie Shin, Garry Choy, and Synho Do. ”How much data is needed to train a medical image deep learning system to achieve necessary high accuracy?.” arXiv preprint arXiv:1511.06348 (2015).
  * [33] Hajibabaee, Parisa, Farhad Pourkamali-Anaraki, and Mohammad Amin Hariri-Ardebili. ”Kernel Matrix Approximation on Class-Imbalanced Data With an Application to Scientific Simulation.” IEEE Access (2021).
  * [34] Mohamadi, Salman, and Hamidreza Amindavar. ”Deep bayesian active learning, a brief survey on recent advances.” arXiv preprint arXiv:2012.08044 (2020).
  * [35] Kingma, Diederik P., and Jimmy Ba. ”Adam: A method for stochastic optimization.” arXiv preprint arXiv:1412.6980 (2014).
  * [36] Steven Eyobu, Odongo, and Dong Seog Han. ”Feature representation and data augmentation for human activity classification based on wearable IMU sensor data using a deep LSTM neural network.” Sensors 18, no. 9 (2018): 2892.
  * [37] Mohamadi, Salman, Farhang Yeganegi, and Nasser M. Nasrabadi. ”Detection and Statistical Modeling of Birth-Death Anomaly.” arXiv preprint arXiv:1906.11788 (2019).
  * [38] Hajibabaee, Parisa, Farhad Pourkamali-Anaraki, and Mohammad Amin Hariri-Ardebili. ”An Empirical Evaluation of the t-SNE Algorithm for Data Visualization in Structural Engineering.” arXiv preprint arXiv:2109.08795 (2021).
  * [39] Mostofa, Moktari, Salman Mohamadi, Jeremy Dawson, and Nasser M. Nasrabadi. ”Deep GAN-Based Cross-Spectral Cross-Resolution Iris Recognition.” arXiv preprint arXiv:2108.01569 (2021).