# Generating multi-type sequences of temporal events to improve fraud
detection in game advertising

Lun Jiang, Nima Salehi Sadghiani, Zhuo Tao  UnitySan FranciscoCA94103 lun,
nimas, zhuo @unity3d.com

(2021)

###### Abstract.

Fraudulent activities related to online advertising can potentially harm the
trust advertisers put in advertising networks and sour the gaming experience
for users. Pay-Per-Click/Install (PPC/I) advertising is one of the main
revenue models in game monetization. Widespread use of the PPC/I model has led
to a rise in click/install fraud events in games. The majority of traffic in
ad networks is non-fraudulent, which imposes difficulties on machine learning
based fraud detection systems to deal with highly skewed labels. From the ad
network standpoint, user activities are multi-type sequences of temporal
events consisting of event types and corresponding time intervals. Time Long
Short-Term Memory (Time-LSTM) network cells have been proved effective in
modeling intrinsic hidden patterns with non-uniform time intervals. In this
study, we propose using a variant of Time-LSTM cells in combination with a
modified version of Sequence Generative Adversarial Generative (SeqGAN)to
generate artificial sequences to mimic the fraudulent user patterns in ad
traffic. We also propose using a Critic network instead of Monte-Carlo (MC)
roll-out in training SeqGAN to reduce computational costs. The GAN-generated
sequences can be used to enhance the classification ability of event-based
fraud detection classifiers.Our extensive experiments based on synthetic data
have shown the trained generator has the capability to generate sequences with
desired properties measured by multiple criteria.

Fraud detection, Temporal events, Generative adversarial network,
Reinforcement learning

††copyright: acmcopyright††journalyear: 2021††submissionid: 2374††ccs:
Computing methodologies Adversarial learning††ccs: Information systems
Computational advertising

##  1\. Introduction

Game developers can monetize their games by selling in-game ad placements to
advertisers. In-game ads can be integrated to the game either through a banner
in the background or commercials during breaks (when a certain part of the
game is completed). There are four main elements in the game advertising
ecosystem: publishers or developers, advertisers 111Advertisers can be
publishers. (demand), advertising network, and users (supply) (Mouawi et al.,
2019). Game advertising networks connect advertisers with game developers and
serve billions of ads to user devices triggering an enormous amount of ad
events. For example, Unity Ads reports 22.9B+ monthly global ad impressions,
reaching 2B+ monthly active end-users worldwide
222https://www.businesswire.com/news/home/20201013005191/en/.

There are multiple types of ad events in the real-world, e.g. request, start,
view, click, install, etc. Each type stands for one specific kind of ad-
related user action happening at a specific time. A complete ad life cycle can
be depicted as a temporal sequence of ad events, each of which is a tuple of
event type with corresponding time interval. Click and install are two kinds
of ad events commonly associated with ad revenue. Pay-Per-Click (Kapoor et
al., 2016) and Pay-Per-Install (Thomas et al., 2016) are the most widely used
advertising models for pricing.

Naturally, as advertisers allocate more of their budgets into this ecosystem,
more fraudsters tend to abuse the advertising networks and defraud advertisers
of their money (Nagaraja and Shah, 2019). Fraudulent ad activities aimed at
generating illegitimate ad revenues or unearned benefits are one of the major
threats to these advertising models. Common types of fraudulent activities
include fake impressions (Haider et al., 2018), click bots (Haddadi, 2010;
Kudugunta and Ferrara, 2018), click farms (Oentaryo et al., 2014), etc. (Zhu
et al., 2017b). Fraud in the advertising ecosystem is top of mind for
advertisers, developers, and advertising networks. Having their reputation and
integrity on the line, a huge amount of effort has been focused on fraud
detection activities from the advertising network’s side (Jianyu et al., 2017;
Dong et al., 2018; Mouawi et al., 2019; Nagaraja and Shah, 2019).

Studying the behaviors of ads temporal sequences helps to identify intrinsic
hidden patterns committed by fake or malicious users in advertising networks.
Given the massive ad activity data in game advertising networks, machine
learning-based approaches have become popular in the industry.

However, it is not a straightforward task to train machine learning models
directly on fraudulent and benign sequences collected from ad activities (Choi
and Lim, 2020). The vast majority of ad traffic is non-fraudulent and data
labeling by human experts is time-consuming, which results in low availability
of labeled fraud sequences and a high class imbalance between the fraud/non-
fraud training data. Simply oversampling the minority fraud class can cause
significant overfitting while undersampling the majority non-fraud may lead to
information loss and yield a tiny training dataset (Ba, 2019). To mitigate
this data availability problem, in this study we present a novel data
generator which is able to learn the intrinsic hidden patterns from sequential
training data and generate emulated sequences with high qualities.

The main contributions of our work can be summarized as follows:

  1. (1)

We build a data generator which is able to generate multi-type temporal
sequences with non-uniform time intervals.

  2. (2)

We present a new application for event-based sequence GAN for fraud detection
in game advertising.

  3. (3)

We propose a new way of sequence GAN training by employing a Critic network.

##  2\. Related Work

Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) have drawn a
significant attention as a framework for training generative models capable of
producing synthetic data with desired structures and properties (Killoran et
al., 2017). Ba, 2019 proposed using GANs to generate data that mimics training
data as an augmented oversampling method with an application in credit card
fraud. The generated data is used to assist the classification of credit card
fraud Ba (2019).

Despite of the remarkable success of GANs in generating real-looking data,
very few studies focus on generating sequence data. This is due to the fact
that taking advantage of GAN to generate temporal sequence data with intrinsic
hidden patterns can be more challenging. Recurrent Neural Network (RNN)
solutions have become state-of-the-art methods on modeling sequential data.
Hyland et al., 2017 developed a Recurrent Conditional GAN (RCGAN) to generate
real-valued multi-dimensional time series, and then used the generated series
for supervised training (Esteban et al., 2017). The time series data in their
study were physiological signals sampled at specific fixed frequencies,
whereas ad events data have a higher complexity in terms of non-uniform time
intervals and discrete event types, and thus can not be modeled as wave
signals. In ad event sequences, two events with a short time interval tend to
be more correlated than events with larger time intervals.

Killoran et al., 2017 proposed a GAN-based generative model for DNA along with
a activation maximization technique for DNA sequence data. Their experiments
have shown that these generative techniques can learn important structure from
DNA sequences and can be used to design new DNA sequences with desired
properties (Killoran et al., 2017). Similar to the previous study, their focus
is on the fixed interval sequences. Zheng et al., 2019 adopted the LSTM-
Autoencoder to encode the benign users into a hidden space. They proposed
using One-Class Adversarial Network (OCAN) for the training process of the GAN
model. In their training framework, the discriminator is trained to be a
classifier for distinguishing benign users and the generator produces samples
that are complementary to the representations of benign users (Zheng et al.,
2019). However, since OCAN has not been trained on the malicious users
dataset, it is hard to measure the quality of the generated sequences and
understand the pattern they follow.

###  2.1. Time-LSTM

The common LSTM cells have shown remarkable success to generate complex
sequences with long-range structures in numerous domains (Graves, 2013).
Recently, a combination of a LSTM cell with a dimension-reducing symbolic
representation was proposed to forecast time series (Elsworth and Güttel,
2020). However, RNN models usually consider the sequence of the events and
ignore their intervals. Thus, these models are not suitable to process non-
uniformly distributed events generated in continuous time. This major drawback
of the traditional recurrent models has led to development of Phased-LSTM
(Neil et al., 2016). An LSTM variant to model event-based sequences. Neil et
al., 2016 proposed adding a new time gate to the traditional LSTM cells. The
time gate is controlled by a parametrized oscillation and has three phases,
i.e. rises from $0$ to $1$ in the first phase, drops from $1$ to $0$ in the
second phase, and it remains inactive in the third phase. Xiao et al., 2017
proposed using an intensity function modulated synergistically by one RNN
(Xiao et al., 2017). Further, the Time-Aware LSTM (T-LSTM) cells were proposed
in (Baytas et al., 2017) to handle non-uniform time intervals in longitudinal
patient records. They used the T-LSTM cell in an auto-encoder to learn a
single representation for sequential records of patients.

Zhu et al., 2017 proposed a LSTM variant named Time-LSTM to model users’
sequential actions in which LSTM cells are equipped with time gates to model
time intervals (Zhu et al., 2017a). In this paper, Time-LSTM was used for
recommending items to users. We find it useful to model event types and time
intervals using Time-LSTM cells because of its ability in capturing the
intrinsic patterns of fraudulent sequences, and thus to understand the
internal mechanisms of how fraudulent activities are generated. We implemented
our version of Time-LSTM cells in Keras and used it in the architecture of the
generators and discriminators of our GAN models.

###  2.2. GAN for Sequence Data

When generating continuous outputs, gradient updates can be passed from the
discriminator to the generator. However, for discrete outputs, due to lack of
differentiability, the backpropagation does not work properly. Yu et al., 2017
addressed the issue of training GAN models to generate sequences of discrete
tokens. They proposed a sequence generation framework, called SeqGAN that
models the data generator as a stochastic policy in Reinforcement Learning
(RL) (Yu et al., 2017). They regarded the generator as a stochastic
parametrized policy. Their policy gradient employs MC search to approximate
the state values, which is a computationally expensive process in the training
loop. Moreover, the SeqGAN is limited only to discrete token generation, in
our work we propose a modified version of seqGAN in combination with Time-LSTM
cells that can generate both discrete tokens and continuous time intervals. To
efficiently train the policy network, we employ a Critic network to
approximate the return given a partially generated sequence to speed up the
training process. This approach also brings the potential to use a trained
Critic network for early fraud detection from partial sequences.

Zhao et al., 2020 presents an application of SeqGAN in recommendation systems.
The paper solves the slow convergence and unstable RL training by using Actor-
Critic algorithm instead of MC roll-outs (Zhao et al., 2020). Their generator
model produces the entire recommended sequences given the interaction history
while the discriminator learns to maximize the score of ground-truth and
minimize the score of generated sequences. In each step, the generator $G$
generates a token by top-k beam search based on the model distribution. In our
work, we directly sample from the distribution of the output probabilities of
the tokens. While our methodologies are close, we are aiming for different
goals. We optimize the generated data to solve the sample imbalance problem
while they optimize for better recommendations. Therefore, different
evaluation metrics are needed. Our methodologies also differ in the training
strategy. For example, we used a Critic network as the baseline whereas they
used Temporal-Difference bootstrap targets. They pre-trained the discriminator
on the generated data to reduce the exposure bias while we pre-trained the
discriminator on the actual training data for improving the metrics we use in
our experiments. More importantly they do not include time intervals as an
attribute in their model while we have time intervals in our models.

Recently, Smith and Smith 2020, proposed using two generators, a convolutional
generator that transforms a single random vector to a RGB spectrogram image,
and a second generator that receives the 2D spectrogram images and outputs a
time series (Smith and Smith, 2020). In our work, we find the RL training
process a more natural way to address the issue of generating discrete
outputs.

##  3\. Methodology

Notation. In this paper, all sequences, sets are denoted by bold letters like
$\bm{A}$. We use $|\bm{A}|$ to refer to the size/length of a sequences or set.

In this section, we introduce a new methodology to generate multi-type
sequences using seqGAN and Time-LSTM cells.

###  3.1. Definitions

An original ad event sequence of length $L$ is composed of two subsequences,
the subsequence of event types $\bm{x}$ and the subsequence of time stamps.
First, we transform the time stamps $\bm{t}$ into time intervals
$\bm{\Delta\textbf{t}}$ and $\Delta t_{m}=t_{m}-t_{m-1},\forall m\in[1,L]$,
and $\Delta t_{1}=t_{1}-0$. Then, we combine the event types and time
intervals into a joint multi-type sequence $\bm{A}$:

$\displaystyle\bm{A}$ $\displaystyle=\bm{A}_{L}=\bm{A}_{1:L}$ (1)
$\displaystyle=\\{(x_{1},\Delta t_{1}),(x_{2},\Delta
t_{2}),\dots,\Delta(x_{m},\Delta t_{m}),\dots,(x_{L},\Delta t_{L})\\}$

where $\bm{A}_{m}=\bm{A}_{1:m}$ denotes a partial sequence from time step $1$
to time step $m$.

###  3.2. Time-LSTM

In this paper, we adopt the type $1$ Time-LSTM cell from (Zhu et al., 2017a).
The update equations of this Time-LSTM cell are as follows:

(2) $\displaystyle i_{m}$ $\displaystyle=$
$\displaystyle\sigma_{i}(x_{m}W_{xi}+h_{m-1}W_{hi}+w_{ci}\odot c_{m-1}+b_{i})$
(3) $\displaystyle f_{m}$ $\displaystyle=$
$\displaystyle\sigma_{f}(x_{m}W_{xf}+h_{m-1}W_{hf}+w_{cf}\odot c_{m-1}+b_{f})$
(4) $\displaystyle T_{m}$ $\displaystyle=$
$\displaystyle\sigma_{t}(x_{m}W_{xt}+\sigma_{\Delta t}(\Delta
t_{m}W_{tt})+b_{t})$ (5) $\displaystyle c_{m}$ $\displaystyle=$ $\displaystyle
f_{m}\odot c_{m-1}$ $\displaystyle+\ i_{m}\odot
T_{m}\odot\sigma_{c}(x_{m}W_{xc}+h_{m-1}W_{hc}+b_{c})$ (6) $\displaystyle
o_{m}$ $\displaystyle=$ $\displaystyle\sigma_{o}(x_{m}W_{xo}+\Delta
tW_{to}+h_{m-1}W_{ho}+w_{co}\odot c_{m}+b_{o})$ (7) $\displaystyle h_{m}$
$\displaystyle=$ $\displaystyle o_{m}\odot\sigma_{h}(c_{m})$

$x_{m}$ is the input feature vector at time step $m$, which in our case would
be the embedding of the event type. $\Delta t_{m}$ is the input time interval
at time step $m$. $i_{m}$, $f_{m}$, $T_{m}$, $o_{m}$ are the activations of
input, forget, time, output gates, respectively. $c_{m}$, $h_{m}$ are the cell
activation and hidden state. $c_{m-1}$ is the cell state of the previous time
step $(m-1)$. $\sigma_{i}$, $\sigma_{f}$, $\sigma_{o}$ are the sigmoid
function and $\sigma_{c}$, $\sigma_{h}$ are the $\tanh$ function. $W_{xi}$,
$W_{hi}$, $W_{xf}$, $W_{hf}$, $W_{xt}$, $W_{xc}$, $W_{tt}$, $W_{xc}$,
$W_{hc}$, $W_{xo}$, $W_{to}$, $W_{ho}$ , $W_{co}$ are the weight parameters of
the cell. $w_{ci}$, $w_{cf}$, $w_{co}$ are peephole parameters.

###  3.3. RL and Policy improvement to train GAN

We implement a modified version of seqGAN model to generate multi-type
temporal sequences. Time-LSTM cells are utilized in our implementations for
both the generator $G$ and the discriminator $D$.

The sequence generation process of our generator $G$ can be modeled as a
sequential decision process in RL. The state at time step $m$ is defined as:

(8) $\displaystyle S_{m}=[T_{m}(\bm{A}_{m},\theta);h_{m}(\bm{A}_{m},\theta)]$

where $T_{m}$ is the time gate activate in (4) and $h_{m}$ is the hidden state
in (7). $\bm{A}_{m}$ is the partial sequence at time step $m$ and $\theta$ is
the trainable parameters of the Time-LSTM cell.

The action at time step $m$ is a combination of two parts:
$a_{m}=\\{a^{x}_{m}$, $a^{\Delta t}_{m}\\}$, where $a^{x}_{m}$ is the action
to find the next event type $x_{m+1}$ and $a^{\Delta t}_{m}$ is the action to
find the next time interval $\Delta t_{m+1}$. Thus a new partial sequence
$\bm{A}_{m+1}$ can be formed step by step, until a complete sequence
$\bm{A}=\bm{A}_{L}$ of length $L$ described in (1) is constructed.

To make decisions in this sequence generation process, we employ a hybrid
policy to represent action spaces with both continuous and discrete dimensions
(similar to the idea in (Neunert et al., 2020)). This policy is designed to
choose discrete event types and continuous time intervals, assuming their
action spaces are independent. Then we use a categorical distribution and a
Gaussian distribution to model the policy distributions for the event types
and the time intervals respectively. So the hybrid generator policy can be
defined as:

$\displaystyle G_{\theta}(a_{m}|S_{m})$
$\displaystyle=\pi^{x}_{\theta}(a^{x}_{m}|S_{m})\cdot\pi^{\Delta
t}_{\theta}(a^{\Delta t}_{m}|S_{m})$ (9)
$\displaystyle=Cat(x|\alpha_{\theta}(S_{m}))\cdot\mathcal{N}(\Delta
t|\mu_{\theta}(S_{m}),\sigma^{2}_{\theta}(S_{m}))$

where $x\in\bm{K},\Delta t\in R_{\geq 0}$. $\bm{K}$ is the set of all possible
event types.

When generating a new event type and time interval at each step, we follow the
generator policy and sample from categorical and normal distributions
independently and concatenate them to obtain the action vector $a_{m}$, then
append them to the current partial sequence $\bm{A}_{m}$ to obtain a new
partial sequence $\bm{A}_{m+1}$. Once a complete sequence of length $L$ has
been generated, we pass the sequence $\bm{A}$ to the Discriminator $D$ which
predicts the probability of the sequence to be real against generated:

(10) $\displaystyle D_{\phi}(\bm{A})$ $\displaystyle=Pr(Y=1|\bm{A};\phi)$

The feedback from $D$ can be used in training $G$ such that $G$ can better
learn how to generate sequences similar to real training data to deceive $D$.
Because the discrete data is not differentiable, gradients can not passed back
to generator like in image-base GANs.

The original seqGAN training uses Policy Gradient method with Monte-Carlo
roll-out to optimize the policy.(Yu et al., 2017). In order to reduce variance
in the optimization process, seqGAN runs the roll-out policy starting from
current state till the end of the sequence for multiple times to get the mean
return. Here we use an Actor-Critic method with a Critic network instead of MC
roll-out to estimate the value of any state, which is computationally more
efficient.(Bhatnagar et al., 2007).

The Critic network models a state-dependent value
$\hat{V}^{G_{\theta}}_{\psi}(S_{m})$ for a partially generated sequence
$A_{m}$ under policy $G_{\theta}$, which is defined as the expected future
return for a complete sequence provided by the Discriminator $D$:

(11)
$\displaystyle\hat{V}^{G_{\theta}}_{\psi}(S_{m})=\mathbb{E}_{\bm{A}_{m+1:L}\sim
G_{\theta}(S_{m})}[D_{\phi}(\bm{A})]$

The value function parameters are updated during training by minimizing the
mean squared error between the true return $D_{\phi}(\bm{A})$ and the value
function $\hat{V}^{G_{\theta}}_{\psi}(S_{m})$:

(12) $\displaystyle
J(\psi)=\mathbb{E}[(D_{\phi}(\bm{A})-\hat{V}^{G_{\theta}}_{\psi}(S_{m}))^{2}]$

The difference between them,
$D_{\phi}(\bm{A})-\hat{V}^{G_{\theta}}_{\psi}(S_{m})$, is named the advantage
function, which is used in $G$ training and helps to reduce variance.

The goal of $G$ training is to choose actions based on a policy that maximizes
expected return. The object function of $G$ follows Policy Gradient method
(Sutton and Barto, 2018) which can be derived as:

(13)

Because of the independence assumption we made, the policy gradient term can
be broken down and written into a categorical cross-entropy and a Gaussian
log-likelihood as follows:

$\displaystyle\nabla_{\theta}\log G_{\theta}(a_{m}|S_{m})$
$\displaystyle=\nabla_{\theta}[\log
Cat(x|\alpha_{\theta}(S_{m}))+\log\mathcal{N}(\Delta
t|\mu_{\theta}(S_{m}),\sigma^{2}_{\theta}(S_{m}))]$
$\displaystyle=\nabla_{\theta}[\mathbb{E}_{x^{\prime}\in\bm{K}}\mathds{1}_{x^{\prime}}(x)\Pr(x=x^{\prime})$
(14) $\displaystyle\quad-\frac{(\Delta
t-\mu_{\theta}(S_{m}))^{2}}{2\sigma^{2}_{\theta}(S_{m})}-\frac{1}{2}\log(2\pi\sigma^{2}_{\theta}(S_{m}))]$

The goal of $D$ training to use distinguish generated sequences with true
sequences from training data.333Definitions of the training data, positive and
negative datasets are in section 4.1. $D_{\phi}$ is updated through minimizing
binary cross-entropy loss.

We keep training $G$ and $D$ alternatively. The Pseudo code of the entire
process is shown in Algorithm 1.

Algorithm 1 Sequence Generative Adversarial Nets

Require: generator policy $G_{\theta}$; Critic $\hat{V}^{G_{\theta}}_{\psi}$;
discriminator $D_{\phi}$; positive dataset
$\bm{\Omega}^{+}=\\{\bm{A}^{+}_{1:L}\\}$, negative dataset
$\bm{\Omega}^{-}=\\{\bm{A}^{-}_{1:L}\\}$

1:Initialize $G_{\theta}$, $D_{\phi}$, $D_{\phi},\hat{V}^{G_{\theta}}_{\psi}$
with random weights $\theta$, $\phi$, $\psi$

2:Pre-train $G_{\theta}$ using MLE on $\bm{\Omega}^{+}\cup\bm{\Omega}^{-}$.

3:Pre-train $D_{\phi}$ via minimizing binary cross-entropy on
$\bm{\Omega}^{+}\cup\bm{\Omega}^{-}$

4:repeat

5: for $G$-steps do

6: Generate a batch of complete sequences $\bm{A}\sim G_{\theta}$

7: Get total rewards $D_{\phi}(\bm{A})$ from discriminator

8: $x_{0}\leftarrow$ initial token

9: $\Delta t_{0}\leftarrow 0$

10: for $m$ in $0:L-1$ do

11: Calculate current state $S_{m}$ via Eq. (8)

12: Sample $x_{m+1}\sim Cat(x|\alpha_{\theta}(S_{m}))$

13: Sample $\Delta t_{m+1}\sim\mathcal{N}(\Delta
t|\mu_{\theta}(S_{m}),\sigma^{2}_{\theta}(S_{m}))$

14: Compute value estimate from Critic $\hat{V}^{G_{\theta}}_{\psi}(S_{m}))$

15: Compute the advantage
$(D_{\phi}(\bm{A})-\hat{V}^{G_{\theta}}_{\psi}(S_{m}))$

16: end for

17: Update Critic param. $\psi$ by minimizing Eq. (12)

18: Update generator param. $\theta$ via policy gradient Eq. (13)

19: end for

20: for $D$-steps do

21: Generate a batch of sequences $\bm{A}_{fake}\sim G_{\theta}$

22: Sample a batch of sequences $\bm{A}_{true}$ from $\bm{\Omega}^{+}$

23: Train discriminator $D_{\phi}$ on $\bm{A}_{fake}\cup\bm{A}_{true}$ and
update param. $\phi$ via minimizing binary cross-entropy

24: end for

25:until terminate condition satisfied

##  4\. Data Experiments

Due to the concerns about data privacy laws (e.g. GDPR 444General Data
Protection Regulation, CCPA 555California Consumer Privacy Act), and to
protect confidential details of the Unity Fraud Detection service, we decide
not to use real-world ad events data and patterns for this study, thus to
avoid data privacy issues and prevent fraudsters from reverse-engineering the
presented algorithms and rules to circumvent fraud detection systems. Instead,
we conduct our experiments on a synthetic dataset emulating real-world ad
events.

###  4.1. Synthetic Dataset

We define the synthetic dataset as $\bm{\Omega}$, and $6$ types of events
$\bm{K}=\\{a,b,c,d,\text{PAD},\text{INI}\\}$ where
$\bm{K^{\prime}}=\\{a,b,c,d\\}$ is the set of hypothetical ad event types; PAD
is reserved for padding and end token; INI is the dummy initial token marking
the beginning of a sequence, which always comes with a zero initial time
interval.

Each sequence in the synthetic dataset has a uniform length $L=21$, including
the dummy initial step $(x_{0},\Delta t_{0})=(\text{INI},0)$. For the
following steps, each event type $x_{m}$ is randomly sampled from
$\bm{K^{\prime}}$ with an equal probability, and each time interval $\Delta
t_{m}$ is sampled from a Chi-Square distribution with the degree of freedom
conditioned on $x_{m}$, i.e.:

(15) $\displaystyle x_{m}\sim\text{Uniform}\\{a,b,c,d\\}$ (20)
$\displaystyle\Delta t_{m}\sim\mathcal{X}^{2}(k),\quad
k=\left\\{\begin{array}[]{ll}8&\quad\text{if}\quad x_{m}=a\\\
16&\quad\text{if}\quad x_{m}=b\\\ 24&\quad\text{if}\quad x_{m}=c\\\
32&\quad\text{if}\quad x_{m}=d\\\ \end{array}\right.$

One example of a complete synthetic sequence is as below:

$\displaystyle\bm{A}_{e.g.}=[$
$\displaystyle(\text{INI},0),(d,28),(a,6),(a,3),(c,18),(c,32),(a,15),$
$\displaystyle(c,26),(c,27),(a,11),(a,3),(a,7),(a,10),(d,26),$
$\displaystyle(b,8),(c,24),(d,22),(a,6),(d,39),(d,19),(c,21)]$

Then, we split the synthetic dataset $\bm{\Omega}$ into a positive dataset
$\bm{\Omega}^{+}$ and a negative dataset $\bm{\Omega}^{-}$, by a set of human
defined rules. These rules are variants of real-world rules observed in ad
activities. In this study, we intentionally avoid using real patterns or rules
that appear in real fraud detection work ,in order to prevent potential
information leakage to fraudsters.

There rules we defined are as follows:

  1. (1)

A sequence starts with an $a$ event.

  2. (2)

There are more than three distinct types of events after the initial token,
and at least one of them is $a$.

  3. (3)

Each $d$ event is paired with one and only one previous $c$ event. Each $c$
event can be paired with at most one later $d$ event.

  4. (4)

The total number of $a$ events is greater or equal to that of $b$ events; The
total number of $b$ events is greater or equal to that of $c$ events; The
total number of $c$ events is greater or equal to that of $d$ events.

  5. (5)

The time delay between any two consecutive same events is no smaller than 10

  6. (6)

The time delay between any two paired $c$ and $d$ events is no greater than
50.

If a sequence follows more than three rules out of six, it is classified as a
positive sequence $\bm{A}^{+}\in\bm{\Omega}^{+}$, otherwise as a negative
sequence $\bm{A}^{-}\in\bm{\Omega}^{-}$.

The goal of GAN training is to teach the generator $D_{\theta}$ to learn the
intrinsic human-defined patterns in the synthetic dataset $\bm{\Omega}$, and
generate sequences satisfying as many above-mentioned rules as possible. In a
real-world application, those patterns can be hidden or unknown to human
experts, but a GAN is expected to learn and reproduce the patterns that are
not intuitive to humans.

###  4.2. Evaluation Metric

In the last few years, several different evaluation metrics for GANs have been
introduced in the literature. Among them, Fréchet Inception Distance (FID)
(Heusel et al., 2017) has been used extensively (DeVries et al., 2019).
However, it is not enough to show the effectiveness of our training on multi-
type sequences using only one metric. This is due to the fact that our
sequences consist of a discrete categorical part (event type) and a continuous
numerical part (time interval). We propose using multiple metrics including a
Rule-Based Quality (RBQ) score (to check if the sequences follow our validity
rules), Mean Absolute Deviation (MAD) metric (event types are diverse), and
Maximum Mean Discrepancy (MMD) (Fortet and Mourier, 1953) (dissimilarity
between event types or time intervals) in addition to FID for time intervals.
The arrows ($\uparrow\downarrow$) show the improvement directions.

RBQ $\uparrow$. The quality of a generated sequence is measured by a metric
derived from the six rules we defined in section 4.1. The general intuition
behind the RBQ score is that it is less probable for a generated sequence to
follow multiple human-defined rules, so that a sequence with more desired
patterns deserves a higher quality score. As a result, in RBQ, rule
combinations are weighted by their length, where an individual rule is
considered as a combination of length $1$. The six rules are considered
equally important in calculation. We employ a geometric series with common
ratio $2$ for weighting. The RBQ score for a sequence $\bm{A}$ is defined as
follows:

(21) $RBQ(\bm{A})=\sum_{\lambda\in\bm{\Lambda}(\bm{A})}2^{|\lambda|}$

where $\bm{\Lambda}(\bm{A})$ is the set of all rule combinations that sequence
$\bm{A}$ follows, $\lambda$ is one rule combination, and $|\lambda|$ is its
length. For example, if a sequences follow rule $1$ and rule $2$ described in
section 4.1, then it contains $3$ different rule combinations
$\\{1,2,(1,2)\\}$, thus yields an RBQ score of $2+2+2^{2}=8$.

MAD $\uparrow$. We propose using MAD to measure statistical dispersion of the
categorical part of the multi-type sequences, i.e., the event types. We use
dispersion as a proxy for diversity of the generated event types. Basically,
we one-hot encode the event types and compute the mean absolute deviation of
each sequence from the median of all sequences. Median is known to be more
robust to noise and fits our need to have categorical values as opposed to
mean. Given the diversity oracle, we compare the MAD score of any batches of
sequences against the MAD score of a batch of sampled sequences from our
$\bm{\Omega}^{+}$ dateset as the comparison base. MAD can be computed using:

(22)
$MAD(\bm{B})=\frac{1}{|\bm{B}|}\sum_{\bm{A}\in\bm{B}}\sum_{m=1}^{L}\left|x_{m}^{\bm{A}}-\tilde{E}_{m}(\bm{B})\right|$

where $\bm{B}$ is a batch of sequences, $|\bm{B}|$ is the batch size, $\bm{A}$
is a sequence of length $L$ in $\bm{B}$, $x_{m}^{\bm{A}}$ is the event type of
step $m$ in $\bm{A}$, $\tilde{E}_{m}(\bm{B})$ is the median of the event types
at step $m$ across the batch $\bm{B}$.

FID $\downarrow$. We use FID to evaluate the numerical part of the multi-type
sequences, i.e., the time intervals. It focuses on capturing certain desirable
properties including the quality and diversity of the generated sequences. FID
performs well in terms of robustness and computational efficiency (Borji,
2019). The Fréchet distance between two Gaussians is defined as:

(23)
$FID(x,g)=\left\lVert\mu_{x}-\mu_{g}\right\rVert^{2}_{2}+Tr\left(\Sigma_{x}+\Sigma_{g}-2\left(\Sigma_{x}\Sigma_{g}\right)^{\frac{1}{2}}\right)$

where $\left(\mu_{x},\Sigma_{x}\right)$ and $\left(\mu_{g},\Sigma_{g}\right)$
are the means and covariances of the samples from the real data distribution
and model distribution, respectively.

MMD $\downarrow$. We also employ MMD to evaluate the time intervals. This
measure computes the dissimilarity between two probability distributions
$P_{r}$ and $P_{g}$ using samples drawn independently from each. We use an
estimator with Radial Basis Function (RBF) kernel $k$, which is:

$\displaystyle MMD_{k}(X,Y)=$ $\displaystyle\frac{1}{{n\choose 2}}\sum_{i\neq
i^{\prime}}k(\bm{x}_{i},\bm{x}^{\prime}_{i})-\sum_{i\neq j}\frac{2}{{n\choose
2}}k(\bm{x}_{i},\bm{y}^{\prime}_{j})$ (24) $\displaystyle+\frac{1}{{n\choose
2}}\sum_{j\neq j^{\prime}}k(\bm{y}_{j},\bm{y}^{\prime}_{j})$

FIDH $\downarrow$. This metric is a variant of FID, which views the Time-LSTM
hidden state $[T_{m}({\phi});h_{m}({\phi})]$ in $D_{\phi}(\bm{A})$ as a
continuous multivariate Gaussian distribution. Then the FID score will be
computed between two hidden states using Eq. 23. Hidden states can be viewed
as representations of the input sequence. FIDH uses the information from both
the discrete part and the continuous part of our multi-typed sequence.

###  4.3. Experiment Setup

We use the $\bm{\Omega}^{+}$ and $\bm{\Omega}^{+}$ datasets defined in section
4.1 for model training and evaluation. Both datasets contain around $150000$
data samples. As described in Algorithm 1, we first pre-train $G$ and $D$
until convergence, and then start RL training for the pre-trained $G$ and $D$.
In this section, we compare the generated sequences from the following models:

  * •

$G0$: Generator with initial random model parameters.

  * •

$G1$: Generator pre-trained using MLE.

  * •

$G2$: Generator trained by algorithm 1.

We monitor the training process and use the metrics defined in section 4.2 to
evaluate model performance during training, which are plotted in Figure 1.

Figure 1. Performance metrics calculated during the training process.

The ratio between $G$-steps and $D$-steps is set to $1:1$. Both $G$ and $D$
have the same batch size $64$, and use the SGD optimizer with learning rate
$10^{-4}$. We save the models during the training process and present the best
performed models for evaluation.

As shown in Figure 1, $G0$ is the randomly initialized model (blue line), $G1$
is the model pre-trained by MLE after $2400$ steps (orange line), and $G2$ is
the generator which is trained for $3000$ steps (green line) using algorithm
1. During the pre-training process, RBQ and MAD increase, while FID and MMD
decrease. After the pre-training finishes and RL training starts, RBQ keeps
increasing, and FID surges dramatically after certain point ($3000$ steps in
Figure 1), where we stop training because it indicates a mode-collapsed
generator. Notably, the FID score measured in training process is calculated
between the training batch sampled from $\bm{\Omega}^{+}$ and the evaluation
batch generated by $G$, both of size $64$, which is different from the FID and
FIDH scores in Table 1.

###  4.4. Experiment Results

To evaluate the performance of a generator after training, we use the trained
generator to create test datasets. Each model generates $30$ batches of size
$10000$. Then, we performed a two-sample t-test to compare each test dataset
with samples of the same size drawn from $\bm{\Omega}^{+}$ and
$\bm{\Omega}^{-}$.

Oracle scores. The Tables 1 show the mean values of the different metrics over
$30$ batches. In particular, the MAD, FID score and FID score for hidden units
(FIDH) are calculated respectively using data sampled from $\bm{\Omega}^{+}$
as the base for the comparisons. The results are presented in 1.

Table 1.  Oracle metrics calculated using $\bm{\Omega^{+}}$ as base Samp. | RBQ $\uparrow$ | MAD $\uparrow$ | FID $\downarrow$ | MMD $\downarrow$ | FIDH $\downarrow$
---|---|---|---|---|---
G0 | 14.7290 | 1.1114 | 9719.9854 | 0.1563 | 5.4784
G1 | 81.0501 | 1.3208 | 103.7343 | 0.0002 | 3.3819
G2 | 123.5881 | 1.2503 | 187.6541 | 0.0002 | 2.6775
$\bm{\Omega^{+}}$ | 122.1438 | 1.2880 | 0.0000 | 0.0000 | —
$\bm{\Omega^{-}}$ | 12.2944 | 1.4534 | 64.6673 | 0.0002 | —

The results demonstrate that the sequences generated by $G2$ network have a
significantly higher RBQ score than that generated by MLE pre-trained
generator $G1$ and randomly initialized generator $G0$. The RBQ score of $G2$
is close to the samples drawn from $\bm{\Omega^{+}}$, which shows the high
quality of these sequences. It indicates the generator $G2$ is able to learn
the intrinsic patterns and rules in $\bm{\Omega^{+}}$ during RL-training, and
then generate sequences that mimics these patterns to deceive the
discriminator.

From the perspective of FID and MAD, $G2$ has lower scores compared to the MLE
pre-trained generator $G1$. As we discussed, in section 4.2, FID evaluates the
continuous distribution of the time interval $\Delta t$, and MAD measures the
dispersion of the discrete event type $x$ in a sequence. In each sequence from
the training datasets, the two features are intrinsically correlated. However,
FID only evaluates the continuous part of the sequences and MAD evaluates the
discrete part of the sequences separately as two independent distributions
without paying attention to their internal correlations. As a result, although
$G1$ have a higher FID score for time intervals and higher MAD score for event
types, it has a lower RBQ score compared to $G2$. This is because the RBQ
score is calculated based on the joint distribution of time intervals and
event types. Same logic applies to FIDH score as well, $G1$ has a lower FIDH
score than $G2$, because FIDH is calculated between the hidden state
representations of two sequences, which utilizes the information from both
time intervals and event types. Moreover, MMD converges fast during the
training and both $G1$ and $G2$ have very similar performance with regards to
MMD.

###  4.5. Discussion

Most metrics such as FID only yield one-dimensional scores and fail to
distinguish between different failure cases. Given that in Table 1 both $G1$
and $G2$ have similar performances, we propose using Precision and Recall for
Distributions (PRD) (Sajjadi et al., 2018) to explain their differences. PRD
is used to compare a distribution $Q$ to a reference distribution $P$. The
intuition behind PRD is that precision measures how much of $Q$ can be
generated by $P$ while recall measures how much of $P$ can be generated by
$Q$. Figure 2 illustrates the comparison of the PRD curves among $G0$, $G1$,
and $G2$:

Figure 2. Comparison of PRD curves of $G0$, $G1$, and $G2$

We can interpret the differences between $G1$ and $G2$ from Figure 2: for a
desired recall level of $\sim 0.8$ and lower, $G1$ generates sequences closer
to the training data. However, if one desires a recall higher than $\sim 0.8$,
$G2$ enjoys higher precision.

We next examine the representations learned by the discriminator that
underpinned the successful performance of it. We use t-SNE (Maaten and Hinton,
2008) to visualize the output of the Time-LSTM cell. Basically, the nearby
points in the representation space have similar rewards given by the
discriminator.

We generate $10000$ sequences from each of $G0$, $G1$, and $G2$ and pass them
to the discriminators that trained with the corresponding generators.

(a) D0

(b) D1

(c) D2

Figure 3. Two-dimensional t-SNE embedding of the representations of the
outputs of Time-LSTM cells

Figure 3 depicts the two-dimensional t-SNE embedding of the representations of
the outputs of the Time-LSTM cells for each discriminator. The points are
colored according to the predicted rewards from the discriminator. Darker
colors mean higher reward. We can clearly see in Figure 3, discriminator $D2$
has more points with relatively darker colors than discriminators $D1$ and
$D0$, which means the discriminator that is trained with $G2$ returns higher
rewards for the generated sequences. On the other hand, the discriminator $D0$
seems to return a mean value for all the sequences (same color). This is due
to the fact that $D0$ is a randomly initialized discriminator. The t-SNE
embeddings can be also used for feature extraction on labeled data.

Radford et al., 2015 has shown a way to build high quality representations by
training a GAN model and reusing parts of the generator and discriminator
networks as feature extractors for other supervised tasks (Radford et al.,
2015). This can be potentially a promising future direction for this work.

##  5\. Conclusions

In this paper, we have described, trained, and evaluated a seqGAN methodology
for generating artificial sequences to mimic the fraudulent user patterns in
ad traffic. We have additionally employed a variant of Time-LSTM cell to
generate synthetic ad events with non-uniform time intervals between events.
As this task poses new challenges, we have presented a new solution by
training seqGAN using a combination of MLE pre-training and a Critic network.
The generator proposed in this paper is capable of generating multi-type
temporal sequences with non-uniform time intervals, which is one of the
novelties of our developed methodology. We have also proposed using multiple
criteria to measure the quality and diversity of the generated sequences.
Through numerous experiments, we have discovered that the generated multi-type
sequences are of desired properties.

Furthermore, we compared the performance of our generator under different
settings with randomly sampled data from our training datasets. We concluded
that the seqGAN-trained generator has a higher performance compared to pre-
trained generators using MLE, measured by multiple criteria including RBQ and
FIDH scores that are appropriate for evaluating multi-type sequences.

##  6\. Acknowledgments

The authors would like to thank Unity for giving the opportunity to work on
this project during Unity’s HackWeek 2020.

## References

  * (1)
  * Ba (2019) Hung Ba. 2019.  Improving Detection of Credit Card Fraudulent Transactions using Generative Adversarial Networks.  _arXiv preprint arXiv:1907.03355_ (2019).
  * Baytas et al. (2017) Inci M Baytas, Cao Xiao, Xi Zhang, Fei Wang, Anil K Jain, and Jiayu Zhou. 2017\.  Patient subtyping via time-aware lstm networks. In _Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining_. 65–74.
  * Bhatnagar et al. (2007) Shalabh Bhatnagar, Richard S Sutton, Mohammad Ghavamzadeh, and Mark Lee. 2007.  Naturalgradient actor-critic algorithms.  _Automatica_ (2007).
  * Borji (2019) Ali Borji. 2019\.  Pros and cons of gan evaluation measures.  _Computer Vision and Image Understanding_ 179 (2019), 41–65.
  * Choi and Lim (2020) Jin-A Choi and Kiho Lim. 2020.  Identifying machine learning techniques for classification of target advertising.  _ICT Express_ (2020).
  * DeVries et al. (2019) Terrance DeVries, Adriana Romero, Luis Pineda, Graham W Taylor, and Michal Drozdzal. 2019.  On the evaluation of conditional GANs.  _arXiv preprint arXiv:1907.08175_ (2019).
  * Dong et al. (2018) Feng Dong, Haoyu Wang, Li Li, Yao Guo, Tegawendé F Bissyandé, Tianming Liu, Guoai Xu, and Jacques Klein. 2018\.  Frauddroid: Automated ad fraud detection for android apps. In _Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. 257–268.
  * Elsworth and Güttel (2020) Steven Elsworth and Stefan Güttel. 2020.  Time Series Forecasting Using LSTM Networks: A Symbolic Approach.  _arXiv preprint arXiv:2003.05672_ (2020).
  * Esteban et al. (2017) Cristóbal Esteban, Stephanie L Hyland, and Gunnar Rätsch. 2017.  Real-valued (medical) time series generation with recurrent conditional gans.  _arXiv preprint arXiv:1706.02633_ (2017).
  * Fortet and Mourier (1953) Robert Fortet and Edith Mourier. 1953.  Convergence de la répartition empirique vers la répartition théorique. In _Annales scientifiques de l’École Normale Supérieure_ , Vol. 70. 267–285.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014\.  Generative adversarial nets. In _Advances in neural information processing systems_. 2672–2680.
  * Graves (2013) Alex Graves. 2013\.  Generating sequences with recurrent neural networks.  _arXiv preprint arXiv:1308.0850_ (2013).
  * Haddadi (2010) Hamed Haddadi. 2010\.  Fighting online click-fraud using bluff ads.  _ACM SIGCOMM Computer Communication Review_ 40, 2 (2010), 21–25.
  * Haider et al. (2018) Ch Md Rakin Haider, Anindya Iqbal, Atif Hasan Rahman, and M Sohel Rahman. 2018.  An ensemble learning based approach for impression fraud detection in mobile advertising.  _Journal of Network and Computer Applications_ 112 (2018), 126–141.
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017\.  Gans trained by a two time-scale update rule converge to a local nash equilibrium. In _Advances in neural information processing systems_. 6626–6637.
  * Jianyu et al. (2017) Wang Jianyu, Wu Chunming, Ji Shouling, Gu Qinchen, and Li Zhao. 2017.  Fraud Detection via Coding Nominal Attributes. In _Proceedings of the 2017 2nd International Conference on Multimedia Systems and Signal Processing_. 42–45.
  * Kapoor et al. (2016) Kawaljeet Kaur Kapoor, Yogesh K Dwivedi, and Niall C Piercy. 2016\.  Pay-per-click advertising: A literature review.  _The Marketing Review_ 16, 2 (2016), 183–202.
  * Killoran et al. (2017) Nathan Killoran, Leo J Lee, Andrew Delong, David Duvenaud, and Brendan J Frey. 2017.  Generating and designing DNA with deep generative models.  _arXiv preprint arXiv:1712.06148_ (2017).
  * Kudugunta and Ferrara (2018) Sneha Kudugunta and Emilio Ferrara. 2018.  Deep neural networks for bot detection.  _Information Sciences_ 467 (2018), 312–322.
  * Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ 9, Nov (2008), 2579–2605.
  * Mouawi et al. (2019) Riwa Mouawi, Imad H Elhajj, Ali Chehab, and Ayman Kayssi. 2019\.  Crowdsourcing for click fraud detection.  _EURASIP Journal on Information Security_ 2019, 1 (2019), 11\.
  * Nagaraja and Shah (2019) Shishir Nagaraja and Ryan Shah. 2019.  Clicktok: click fraud detection using traffic analysis. In _Proceedings of the 12th Conference on Security and Privacy in Wireless and Mobile Networks_. 105–116.
  * Neil et al. (2016) Daniel Neil, Michael Pfeiffer, and Shih-Chii Liu. 2016\.  Phased lstm: Accelerating recurrent network training for long or event-based sequences. In _Advances in neural information processing systems_. 3882–3890.
  * Neunert et al. (2020) Michael Neunert, Abbas Abdolmaleki, Markus Wulfmeier, Thomas Lampe, Jost Tobias Springenberg, Roland Hafner, Francesco Romano, Jonas Buchli, Nicolas Heess, and Martin Riedmiller. 2020\.  Continuous-Discrete Reinforcement Learning for Hybrid Control in Robotics.  _arXiv preprint arXiv:2001.00449_ (2020).
  * Oentaryo et al. (2014) Richard Oentaryo, Ee-Peng Lim, Michael Finegold, David Lo, Feida Zhu, Clifton Phua, Eng-Yeow Cheu, Ghim-Eng Yap, Kelvin Sim, Minh Nhut Nguyen, et al. 2014\.  Detecting click fraud in online advertising: a data mining approach.  _The Journal of Machine Learning Research_ 15, 1 (2014), 99–140.
  * Radford et al. (2015) Alec Radford, Luke Metz, and Soumith Chintala. 2015.  Unsupervised representation learning with deep convolutional generative adversarial networks.  _arXiv preprint arXiv:1511.06434_ (2015).
  * Sajjadi et al. (2018) Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. 2018.  Assessing generative models via precision and recall. In _Advances in Neural Information Processing Systems_. 5228–5237.
  * Smith and Smith (2020) Kaleb E Smith and Anthony O Smith. 2020.  Conditional GAN for timeseries generation.  _arXiv preprint arXiv:2006.16477_ (2020).
  * Sutton and Barto (2018) Richard S Sutton and Andrew G Barto. 2018.  _Reinforcement learning: An introduction_.  MIT press.
  * Thomas et al. (2016) Kurt Thomas, Juan A Elices Crespo, Ryan Rasti, Jean-Michel Picod, Cait Phillips, Marc-André Decoste, Chris Sharp, Fabio Tirelo, Ali Tofigh, Marc-Antoine Courteau, et al. 2016\.  Investigating commercial pay-per-install and the distribution of unwanted software. In _25th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 16)_. 721–739.
  * Xiao et al. (2017) Shuai Xiao, Junchi Yan, Mehrdad Farajtabar, Le Song, Xiaokang Yang, and Hongyuan Zha. 2017\.  Joint modeling of event sequence and time series with attentional twin recurrent neural networks.  _arXiv preprint arXiv:1703.08524_ (2017).
  * Yu et al. (2017) Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017\.  Seqgan: Sequence generative adversarial nets with policy gradient. In _Thirty-first AAAI conference on artificial intelligence_.
  * Zhao et al. (2020) Pengyu Zhao, Tianxiao Shui, Yuanxing Zhang, Kecheng Xiao, and Kaigui Bian. 2020.  Adversarial Oracular Seq2seq Learning for Sequential Recommendation. In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI_. 1905–1911.
  * Zheng et al. (2019) Panpan Zheng, Shuhan Yuan, Xintao Wu, Jun Li, and Aidong Lu. 2019.  One-class adversarial nets for fraud detection. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 1286–1293.
  * Zhu et al. (2017b) Xingquan Zhu, Haicheng Tao, Zhiang Wu, Jie Cao, Kristopher Kalish, and Jeremy Kayne. 2017b.  _Fraud prevention in online digital advertising_.  Springer.
  * Zhu et al. (2017a) Yu Zhu, Hao Li, Yikang Liao, Beidou Wang, Ziyu Guan, Haifeng Liu, and Deng Cai. 2017a.  What to Do Next: Modeling User Behaviors by Time-LSTM.. In _IJCAI_ , Vol. 17. 3602–3608.