# Application of Deep Reinforcement Learning to Payment Fraud

Siddharth Vimal<EMAIL_ADDRESS>AI Garage, MastercardIndia ,
Kanishka Kayathwal<EMAIL_ADDRESS>AI Garage,
MastercardIndia ,  Hardik Wadhwa<EMAIL_ADDRESS>AI Garage,
MastercardIndia and  Gaurav Dhama<EMAIL_ADDRESS>AI Garage,
MastercardIndia

###### Abstract.

The large variety of digital payment choices available to consumers today has
been a key driver of e-commerce transactions in the past decade.
Unfortunately, this has also given rise to cybercriminals and fraudsters who
are constantly looking for vulnerabilities in these systems by deploying
increasingly sophisticated fraud attacks. A typical fraud detection system
employs standard supervised learning methods where the focus is on maximizing
the fraud recall rate. However, we argue that such a formulation can lead to
sub-optimal solutions. The design requirements for these fraud models requires
that they are robust to the high-class imbalance in the data, adaptive to
changes in fraud patterns, maintain a balance between the fraud rate and the
decline rate to maximize revenue, and be amenable to asynchronous feedback
since usually there is a significant lag between the transaction and the fraud
realization. To achieve this, we formulate fraud detection as a sequential
decision-making problem by including the utility maximization within the model
in the form of the reward function. The historical decline rate and fraud rate
define the state of the system with a binary action space composed of
approving or declining the transaction. In this study, we primarily focus on
utility maximization and explore different reward functions to this end. The
performance of the proposed Reinforcement Learning system has been evaluated
for two publicly available fraud datasets using Deep Q-learning and compared
with different classifiers. We aim to address the rest of the issues in future
work.

Payment Fraud, Deep Reinforcement Learning, Neural Networks

††copyright: none††conference: Marble-KDD 21; August 16, 2021; Singapore††ccs:
Applied computing Online banking††ccs: Applied computing Secure online
transactions††ccs: Applied computing Online shopping

##  1\. Introduction

With the increasing involvement of businesses and consumers in the digital
ecosystem, there has been an exponential rise in digital payments in the past
decade fuelled by the variety of payment choices launched by the payments
industry. Due to the increasing reach and complexity of the technology
involved, fraudsters constantly devise new methods to attack these systems.
Many machine learning techniques have already been proposed to tackle this
problem, like neural networks (Kazemi and Zarrabi, 2017) and decision trees
(Varmedja et al., 2019), however these techniques can be sensitive to high-
class imbalance ratios, changing distributions, and might require re-training.
The traditional paradigm of fraud detection solutions in financial
institutions consists of formulating it as a classification problem with a
focus on improving the fraud recall rates of these classification models.
Several papers in the literature have proposed different methods for creating
robust features that are immune to model/concept drift using statistical, deep
learning, and unsupervised techniques ((Lucas et al., 2020),(Carcillo et al.,
2019),(Zhang et al., 2019),(Bahnsen et al., 2016),(Dastidar et al., 2020)).
However, these methods of problem formulation ignore a number of issues:

  * •

Utility Maximization – the models are not optimized to maximize the utility
function and might end up declining genuine transactions, ultimately leading
to revenue loss.

  * •

Non-Stationarity – the distribution of fraudulent transactions is constantly
changing owing to the emergence of new types of fraud. Also, large-scale fraud
events significantly distort these distributions.

  * •

Asynchronous Feedback – the labels for fraud are usually delayed, and the
financial institution recognizes the fraud much later after the transaction
has already been processed.

  * •

Counterfactuals – the financial institution loses the label for the
transaction once it declines the transaction.

Moreover, there are practical issues with deploying offline classifier models
for fraud since a model trained on historical data might see a loss in
performance due to the time required between training and deployment, i.e.,
the data might become stale.

With the great success that Deep Reinforcement Learning (DRL) methods have
achieved in problems with sequential decision making, ranging from achieving
professional human-like performance in Atari Games (Mnih et al., 2015) to
applications in cyber-security (Nguyen and Reddi, 2019) and recommender
systems (Zheng et al., 2018), application of Deep Reinforcement Learning to
real-world applications like fraud detection deserves more attention. In this
paper, we aim to study the problem of utility maximization in fraud by
formulating it as a DRL problem and evaluating different reward functions. The
other issues of non-stationarity, asynchronous feedback, and counterfactuals
can potentially be solved by various methods available in the DRL literature
(Foerster et al., 2018)(Walsh et al., 2007)(Igl et al., 2020), and it further
serves as the motivation to use DRL for fraud detection. We haven’t explored
these issues in the current study and will consider these in future works.

However, there are two non-trivial issues that we need to address while
attempting this formulation. Firstly, the definition of the reward function
must incorporate the utility of money defined for the financial institution
deploying the model. Smaller financial institutions with limited budgets might
be more risk-averse to potentially fraudulent transactions than larger
financial institutions that might prioritize customer experience over fraud
costs (especially those offering premium financial products). The famous
example which demonstrates the non linearity of the utility of money is the
Saint Petersburg Paradox (Todhunter, 1865). Various methods in the literature
try to estimate this function based on utility elicitation, such as the
standard gamble method, time trade-off, and visual analog methods (see chapter
22 of (Koller and Friedman, 2009) for a detailed discussion). However, a
detailed comparison of utility estimation methods in the context of fraud
detection would probably need a paper of its own hence we will not dwell
further into the matter. Moreover, publicly available fraud datasets do not
have information that can quantify the utility functions; therefore, we assume
that the utility function of the financial institution is risk-neutral, is not
dependent upon historically accumulated rewards, and is directly proportional
to the revenue earned in the transaction. However, in the real-life scenario,
we would need to place a utility distribution over different customer segments
(depending on the preferences of the financial institution) and the
historically accumulated rewards (depending on the utility of money curve).

The second issue that we need to address is the definition of state for this
problem. While it is intuitive that the historical false decline rate and the
fraud rate need to be part of the state, their form of inclusion is not clear.
This is because as time passes and we have accumulated a large number of
transactions, these metrics will tend to approach a constant value
asymptotically, and actions by the agent, even on a large set of transactions,
might not significantly alter the state, thus stalling agent learning. Hence
it is necessary to introduce some form of decay where older transactions are
discarded to calculate false decline rates and fraud rates after some point.
We adopt a piece-wise approach where these rates are only calculated only on
the current episode and recomputed on the new episode upon completion, and the
episode size is a tunable parameter. More details are available in the
methodology section.

Based on the factors described above, the contributions of this paper can be
summarized as below :

  * •

Sensitizing the research community around the problems faced in industrial
applications of fraud detection

  * •

Formulation of the fraud detection problem as a DRL

  * •

A novel tunable reward function that tries to maximize the revenue and also
rewards the agent for a reasonable balance between fraud rate and decline rate

  * •

Comparison between the proposed reward function vs. others proposed in the
literature along with standard classifiers

The rest of this paper is organized as follows. We summarize the related
literature in Section 2, and describe the detailed methodology and
architecture in Section 3. We report and discuss experimental results in
Section 4. We talk about future directions and conclude in Section 5.

##  2\. Related Work

Supervised machine learning approaches involve building a classification model
to identify fraudulent transactions. A number of studies (Varmedja et al.,
2019)(Mishra and Ghorpade, 2018)(Lakshmi and Kavilla, 2018)(Kazemi and
Zarrabi, 2017) compare the performance of multiple machine learning algorithms
such as Gradient Boosting, Logistic Regression, Support Vector Machines,
XGBoost, Multilayer Perceptron (MLP) on transaction fraud detection task.
(Jurgovsky et al., 2018) (Roy et al., 2018) employed Long Short-Term Memory
(LSTM) networks to capture the sequential behaviour in fraud detection.
Likewise, Convolutional Neural Networks (CNN) based framework is proposed in
(Fu et al., 2016) to capture fraud patterns from labeled data. A detailed
study presented in (Nguyen et al., 2020) shows that CNN and LSTM perform
better than traditional machine learning algorithms in credit card fraud
detection (Heryadi and Warnars, 2017) does a similar comparative analysis
between CNN, Stacked LSTM, and a hybrid of CNN-LSTM on credit card data of an
Indonesian bank. Recently developed attention mechanism is explored in (Li et
al., 2019)(Cheng et al., 2020) to detect fraudulent transactions.

Fraud detection can also be formulated as anomaly detection due to the rare
occurrence of fraud in the transaction data. Most traditional approaches use
distance and density-based methods to detect anomalies such as local outlier
factor (Breunig et al., 2000), isolation forest (Liu et al., 2012), K-nearest
neighbourhood (Angiulli and Pizzuti, 2002). Deep learning is used in anomaly
detection by using autoencoders (Chen et al., 2017)(Zhou and Paffenroth, 2017)
or generative adversarial networks (Schlegl et al., 2017) to compute anomaly
score using reconstruction error. Some studies (Pang et al., 2019)(Ruff et
al., 2019) show improvement in deep anomaly detection with the use of some
labeled anomalies in a semi-supervised manner. Few recent papers (Pang et al.,
2020)(Oh and Iyengar, 2019) investigate use of reinforcement learning in
anomaly detection.

Although there are many deep learning-based methods proposed in fraud
detection, the application of reinforcement learning in building fraud systems
has not found traction among researchers. (Zhinin-Vera et al., 2020) uses an
autoencoder to learn a latent representation of features and pass them to an
agent , which is then trained with a Q-learning algorithm to identify fraud in
credit cards. We compare the performance of our reward function with the
reward function proposed in this paper. In (El Bouchti et al., 2017), authors
present application of DRL in financial risk analysis and fraud detection
while (Shen and Kurshan, 2020) proposes alert threshold selection policy in
fraud systems using Deep Q-Network.

##  3\. Methodology

###  3.1. Problem Definition

Given a dataset D =
{$(x_{1},y_{1}),(x_{2},y_{2}),(x_{3},y_{3}),...,(x_{n},y_{n})$}, where $x_{i}$
is the feature vector for the $i^{th}$ transaction in the dataset and $y_{i}$
represents the corresponding fraud label. Fraud transaction forms the positive
class in our datasets i.e $y=1$ for a fraudulent transaction. We sort the data
with respect to time, preserving the sequential aspect and formulate the fraud
classification problem as a Sequential Decision Making problem. The agent is
given transaction $x_{t}$ at timestep t, the agent takes an action of either
approving the transaction ($a_{t}=0$) or declining the transaction
($a_{t}=1$). In return the environment provides the agent with a reward based
on the current classification performance and the next transaction $x_{t+1}$.
The aim of the agent is to be able to classify the transactions such that the
utility is maximised such that significant monetary losses due to fraud
transactions are avoided and genuine transactions are not declined while also
maintaining an optimal balance between the fraud rate (fr) and decline rate
(dr). This is being done using reward function $\mathcal{R}$. Using a Markov
Decision Process (MDP) we represent the environment as
$\big{<}\mathcal{S,A,R,T}\big{>}$ (Lin et al., 2020) (Zhinin-Vera et al.,
2020) with the following definitions:

  * •

State $\mathcal{S}$: At time step t, the state is $s_{t}$ is the $t^{th}$
transaction $x_{t}$ in the dataset (along with dr and fr at time t). Since,
$x_{t}$ contains the attributes of the $t^{th}$ transaction, we will call it
the feature vector for $x_{t}$

  * •

Action $\mathcal{A}$: The action space for this MDP is discrete. We define
$\mathcal{A}$ = {0, 1}, where the agent can approve ($a_{t}=0$) or decline
($a_{t}=1$) a transaction.

  * •

Reward $\mathcal{R}$: A reward $r_{t}$ is a scalar which measures the goodness
of the action $a_{t}$ taken by the agent in the state $s_{t}$. Usually, the
reward is positive when the agent takes a preferable action and negative when
the action is not desirable. For example, approving a fraud transaction is not
preferred, so the agent must be rewarded negatively by the environment. The
reward function for the MDP is described in detail in the next subsection.

  * •

Transition Probability $\mathcal{T}$: The agent takes a decision in the
current state $s_{t}$ and is given a new state $s_{t+1}$ by the environment.
The new state $s_{t+1}$ is the transaction that occurred just after the
transaction $s_{t}$ in the data. We can say that the transition probability is
deterministic.

  * •

Episode: An episode refers to an iteration of the agent interacting with the
environment, which includes getting a state, taking action, receiving a reward
for the action, and then moving to the next state. An episode ends when the
agent reaches a terminal state. For our case, the agent processes transactions
one-by-one and reaches a terminal state once it has taken action on l (l =
500) transactions. This way the agent takes action on
$x_{1},x_{2},x_{3},...,x_{l}$ in the first episode,
$x_{l+1},x_{l+2},x_{l+3},...,x_{2l}$ in the second episode and so on.

  * •

Decline Rate (dr): It is defined as the percent of non-fraud transactions
declined during the last k transactions processed by the agent.

  * •

Fraud Rate (fr): It is defined as the percent of fraud transactions approved
by the agent during the last k transactions processed by the agent.

The decline rate and the fraud rate are appended to the feature vector of the
$t^{th}$ transaction $x_{t}$. This complete vector represents the state
$s_{t}$ of our environment at time step t. For the experiments, we take k =
4000. Figure 1 shows the environment for agent training.

Figure 1. Environment design for agent training

###  3.2. Agent

The objective of a reinforcement learning agent is to find an optimal policy
$\pi^{*}$ such that it is able to maximise the cumulative rewards $G_{t}$,

(1) $G_{t}=\sum_{m=0}^{\infty}\gamma^{m}r_{t+m}$

The choice of agent for experiments is the Deep Q-network (DQN) (Mnih et al.,
2015), which uses a deep neural network to approximate the optimal action-
value function given by:

(2) $Q^{*}(s,a)=\max_{\pi}\mathbb{E}[r_{t}+\gamma
r_{t+1}+\gamma^{2}r_{t+2}+...|s_{t}=s,a_{t}=a,\pi],$

which gives the maximum sum of rewards $r_{t}$ discounted by $\gamma$ at each
time step t and following the policy $\pi$ = P(a—s). The neural network learns
the parameters $\theta$ by performing Q-learning updates iteratively. At an
iteration i, the loss function is:

(3) $L_{i}(\theta_{i})=\mathbb{E}_{(s,a,r,s^{\prime})\sim
U(M)}\Big{[}\Big{(}r+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime};\theta_{i}^{-})-Q(s,a;\theta_{i})\Big{)}^{2}\Big{]}$

where $\gamma$ is the discount factor, and $\theta_{i}$ are the parameters of
the Q-network at $i^{th}$ iteration and $\theta_{i}^{-}$ are the parameters of
the target network, which is used to calculate the target. The target network
parameters are set equal to the parameters of the Q-network after every K
steps. M denotes the memory pool of the agent where it stores its experiences
as $m_{t}=(s_{t},a_{t},r_{t},s_{t+1})$ at time t. We chose deep Q-learning
because the action space is discrete. The Q-learning updates take place on
mini batches (batch size = 32) drawn uniformly at random from the memory pool.
This process where the agent learns from its past experiences is also termed
experience replay. We use a double ended queue of fixed length as the memory
pool for the agent. This way, the older samples get discarded, and more recent
samples are stored in the memory pool M. The agent is trained using the
$\epsilon$-greedy policy. The training process begins with $\epsilon$ = 1 and
using decay rate $\delta$ we linearly anneal $\epsilon$ to 0.01.

###  3.3. Reward Function

The agent is rewarded with two rewards $r^{m}_{t}$ and $r^{b}_{t}$ after it
takes action $a_{t}$ in state $s_{t}$, which guide the agent to find an
optimal policy for maximizing the revenue and also maintaining a balance
between fraud rate (fr) and decline rate (dr). The reward function is defined
as:

(4)
$\mathcal{R}^{m}(s_{t},a_{t},y_{t})=\begin{cases}+\alpha*log(amount)&\text{if
$a_{t}=0$ and $y_{t}=0$},\\\ -1*\alpha*log(amount)&\text{if $a_{t}=1$ and
$y_{t}=0$},\\\ -1*log(amount)&\text{if $a_{t}=0$ and $y_{t}=1$},\\\
+log(amount)&\text{if $a_{t}=1$ and $y_{t}=1$}\end{cases}$

The above reward function is inspired from the working of the four party model
for payments, where the merchant, acquirer, payment network and the issuer
bank involved in a payment keep a small cut of the payment they process. The
issuer takes the largest cut (close to 2% of the transaction amount), but also
takes the highest risk. This serves as the base for the monetary reward
$r^{m}_{t}$. The reward $r^{m}_{t}$ is defined in a manner to be consistent to
a banks revenue model on credit/debit cards, where the banks charge an
interchange fee to the merchant for every non-fraud transaction approved but
suffer a complete loss in case of an approved fraudulent transaction (except
for cases of merchant fraud). We keep $\alpha=0.02$ for our experiments.
Second reward $r^{b}_{t}$ is given based on the balance between the fraud rate
and the decline rate of the system. This reward is defined as:

(5)
$\mathcal{R}^{b}(dr_{t},fr_{t})=\frac{1}{8}*\frac{(1+\beta^{2})*(1-dr_{t})*(1-fr_{t})}{\beta^{2}*(1-dr_{t})+(1-fr_{t})}$

where $\beta$ is a hyper-parameter with a preferred value close to 1. The
reward function $\mathcal{R}^{b}$ is a scaled down weighted harmonic mean of
the two quantities we want the agent to minimize. We scale down the harmonic
mean by a factor of 8 so that the maximum possible reward from
$\mathcal{R}^{b}$ and $\mathcal{R}^{m}$ is approximately same for each
episode. This resulted in a more stable training process.

We further compare the performance of our reward function
$\mathcal{R}=\mathcal{R}^{m}(s_{t},a_{t},l_{t})+\mathcal{R}^{b}(dr_{t},fr_{t})$
with reward functions proposed like $\mathcal{R^{\prime}}$ (Lin et al., 2020)
and $\mathcal{R^{\prime\prime}}$ (Zhinin-Vera et al., 2020) for classification
using reinforcement learning. The reward function $\mathcal{R^{\prime}}$ is
defined as:

(6) $\mathcal{R^{\prime}}(s_{t},a_{t},y_{t})=\begin{cases}+1&\text{if
$a_{t}=y_{t}$ and $s_{t}\in D_{P}$}\\\ -1&\text{if $a_{t}\neq y_{t}$ and
$s_{t}\in D_{P}$}\\\ +\lambda&\text{if $a_{t}=y_{t}$ and $s_{t}\in D_{N}$}\\\
-\lambda&\text{if $a_{t}\neq y_{t}$ and $s_{t}\in D_{N}$}\end{cases}$

For $\mathcal{R^{\prime}}$, authors suggest $\lambda\ =\rho$, where $\rho$ is
the class imbalance ratio given by $\rho=\frac{|D_{P}|}{|D_{N}|}$. $D_{P}$ is
the minority (fraud) sample set and $D_{N}$ is the majority (non-fraud) sample
set. For the other proposed reward $\mathcal{R^{\prime\prime}}$ the suggested
value for $\lambda$ = 0.1.

###  3.4. Training the agent

We construct an environment according to the MDP defined above. During
training, the agent uses $\epsilon$-greedy policy for selecting its actions.
The training stops when the agent has taken action on all the transactions in
the training data once. Therefore, the number of training episodes is
$\frac{|D_{T}|}{l}$, where $l$ is the length of an episode. The agent training
process is given in Figure 1. The test environment is similar to the train
environment, but the agent doesn’t use the experience replay in the test
environment. Also the actions for test observations are chosen based on the
q-values predicted by the Q-network (action with the max q-value is taken by
the agent).

Data: Training Data

initialize environment variables fr, dr to 0;

initialize memory pool M;

initialize parameters $\theta$;

initialize $s_{1}=(x_{1},fr,dr)$;

for  _episode $e=1$ to N_ do

for  _t=1 to l_ do

select $a_{t}$ with $\epsilon$-greedy policy;

$r_{t}$, $s_{t+1}$ = step($s_{t},a_{t},y_{t}$);

update $fr,dr$ and $s_{t}=s_{t+1}$;

store ¡$s_{t},a_{t},r_{t},s_{t+1}$¿ to $M$;

randomly sample $batch$ from $M$;

experience replay on $batch$;

end for

end for

Algorithm 1 Training Environment Algorithm

##  4\. Experimental Results

###  4.1. Datasets

We have used two open datasets for credit card fraud transactions. First is
European card data (ECD) on Kaggle (Dal Pozzolo et al., 2015). This dataset
contains 284,807 transactions and 31 features. It contains 492 fraud
transactions which make the dataset highly imbalanced (0.172%). The 28
numerical features in this dataset are the result of PCA transformation. This
has been done due to confidentiality and privacy reasons. The data also
provides the time and amount column. The time column is the seconds elapsed
from the first transaction in the data, and the amount is the transaction
amount.

The second dataset is the IEEE-CIS fraud dataset (IEEE) on Kaggle 111Available
at https://www.kaggle.com/c/ieee-fraud-detection. It contains real-world
e-commerce transactions with a variety of numerical and categorical features.
There are 590,540 transactions, out of which 20,663 are fraud transactions, so
the class imbalance ratio is roughly 3.5% for this dataset. This dataset also
contains the time and amount columns.

###  4.2. Preprocessing

All the numerical variables are normalized using the MinMaxScaler. For the
IEEE data, we use only the numerical columns along with a few aggregated
features created. We target encoded three categorical variables.

The data is sorted by the time column, the first 70% of the transactions
become the training set. The following 10% data is used as a validation set
for the supervised algorithms, and the last 20% is used as the test set for
evaluation purposes.

###  4.3. Evaluation Metrics

Our evaluation metrics can be categorized broadly into two categories. First,
we rely on standard classification metrics such as precision, recall, and F1
score. A high precision, high recall, and high F1 score is the primary step
towards model efficacy. In addition, these metrics are helpful in comparative
evaluation with other models out there in the literature. Second, in line with
our argument that fraud detection should be framed as a utility maximization
problem, we evaluate our performance on dollar values of Non-Fraud, Fraud
approve and decline numbers. To explain, a financial institution with high
Non-Fraud declines might be at the risk of poor customer experience. On the
other hand, an institution with high Fraud approvals is at the risk of losing
a massive amount of money. Further, we include two additional metrics:
Approval percentage and Fraud(in bps). A low approval percentage means that
the institution is declining many genuine transactions to catch frauds, which
is unacceptable in a real-world scenario and may lead to reputation loss of
the institution. Also, high Fraud(in bps) means the institution is approving
many fraud transactions.

  * •

App%: No. of approved transactions per 100 transactions.

  * •

F(bps): Approved fraud transactions per basis points.

  * •

$F_{N}$ app: $ amt of approved genuine (non-fraud) transactions.

  * •

$F_{N}$ dec: $ amt of declined genuine (non-fraud) transactions.

  * •

F app: $ amt of approved Fraud transactions.

  * •

F dec: $ amt of declined Fraud transactions.

(a) Decline Rates ($dr$)

(b) Fraud Rates ($fr$)

Figure 2. Performance of different agents and XGBoost in test environment of
IEEE data

###  4.4. Methods and Performance Evaluation

To evaluate the performance of a fraud model, we will use precision, recall,
and F1 scores. We also compare some business-related metrics to judge the
performance of these models.

  * •

Agent: We train three agents $DQNR$, $DQNR^{\prime}$ and $DQNR^{\prime\prime}$
with reward functions $\mathcal{R}$, $\mathcal{R^{\prime}}$,
$\mathcal{R^{\prime\prime}}$ respectively. The Q-network contains two hidden
layers with 128 nodes each. For the environment, each episode is of length
$l=500$, the discount rate $\gamma=0.99$, $\beta=0.5$. The decay rate $\delta$
is 0.000008 and 0.000004 for ECD and IEEE data respectively. We update the
target network every 25 episodes and the length of the memory pool $M$ for the
agent is 75000. The Q-network uses huber loss with Adam optimizer, and
learning rate 0.005.

  * •

Deep neural networks (NN): The neural network is similar to the Q-network but
is trained using binary cross-entropy loss. The optimizer used is Adam and the
learning rate is 0.0002. We use the validation data for the early stopping of
the training process.

  * •

CNN (LeCun et al., 1998): We applied 1-D CNN on the data represented as a
feature matrix. The network consists of two convolution layers followed by a
dense layer.

  * •

LSTM (Hochreiter and Schmidhuber, 1997): Transaction sequences are first
created using a rolling window with a window size of 30. A 2 layer stacked
LSTM is used to capture the sequential information. This information is passed
to a dense layer for the final prediction.

  * •

Random forest classifier (RF) (Liaw et al., 2002): It is a popular choice for
classification problems. We select the best parameters using a randomized grid
search.

  * •

XGBoost (Chen and Guestrin, 2016): Parameters for the XGBoost classifier are
selected from a randomized grid search.

Table 1. Experiment Results for ECD Model | Precision | Recall | F1 Score | App% | F(bps)
---|---|---|---|---|---
DQNR(ours) | 83% | 76% | 79% | 99.88% | 3.16
DQNR’ | 98% | 68% | 80.3% | 99.91% | 4.22
DQNR” | 100% | 37% | 54% | 99.95% | 8.26
NN | 98% | 61% | 75% | 99.93% | 6.15
CNN | 100% | 60% | 75% | 99.92% | 5.27
LSTM | 92% | 59% | 72% | 99.92% | 5.45
RF | 80% | 75% | 77% | 99.88% | 3.34
XGBoost | 95% | 72% |  81.9% | 99.9% | 3.69
Table 2. Experiment Results for IEEE data Model | Precision | Recall | F1 Score | App% | F(bps)
---|---|---|---|---|---
DQNR | 48% | 35% | 41% |  97.5% | 229
DQNR’ | 32% | 41% | 36% | 95.5% | 211
DQNR” | 23% | 47% | 31% | 93.1% | 197
NN | 52% | 30% | 38% | 98% | 246
CNN | 43% | 38% | 40% | 97% | 221
LSTM | 35% | 34% | 35% | 96.6% | 234
RF | 32% | 40% | 36% | 95.8% | 216
XGBoost | 58% | 43% | 49% |  97.5% | 203
Table 3. Amount approved and declined ECD Model |  $F_{N}$ app |  $F_{N}$ dec |  $F$ app |  $F$ dec
---|---|---|---|---
DQNR | $4,459,459 | $1,398 | $2,402 | $5,327
DQNR’ | $4,435,166 | $25,691 | $2,817 | $4,912
DQNR” | $4,460,857 | $0 | $7,286 | $444
NN | $4,435,166 | $25,691 | $3,871 | $3,858
CNN | $4,460,857 | $0 | $4,564 | $3,166
LSTM | $4,435,164 | $25,693 | $3,620 | $4,110
RF | $4,433,783 | $27,074 | $2,442 | $5,288
XGBoost | $4,435,164 | $25,693 | $2,724 | $5,005
Table 4. Amount approved and declined for IEEE data Model |  $F_{N}$ app |  $F_{N}$ dec |  $F$ app |  $F$ dec
---|---|---|---|---
DQNR | $15,487,021 | $146,196 | $481,757 | $128,177
DQNR’ | $14,981,863 | $651,355 | $391,945 | $217,989
DQNR” | $14,153,283 | $1,479,935 | $323,505 | $286,429
NN | $15,442,216 | $191,002 | $483,072 | $126,863
CNN | $15,278,589 | $354,909 | $429,033 | $180,901
LSTM | $14,984,025 | $649,473 | $422,208 | $187,727
RF | $15,260,220 | $372,998 | $458,605 | $151,329
XGBoost | $15,389,617 | $243,880 | $385,909 | $224,026

We use the validation data to adjust the probability thresholds to get the
maximum F1 score from the supervised algorithms. We don’t require any
threshold adjustment for our agent DQNR, and the environment parameters are
also fixed for the two datasets. Although a comparison between supervised
learning methods and DRL might not make sense from a theoretical perspective
because of the different nature of the two-class of methods (supervised
learning being an ”instructive” algorithm vs. DRL being an ”evaluative”
algorithm), the purpose is to compare the utility of these methods for the
organization using these methods in production.

###  4.5. Discussion

We report the performance of different algorithms in Table 1 and Table 2 on
precision, recall, F1 score, approval and fraud rate on ECD and IEEE datasets
respectively. The proposed DQNR method performs better/at par with all other
models on the F1 score except XGBoost on both datasets. XGBoost outperforms
all other methods on the F1 score. This is primarily because XGBoost being an
”instructive” process, has access to complete data during training which
allows it to learn a better representation of the data compared to a DRL agent
trained in an episodic manner. These problems can be potentially be resolved
by handling the distribution shift in offline reinforcement learning (Kumar et
al., 2019), using a better curriculum strategy (Narvekar et al., 2020) or by
solving for the representation learning problem (Stooke et al., 2020).

An inherent requirement of fraud models is to maintain an optimal balance
between approval and fraud rates as they directly affect the customer
experience and fraud cost, respectively. Large financial institutions
generally have a high-risk appetite and they would want to keep high approval
rates at the expense of incurring losses due to fraud, as this is directly
proportional to better customer experience. For both the datasets, DQNR
maintains a balance between Approval and Fraud rate. In addition, it has the
lowest F(bps) for the ECD dataset among all other models while maintaining a
comparable approval rate with the XGBoost model.

We also provide results on monetary metrics in Table 3 and Table 4 for these
models. An optimal combination of Non-Fraud (approvals/declines) and Fraud
(approvals/declines) dollar amounts would provide us with the right business
metrics to further study each of the algorithms above.

1\. In the case of Non-Fraud approvals and declines, in both the datasets, the
performance of the DQNR model is better/at par. In the Non-Fraud declines of
the IEEE dataset, there is a substantial difference in dollar amounts compared
to other models. As Non-Fraud declines are the genuine transactions that
affect customer experience, this is a priority for major financial
institutions.

2\. In case of Fraud approvals and declines, the performance of the DQNR model
on ECD beats all other models. For IEEE dataset, for DQNR model, the Fraud
approval/decline numbers are not able to beat DQNR’ and DQNR”. In the next
section, we perform a hyperparameter study to understand how $\beta$ is
controlling our approval rate, non-fraud declines, and Fraud declines.

We can also compare the stability of three agents - DQNR, DQNR’, DQNR” and
XGBoost (XGBoost evaluated in episodic manner) based on Figure 2 where we draw
the actual fraud rate against the agent fraud (fr) and decline rate (dr) as we
progress in terms of episodes for the test set. We can see DQNR’ and DQNR”
behave erratically with high decline rates compared to DQNR and XGBoost on the
IEEE dataset. This may result in undesirable performance as the data
distribution changes or new types of fraud are encountered by these agents.

Figure 3 shows how the total no of frauds changes in each episode for the
datasets. This type of behavior will be prevalent when these models are
trained in an online setting, and our proposed method DQNR has proved to be
robust against any such distribution change.

There are certain limitations to using the DRL framework for a fraud detection
task. Agents trained on previously collected datasets without any active
environment interaction are prone to overfitting as a result of excessive
training (Singh et al., 2020). Their performance is bound by the size of the
dataset and highly dependent upon the state and reward definition (Wang,
[n.d.]). Further, transaction embedding learned via better representational
learning methods (Liu et al., 2021) can provide a better state representation
and can help the agent to reach the high reward regions of the state space.

(a) IEEE data

(b) ECD

Figure 3. No. of fraud examples in test episodes for datasets

###  4.6. Hyperparameter Analysis

The proposed reward function has two hyper-parameters $\alpha$ and $\beta$.
$\alpha$ is analogous to the cut per transaction that the financial institute
gets for each non-fraud transaction it facilitates. $\beta$ is a parameter to
adjust the strictness of the agent. Table 5 contains the effects of $\beta$ on
the agent’s ($DQNR$) performance on IEEE data. This provides a user the
flexibility to change the agent’s behavior according to their risk appetite.
$\beta$ less than one will act loosely for fraud detection and approve most
transactions. A higher $\beta$ will be very vigilant for fraud but at the cost
of declining genuine transactions.

Table 5. Effect on agent DQNR by varying $\beta$ for IEEE data $\beta$ | Precision | Recall | F1 Score | App% | F(bps)
---|---|---|---|---|---
0.5 | 48% | 35% | 41% | 97.5% | 229
1 | 41% | 38% | 39% | 96.8% | 222
3 | 35% | 41% | 38% | 96.0% | 213

##  5\. Future Work and Conclusion

Fraud detection in payment networks is formulated as a classification problem
with a focus on improving the fraud recall rates of these classification
models. In this paper, we frame it as a DRL problem and propose a reward
function that aims to maximize utility such that significant monetary losses
due to fraud transactions are controlled and keeping a check on the decline
rate of genuine transactions. We train a RL agent to detect fraud in
transaction data while maintaining a balance between the fraud and decline
rates. We show that the agent performs well on both the credit card
dataset(ECD) and the e-commerce dataset(IEEE) with different class imbalance
ratios without the need for aggressive parameter tuning or threshold
adjustments. With some modifications, the agent can be used for streaming data
and can adapt to changing distributions in a better way. This can solve the
issue of re-training fraud models, which is an inherent problem with most
classifiers. Furthermore, better algorithms coupled with a more advanced
environment and state design might help to improve the performance. The
availability of better datasets will also be beneficial for future research.

## References

  * (1)
  * Angiulli and Pizzuti (2002) Fabrizio Angiulli and Clara Pizzuti. 2002.  Fast outlier detection in high dimensional spaces. In _European conference on principles of data mining and knowledge discovery_. Springer, 15–27.
  * Bahnsen et al. (2016) Alejandro Correa Bahnsen, Djamila Aouada, Aleksandar Stojanovic, and Björn Ottersten. 2016\.  Feature engineering strategies for credit card fraud detection.  _Expert Systems with Applications_ 51 (2016), 134–142.
  * Breunig et al. (2000) Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. 2000.  LOF: identifying density-based local outliers. In _Proceedings of the 2000 ACM SIGMOD international conference on Management of data_. 93–104.
  * Carcillo et al. (2019) Fabrizio Carcillo, Yann-Aël Le Borgne, Olivier Caelen, Yacine Kessaci, Frédéric Oblé, and Gianluca Bontempi. 2019\.  Combining unsupervised and supervised learning in credit card fraud detection.  _Information Sciences_ (2019).
  * Chen et al. (2017) Jinghui Chen, Saket Sathe, Charu Aggarwal, and Deepak Turaga. 2017.  Outlier detection with autoencoder ensembles. In _Proceedings of the 2017 SIAM international conference on data mining_. SIAM, 90–98.
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin. 2016.  Xgboost: A scalable tree boosting system. In _Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining_. 785–794.
  * Cheng et al. (2020) Dawei Cheng, Sheng Xiang, Chencheng Shang, Yiyi Zhang, Fangzhou Yang, and Liqing Zhang. 2020\.  Spatio-temporal attention-based neural network for credit card fraud detection. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 362–369.
  * Dal Pozzolo et al. (2015) Andrea Dal Pozzolo, Olivier Caelen, Reid A Johnson, and Gianluca Bontempi. 2015.  Calibrating probability with undersampling for unbalanced classification. In _2015 IEEE Symposium Series on Computational Intelligence_. IEEE, 159–166.
  * Dastidar et al. (2020) Kanishka Ghosh Dastidar, Johannes Jurgovsky, Wissam Siblini, Liyun He-Guelton, and Michael Granitzer. 2020.  NAG: Neural feature aggregation framework for credit card fraud detection. In _2020 IEEE International Conference on Data Mining (ICDM)_. IEEE, 92–101.
  * El Bouchti et al. (2017) Abdelali El Bouchti, Ahmed Chakroun, Hassan Abbar, and Chafik Okar. 2017.  Fraud detection in banking using deep reinforcement learning. In _2017 Seventh International Conference on Innovative Computing Technology (INTECH)_. IEEE, 58–63.
  * Foerster et al. (2018) Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. 2018\.  Counterfactual multi-agent policy gradients. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 32.
  * Fu et al. (2016) Kang Fu, Dawei Cheng, Yi Tu, and Liqing Zhang. 2016\.  Credit card fraud detection using convolutional neural networks. In _International Conference on Neural Information Processing_. Springer, 483–490.
  * Heryadi and Warnars (2017) Yaya Heryadi and Harco Leslie Hendric Spits Warnars. 2017.  Learning temporal representation of transaction amount for fraudulent transaction recognition using CNN, Stacked LSTM, and CNN-LSTM. In _2017 IEEE International Conference on Cybernetics and Computational Intelligence (CyberneticsCom)_. IEEE, 84–89.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Igl et al. (2020) Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. 2020\.  The impact of non-stationarity on generalisation in deep reinforcement learning.  _arXiv preprint arXiv:2006.05826_ (2020).
  * Jurgovsky et al. (2018) Johannes Jurgovsky, Michael Granitzer, Konstantin Ziegler, Sylvie Calabretto, Pierre-Edouard Portier, Liyun He-Guelton, and Olivier Caelen. 2018.  Sequence classification for credit-card fraud detection.  _Expert Systems with Applications_ 100 (2018), 234–245.
  * Kazemi and Zarrabi (2017) Zahra Kazemi and Houman Zarrabi. 2017.  Using deep networks for fraud detection in the credit card transactions. In _2017 IEEE 4th International Conference on Knowledge-Based Engineering and Innovation (KBEI)_. IEEE, 0630–0633.
  * Koller and Friedman (2009) Daphne Koller and Nir Friedman. 2009.  _Probabilistic graphical models: principles and techniques_.  MIT press.
  * Kumar et al. (2019) Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. 2019\.  Stabilizing off-policy q-learning via bootstrapping error reduction.  _arXiv preprint arXiv:1906.00949_ (2019).
  * Lakshmi and Kavilla (2018) SVSS Lakshmi and SD Kavilla. 2018.  Machine learning for credit card fraud detection system.  _International Journal of Applied Engineering Research_ 13, 24 Pt. 1 (2018), 16819–16824.
  * LeCun et al. (1998) Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998.  Gradient-based learning applied to document recognition.  _Proc. IEEE_ 86, 11 (1998), 2278–2324.
  * Li et al. (2019) Longfei Li, Ziqi Liu, Chaochao Chen, Ya-Lin Zhang, Jun Zhou, and Xiaolong Li. 2019\.  A Time Attention based Fraud Transaction Detection Framework.  _arXiv preprint arXiv:1912.11760_ (2019).
  * Liaw et al. (2002) Andy Liaw, Matthew Wiener, et al. 2002\.  Classification and regression by randomForest.  _R news_ 2, 3 (2002), 18–22.
  * Lin et al. (2020) Enlu Lin, Qiong Chen, and Xiaoming Qi. 2020.  Deep reinforcement learning for imbalanced classification.  _Applied Intelligence_ (2020), 1–15.
  * Liu et al. (2012) Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. 2012\.  Isolation-based anomaly detection.  _ACM Transactions on Knowledge Discovery from Data (TKDD)_ 6, 1 (2012), 1–39.
  * Liu et al. (2021) Guoqing Liu, Chuheng Zhang, Li Zhao, Tao Qin, Jinhua Zhu, Jian Li, Nenghai Yu, and Tie-Yan Liu. 2021\.  Return-Based Contrastive Representation Learning for Reinforcement Learning.  _arXiv preprint arXiv:2102.10960_ (2021).
  * Lucas et al. (2020) Yvan Lucas, Pierre-Edouard Portier, Léa Laporte, Liyun He-Guelton, Olivier Caelen, Michael Granitzer, and Sylvie Calabretto. 2020\.  Towards automated feature engineering for credit card fraud detection using multi-perspective HMMs.  _Future Generation Computer Systems_ 102 (2020), 393–402.
  * Mishra and Ghorpade (2018) Ankit Mishra and Chaitanya Ghorpade. 2018.  Credit card fraud detection on the skewed data using various classification and ensemble techniques. In _2018 IEEE International Students’ Conference on Electrical, Electronics and Computer Science (SCEECS)_. IEEE, 1–5.
  * Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015\.  Human-level control through deep reinforcement learning.  _nature_ 518, 7540 (2015), 529–533.
  * Narvekar et al. (2020) Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. 2020\.  Curriculum learning for reinforcement learning domains: A framework and survey.  _Journal of Machine Learning Research_ 21, 181 (2020), 1–50.
  * Nguyen and Reddi (2019) Thanh Thi Nguyen and Vijay Janapa Reddi. 2019.  Deep reinforcement learning for cyber security.  _arXiv preprint arXiv:1906.05799_ (2019).
  * Nguyen et al. (2020) Thanh Thi Nguyen, Hammad Tahir, Mohamed Abdelrazek, and Ali Babar. 2020.  Deep Learning Methods for Credit Card Fraud Detection.  _arXiv preprint arXiv:2012.03754_ (2020).
  * Oh and Iyengar (2019) Min-hwan Oh and Garud Iyengar. 2019.  Sequential anomaly detection using inverse reinforcement learning. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 1480–1490.
  * Pang et al. (2020) Guansong Pang, Anton van den Hengel, Chunhua Shen, and Longbing Cao. 2020.  Deep Reinforcement Learning for Unknown Anomaly Detection.  _arXiv preprint arXiv:2009.06847_ (2020).
  * Pang et al. (2019) Guansong Pang, Chunhua Shen, and Anton van den Hengel. 2019\.  Deep anomaly detection with deviation networks. In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_. 353–362.
  * Roy et al. (2018) Abhimanyu Roy, Jingyi Sun, Robert Mahoney, Loreto Alonzi, Stephen Adams, and Peter Beling. 2018\.  Deep learning detecting fraud in credit card transactions. In _2018 Systems and Information Engineering Design Symposium (SIEDS)_. IEEE, 129–134.
  * Ruff et al. (2019) Lukas Ruff, Robert A Vandermeulen, Nico Görnitz, Alexander Binder, Emmanuel Müller, Klaus-Robert Müller, and Marius Kloft. 2019\.  Deep semi-supervised anomaly detection.  _arXiv preprint arXiv:1906.02694_ (2019).
  * Schlegl et al. (2017) Thomas Schlegl, Philipp Seeböck, Sebastian M Waldstein, Ursula Schmidt-Erfurth, and Georg Langs. 2017.  Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In _International conference on information processing in medical imaging_. Springer, 146–157.
  * Shen and Kurshan (2020) Hongda Shen and Eren Kurshan. 2020.  Deep Q-Network-based Adaptive Alert Threshold Selection Policy for Payment Fraud Systems in Retail Banking.  _arXiv preprint arXiv:2010.11062_ (2020).
  * Singh et al. (2020) Avi Singh, Albert Yu, Jonathan Yang, Jesse Zhang, Aviral Kumar, and Sergey Levine. 2020\.  COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning.  _arXiv preprint arXiv:2010.14500_ (2020).
  * Stooke et al. (2020) Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. 2020\.  Decoupling representation learning from reinforcement learning.  _arXiv preprint arXiv:2009.08319_ (2020).
  * Todhunter (1865) I Todhunter. 1865\.  A History of the Mathematical Theory of Probability, Cambridge.  _Reprinted (1949, 1965), New York: Chelsea_ (1865).
  * Varmedja et al. (2019) Dejan Varmedja, Mirjana Karanovic, Srdjan Sladojevic, Marko Arsenovic, and Andras Anderla. 2019\.  Credit card fraud detection-machine learning methods. In _2019 18th International Symposium INFOTEH-JAHORINA (INFOTEH)_. IEEE, 1–5.
  * Walsh et al. (2007) Thomas J Walsh, Ali Nouri, Lihong Li, and Michael L Littman. 2007.  Planning and learning in environments with delayed feedback. In _European Conference on Machine Learning_. Springer, 442–453.
  * Wang ([n.d.]) Mengdi Wang. [n.d.].  Reinforcement Learning from small data.  https://faculty.nps.edu/joroyset/docs/bayopt_2019/mdp-compression-20190517baypopt-shared.pdf
  * Zhang et al. (2019) Xinwei Zhang, Yaoci Han, Wei Xu, and Qili Wang. 2019\.  HOBA: A novel feature engineering methodology for credit card fraud detection with a deep learning architecture.  _Information Sciences_ (2019).
  * Zheng et al. (2018) Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, and Zhenhui Li. 2018.  DRN: A deep reinforcement learning framework for news recommendation. In _Proceedings of the 2018 World Wide Web Conference_. 167–176.
  * Zhinin-Vera et al. (2020) Luis Zhinin-Vera, Oscar Chang, Rafael Valencia-Ramos, Ronny Velastegui, Gissela E Pilliza, and Francisco Quinga-Socasi. 2020.  Q-Credit Card Fraud Detector for Imbalanced Classification using Reinforcement Learning.. In _ICAART (1)_. 279–286.
  * Zhou and Paffenroth (2017) Chong Zhou and Randy C Paffenroth. 2017.  Anomaly detection with robust deep autoencoders. In _Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining_. 665–674.