# Early Churn Prediction from Large Scale User-Product Interaction Time Series

Shamik Bhattacharjee
Dream11
Mumbai, India
<EMAIL_ADDRESS>
&Utkarsh Thukral
Dream11
Mumbai, India
<EMAIL_ADDRESS>
&Nilesh Patil
Dream11
Mumbai, India
<EMAIL_ADDRESS>

(July 1, 2022)

###### Abstract

Churn, defined as the phenomenon of customers discontinuing their relationship
with a business, poses significant economic implications for a variety of
Business-to-Customer settings. Success of most system to user actions like
promotional discounts, retention campaigns, communications & recommendations
often depends on accurately predicting potential churners as the first goal.
This is even more important for fast-moving and transaction heavy domains like
fantasy sports, where even regular spending patterns are driven by factors
completely out of control ( e.g. high value international sports events
affecting smaller parallel events via unexpected secondary and tertiary
interactions, user’s win/loss in a given contest etc ). Therefore, features
that represent transaction history and interaction of a given user with
product are good indicators to predict churn, but require deep domain
understanding and extensive feature engineering. Feature creation for churn
prediction systems also cause large time and resource constraints in
production environment where inference pipelines take $>$ 70% of total time
for feature engineering at the scale of $\sim 10^{8}$ users. This paper
presents a comprehensive study focused on predicting customer churn using
historical data. Through these analyses, our objective is to develop a model
that predicts the likelihood of customer churn, thus enabling businesses to
better understand attrition patterns and subsequently devise effective
customer retention strategies. We formulate churn prediction as multivariate
time series classification and show that predicting churn with user behavior
is a powerful method for noisy Business-to-Customer settings. We show that the
approach reduces need for extensive feature engineering and outperforms other
classical methods prevalent for similar tasks. We present results from our
experiments and show that our Transformer based models using limited features,
improve upon traditional churn prediction methods.

_Keywords_ churn, neural networks, transformers, time series, fantasy sports,
machine-learning systems

##  1 Introduction

The digital revolution has dramatically changed the landscape of sports, and
in particular, the growth of fantasy sports has been significant. As of 2023,
millions of people participate in online fantasy sports games worldwide,
highlighting its transformation from a niche hobby into a mainstream form of
entertainment. However, amidst this growing market, retaining customers
remains a challenging task, rendering the need to understand and predict
customer churn paramount.

Customer churn, defined as the scenario when a customer stops doing business
or ends the relationship with a company, represents a considerable risk and
cost to any business, especially in non-subscription-based industries like
fantasy sports. In any direct Business-to-Customer setting, acquiring new
users is typically much more expensive than retaining existing users. High
churn rates not only result in lost revenue but also increase the acquisition
cost for new customers. In the fiercely competitive fantasy sports industry,
where customer loyalty is frequently transient, the ability to predict and
proactively address churn is crucial. Retaining existing users requires the
ability to know in advance which users are likely to stay and which users are
likely to churn out of the platform. With increasing data volumes across
multiple touch points and progress of machine learning algorithms in recent
years to learn from large datasets, user churn prediction is one of the most
direct application areas.

Churn prediction has emerged as a critical branch of data analysis in many
industries, employing statistical modeling, machine learning, and AI
techniques to identify the likelihood of customer attrition. This study aims
to extend the application of these methods to the fantasy sports industry. By
leveraging historical customer data, we seek to create a model that can
predict customer churn. This predictive model could offer valuable insights to
companies seeking to enhance their customer retention strategies and foster
long-term loyalty.

At Dream11, classical ML models for churn prediction are already used at the
scale of $~{}10^{8}$ users for short future horizon scenarios. The focus now
is to not only improve their performance further, but also increase prediction
time horizon and reduce complexity of computation required for data pipelines.
Some of the classical ML techniques lose out on rich information present in
the data by ignoring sequential or contextual information. Modern state-of-
the-art techniques in the domain of Natural Language Processing or Computer
Vision have shown strong performance in their respective domains since they
are backed by automated feature extraction stems which learn how to extract
complex features from underlying data given a particular target outcome.

This paper will detail the development and testing of a churn prediction model
for the fantasy sports industry. Here, we show that the current state of the
art deep-learning techniques, with some architectural modifications, can be
directly applied for churn prediction. The goal is to provide a valuable tool
for industry stakeholders to mitigate churn, optimize their customer retention
strategies, and ultimately, sustain the growth and profitability of their
businesses in the highly competitive landscape of fantasy sports.

##  2 Related Work

The problem of predicting customer churn is a well-studied area in data-
science with many different approaches proposed in the literature. These
approaches span various industries, from telecommunications and financial
services to e-commerce and online gaming.

Traditional (or classical) machine learning techniques like the Support Vector
Machines (SVM) Kim et al. (2005); Zhao et al. (2005); Vafeiadis et al. (2015);
Shaaban et al. (2012), Logistic Regression ( LR ) De Caigny et al. (2018);
Jain et al. (2020), Decision Trees ( DT ) De Caigny et al. (2018); Hur and Lim
(2005) or shallow Artificial Neural Networks (ANN)s have been successful in
formulating churn prediction as classification Ahn et al. (2020). In Kumar and
Chandrakala (2017), the authors used a combination of the support vector
machine (SVM) and adaptive boosting (AdaBoost) to classify users with high
churn probability. Logistic Regression and Decision Trees are two widely used
algorithms for churn prediction, due to their highly interpretability and ease
of training. In De Caigny et al. (2018), the authors used Logistic Regression
and Decision Trees to create a hybrid stacked model and the stacked model
shows improved performance compared to individual building block models (LR
and DT).

Ensemble algorithms improve upon the performance of constituent learning
algorithms. Ensemble techniques combine the prediction outputs of various
individual algorithms to provide improved and robust predictions Dong et al.
(2020). Ensemble algorithms like Bagging Hu (2005) and Boosting Jinbo et al.
(2007) have widely been used in churn prediction tasks. Bagging techniques,
like the Random Forest classifier Kumar et al. (2008); Pamina et al. (2019);
Ullah et al. (2019) or its improved variants Xie et al. (2009) have been
successful in churn classification problems. Amongst boosting based
algorithms, the XGBoost Pamina et al. (2019); Tang et al. (2020); Çelik and
Osmanoglu (2019) is very successful in this domain.

The literature on the application of deep learning methods for customer churn
prediction has traditionally been relatively scarce. However, there has been a
recent surge in this research area, indicating a growing recognition of the
potential of deep learning in understanding churn behavior. Deep learning
models work as powerful feature extractors, that automatically extract highly
non-linear features from datasets. This minimizes the need for manual feature
engineering. Domingos et al. Domingos et al. (2021) applied a Multilayer
Perceptron (MLP) and a Deep Neural Network (DNN) on public customer churn data
set of a fictitious bank. Cenggoro et al. Cenggoro et al. (2021) used deep-
learning methods for creating embedding vectors which were highly
discriminative between potential churning and loyal users. For this task the
authors used a publicly available telecommunications dataset of 3333
customers. Their best model achieved a F1 score of 81.16%. Panjasuchat and
Limpiyakorn (2020) Panjasuchat and Limpiyakorn (2020) for the first time
introduced reinforcement learning for predicting churn. The authors used a
Deep-Q Network (DQN) to train on a public telecommunication dataset of $100K$
rows and 99 attributes. To train the DQN agent, the authors used the features
vectors as the states and the class labels (churn / no-churn) as the actions.
The authors compared their model with other classical-ML models (XGBoost,
Random Forest, kNN) and showed that the DQN model was more robust compared
with other machine learning models wrt data pattern changes. Cao et al. Cao et
al. (2019) used a Stacked Autoencoder Network (SAE) to extract salient churn-
related features from user interaction data. Subsequently, they employed a
logistic regression-based classification head for churn classification on the
extracted features. In their subscription based service, the authors defined
churn as the successive three months of non-subscription window.

In Wangperawong et al. (2016) , authors represented user interaction data as
images and applied Convolutional Neural Networks (CNNs) to predict churn
labels based on historical user behavior. The authors evaluated two distinct
CNN architectures—DL1 and DL2—which achieved holdout AUC scores of 0.706 and
0.743, respectively. This approach shows a fusion of computer-vision
techniques with churn analysis. Furthermore, Fujo et al. (2022) Fujo et al.
undertook an extensive examination of churn prediction within the
telecommunication sector. Leveraging a Deep Learning (DL) model on the IBM-
Watson public datasets (IBM Telco & Cell2Cell), they conducted a comprehensive
comparative analysis against classical Machine Learning (ML) models. For the
IMB Telco dataset, their DL model achieved a 10-fold cross-validation AUC of
86.57% and a holdout set AUC of 88.11%. Whereas for the Cell2Cell dataset, the
numbers were 73.90% and 79.38%. These results were a significant improvement
over the classical-ML models applied on the same dataset. This underscores the
efficacy of Deep-Learning techniques in enhancing churn prediction accuracy,
highlighting its potential in churn prediction studies. In the paper by
Umayaparvath et al. Umayaparvathi and Iyakutti (2017), the authors developed
three deep neural network architectures to predict churn on two public
datasets, Cell2Cell & CrowdAnalytix. Performance of their deep-learning based
models was as good as traditional classification models.

##  3 Formal Problem Formulation

In our study, we aim to predict user churn in a non-subscription based fantasy
sports platform. Leveraging the past thirty days of user data, we attempt to
predict the churn probabilities for four subsequent weeks. Given a sequence of
user activities over the past 30 days, represented as
$X_{i}=\left[\vec{f}_{t-30},\vec{f}_{t-29},\ldots,\vec{f}_{t-1}\right]$, our
objective is to predict churn probabilities $y_{1},y_{2},y_{3},y_{4}$ for the
next four weeks, where $y_{i}=1$ represents churn and $y_{i}=0$ represents no
churn in the $i^{th}$ week.

The aim is to train a model $y=g(\theta;x)$ on a training set of user activity
sequences and their corresponding churn labels so that the model minimizes the
difference between the predicted churn probabilities and the actual churn
labels. This can be represented as:

$\min_{\theta}\sum_{i=1}^{N}\sum_{j=1}^{4}(y_{ij}-g_{j}(\theta;X_{i}))^{2}$

Where, $N$ is the number of users in the training set, $y_{ij}$ is the actual
churn label for the $i^{th}$ user in the $j^{th}$ week and
$g_{j}(\theta;X_{i})$ is the predicted churn probability for the $i^{th}$ user
in the $j^{th}$ week.

###  3.1 Defining Churn

Defining the target variable — or churn — is a critical step, and the
definition can vary depending on the user’s category or behavior pattern.
Here, we are dealing with a situation where churn does not have a uniform
definition across all users. Churn in non-contractual systems is defined using
consecutive periods of user inactivity. Selection of the exact period depends
on the business needs and how user transaction data is distributed. In the
absence of explicit indications like subscription cancellations, setting clear
and measurable criteria for defining churn is crucial in a non-subscription
model. In the context of our study, we have specifically defined churn within
the non-subscription based fantasy sports industry as a week of inactivity on
our platform. This time frame was chosen as it represents a significant period
of inactivity that could indicate a user’s decreased engagement or intention
to discontinue their involvement in the platform.

This definition of churn – a full week of inactivity – serves as the target
variable that our predictive model, built on Transformers and Convolutional
Neural Networks, will aim to predict. It’s crucial to note that this
definition might not capture all forms of churn, such as users who
significantly reduce, but do not entirely cease their activity. However, it
provides a practical and measurable churn definition that can be universally
applied across the user base for consistent model training and evaluation.

###  3.2 Reformulation as a Classification Problem :

In churn prediction, the problem can be framed as a classification task. We
aim to predict the churn labels for the next four weeks based on the past 30
days of user vector features. Here, each input and output pair $(X,Y)$ is
represented by

$([\vec{f}_{t-30},\vec{f}_{t-29},\ldots,\vec{f}_{t-1}],[y_{\text{week1}},y_{\text{week2}},y_{\text{week3}},y_{\text{week4}}])$

Here, the input features consist of the past 30 days’ worth of user vector
features $\vec{f}$, and the output vectors represents the output churn labels
$y$ for the four future weeks.

Traditionally, churn prediction tasks require extensive feature engineering to
extract meaningful representations from raw data. However, deep neural
networks have a potential advantage of learning relevant features directly
from input data representation, alleviating the need for human knowledge
driven feature engineering. This is particularly beneficial in scenarios where
complex relationships exist within user activity tracking dataset and output
labels.

By leveraging deep learning models, such as convolutional neural networks
(CNNs) or transformer models, we can capture intricate dependencies and
temporal dynamics within the input sequences. These models can effectively
learn and extract high-level representations from the raw data, leading to
competitive precision and recall without the need for explicit feature
engineering.

Overall, by utilizing deep learning models for churn prediction, we can
overcome the limitations of manual feature engineering and achieve competitive
accuracy, precision, and recall in the classification task, as the models are
adept at learning and extracting meaningful representations from the input
data.

###  3.3 Modeling Techniques

Our methodology comprised two distinct approaches, each of which leveraged a
different form of user activity data to predict user churn.

Approach 1 - Classical Machine Learning:

In our first approach, we employed Classical Machine Learning models,
utilizing aggregated feature vectors to predict the probability of user churn.
This approach requires extensive feature engineering since all input features
are domain expertise driven and can not be learnt by the learning algorithm.

Let $G_{i}$ represent the feature vector for user $i$ that we’ve constructed
from aggregated data. Our Classical Machine Learning models then predict
future churn probabilities $P(C_{i}|G_{i})$ for user $i$ over the next four
weeks. This can be written as follows:

$P(C_{i}|G_{i})=Model(G_{i})$ (1)

where $Model$ refers to our Classical Machine Learning models (Logistic
Regression, Random Forest, or Gradient Boosting Trees), $G_{i}$ is the feature
vector for user $i$, and $C_{i}$ is the churn label vector for user $i$ for
the next four weeks.

Approach 2 - Deep Learning Models:

The second approach involved Deep Learning models, specifically Convolutional
Neural Networks (CNNs) and attention based models which were trained on the
users’ past 30 days of time-series feature data to predict churn. This can be
mathematically represented as:

Let $F_{i}==\left[\vec{f}_{t-30},\vec{f}_{t-29},\ldots,\vec{f}_{t-1}\right]$
represent the sequence of 30-day feature vectors for user $i$ that we’ve
constructed from daily transaction data. Our Deep Learning models then predict
the churn probabilities $P(C_{i}|F_{i})$ for user $i$ over the next four
weeks. This can be written as follows:

$P(C_{i}|F_{i})=Model(F_{i})$ (2)

Here, Model refers to our Deep Learning models: Vanilla CNN, VGG-like
Krizhevsky et al. (2012), CNN(filter-width=nfeatures), CNN(filter-
height=$\tau$), Inception-Resnet Szegedy et al. (2015, 2016, 2017), ConvNeXt
Li et al. (2022), Transformer Vaswani et al. (2017); Liu et al. (2021)

The comparative analysis of these two approaches facilitates the evaluation of
the impact of feature representation and model complexity on the prediction of
user churn.

Our quest to improve predictive performance in the task of customer churn
prediction led us to explore a wide array of deep learning models. These
models, each with their own unique capabilities for processing and learning
from time-series data, ranged from simpler architectures like Convolutional
Neural Networks (CNNs) to more intricate designs such as custom Inception-
ResNet hybrids and Transformer Encoders.

We commenced our journey with the traditional CNN Krizhevsky et al. (2012);
Simonyan and Zisserman (2014); Szegedy et al. (2015); He et al. (2016), a
popular choice for time-series analysis due to its capacity to discern
patterns over fixed-sized windows. Convolution neural networks have been
extensively used in problems involving time-series forecasting or
classification tasks Cui et al. (2016); Chen (2015); Ismail Fawaz et al.
(2019). Time-series classification or forecasting has wide range of
applications ranging from stock market prediction Hoseinzade and Haratizadeh
(2019), weather forecasting Weyn et al. (2020), disease detection Dobko et al.
(2020), to customer churn prediction. We benchmark with vanilla CNN ( VGG-like
) for our classification task involving the multivariate user time-series
data. As we sought to enhance our model’s performance, we extended the vanilla
CNN in two distinct ways: First, we modified the kernel width to align with
the dimension of the feature vectors, enabling our model to detect
interactions among all features within a given day. Second, we adjusted the
kernel height to span the entirety of the 30-day time window, thus allowing
our model to capture long-term temporal patterns within the user activity
data.

To obtain even higher predictive performance, We developed a custom hybrid of
Inception Szegedy et al. (2017) and ResNet He et al. (2016) models, designed
to extract multi-scale feature patterns while addressing the vanishing
gradient problem common in deep networks Szegedy et al. (2017); He et al.
(2016). Variations of the Inception-ResNet architecture has also been used for
time-series classification tasks. We explore the InceptionTime Ismail Fawaz et
al. (2020) architecture for our use case. However, we observe a deterioration
in performance.

Following this, we implemented a ConvNeXt architecture Liu et al. (2022).
ConvNeXt is renowned for its ability to capture intricate feature interactions
and reduce computational overhead through its unique grouped convolution
design, making it an attractive choice for our task. In the present work, we
used a scaled-down version of the ConvNeXt model for the classification task
on our reshaped two-dimensional multivariate time-series data.

Finally, we utilized a variant of the Transformer architecture Vaswani et al.
(2017), specifically a scaled-down version of the Vision Transformer (ViT)
model proposed by Dosovitskiy et al Dosovitskiy et al. (2020). This version
retained the self-attention mechanism of the original model, allowing it to
focus on different parts of the input sequence and recognize intricate
temporal patterns across the 30-day user activity window. By only using the
encoder mechanism, we could efficiently handle time-series data, while
reducing the complexity introduced by the decoder mechanism in the full
transformer model.

This broad spectrum of deep learning models provided us with an array of tools
to understand and predict user churn. Each model, with its unique strengths
and features, lent different insights into the user activity data. In the
following sections, we will delve into a more detailed comparison of these
models, and discuss the implications of our findings.

##  4 Data Preparation

The process of preparing the dataset for our predictive models was a
meticulous one, encompassing several crucial stages. Our primary objective was
to construct a detailed yet comprehensible representation of each user’s
interaction history that could be effectively processed by both classical
Machine Learning and Deep Learning models. Given the scale required for
preparing this dataset, we split data preparation into 2 stages : (1) Level-01
Features from transaction level data (2) Level-02 Features from Level-01
features. The raw data form is shown in Table 1 and we use these transactions
data to create Level-01 features.

Representing user transactional data in a table, we might have a structure
similar to the following:

Transaction ID | User ID | Timestamp | Feature Vector
---|---|---|---
$1$ | 1 | $t_{1,1}$ | $\vec{f}_{1,1}$
$2$ | 1 | $t_{2,1}$ | $\vec{f}_{2,1}$
⋮ | ⋮ | ⋮ | ⋮
$n_{1}$ | 1 | $t_{n_{1},1}$ | $\vec{f}_{n_{1},1}$
⋮ | ⋮ | ⋮ | ⋮
$N$ | $M$ | $t_{1,M}$ | $\vec{f}_{1,M}$
$N+1$ | $M$ | $t_{2,M}$ | $\vec{f}_{2,M}$
⋮ | ⋮ | ⋮ | ⋮
$N+n_{M}$ | $M$ | $t_{n_{M},M}$ | $\vec{f}_{n_{M},M}$
Table 1: Example of raw transactional data

In Table 1, each row is representative of a unique transaction, identified
distinctly by a Transaction ID. The User ID field pertains to the specific
user who was involved in the transaction, while the Timestamp field indicates
the precise time at which the transaction occurred. The array of features tied
to each transaction is denoted in the Feature Vector column. These
transactional data sets form the cornerstone for generating the Level-1
features that will further be used in the modeling process.

This raw transaction data is then aggregated on a day level to form the
Level-01 features that feed into the model (Table 2). The Level-01 features
will be vectors of aggregated feature values (such as count, sum, mean, etc.)
for each user for each day.

Here’s how the Level-01 features can be represented for each user:

User ID | Date | Feature Vector | No. of Transactions
---|---|---|---
1 | $d_{1,1}$ | $\vec{f}_{1,1}$ | $n_{1,1}$
| $d_{1,2}$ | $\vec{f}_{1,2}$ | $n_{1,2}$
| $\vdots$ | $\vdots$ | $\vdots$
| $d_{1,D_{1}}$ | $\vec{f}_{1,D_{1}}$ | $n_{1,D_{1}}$
2 | $d_{2,1}$ | $\vec{f}_{2,1}$ | $n_{2,1}$
| $d_{2,2}$ | $\vec{f}_{2,2}$ | $n_{2,2}$
| $\vdots$ | $\vdots$ | $\vdots$
| $d_{2,D_{2}}$ | $\vec{f}_{2,D_{2}}$ | $n_{2,D_{2}}$
$\vdots$ | $\vdots$ | $\vdots$ | $\vdots$
$M$ | $d_{M,1}$ | $\vec{f}_{M,1}$ | $n_{M,1}$
| $d_{M,2}$ | $\vec{f}_{M,2}$ | $n_{M,2}$
| $\vdots$ | $\vdots$ | $\vdots$
| $d_{M,D_{M}}$ | $\vec{f}_{M,D_{M}}$ | $n_{M,D_{M}}$
Table 2: Level-1 User Feature Table. Each row under a user ID represents a
day’s aggregated features of that user.

Using the Level-01 aggregates, we create the training data. For each user, we
consider the past $30$ days of data as the time series input and generate the
corresponding churn label for the next four weeks. The churn label for each
week indicates whether the user has churned (inactive) or not.

To clarify this in the context of the table, we can define the training data
as follows:

\- Input (X): For each user, we extract the feature vectors from the past 30
days, denoted as $\vec{f}_{t-30},\vec{f}_{t-29},\ldots,\vec{f}_{t-1}$, where
$t$ represents the timestamp of the last transaction in the training data.

\- Output (Y): The churn label for each week is represented as a binary value,
indicating whether the user has churned (1) or not churned (0) in that
specific week. Let’s denote these labels as
$y_{\text{week1}},y_{\text{week2}},y_{\text{week3}},y_{\text{week4}}$.

Hence, the training data can be represented as a pair of input-output
sequences:

$\left(\left[\vec{f}_{t-30},\vec{f}_{t-29},\ldots,\vec{f}_{t-1}\right],\left[y_{\text{week1}},y_{\text{week2}},y_{\text{week3}},y_{\text{week4}}\right]\right)$

This training data can then be used to train models for churn prediction using
various techniques, such as classical machine learning models or deep learning
models.

In the next stage, we leveraged the Level 01 features to create Level-02 data
by conducting various aggregations such as mean ($\mu$), standard deviation
($\sigma$), and other statistical measures over the 30-day window. This
higher-level aggregation aimed to capture different aspects of the user’s
behavior, including their consistency and variability in their activity over
time. Formally, for a user $i$ and a feature $j$, the Level-02 features were
defined as:

$\mu_{i,j}=\frac{1}{30}\sum_{k=0}^{d}f_{i,k,j}$
$\sigma_{i,j}=\sqrt{\frac{1}{30}\sum_{k=0}^{d}(f_{i,k,j}-\mu_{i,j})^{2}}$

The resulting Level-02 data offered a more condensed and generalized view of
the user’s historical activity, which was particularly suited for our
classical Machine Learning models. The Level-02 feature vector $G_{i}$ for
user $i$ was then formed as:

$G_{i}=\\{\mu_{i,1},\sigma_{i,1},...,\mu_{i,N},\sigma_{i,N}\\}$

Since our model is designed to predict future user churn, we construct a
binary target vector $Y_{i}$ for each $user_{i}$, where each element $y_{i,w}$
of the vector corresponds to whether the $i^{th}$ user churns in week $w$ (1
to 4) after the end of the 30-day activity window:

$Y_{i}=\\{y_{i,1},y_{i,2},y_{i,3},y_{i,4}\\}$

Formally, for each week $w_{j}$ after the 30-day window, $y_{i,w}$ is defined
as:

$y_{i,w}=\begin{cases}1&\text{{if user i has no activity in week } w}\\\
0&\text{otherwise}\end{cases}$

It is important to note that by designing the target as a vector, we enable
our model to provide a more granular prediction, indicating not just if a user
will churn, but also when they are likely to churn within a four-week horizon.
This additional information can be of significant value to various business
operations, such as customer engagement and retention strategies.

For our current work, we selected eleven features from a pool of a large
number of available features. These eleven features were shortlisted through
rigorous exploratory data analysis (EDA) and domain knowledge.

###  4.1 Data Sampling

We run our experiments on sampled data from Dream11’s fantasy sports user
base. We use user transaction history records between 2018-01-01 and
2020-12-31. The setting here is non-contractual churn. We first create a
random sample of Dream11 users and process their transactions for creating
modeling dataset for our experiments. This modeling dataset has $\sim 10^{6}$
distinct users and $\sim 10^{8}$ raw transactions with various different types
of transactions. This large dataset provides a comprehensive historical
perspective of user interactions with our fantasy sports platform. Using this
dataset, we created the train, validation and test datasets. The Train,
Validation and Test ratio was chosen to be $0.75$, $0.05$ and $0.20$
respectively.

Here, We are not mentioning individual sporting events because we are not
doing analysis for individual sporting events. We are considering users’ past
history irrespective of individual sporting events across all models. Any such
system-level features can be included in the same formulation by adding
additional covariates.

##  5 Training Setup

Leveraging this vast dataset, our deep learning models were trained using the
PyTorch framework, a popular open-source machine learning library for Python.
We follow the same training protocols for all models. All models were
initialized with random weights and trained for 100 epochs. In brief, we used
the Adam optimizer Kingma and Ba (2014) to train our models.

In terms of computational setup, our training process was conducted in a
distributed manner across multiple GPUs, ensuring efficient handling of our
large-scale data and complex models. We utilized Horovod, a distributed deep
learning training framework, to coordinate the training process across these
GPUs. Horovod facilitates synchronous distributed training, enabling each
participating GPU to independently compute the gradients for the subset of
data it was allocated and then collectively averaging these gradients to
update the model weights. This method ensures model consistency and
accelerates the training process by allowing concurrent computations.

The process of loading data into our training environment was facilitated by
Petastorm, an open-source data access library. It enabled efficient and high-
speed loading of our large-scale data directly from Amazon S3 into our PyTorch
models. Petastorm creates a streamlined interface between the data stored in
S3 and the PyTorch DataLoader, simplifying the data ingestion process and
enhancing the performance of our training operations.

Given the complex nature of our models and the size of our data, the training
process was set to run for 100 epochs. This balance was struck to allow
sufficient learning time for our models to capture intricate patterns in the
data while avoiding excessive training that could lead to overfitting. Over
the course of these epochs, the model performance was continuously monitored,
and adjustments were made as necessary to optimize the prediction accuracy.

We conducted grid search over all the tunable hyper-parameters on the held-out
validation set for each of the models. The global batch size was chosen $16$K
training samples. During training, each GPU was allocated $\sim 4$K training
samples. This batch size was carefully chosen to achieve best training
performance at the shortest possible training time. The learning-rate
parameter ($lr$) for all the models was kept fixed at $10^{-4}$. The number of
trainable parameters for different models varied significantly and hence total
training time also differed for each. Our custom transformer architecture has
approximately $1.8$ million trainable parameters, featuring a composition of
eight self-attention blocks. The custom Inception-Resnet comprises nine
inception blocks, each encompassing convolution operations with three
different kernel sizes: $3\times 3$, $5\times 5$ and $7\times 7$. The
architecture contained a total of approximately $2.4$ million trainable
parameters. Our adapted ConvNeXt architecture featured three convolution
layers with approximately $0.8$ million trainable parameters. For our Long
Short-Term Memory (LSTM) model, we designed a five-layer architecture,
followed by a classification head comprised of two dense layers. The total
count of trainable parameters here was approximately one million. Both the
Convolutional Neural Network architectures have approximately one million
trainable parameters. Here also a two layer dense neural network has been used
as the classification head for predicting the churn label.

For all models, we used a Batch-Normalization Ioffe and Szegedy (2015) layer
to improve stability and ensure faster training convergence. To reduce model
over-fitting, Dropout layers Srivastava et al. (2014) were used. The Dropout
probability ($p$) was chosen carefully for the individual models and it varied
from $0.1$ to $0.4$. Training time for CNN was least ( 6 hours) while training
a transformer model took the longest ( 13 hours). We experimented with
N-epochs = [10, 20, 50, 75, 100] and observed that model metrics improvement
slowed down after 20 epochs and did not benefit from any additional epochs
after 50.

4 Nvidia-A10 GPUs were used for training Deep learning models while Spark was
used for creating features at different aggregation levels over the complete
dataset and instrumenting distributed training using Horovod Sergeev and Balso
(2018) and Petastorm .

##  6 Results

A receiver operating characteristic curve, or $ROC\ curve$, is used to measure
model performance of a binary classifier as the discrimination threshold on
top of predicted scores is varied. We calculate True Positive Rate ($TPR$) and
False Positive Rate ($FPR$) for each threshold $\in[0,1]$ as shown below.

$TPR=\frac{TP}{TP+FN}$ (3) $FPR=\frac{FP}{FP+TN}$ (4)

A high level summary of results from all of our experiments is presented in
Table 3. Our best performing model was the transformer architecture which
shows  $\sim\textbf{6\%}$ improvement on an average across all time periods
compared to best performing GBT model. Complete $ROC$ Curves for each model
are also presented (Fig. 1) which are used to calculate the $AUC$ benchmark.

We also present $Precision-Recall$ Curves for all of our models (1), showing
substantial difference between each model variant. In our use case, there is
always a trade-off involved between high false $+ve$ rate vs retention
budgets. The goal is to get as many as users as we can without wasting
treatment budgets. This trade-off has different thresholds for each
application, e.g. for giving promotional discounts using predicted scores, we
have to be more stringent while for communication related actions, we have
substantial leeway.

Classifier | $AUC_{W_{01}}$ | $AUC_{W_{02}}$ | $AUC_{W_{03}}$ | $AUC_{W_{04}}$
---|---|---|---|---
LR | 0.644 | 0.749 | 0.672 | 0.662
RF | 0.640 | 0.775 | 0.699 | 0.687
GBT | 0.646 | 0.773 | 0.703 | 0.694
CNN | 0.797 | 0.705 | 0.686 | 0.675
ConvNext | 0.835 | 0.751 | 0.729 | 0.717
Inception-Resnet | 0.819 | 0.729 | 0.704 | 0.696
$CNN_{H=\tau}$ | 0.807 | 0.742 | 0.725 | 0.716
$CNN_{W=N}$ | 0.822 | 0.747 | 0.728 | 0.729
LSTM | 0.800 | 0.742 | 0.727 | 0.721
Transformer | 0.858 | 0.756 | 0.732 | 0.716
Table 3: Model Performance |  |  |
---|---|---|---
|  |  |
Figure 1: Model Performance

###  6.1 Results from Classical Models :

As shown in Table 3, results from Logistic Regression are worse across all
time periods (i.e. Week - 01 / 02 / 03 / 04 churn ) but it is a sensible
benchmark against which other modeling approaches can be tested. Ensemble
models perform better compared to the LR model. The PR Curve for GBT vs LR
clearly shows performance improvement as we move from LR $\longrightarrow$
GBT. Overall, these methods under-perform substantially compared to
Transformer based model as shown in Table 3.

###  6.2 Results from Deep Networks :

Our Transformer model has consistently higher performance compared to others
with an additional training time of $15\%$ for the current version. Based on
ROC and PR curves in Model Performance, there are certain instances where ROC
curve for classical algorithms is better compared to Deep learning based
models. Taking a deeper look through the PR curves though, shows substantially
poor performance. This is primarily caused by data imbalance in churn problem
statements, where label distribution is heavily skewed towards one label. In
our case, different weeks have different skews ranging from $2$x to $10$x
which shows up in poor performance of models with lower learning capacity.

##  7 Conclusions and Summary

In the current paper, we presented our work on User Churn Prediction. We first
formulated this problem as a classification problem statement and proposed two
distinct approaches to solve it for real world datasets. In the first
approach, we aggregated a user’s activity time series into a row vector and
build a classifier on top of this representation. To improve performance of
this approach, extensive domain knowledge is required to create new features
from a user’s time series. With this approach, results from GBTs perform
better compared to other classical ML methods.

Second approach creates a 2D array of chronologically arranged observations
($X_{k,\tau}$) as input for the DL models. Here, we used DL models which take
advantage of the sequential nature as well as interaction between the
features. The highest performing model was the Transformer architecture which
outperforms prediction by the classical models by $\sim 6\%$ on an average.
With this improved approach, there’s a substantial scope for exploring
seq-2-seq applications in other time series applications ( e.g. CLTV
prediction, Forecasting, sequential product recommendations being some of the
top use-cases ).

A major limitation of this approach is requirement of large training data
volume and long training duration. The first limitation is usually less
relevant for forecasting user metrics - given high user counts in most
Business to Customer settings, but could prove to be a problem for other
entities used in forecasting.

## References

  * Kim et al. [2005] Sun Kim, Kyung-shik Shin, and Kyungdo Park.  An application of support vector machines for customer churn analysis: Credit card case.  In _International Conference on Natural Computation_ , pages 636–647. Springer, 2005.
  * Zhao et al. [2005] Yu Zhao, Bing Li, Xiu Li, Wenhuang Liu, and Shouju Ren.  Customer churn prediction using improved one-class support vector machine.  In _International conference on advanced data mining and applications_ , pages 300–306. Springer, 2005.
  * Vafeiadis et al. [2015] Thanasis Vafeiadis, Konstantinos I Diamantaras, George Sarigiannidis, and K Ch Chatzisavvas.  A comparison of machine learning techniques for customer churn prediction.  _Simulation Modelling Practice and Theory_ , 55:1–9, 2015.
  * Shaaban et al. [2012] Essam Shaaban, Yehia Helmy, Ayman Khedr, and Mona Nasr.  A proposed churn prediction model.  _International Journal of Engineering Research and Applications_ , 2(4):693–697, 2012.
  * De Caigny et al. [2018] Arno De Caigny, Kristof Coussement, and Koen W De Bock.  A new hybrid classification algorithm for customer churn prediction based on logistic regression and decision trees.  _European Journal of Operational Research_ , 269(2):760–772, 2018.
  * Jain et al. [2020] Hemlata Jain, Ajay Khunteta, and Sumit Srivastava.  Churn prediction in telecommunication using logistic regression and logit boost.  _Procedia Computer Science_ , 167:101–112, 2020.
  * Hur and Lim [2005] Yeon Hur and Sehun Lim.  Customer churning prediction using support vector machines in online auto insurance service.  In _International Symposium on Neural Networks_ , pages 928–933. Springer, 2005.
  * Ahn et al. [2020] Jaehyun Ahn, Junsik Hwang, Doyoung Kim, Hyukgeun Choi, and Shinjin Kang.  A survey on churn analysis in various business domains.  _IEEE Access_ , 8:220816–220839, 2020.
  * Kumar and Chandrakala [2017] A Saran Kumar and D Chandrakala.  An optimal churn prediction model using support vector machine with adaboost.  _Int. J. Sci. Res. Comput. Sci. Eng. Inf. Technol_ , 2(1):225–230, 2017.
  * Dong et al. [2020] Xibin Dong, Zhiwen Yu, Wenming Cao, Yifan Shi, and Qianli Ma.  A survey on ensemble learning.  _Frontiers of Computer Science_ , 14(2):241–258, 2020.
  * Hu [2005] Xiaohua Hu.  A data mining approach for retailing bank customer attrition analysis.  _Applied Intelligence_ , 22(1):47–60, 2005.
  * Jinbo et al. [2007] Shao Jinbo, Li Xiu, and Liu Wenhuang.  The application ofadaboost in customer churn prediction.  In _2007 International Conference on Service Systems and Service Management_ , pages 1–6. IEEE, 2007.
  * Kumar et al. [2008] Dudyala Anil Kumar, Vadlamani Ravi, et al.  Predicting credit card customer churn in banks using data mining.  _Int. J. Data Anal. Tech. Strateg._ , 1(1):4–28, 2008.
  * Pamina et al. [2019] Jeyakumar Pamina, Beschi Raja, S SathyaBama, MS Sruthi, Aiswaryadevi VJ, et al.  An effective classifier for predicting churn in telecommunication.  _Jour of Adv Research in Dynamical & Control Systems_, 11, 2019.
  * Ullah et al. [2019] Irfan Ullah, Basit Raza, Ahmad Kamran Malik, Muhammad Imran, Saif Ul Islam, and Sung Won Kim.  A churn prediction model using random forest: analysis of machine learning techniques for churn prediction and factor identification in telecom sector.  _IEEE access_ , 7:60134–60149, 2019.
  * Xie et al. [2009] Yaya Xie, Xiu Li, EWT Ngai, and Weiyun Ying.  Customer churn prediction using improved balanced random forests.  _Expert Systems with Applications_ , 36(3):5445–5449, 2009.
  * Tang et al. [2020] Qi Tang, Guoen Xia, Xianquan Zhang, and Feng Long.  A customer churn prediction model based on xgboost and mlp.  In _2020 International Conference on Computer Engineering and Application (ICCEA)_ , pages 608–612. IEEE, 2020.
  * Çelik and Osmanoglu [2019] Ozer Çelik and Usame O Osmanoglu.  Comparing to techniques used in customer churn analysis.  _Journal of Multidisciplinary Developments_ , 4(1):30–38, 2019.
  * Domingos et al. [2021] Edvaldo Domingos, Blessing Ojeme, and Olawande Daramola.  Experimental analysis of hyperparameters for deep learning-based churn prediction in the banking sector.  _Computation_ , 9(3):34, 2021.
  * Cenggoro et al. [2021] Tjeng Wawan Cenggoro, Raditya Ayu Wirastari, Edy Rudianto, Mochamad Ilham Mohadi, Dinne Ratj, and Bens Pardamean.  Deep learning as a vector embedding model for customer churn.  _Procedia Computer Science_ , 179:624–631, 2021.
  * Panjasuchat and Limpiyakorn [2020] M Panjasuchat and Y Limpiyakorn.  Applying reinforcement learning for customer churn prediction.  In _Journal of Physics: Conference Series_ , volume 1619, page 012016. IOP Publishing, 2020.
  * Cao et al. [2019] Shulin Cao, Wei Liu, Yuxing Chen, and Xiaoyan Zhu.  Deep learning based customer churn analysis.  In _2019 11th International Conference on Wireless Communications and Signal Processing (WCSP)_ , pages 1–6. IEEE, 2019.
  * Wangperawong et al. [2016] Artit Wangperawong, Cyrille Brun, Olav Laudy, and Rujikorn Pavasuthipaisit.  Churn analysis using deep convolutional neural networks and autoencoders.  _arXiv preprint arXiv:1604.05377_ , 2016.
  * Fujo et al. [2022] Samah Wael Fujo, Suresh Subramanian, Moaiad Ahmad Khder, et al.  Customer churn prediction in telecommunication industry using deep learning.  _Information Sciences Letters_ , 11(1):24, 2022.
  * Umayaparvathi and Iyakutti [2017] V Umayaparvathi and K Iyakutti.  Automated feature selection and churn prediction using deep learning models.  _International Research Journal of Engineering and Technology (IRJET)_ , 4(3):1846–1854, 2017.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ , 25, 2012.
  * Szegedy et al. [2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1–9, 2015.
  * Szegedy et al. [2016] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2818–2826, 2016.
  * Szegedy et al. [2017] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi.  Inception-v4, inception-resnet and the impact of residual connections on learning.  In _Thirty-first AAAI conference on artificial intelligence_ , 2017.
  * Li et al. [2022] Jiachen Li, Chixin Wang, Banban Huang, and Zekun Zhou.  Convnext-backbone hovernet for nuclei segmentation and classification.  _arXiv preprint arXiv:2202.13560_ , 2022.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10012–10022, 2021.
  * Simonyan and Zisserman [2014] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Cui et al. [2016] Zhicheng Cui, Wenlin Chen, and Yixin Chen.  Multi-scale convolutional neural networks for time series classification.  _arXiv preprint arXiv:1603.06995_ , 2016.
  * Chen [2015] Yahui Chen.  Convolutional neural network for sentence classification.  Master’s thesis, University of Waterloo, 2015.
  * Ismail Fawaz et al. [2019] Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller.  Deep learning for time series classification: a review.  _Data mining and knowledge discovery_ , 33(4):917–963, 2019.
  * Hoseinzade and Haratizadeh [2019] Ehsan Hoseinzade and Saman Haratizadeh.  Cnnpred: Cnn-based stock market prediction using a diverse set of variables.  _Expert Systems with Applications_ , 129:273–285, 2019.
  * Weyn et al. [2020] Jonathan A Weyn, Dale R Durran, and Rich Caruana.  Improving data-driven global weather prediction using deep convolutional neural networks on a cubed sphere.  _Journal of Advances in Modeling Earth Systems_ , 12(9):e2020MS002109, 2020.
  * Dobko et al. [2020] Mariia Dobko, Bohdan Petryshak, and Oles Dobosevych.  Cnn-cass: Cnn for classification of coronary artery stenosis score in mpr images.  _arXiv preprint arXiv:2001.08593_ , 2020.
  * Ismail Fawaz et al. [2020] Hassan Ismail Fawaz, Benjamin Lucas, Germain Forestier, Charlotte Pelletier, Daniel F Schmidt, Jonathan Weber, Geoffrey I Webb, Lhassane Idoumghar, Pierre-Alain Muller, and François Petitjean.  Inceptiontime: Finding alexnet for time series classification.  _Data Mining and Knowledge Discovery_ , 34(6):1936–1962, 2020.
  * Liu et al. [2022] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11976–11986, 2022.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Ioffe and Szegedy [2015] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _International conference on machine learning_ , pages 448–456. PMLR, 2015.
  * Srivastava et al. [2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ , 15(1):1929–1958, 2014.
  * Sergeev and Balso [2018] Alexander Sergeev and Mike Del Balso.  Horovod: fast and easy distributed deep learning in TensorFlow.  _arXiv preprint arXiv:1802.05799_ , 2018.