[style=theoremstyle]claimClaim

# A Hybrid Statistical-Machine Learning Approach for Analysing Online Customer
Behavior: An Empirical Study

Saed Alizamir School of Management, Yale University, New Haven, CT, USA.
Email<EMAIL_ADDRESS>Kasun Bandara School of Computing and
Information Systems, Melbourne Centre for Data Science, University of
Melbourne, VIC, Australia. Email<EMAIL_ADDRESS>Ali Eshragh
Carey Business School, Johns Hopkins University, MD, and International
Computer Science Institute, Berkeley, CA, USA. Email<EMAIL_ADDRESS>Foaad Iravani Uber Freight, San Francisco, CA, USA. Email<EMAIL_ADDRESS>
###### Abstract

We apply classical statistical methods in conjunction with the state-of-the-
art machine learning techniques to develop a hybrid interpretable model to
analyse 454,897 online customers’ behavior for a particular product category
at the largest online retailer in China, that is JD. While most mere machine
learning methods are plagued by the lack of interpretability in practice, our
novel hybrid approach will address this practical issue by generating
explainable output. This analysis involves identifying what features and
characteristics have the most significant impact on customers’ purchase
behavior, thereby enabling us to predict future sales with a high level of
accuracy, and identify the most impactful variables. Our results reveal that
customers’ product choice is insensitive to the promised delivery time, but
this factor significantly impacts customers’ order quantity. We also show that
the effectiveness of various discounting methods depends on the specific
product and the discount size. We identify product classes for which certain
discounting approaches are more effective and provide recommendations on
better use of different discounting tools. Customers’ choice behavior across
different product classes is mostly driven by price, and to a lesser extent,
by customer demographics. The former finding asks for exercising care in
deciding when and how much discount should be offered, whereas the latter
identifies opportunities for personalized ads and targeted marketing. Further,
to curb customers’ batch ordering behavior and avoid the undesirable Bullwhip
effect, JD should improve its logistics to ensure faster delivery of orders.

Keywords: statistical data analysis, machine learning, customer behavior
prediction, Shapley values, interpretability

##  1 Introduction

Accurate sales forecasts are crucial to managing supply chains effectively.
Reliable sales forecasts give businesses insights about how they should
utilize resources, control production, inventory, and logistics, manage risks,
and plan for the foreseeable future. Furthermore, accurate sales forecasts
drive better performance, and help managers set realistic goals for the
business.

There are three main approaches for forecasting, including classical
statistical models [27], traditional judgmental methods [29], and state-of-
the-art machine learning techniques [26, 18]. While the first two approaches
have been extensively developed and studies in the literature [1], the third
approach has recently gained popularity [8].

Statistical predictive models (e.g., linear and logistic regression models)
are easy to interpret but typically make some assumptions which might be
violated in practice. Alternatively, one may use advanced machine learning
(ML) methods (e.g., neural networks and support vector machines). These
sophisticated methods are more flexible and adaptable to complex (but
intuitive) behaviors in reality, such as nonlinear relationships or
interactions between explanatory variables. While ML methods can usually
provide more accurate predictions, the “black-box” nature of these methods
makes it difficult to interpret and explain how they generate a prediction or
make a recommendation in a way that is understandable by humans. This issue
creates a trade-off between accuracy and interpretability [25].

In a recent review of data analytics applications in operations management,
[34] highlight the importance of using interpretable ML models in OM. However,
there are only a few recent papers on interpretable models in OM. [12] propose
an algorithm for optimal stopping problems, in the form of a binary tree. They
show that their tree policies are simple and intuitive and outperform the non-
interpretable ML methods. [9] develop an ML approach, called mining optimal
policies (MinOP), that extracts the structural properties of optimal policies
for stochastic dynamic programs from numerical solutions to problem instances.
[5] provide a decision-tree framework for segmenting users in a population-
based on differences in their response patterns.

Explainable AI (stands for Artificial Intelligence) (or Trusted AI) is an
emerging development within AI that proposes methods to help interpret the
predictions of a complex model, while maintaining the high level of prediction
accuracy [35]. The main objective of Explainable AI is to build the trust of
users in a model by offering transparent reasons for why the model makes a
prediction. The ML methods are used for decision making in a wide range of
industries, such as manufacturing, healthcare, education, public policy, and
finance. Users will be reluctant to accept and act upon model outcomes if they
cannot understand how the model makes recommendations, especially if the
consequences can be catastrophic (e.g., medical diagnosis or counter-
terrorism). Moreover, regulators are increasingly passing legislation (e.g.,
the General Data Protection Rules in the European Union), requiring companies
to provide an explanation of their data-driven decisions to consumers [14,
22]. Currently, the state-of-the-art algorithm in Explainable AI is SHAP
(SHapley Additive exPlanations) developed by [32]. The SHAP framework has
proved to be an important advancement in the field of machine learning model
interpretation and provides an intuitive unified approach to explaining the
output of any ML model.

Our paper makes three contributions to the burgeoning field of ML models in
OM. First, we use a highly-efficient recent algorithm (LightGBM) to predict
JD’s sales. We show that LightGBM performs better than LASSO Regression, Ridge
Regression and KNN Regressor. Second, we use SHAP to interpret these
predictions. Using SHAP’s global interpretability property, we show the
importance of each variable in predicting sales. We also present individual
SHAP value plots that describe each variable’s marginal contribution to the
predicted purchase volume of any individual user. Third, we present some of
the significant interaction terms that are identified by SHAP.

To the best of our knowledge, this work is the first attempt in the OM
community to use modern ML algorithms to predict online users’ purchase
behavior and, more importantly, offer clear and insightful interpretations
about the prediction model’s outcomes. Our analysis helps JD managers identify
features/attributes that have the highest impact on sales volume and the
direction of the association between these features/attributes and sales.
Using this information, the managers can increase sales volume by focusing
their efforts and investments on the most impactful variables. When our
analytical framework is used to predict a given user’s total purchase, the
managers can clearly measure and interpret the portion of the forecasted sales
that is attributable to each variable. In other words, our analysis helps
managers explain why an individual prediction is obtained.

The rest of the paper is organized as follows. Section 2 reviews the
literature. Section 3 describes the data. Section 4 analyzes product sales and
Section 5 analyzes customer choice behavior. We conclude the paper with a
discussion in Section 6.

##  2 Literature

Forecasting is the art of utilizing the available data to predict the future
values of a dynamic system evolving in time. It has a wide range of
applications spanning from business and health to science and engineering [10,
41, 16, 33, 1, 15]. In particular, in supply chain management (SCM), product
demand forecasting plays a crucial role as numerous decisions such as
production planning, logistics, and inventory management heavily rely on
demand forecasts [20]. Therefore, providing accurate demand forecasts can
directly result in better operational efficiency, customer satisfaction, and
financial savings throughout the entire supply chain [28, 43].

In general, there are three main approaches for demand forecasting as listed
and explained below:

(i) Statistical Models.

Statistical forecasting models are the time series models that use merely
historical data to predict future values. Some common statistical forecasting
models include (but are not limited to) exponential smoothing, and auto-
regressive integrated moving average (ARIMA) model (also known as Box-Jenkins
model) along with its numerous variants (e.g., SARIMA, ARIMAX, and GARCH) [24,
17].

In the context of SCM, statistical forecasting models have been developed with
advancements in computational power, software and information system
technologies such as enterprise resource planning systems, electronic data
interchange, and point of sale scanning [39]. Such advancements have enabled
vast amounts of data to be easily collected, analyzed, and shared throughout
the supply chain. A more advanced form of these models are hybrid time series-
regression models which incorporate the historical data of the primary time
series (e.g., demand) along with one or more explanatory independent variables
influencing the primary time series (e.g., promotions, holidays and special
events that can impact customer demand) [44, 1].

Although, in practice, there exist several potential candidates for
explanatory independent variables, it is prudent to keep models as
parsimonious as possible while maintaining desired accuracy [11]. This is
because implementing hybrid time series-regression models with a high number
of explanatory variables would be costly in terms of time and memory.
Therefore, they are not frequently utilized in industry due to high costs,
lack of internal expertise and resources, as well as other organizational
barriers [43].

(ii) Judgmental Methods.

Judgmental forecasting methods refer to traditional techniques incorporating
intuitive judgment, opinions and subjective probability estimates provided by
experts. Although it has been shown that there are usually high biases in
forecasts provided by human [45], there has been recent increase in
integrating these methods with statistical models so that complementary
benefits can be realized to alleviate the drawbacks of each approach [4, 6].

[30] classify the useful data for demand forecasting into two classes: (1)
historical data, and (2) contextual knowledge. While the former data is simply
the recorded time series of historical product sales, contextual knowledge is
any other information relevant to interpreting, explaining and anticipating
time series behavior, such as promotional plans, competitor activities, and
sudden climate changes. Statistical methods are well suited to handle vast
amounts of historical data, but when the effects of discontinuities which are
often caused by contextual factors, cannot be estimated from historical data,
human judgment can be utilized to overcome this issue and incorporate valuable
contextual information by adjusting baseline statistical forecasts [23, 29,
28]. The demonstrated accuracy improvements through judgmental forecast
adjustments have inevitably resulted in its widespread use in industry [39,
19, 36].

(iii) State-of-the-art Machine Learning Techniques.

Machine learning techniques are increasing in popularity with applications in
many different areas. In particular, data scientists have applied these state-
of-the-art methods to develop novel models and algorithms for time series
forecasting. In this new approach, historical data are used to train a (deep)
neural network (NN) to predict the future values. Due to the dependency
structure of time series data, recurrent neural networks (RNNs) are the most
common class of NNs used in forecasting [26]. A few packages have been
developed for utilizing RNNs in predicting time series data [2].

In supply chain demand forecasting problems, the requirement is often to
produce forecasts for several product demands that may have similar patterns.
In such scenarios, global models can learn across input time series to
incorporate more information, in comparison to local methods [43]. Examples
include [37, 7, 38]. For a more comprehensive literature review refer to [8].

Although ML methods can provide accurate forecasts, unlike statistical models,
they may not be easily explained and interpreted. The address this major
drawback of ML methods, Explainable AI has been emerged. A novel algorithm in
this area is called SHAP [32]. The SHAP algorithm provides an intuitive
approach to interpreting the output of ML models. The idea of SHAP comes from
the classic Shapley values that determine the contribution of each player in a
cooperative game [40]. The SHAP values quantify the contribution that each
feature (variable)111We will be using the terms “variable” and “feature”,
interchangeably. brings to the prediction made by a model. The SHAP values has
three important properties:

  1. 1.

Global Interpretability: SHAP values collectively can show how much each
predictor variable contributed to the target variable, either positively or
negatively.

  2. 2.

Local Interpretability: Each prediction receives its own set of SHAP values.
This is a useful property that significantly enhances the interpretability of
ML algorithms. Traditional algorithms that analyze variable importance only
measure the results over the entire population, but not on individual
predictions. Local interpretability enables users to compare the impacts of
different features across different predictions.

  3. 3.

Computability: SHAP values can be calculated for any tree-based model, whereas
other methods use linear or logistic regression models.

In this paper we mainly apply statistical methods and ML techniques to analyze
JD’s data. To provide interpretability to our ML outcome, we utilize the SHAP
algorithm in our analysis. The details of our analysis are elaborated in
Sections 3, 4, and 5.

##  3 JD Data

We analyze JD.com’s user transaction data for March $2018$. The MSOM Society
provided the data for the $2020$ Data-Driven Challenge. JD is China’s largest
online retailer and its biggest retailer overall. A member of the Fortune
Global $500$, JD was founded in $1998$ and its retail platform moved online in
$2004$. JD’s logistic network provides standard same- and next-day delivery of
a wide range of products to $99\%$ of China’s population (www.JD.com). This
section provides data description as well as empirical statistical analysis on
users spending behavior.

###  3.1 Data Description

The dataset consists of $20.2$ million click records from $2.56$ million known
users who explored $31,868$ SKUs from one product category during the month of
March in $2018$. Out of these numbers, only $454,897$ users (i.e., $17.78\%$
of known users) ordered at least one item during the month. Furthermore, these
$454,897$ buyer users consists of $374,124$ users who explored the site before
placing their order (i.e., there exist some click records for each of them
before placing an order), $20,573$ who appear to have made their purchase
decisions before logging in (no pre-order interaction data is available), and
$60,200$ users for whom we have no interaction information. Thus, in total
$80,773$ users’ first interaction with the site was an order (i.e., $17.76\%$
of the total buyer users).

It should be noted that although there are $457,298$ users in the data file JD
user data.csv, the order data indicates that only $454,897$ users ordered at
least one item during March $2018$ and $2401$ users in the users data file did
not place any order during this month.

Number of orders. Table 1 displays the number of orders made by buyer users in
terms of both the actual number and percentage. More specifically, each column
shows the number of buyer users and the corresponding percentage for the given
number of orders varying from $1$ to $9$ units. The last column labeled by
$10+$ includes the orders of $10$ units or more. It is readily seen that the
majority of buyer users, that is $430,743$ users comprising $94.69\%$ of the
total buyer users, ordered only $1$ unit. Note that the largest order size
made by a user was $521$ units.

Number of orders | $1$ | $2$ | $3$ | $4$ | $5$ | $6$ | $7$ | $8$ | $9$ | $10+$
---|---|---|---|---|---|---|---|---|---|---
Number of buyers | $430,743$ | $21,688$ | $1,832$ | $284$ | $99$ | $42$ | $37$ | $22$ | $19$ | $131$
Percentage $\%$ | $94.69$ | $4.77$ | $0.40$ | $0.06$ | $0.02$ | $0.01$ | $0.01$ | $<0.01$ | $<0.01$ | $0.03$
Table 1: Number of orders made by buyer users

Total spent money. Figures 1 and 2 display the histogram and cumulative
distribution function of the total amount of money (in RMB) spent by buyer
user in March $2018$. Figure 1 shows that the distribution of the total amount
spent by users is skewed to the right with a mode around RMB $60$ (after
smoothing out the plot). Furthermore, Figure 2 illustrates that while the
median (i.e., $50\%$ percentile) of the total amount spent by buyer user is
RMB $80$, around $90\%$ of buyer users spent at most RMB $210$ in this product
category in March $2018$.

Figure 1: Histogram of amount of money (in RMB) spent by buyer users in March
$2018$ Figure 2: Cumulative distribution function of amount of money (in RMB)
spent by buyer users in March $2018$

User interaction time. Figures 3 and 4 display the probability and cumulative
distribution function of user interaction time with the site for both buyer
and non-buyer users (via whichever channel), respectively. More specifically,
they show the time elapsed between a user’s first and last record over the
whole month. A typical user interaction involves a few clicks on different
SKUs which may result in ordering one of them. In both figures, the bin size
is $10$ minutes and the x-axis is log scale. The x-axis labels are hours on
the left varying from $1$ to $24$ hours, and then become days covering all
$30$ days. Such scaling illustrates the detail of early times while still
spreading out the rest of the month enough. Figure 3 exhibits a $24$-hour
frequency component to the buyer users’ visits, that is people were often
coming back to the site at about the same time each day. Moreover, Figure 4
reveals that $40\%$, $54\%$ and $78\%$ of buyer users had less than $1$
hour’s, $1$ days’, and $1$ week’s interaction with the site, respectively.
These percentages for non-buyer users are $70\%$, $77\%$ and $88\%$,
respectively. This implies that in average, buyer users spend more time on the
site comparing with non-buyer users.

Figure 3: Probability distribution function of the total duration of
interaction Figure 4: Cumulative distribution function of the total duration
of interaction

Although the click data are bearing valuable information for making
predictions, considering them as independent samples may not be an accurate
approach. Because there are some obvious significant correlations between
multiple clicks of a buyer user led to an order. To address this issue, for
each buyer user, we eliminate any click activity that they had before placing
their order and keep only the last click record on the SKU in which they
ordered. However, $80,773$ buyer users’ whose first interaction with the site
was an order did not have any pre-click history. For these users, we keep
their order records as their successful clicks. Although the click records
include the click channel data, the order records do not provide this
information. Therefore, for these buyer users, we introduce a new channel,
namely unknown.

###  3.2 Empirical Statistical Analysis on Users’ Spending Behavior

This section studies the impact of six categorical features consisting of
“Age”, “Gender”, “Education”, “Marital status”, “Plus status”, and “User
level” on three numerical variables including total spending money by buyer
users, total discount that buyer users received, and total number of units
that buyer users ordered. For this purpose, we construct the empirical
probability distribution for each level of the given categorical variables and
display them in two-dimensional histograms, with each row is normalized, so
the sum across rows is $100\%$. In all Figures 5–10, the lighter/darker colors
imply the higher/lower percentage. In addition, total spending and total
discount bins are $0$ to $500$ in steps of $10$, and units per order bins are
$1$ to $10$. The last bin labeled $10$, includes all orders of 10 units or
more. Furthermore, the level “Unknown” refers to those buyer users whose
values for the corresponding features are not provided in the dataset.

Age. The feature age is categorized into seven levels consisting of “less than
$16$”, “$16-25$”, “$26-35$”, “$36-45$”, “$46-55$”, “greater than $55$”, and
“Unknown”. Figure 5 indicates that for all age groups varying between $16-55$
as well as the unknown group, their total spending distributions are similar,
achieving their spike at RMB $60$. However, the youngest age group (i.e., less
than $16$) total spending distribution has an exotic pattern with a mode at
RMB $60$ and several peaks, and the oldest age group (i.e., greater than $55$)
total spending distribution achieves its spike at RMB $90$. A similar behavior
appears for the total discount distribution. While the age groups between
$16-55$ as well as the unknown group attain their maximum at RMB zero, the
youngest group reaches its maximum at RMB $20$ and the oldest age group has
two nice peaks one at RMB zero and the other one at RMB $340$. The behavior of
buyer users in all age groups for the order volume seem to be similar
achieving their peaks at one unit.

Figure 5: The distribution of total spending money, total discount, and number
of orders vs. age groups

Gender. The feature gender is categorized into three levels consisting of
“Female”, “Male”, and “Unknown”. Figure 6 shows that the distribution of total
spending , total discount, and order volume for all three gender groups are
skewed to the right with peaks at RMB $60$, RMB zero, and one unit,
respectively. Furthermore, the total spending distribution of the male and
female groups achieves a second spike at RMB $210$. It appears that the
unknown group spends more than the other two gender groups in average.

Figure 6: The distribution of total spending money, total discount, and number
of orders vs. gender

Education. The feature education is categorized into five levels consisting of
“$1$: less than high school”, “$2$: high school diploma or equivalent”, “$3$:
Bachelor’s degree”, “$4$: post-graduate degree”, and “$-1$: unknown”. Figure 7
shows that the total spending distribution for all education groups is skewed
to the right and achieves its spike at RMB $60$, but the highest two education
groups $3$ and $4$ posses a couple of further peaks at RMB $100$ and RMB
$210$. Furthermore, while the education groups $-1$, $1$ and $2$ have a
heavier low spending tail, the last two education groups $3$ and $4$ have a
longer high spending tail. It appears that the two education groups $1$ and
$2$ spend more than the other groups, in average. Similar to the previous
features, most discounts are small (peaked at zero for all education groups)
and pretty much everyone orders only one unit at a time.

Figure 7: The distribution of total spending money, total discount, and number
of orders vs. education levels

Marital status. The feature marital status is categorized into three levels
consisting of “Single”, “Married”, and “Unknown”. Figure 8 shows that the
total spending distribution for all marital status groups is skewed to the
right and achieves its spike at RMB $60$, but the two single and married
groups posses one further peak at RMB $210$ and have a longer high spending
tail. In average, the unknown group spends more than the other two marital
status groups. Similar to the previous features, most discounts are small
(peaked at zero for all marital status groups) and pretty much everyone orders
only one unit at a time.

Figure 8: The distribution of total spending money, total discount, and number
of orders vs. marital status

Plus status. The feature plus status is categorized into two levels consisting
of “Plus” and “Non-plus”. Figure 9 shows that the total spending distribution
for both plus status groups is skewed to the right and achieves its spike at
RMB $60$. While the plus group has a smaller variance and lighter tails, the
non-plus group has a flatter distribution with heavier tails. In addition, the
plus group spends more than the non-plus group, in average. Unlike the
previous features, while the total discount distribution for the plus group
achieves its maximum at RMB zero and a second peak at RMB $20$, the
corresponding distribution for the non-plus group achieves its maximum at RMB
$20$ with a second peak at RMB $70$. This implies that discount is more
enticing for non-plus users than the plus users. Again, in consistent with the
previous results, pretty much everyone orders only one unit at a time.

Figure 9: The distribution of total spending money, total discount, and number
of orders vs. plus status levels

User level. The feature user level is categorized into seven levels consisting
of “$-1$”, “$0$”, “$1$”, “$2$”, “$3$”, “$4$”, and “$10$”, where $-1$
corresponds to new buyer users, $0-4$ correspond to existing users where a
higher user level is associated with a higher total purchase value in the
past, and $10$ corresponds to enterprise users. Figure 10 shows that the total
spending distribution for all user levels $-1$ and $1-4$ is skewed to the
right, has light tails, with the maximum achieved at RMB $60$, the zero user
level has a very low variance with the maximum achieved at RMB zero, and the
user level $10$ is flat with a big spike at RMB $500$. Furthermore, while the
user level $-1$ has a heavier low spending tail, the user levels $1-4$ have a
longer high spending tail. In addition, buyer users in level $0$ spends more
than the other levels, in average. Such a similar pattern is observed for the
total discount. The total discount distribution for all user levels $-1$ and
$1-4$ has two peaks at RMB zero and RMB $60$, the zero user level has a few
peaks at RMB zero, RMB $80$, RMB $130$, and RMB $190$, and the enterprise
users (i.e., user level $10$) has a maximum at $0$ level. This observation
implies that while the behavior of new users and more established individual
users are similar, those users who have recently purchased through the site
for the first time (i.e., user level $0$) behave more sensitively to the
discounts offered by the site. As expected, the enterprise users are
insensitive to discounts. Regarding the order volume distribution, while most
individual users order only one unit at a time, the corresponding distribution
for the enterprise users attains two spikes at $1$ unit and $10+$ units.

Figure 10: The distribution of total spending money, total discount, and
number of orders vs. user levels

Figure 11 shows the month of the first order placed by buyer users in March
$2018$. It reveals that while the oldest month of the first order placed by a
user, was December $2003$, there has been an increasing trend in the number of
new users such that $28,497$ users (i.e., $6.26\%$ of the total $454,897$
buyer users in this month) made their first order in March $2018$. These are
users with level $-1$.

Figure 11: The first order placed by buyer users in March $2018$

######  Observation 1.

In summary, while the spending behavior of buyer users with respect to the
features age, gender, education, and marital status appear similar, their
behavior with respect to the two features plus status and user levels are
totally different. Such observation is made for the total discount
distribution as well. However, irrespective of the chosen feature, the
majority of buyer users order only one unit at a time which is consistent with
the results reported in Table 1.

##  4 Sales Volume Analysis

In this and the next sections, we examine JD customers’ behavior and identify
the most important factors that influence their purchase behavior. In this
section, we build a model to predict customers’ purchase volume (sales). In
Section 5, we build a classification model to explore customers’ product
choice decision. We now describe the steps we took to build our sales
prediction model.

###  4.1 Variable Selection

The first step of our analysis was choosing the explanatory variables for
predicting sales. After reviewing the data , we decided to use the following
explanatory variables.

####  4.1.1 Customer Variables:

We use Gender, Marital Status, Education, User Level, and User Plus variables,
which were defined in Section 3.1. We also use the following variables:

  1. 1.

City Level: JD has classified customer cities based on their most commonly
used shipping address. City level takes on a value from $\\{$-1, 1, 2, 3, 4,
5$\\}$, where level 1 corresponds to highly industrialized cities such as
Beijing and Shanghai, level 2 cities correspond to provincial capitals, and
level 3 to 5 cities are smaller cities. If there are no data, then city level
is set to -1.

  2. 2.

Purchase Power: customer’s estimated purchase power indexed $\\{$-1, 1, 2, 3,
4, 5$\\}$, where -1 means no estimation, 1 represents the highest, and 5
represents the lowest purchase power.

####  4.1.2 Product Variables:

We use Attribute 1 and Attribute 2, which are numerical integer variables
between 1 and 4, and between 30 and 100, respectively. Both attributes
characterize the functionality of a product, and a higher attribute value
indicates better product performance.

####  4.1.3 Seller Variables:

The following variables capture information about the seller.

  1. 1.

Product Type: If the product is sold by JD, product type is 1. If the product
is sold by a third-party seller on JD, product type is 2.

  2. 2.

Original Price: The original list price of the SKU before any discounts are
applied.

  3. 3.

Promise: shows the expected delivery time of the order in days. In the data,
Promise could take a value from $\\{$N/A,1, 2, 3, 4, 5, 6, 7, 8$\\}$, where
N/A is Not Available.

####  4.1.4 Promotion Variables:

These variables capture sellers’ promotion strategies.

  1. 1.

Number of Gifts: the total number of free gift items that a customer receives
in an order. This variable did not exist in the data set and we defined it
using the orders table. For each order ID, we identified the items that
belonged to the order. If all items were free gifts, the order was eliminated
from the analysis. Otherwise, we calculated the total number of free gifts in
the order and assigned this number to Number of Gifts associated with the item
that the customer paid for (all orders that included some free gifts only had
one paid item). We then erased the free gifts from the order. For example, if
one order included SKU #1 (paid), 2 units of SKU #2 (free gift), and 2 units
of SKU #3 (free gifts), we set Number of Gifts to 4 for SKU #1, and deleted
SKUs #2 and #3. The purpose of this variable is to capture the added value
that a paid item generates for a customer when it comes with free gifts.

  2. 2.

Percentages of Direct Discount, Quantity Discount, Bundle Discount, and Coupon
Discount per unit: These variables show the value of each discount as a
percentage of the original price. In Python, the discount variables were coded
as ratios. One can simply multiply the ratios by 100 to get percentages.

Initially, we wanted to use the total value (price) of the gift items as a
predictive variable. However, the original and final prices of gift items were
both recorded as zero in the data set. As a result, it was not possible to
perform this analysis. Before providing the data, JD had applied an allocation
rule to calculate the contribution of each unit in an order to quantity and
bundle discounts, so we did not need to adjust the discount values for the
order quantities.

####  4.1.5 Other Variables:

These variables convey other aspects of orders. We use the variable Channel
from Section 3.1. We also use categorical variables Day of Week and Hour of
Day.

We used LightGBM Regression to predict sales. Tabel 2 reports the accuracy of
LightGBM against alternative methods LASSO, Ridge, and KNN regression. LASSO
regression uses the $\ell_{1}$ norm of the regression coefficients to minimize
overfitting. Ridge regression uses the squared $\ell_{2}$ norm for
regularization. KNN regression uses the weighted average of the $k$ nearest
points in the training set to predict the response variable for a given
observation. Accuracy is measured by RMSE (Root Mean Squared Error). We see
that LightGBM regression performs significantly better.

Method | LightGBM Regression | LASSO Regression | Ridge Regression | KNN Regression
---|---|---|---|---
RMSE | 0.9367 | 1.1512 | 1.1337 | 1.0130
Table 2: Benchmarking LightGBM Regression against other regression methods

We now apply the SHAP method to interpret the results of the regression model
and provide managerial insights to JD. To understand how SHAP values are
calculated for a model, it is worth reviewing the concept of Shapley values.
Consider a cooperative game with $N$ players and pay-off function $v$ that
maps subsets of players to real numbers. The Shapley value for player $i$
determines the contribution that the player brings to the game and is
calculated as follows

$\displaystyle\varphi_{i}(v)=\sum_{S\subseteq
N\setminus\\{i\\}}\frac{|S|!(N-|S|-1)!}{N!}\bigl{(}v(S\cup\\{i\\})-v(S)\bigr{)}$
(1)

where $S$ is a subset of players of size $|S|$ that excludes player $i$.
Essentially, the formula calculates the average additional value that player
$i$ brings to coalition $S$ over all possible permutations.

Think of our ML model as a game, the predictions as the pay-off function, and
the variables as players. Then Shapley values can be used to quantify the
average contribution of each variable to the model. Note that the marginal
contribution of variables to the model’s prediction for an observation depends
on the order in which the variables enter the model. In other words, the
contribution of variable $i$ to the model depends on the set of variables $S$
that already exist in the model. Thus, the overall contribution of each
variable to the model (global interpretability) is measured by the average
contribution (Shapley value) of the variable to each prediction.

###  4.2 Global Interpretation

As the first step of SHAP analysis, we present the SHAP Variable Importance
Plot in Figure 12, which illustrates the impact of each variable on the
model’s predictions.222We would like to clarify that the word ”impact” does
not imply causality. Variable importance plots are useful in identifying and
prioritizing the features that play a more important role in determining model
outputs. The top five impactful variables for sales are: (1) expected delivery
time in days (Promise); (2) per-unit direct discount as a percentage of the
original price; (3) per-unit quantity discount as a percentage of the original
price; (4) original price; (5) per-unit coupon discount as a percentage of the
original price.

Figure 12: Variable importance plot for sales prediction

Naturally, the importance plots raise a follow-up question: How are the most
impactful variables associated with the model’s target variable? SHAP Value
Plots can answer these questions. The value plots are made of the training
data and demonstrate the following information: (I) Impact: The horizontal
location shows whether the effect of that value is associated with a higher or
lower prediction; (II) Original value: Color shows whether that variable is
high (in red) or low (in blue) for that observation; (III) Association: Color
and horizontal location of data points show whether a variable is positively
or negatively correlated with the target variable.

Figure 13: SHAP value plot for sales prediction

To understand how these top variables impact sales in detail, we can examine
the SHAP dependency plots. First, we look at the dependency plot for Promise
in Figure 14. It illustrates how the SHAP value for Promise varies with
Promise, with each dot representing an observation in the data. The vertical
bar on the right hand side displays the variable in the model that Promise
interacts with most frequently using Friedman’s H-statistic [35, 21].

Let us ignore the interaction aspect for a moment (as if all the dots had the
same color). The plot shows that when Promise increases from one day, overall,
there is an upward trend in customers’ purchase volume. One plausible theory
for this observation is customers’ hoarding behavior. When delivery takes
longer, customers tend to get more anxious as they need to spend more time and
effort to monitor their home inventories more frequently and plan for the
subsequent orders to avoid stockouts. To avoid this inconvenience, customers
decide to increase their order size and stockpile larger inventories. The
interaction visualization in Figure 14 further supports our theory. User level
has the strongest interaction with Promise, and high SHAP values for Promise
belong to high user levels. In other words, as the promised delivery takes
longer, customers that historically make large purchases are the first to
batch their orders.

Such a batch-ordering behavior has been repeatedly documented in the supply
chain literature through the famous Beer Game [42]. When lead times increase,
supply chain members use order batching to maintain their stocks. We
conjecture that JD customers exhibit similar behavior. Although customers’
stockpiling behavior increases sellers’ revenue in the short-term, we know
from the literature that it leads to the bullwhip effect that bears many
negative consequences such as poor customer service, poor capacity
utilization, and unstable production schedules. Thus, our first recommendation
to the sellers on JD.com is as follows.

######  Recommendation 1.

Given that the promised delivery is the most impactful feature for purchase
volumes, JD sellers should achieve and maintain quick delivery to alleviate
customers’ batch ordering and prevent the bullwhip effect.

Next, we look at the SHAP dependency plot for the per-unit direct discount as
a percentage of the original price in Figure 15(a). We observe that direct
discount has a positive impact on sales in general, even when the amount of
discount is a small percentage of the listed price.

######  Recommendation 2.

Sellers should utilize the effectiveness of direct discounts in boosting
sales. A little direct discount goes a long way.

Now we examine Figure 15(b) for the effect of quantity discounts. The plot
shows that quantity discounts are effective, even when they are a small
fraction of the listed price. Also, we generally observe that higher values of
Attribute 2 increase the positive impact of quantity discounts.

######  Recommendation 3.

As the second most impactful promotion strategy, sellers should take advantage
of quantity discounts. In particular, sellers should offer quantity discounts
on SKUs whose Attribute 2 values are higher.

Next, the value plot shows that, as expected, higher prices reduce sales.
Figure 15(c) allows us to see this trend in detail. The most salient
observation in this plot is that the SHAP value for price turns negative near
$200$ RMB.

######  Recommendation 4.

List prices that exceed 200 RMB hinder sales, unless the seller offers
discounts to reduce the net cost. In doing so, sellers should use the discount
methods that are deemed effective in the SHAP analysis.

Finally, Figure 15(d) shows the association between sales and coupon discount
percentages. The dominant trend in the SHAP values and the interaction effect
lead us to the next recommendation.

######  Recommendation 5.

In addition to direct and quantity discounts, sellers should use coupon
discounts to increase sales. In particular, sellers should extend coupons to
customers who have made larger total purchases in the past.

Figure 14: SHAP dependency plot for Promise.

a) b)

c) d)

Figure 15: SHAP dependency plots for direct, quantity, and coupon discounts,
and for original price.

###  4.3 Local Interpretation

In this section, we describe SHAP’s local interpretation capabilities. This
feature of SHAP helps model users understand how the prediction model arrives
at a particular prediction and illustrates the contribution of each variable.

SHAP Decision Plots are created to effectively explain a model’s prediction
for any observation. Figure 16 shows an example of a decision plot for our
sales model. The straight vertical line marks the model’s base value. The
colored line is the prediction. Feature values are printed next to the
prediction line. The product has attributes are 3 and 80, is sold by JD at 191
RMB for next-day delivery, and does not come with any gifts. The unit
discounts are 2.1% (quantity discount) and 0.5% (direct discount) off the list
price. The customer is married and female, age 26-35, holds a Bachelor’s
degree, lives in a highly-industrialized city, has a relatively high purchase
power, is of user level 3, and is not a PLUS member. She is visiting JD
through her PC at 9 p.m. on a Thursday. Starting at the bottom, the prediction
line shows how the SHAP values (i.e., the feature effects) accumulate from the
”base value” (0.4767) to reach the model’s final prediction (4.66) at the top
of the plot. The base value marks the model’s average prediction over the
training set.

Figure 16:  Decision plot for a prediction. The base value is 0.4767. The
sales prediction is 4.66

##  5 Product Choice Analysis

The purpose of this section is to build a classification model that would
predict the probability that a customer would purchase a particular SKU. We
use the same explanatory variables as in Section 4, except Attribute 1 and
Attribute 2, which were used for clustering.

###  5.1 Clustering SKUs

We filtered the 31,868 SKUs in the data set based on whether or not their
Attribute 1 and Attribute 2 values were recorded. Only 13,725 SKUs had both
attributes recorded. We dropped the 18,143 SKUs whose Attribute 1 and/or
Attribute 2 values were missing. We did not use imputation for two reasons.
First, the number of SKUs with missing values is much larger than the number
of SKUs with recorded values. Thus, any imputation method would entail
significant errors and adversely impact the reliability of our analysis.
Second, the number of SKUs with available data is sufficiently large to
conduct a meaningful study on customers’ product choice behavior without
requiring to estimate the missing data.

We clustered the 13,725 SKUs into classes based on the two attributes, using
the $K$-means clustering algorithm [31]. The Elbow Method indicated that using
four clusters was appropriate. Table 3 shows the number of SKUs assigned to
each cluster.

Cluster | 0 | 1 | 2 | 3
---|---|---|---|---
SKUs | 3754 | 3365 | 4685 | 1921
Table 3: Number of SKUs assigned to each cluster

The LightGBM algorithm can be used for both classification and regression.
Table 4 reports the precision of LightGBM against alternative classification
methods. The idea of Naïve Bayes is to calculate $P(\mathbf{X}|C_{k})$ for
each observation, the conditional probability of a given set of feature values
$\mathbf{X}$, given the response value belongs to class $k$. Then each
observation is assigned to the class with the highest $P(\mathbf{X}|C_{k})$.
Once the model is trained, for a new observation, these conditional
probabilities are calculated for its $\mathbf{X}$ to classify the observation.
However, calculating the conditional probabilities using the Bayes’ Rule
requires calculating joint probabilities that are difficult to estimate in
practice. Naïve Bayes assumes that the elements of $\mathbf{X}$ are mutually
independent to make the calculation of $P(\mathbf{X}|C_{k})$ easier. In the
K-Nearest Neighbors (KNN) Classifier, an observation is classified by a
majority vote of its neighbors, with the observation being assigned to the
class most common amongst its $K$ nearest neighbors [13, 3].

Method | LightGBM Classifier | Naïve Bayes Classifier | KNN Classifier
---|---|---|---
Precision | 0.8653 | 0.3967 | 0.6920
Table 4: Performance of LightGBM classifier against other classification
methods

###  5.2 Global Interpretation

We now examine the SHAP importance, value, and dependency plots in Figures 17
to 23.333Due to space limitations, we moved the plots to the Online Companion.
The importance plots show that, for all four classes, financial factors such
as price and discount play a much more critical role in influencing customers’
choice behavior.

In interpreting the SHAP plots, one should differentiate between two different
types of variables. The first group of variables includes those that
characterize an incoming visit to the platform, but do not depend on the
purchased product. We refer to these variables as customer-specific variables
(such as education, channel, hour of the day, etc.). In contrast, other
variables – referred to as platform-specific variables – vary by product and
are set by the platform for each SKU before the customer arrives (such as
price, product type, promise, and various discounts).

For the first variable group, any change in the variable can be directly
linked to the likelihood of a purchase from one of the classes. For instance,
one can infer whether a higher level of education (while keeping everything
else constant) leads to a higher or lower chance of purchasing an SKU from
class 2. However, the same conclusion does not hold about the second variable
group because these variables correspond to many different SKUs that belong to
a particular class. Therefore, any changes in the value of these variables and
their corresponding SHAP values should be used to explain how customers are
choosing different SKUs within that class. For instance, we may observe that
among all SKUs that belong to a particular class, those with a higher price
also have a higher probability of purchase (e.g., because customers prefer
them for reasons other than price).

To draw insights about how customers’ choice behavior, we distinguish between
these two variable groups and focus on each group’s most influential variables
separately for each of the four classes. Based on the SHAP plots, we present
the following observations:

######  Observation 2.

Customer-specific variables seem to have low predictive power for all four
classes. Therefore, the incoming customer’s demographic information, the day
and time of her visit, and the channel that she is using does not reveal a
significant amount of information about the customers’ preference for one
class over the other. Put differently, the probability of purchase from any of
the four classes remain largely unaffected by customer-specific variables. The
only exceptions to this statement is Education Level for Class 3 products and
Day of the Week for Class 1 products. More precisely, keeping everything else
constant, an incoming customer to JD.com is more likely to prefer an SKU from
Class 3 over other classes if the customer has a lower education level.
Similarly, a new visitor is more likely to prefer an SKU from Class 1 if her
visit happens during earlier days of the week (keeping everything else
constant).

This observation can be incorporated into JD’s demand forecast system and
improve inventory management decisions, both temporally and spatially.
Demographic information can also be used toward targeted marketing, and
identifying customers that are more likely to be persuaded by personalized
ads. The platform can utilize such information in its targeting ads and
customize its product portfolio (i.e., the items shown to the customer in the
first page upon their visit to the platform) according to customers’ specific
characteristics.

######  Observation 3.

Among SKUs in Class 0 and Class 3, those with lower prices seem to be more
popular than more expensive ones. This conclusion, however, does not hold
about SKUs in Class 1 and Class 2. Customers’ preference in choosing between
Class 1 and Class 2 products is mostly driven by factors other than price.

######  Observation 4.

Direct Discount seems to be effective for Class 0 items only if it is
sufficiently high (i.e., higher than 40%). However, the opposite is true for
Class 3, so that items receiving a moderate level of Direct Discount are more
popular with the customers who have purchased from this class. More
interestingly, the low customer price sensitivity for Class 1 and Class 2
items is also reflected in their reaction to Direct Discount for these
classes. Specifically, Direct Discount is largely ineffective in persuading
customers to buy discounted items in Class 1 and Class 2. In fact, SKUs that
have received large Direct Discount in Class 2 remain highly unpopular.

######  Observation 5.

Quantity Discount is an effective tool to influence purchase for Class 0
products and Class 3 products (if the discount is large enough). However,
similar to Direct Discount, Quality Discount is not the main factor
influencing customers’ preference for items that belong to Class 1 and Class
2.

Based on this observation, JD should use Direct Discount and/or Quantity
Discount to increase the demand for items in Class 0 and Class 3. These
discounting methods, however, are mostly ineffective for Class 1 and Class 2
items (For SKUs in Class 2, Bundle discount is the only effective discounting
strategy, and hence should be preferred over Direct discounting and Quantity
discounting). For Class 1 and Class 2 items, customers’ preferences seem to be
less driven by price and discount considerations. Therefore, targeted
marketing is a more effective tool for these items compared to various forms
of discount.

######  Observation 6.

Customers who have purchased from Class 1, Class 2, and Class 3 are largely
insensitive about Product Type. In other words, whether JD or a third party
has fulfilled an order has not been a major driver for customers’ product
choice. For Class 0, on the other hand, customers seem to have a strong
preference for SKUs sold by third-party sellers.

######  Observation 7.

Customers’ product choice is largely unaffected by the promised delivery time.
That is, customers are unlikely to switch their favorite product choice due to
a longer delivery time.

##  6 Discussion and Conclusion

We analyzed 454,897 JD buyer users who ordered at least one item from a
particular product category in March 2018. Our descriptive statistical
analysis reveals that $50\%$ of buyer users spent less than $80$ RMB and
around $90\%$ spent less than $210$ RMB. Most buyer users (i.e., $99.46\%$)
ordered either one or two units. While $40\%$, $54\%$ and $78\%$ of buyer
users had less than $1$ hour, $1$ day, and $1$ week interaction with the site,
these percentages for non-buyer users grow significantly to $70\%$, $77\%$ and
$88\%$. Thus, on average, buyer users spend more time than non-buyer users.

We also performed empirical statistical analyses of users’ spending behavior
More precisely, we constructed the empirical probability distribution of total
spending, total discount, and number of units per order for different levels
of six features consisting of age, gender, education, marital status, PLUS
status, and user level. Our analysis illustrates that while the distribution
of total spending and total discount for buyer users with respect to the
features age, gender, education, and marital status appear similar, they could
be different for the PLUS status and user level features. More precisely,
while user behavior in different levels of the first four features did not
vary dramatically, there were considerable changes in behavior at different
levels of PLUS status and user level. However, for all given six features,
most buyer users ordered only one unit at a time, which is consistent with the
descriptive statistical output.

We also built models to predict sales and customers’ product choice. We
applied a novel framework to interpret the predictions. Our findings lead to
an important insight into the effect of delivery timing on customers’ purchase
decisions. Specifically, even though customers’ product choice is highly
insensitive to the promised delivery time, this factor significantly impacts
customers’ order quantity. In other words, longer delivery times do not induce
customers to switch their preferred product but persuade them to order more,
which can potentially entail an undesirable bullwhip effect for the platform.
Furthermore, the effectiveness of various discounting methods depends on the
specific product and the magnitude of the discount. Therefore, JD should
exercise carefully utilize the right method of discounting as well as its
amount. For instance, we observe that direct discount and quantity discounts
are highly ineffective for persuading customers to purchase from certain
classes of products, while their effectiveness for other classes depends on
the magnitude of the discount. When it comes to the purchase quantity (i.e.,
order size), even a small level of discount can have a substantial effect.
Finally, 200 RMB seems to be an anchor point for customers who are buying this
product category, and the shift in their behavior in the vicinity of this
price point is the largest.

## Declarations

  * •

Funding: Not applicable

  * •

Competing interests: Not applicable

  * •

Ethics approval: Not applicable

  * •

Consent to participate: Not applicable

  * •

Consent for publication: Not applicable

  * •

Availability of data and materials: Not applicable

  * •

Code availability: Not applicable

  * •

Authors’ contributions: Not applicable

## References

  * [1] M. Abolghasemi, J. Hurley, A. Eshragh, and B. Fahimnia.  Demand forecasting in the presence of systematic events: Cases in capturing sales promotions.  International Journal of Production Economics, 230:107892, 2020\.
  * [2] A. Alexandrov, K. Benidis, M. Bohlke-Schneider, V. Flunkert, J. Gasthaus, T. Januschowski, D.C. Maddix, S.S. Rangapuram, D. Salinas, J. Schulz, L. Stella, A.C. Türkmen, and Y. Wang.  Probabilistic time series models in python.  arXiv preprint arXiv:1906.05264, 2019.
  * [3] N.S. Altman.  An introduction to kernel and nearest-neighbor nonparametric regression.  The American Statistician, 46(3):175–185, 1992.
  * [4] J. Alvarado-Valencia, L.H. Barrero, D. Önkal, and J.T. Dennerlein.  Expertise, credibility of system forecasts and integration methods in judgmental demand forecasting.  International Journal of Forecasting, 33(1):298–313, 2017.
  * [5] A. Aouad, A.N. Elmachtoub, K.J. Ferreira, and R. McNellis.  Market segmentation trees.  arXiv preprint arXiv:1906.01174, 2019.
  * [6] P. Baecke, S. De Baets, and K. Vanderheyden.  Investigating the added value of integrating human judgement into statistical demand forecasting systems.  International Journal of Production Economics, 191:85–96, 2017\.
  * [7] K. Bandara, P. Shi, C. Bergmeir, H. Hewamalage, Q. Tran, and B. Seaman.  Sales demand forecast in e-commerce using a long short-term memory neural network methodology.  In Neural information processing, pages 462–474, 2019.
  * [8] K. Benidis, S.S. Rangapuram, V. Flunkert, B. Wang, D. Maddix, C. Turkmen, J. Gasthaus, M. Bohlke-Schneider, D. Salinas, L. Stella, L. Callot, and T. Januschowski.  Neural forecasting: Introduction and literature overview.  arXiv preprint arXiv:2004.10240, 2020.
  * [9] F. Bravo and Y. Shaposhnik.  Mining optimal policies: A pattern recognition approach to model analysis.  INFORMS Journal on Optimization, 2(3):145–166, 2020.
  * [10] N. Chakravarthy, A. Spanias, L.D. Iasemidis, and K. Tsakalis.  Autoregressive modeling and feature analysis of dna sequences.  EURASIP Journal on Advances in Signal Processing, 2004:13–28, 2004\.
  * [11] H. Charkhgard and A. Eshragh.  A new approach to select the best subset of predictors in linear regression modelling: Bi-objective mixed integer linear programming.  The ANZIAM Journal, 61(1):64–75, 2019.
  * [12] D.F. Ciocan and V.V. Mišić.  Interpretable optimal stopping.  Management Science, Articles in Advance, 2020.
  * [13] T. Cover and P. Hart.  Nearest neighbor pattern classification.  IEEE transactions on information theory, 13(1):21–27, 1967.
  * [14] F. Doshi-Velez and B. Kim.  Towards a rigorous science of interpretable machine learning.  arXiv preprint arXiv:1702.08608, 2017.
  * [15] A. Eshragh, S. Alizamir, P. Howley, and E. Stojanovski.  Modeling the dynamics of the COVID-19 population in Australia: A probabilistic analysis.  PLoS ONE, 15(10):e0240153, 2020.
  * [16] A. Eshragh, B. Ganim, T. Perkins, and K. Bandara.  The importance of environmental factors in forecasting australian power demand.  Environmental Modeling & Assessment, 27:1–11, 2022.
  * [17] A. Eshragh, G. Livingston, T.M.C. McCann, and L. Yerbury.  Rollage: Efficient rolling average algorithm to estimate ARMA models for big time series data.  arXiv preprint arXiv:2103.09175, 2021.
  * [18] A. Eshragh, F. Roosta, A. Nazari, and M.W. Mahoney.  LSAR: Efficient leverage score sampling algorithm for the analysis of big time series data.  Journal of Machine Learning Research, 23:1–36, 2022.
  * [19] R. Fildes, P. Goodwin, M. Lawrence, and K. Nikolopoulos.  Effective forecasting and judgmental adjustments: an empirical evaluation and strategies for improvement in supply-chain planning.  International Journal of Forecasting, 25(1):3–23, 2009.
  * [20] R. Fildes, P. Goodwin, and D. Önkal.  Use and misuse of information in supply chain forecasting of promotion effects.  International Journal of Forecasting, 35(1):144–156, 2019.
  * [21] J.H. Friedman, B.E. Popescu, et al.  Predictive learning via rule ensembles.  The Annals of Applied Statistics, 2(3):916–954, 2008.
  * [22] B. Goodman and S. Flaxman.  European union regulations on algorithmic decision-making and a “right to explanation”.  AI magazine, 38(3):50–57, 2017.
  * [23] P. Goodwin.  Integrating management judgment and statistical methods to improve short-term forecasts.  Omega, 30(2):127–135, 2002.
  * [24] J.D. Hamilton.  A new approach to the economic analysis of nonstationary time series and the business cycle.  Econometrica, 57(2):357–384, 1989.
  * [25] T. Hastie, R. Tibshirani, and J. Friedman.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction.  Springer, 2009.
  * [26] H. Hewamalage, C. Bergmeir, and K. Bandara.  Recurrent neural networks for time series forecasting: Current status and future directions.  International Journal of Forecasting, To Appear, 2020.
  * [27] R.J. Hyndman, A.B. Koehler, J.K. Ord, and R.D. Snyder.  Forecasting with exponential smoothing: the state space approach.  Springer Science & Business Media, 2008.
  * [28] M. Kremer, E. Siemsen, and D.J. Thomas.  The sum and its parts: Judgmental hierarchical forecasting.  Management Science, 62(9):2745–2764, 2015.
  * [29] M. Lawrence, P. Goodwin, M. O’Connor, and D. Önkal.  Judgmental forecasting: A review of progress over the last 25 years.  International Journal of Forecasting, 22(3):493–518, 2006.
  * [30] M. Lawrence, M. O’connor, and B. Edmundson.  A field study of sales forecasting accuracy and processes.  European Journal of Operational Research, 122(1):151–160, 2000\.
  * [31] S. Lloyd.  Least squares quantization in pcm.  IEEE transactions on information theory, 28(2):129–137, 1982.
  * [32] S.M. Lundberg and S. Lee.  A unified approach to interpreting model predictions.  Advances in neural information processing systems, pages 4765–4774, 2017.
  * [33] J.W. Messner and P. Pinson.  Online adaptive lasso estimation in vector autoregressive models for high dimensional wind power forecasting.  International Journal of Forecasting, 35(4):1485–1498, 2019.
  * [34] V.V. Mišić and G. Perakis.  Data analytics in operations management: A review.  Manufacturing & Service Operations Management, 22(1):158–169, 2020\.
  * [35] C. Molnar.  Interpretable Machine Learning.  Lulu. com, 2020.
  * [36] B. Moritz, E. Siemsen, and M. Kremer.  Judgmental forecasting: Cognitive reflection and decision speed.  Production and Operations Management, 23(7):1146–1160, 2014.
  * [37] S. Mukherjee, D. Shankar, A. Ghosh, N. Tathawadekar, P. Kompalli, S. Sarawagi, and C. Krishnendu.  ARMDN: Associative and recurrent mixture density net- works for retail demand forecasting.  arXiv preprint arXiv:1803.03800, 2018.
  * [38] D. Salinas, V. Flunkert, J. Gasthaus, and T. Januschowski.  DeepAR: Probabilistic forecasting with autoregressive recurrent networks.  International Journal of Forecasting, 36(3):1181–1191, 2020.
  * [39] N.R. Sanders and K.B. Manrodt.  The efficacy of using judgmental versus quantitative forecasting methods in practice.  Omega, 31(6):511–522, 2003.
  * [40] L.S. Shapley.  A value for n-person games.  Contributions to the Theory of Games, 2(28):307–317, 1953.
  * [41] X. Shen and Q. Lu.  Joint analysis of genetic and epigenetic data using a conditional autoregressive model.  BMC Genetics, 16(Suppl 1):51–54, 2018.
  * [42] D. Simchi-Levi, P. Kaminsky, E. Simchi-Levi, and R. Shankar.  Designing and managing the supply chain: concepts, strategies and case studies.  Tata McGraw-Hill Education, 2008.
  * [43] J.R. Trapero, N. Kourentzes, and R. Fildes.  On the identification of sales forecasting models in the presence of promotions.  Journal of the operational Research Society, 66(2):299–307, 2015\.
  * [44] J.R. Trapero, D.J. Pedregal, R. Fildes, and N. Kourentzes.  Analysis of judgmental adjustments in the presence of promotions.  International Journal of Forecasting, 29(2):234–243, 2013.
  * [45] A. Tversky and D. Kahneman.  Judgment under uncertainty: Heuristics and biases.  Science, 185:1124–1131, 1974.

##  Appendix A Online Appendix: Supplementary Plots

Figure 17: Variable importance plots for classes 0 and 1.

Figure 18: Variable importance plots for classes 2 and 3.

Figure 19: SHAP value plots for classes 0 and 1.

Figure 20: SHAP value plots for classes 2 and 3.

Figure 21: SHAP dependency plots for price (the first four) and direct
discount (the second four). Classes 0 to 3 are placed clockwise.

Figure 22: SHAP dependency plots for quantity discount (the first four) and
product type (the second four). Classes 0 to 3 are placed clockwise.

Figure 23: SHAP dependency plots for Promise (classes 0 and 1 in the first
row), Bundle discount (class 2, bottom left), and Education (class 3, bottom
right). Figure 24: Aggregate variable importance plot for the classification
model Figure 25: A Decision Plot for the classification analysis. The
predicted probability is 0.63.


#  $\mathbb{Z}_{2}\times\mathbb{Z}_{2}$ Equivariant Quantum Neural Networks:
Benchmarking against Classical Neural Networks

Zhongtian Dong
Dep. Physics & Astronomy
University of Kansas
Lawrence, KS 66045
<EMAIL_ADDRESS>
&Marçal Comajoan Cara
Department of Signal Theory and Communications
Polytechnic University of Catalonia
Barcelona, Barcelona 08034, Spain
<EMAIL_ADDRESS>
&Gopal Ramesh Dahale
Indian Inst. of Technology Bhilai
Kutelabhata, Khapri,
Chhattisgarh – 491001, India
<EMAIL_ADDRESS>
&Roy T. Forestano
IFT, Physics Department,
University of Florida
Gainesville, FL 32611
<EMAIL_ADDRESS>&Sergei Gleyzer
Dep. Physics & Astronomy
University of Alabama
Tuscaloosa, AL 35487
<EMAIL_ADDRESS>
&Daniel Justice
Software Engineering Institute
Carnegie Mellon University
Pittsburgh, PA 15213
<EMAIL_ADDRESS>&Kyoungchul Kong
Dep. Physics & Astronomy
University of Kansas
Lawrence, KS 66045
<EMAIL_ADDRESS>
&Tom Magorsch
Physik-Department
Technische Univ. München
85748 Garching, Germany
<EMAIL_ADDRESS>
&Konstantin T. Matchev
IFT, Physics Department,
University of Florida
Gainesville, FL 32611
<EMAIL_ADDRESS>
&Katia Matcheva
IFT, Physics Department,
University of Florida
Gainesville, FL 32611
<EMAIL_ADDRESS>
&Eyup B. Unlu
IFT, Physics Department,
University of Florida
Gainesville, FL 32611
<EMAIL_ADDRESS>
###### Abstract

This paper presents a comprehensive comparative analysis of the performance of
Equivariant Quantum Neural Networks (EQNN) and Quantum Neural Networks (QNN),
juxtaposed against their classical counterparts: Equivariant Neural Networks
(ENN) and Deep Neural Networks (DNN). We evaluate the performance of each
network with two toy examples for a binary classification task, focusing on
model complexity (measured by the number of parameters) and the size of the
training data set. Our results show that the
$\mathbb{Z}_{2}\times\mathbb{Z}_{2}$ EQNN and the QNN provide superior
performance for smaller parameter sets and modest training data samples.

##  1 Introduction

Classical Equivariant Neural Networks (ENNs) exploit the underlying symmetry
structure of the data, ensuring that the input and output transform
consistently under the symmetry [1]. ENNs have been widely used in various
applications including deep convolutional neural networks for computer vision
[2], AlphaFold for protein structure prediction [3], Lorentz equivariant
neural networks for particle physics [4], etc. In recent years, significant
development has been made in their quantum counterparts, Equivariant Quantum
Neural Networks (EQNNs) [5, 6, 7, 8].

In this paper we benchmark the performance of EQNNs against various classical
and/or non-equivariant alternatives for two-dimensional toy data sets, which
exhibit a $\mathbb{Z}_{2}\times\mathbb{Z}_{2}$ symmetry structure. Such
patterns often appear in high-energy physics data, e.g., as kinematic
boundaries in the high-dimensional phase space describing the final state [9,
10]. By a clever choice of the kinematic variables for the analysis, these
boundaries can be preserved in projections onto a lower-dimensional feature
space [11, 12, 13, 14, 15]. In this study, we consider simplified two-
dimensional data sets that mimic the data arising in such projections.

##  2 Dataset Description

In both examples, we consider two-dimensional data $(x_{1},x_{2})$ on the unit
square ($-1\leq x_{i}\leq 1$) (see Fig. 1). The data points belong to two
classes: $y=+1$ (blue points) and $y=-1$ (red points). In the first example
(left panel of Fig. 1), the labels are generated by the function

$y(x_{1},x_{2})=2H\left(R-\sqrt{(x_{1}+1)^{2}+(x_{2}-1)^{2}}\right)+2H\left(R-\sqrt{(x_{1}-1)^{2}+(x_{2}+1)^{2}}\right)-1,$
(1)

where $H(x)$ is the Heaviside step function and for definiteness we choose
$R=1.1$. The function (1) respects a $\mathbb{Z}_{2}\times\mathbb{Z}_{2}$
symmetry, where the first $\mathbb{Z}_{2}$ is given by a reflection about the
$x_{1}=x_{2}$ diagonal

$x_{1}\to x_{2},\qquad x_{2}\to x_{1},\qquad y\to y,$ (2)

while the second $\mathbb{Z}_{2}$ corresponds to a reflection about the
$x_{1}=-x_{2}$ diagonal

$x_{1}\to-x_{2},\qquad x_{2}\to-x_{1},\qquad y\to y.$ (3)

This $\mathbb{Z}_{2}\times\mathbb{Z}_{2}$ example was studied in Ref. [6] and
we shall refer to it as the symmetric case, since the $y$ label is invariant.

Figure 1: Pictorial illustration of the datasets used in this study - the
symmetric case (1) in the left panel and the anti-symmetric case (4) in the
right panel.

The second example is illustrated in the right panel of Fig. 1. The labels are
generated by the function

$\displaystyle y(x_{1},x_{2})$
$\displaystyle=H\left(-x_{1}\right)H\left(-x_{2}\right)+H\left(-x_{1}\right)H\left(x_{2}\right)H(x_{1}+x_{2})$
$\displaystyle-H\left(x_{1}\right)H\left(x_{2}\right)+H\left(x_{1}\right)H\left(-x_{2}\right)H(x_{1}+x_{2}).$
(4)

The first $\mathbb{Z}_{2}$ is still realized as in (2). However, this time the
labels are flipped under a reflection along the $x_{1}=-x_{2}$ diagonal:

$x_{1}\to-x_{2},\qquad x_{2}\to-x_{1},\qquad y\to-y,$ (5)

which is why we shall refer to this case as anti-symmetric.

##  3 Network Architectures

To assess the importance of embedding the symmetry in the network, and to
compare the classical and quantum versions of the networks, we study the
performance of the following four different architectures: (i) Deep Neural
Network (DNN), (ii) Equivariant Neural Network (ENN), (iii) Quantum Neural
Network (QNN), and (iv) Equivariant Quantum Neural Network (EQNN). In each
case, we adjust the hyperparameters to ensure that the number of network
parameters is roughly the same.

Deep Neural Networks. In our DNN, for the symmetric (anti-symmetric) case we
use 1 (2) hidden layer(s) with 4 neurons. For both types of classical
networks, we use softmax activation function, Adam optimizer and learning rate
of $0.1$. We use the binary cross entropy for both DNN and ENN.

Equivariant Neural Networks. A given map $f:x\in X\to f(x)\in Y$ between an
input space $X$ and an output space $Y$ is said to be equivariant under a
group $G$ if it satisfies the following relation:

$f(g_{\text{in}}(x))=g_{\text{out}}(f(x)),$ (6)

where $g_{\text{in}}$ ($g_{\text{out}}$) is a representation of a group
element $g\in G$ acting on the input (output) space. In the special case when
$g_{\text{out}}$ is the trivial representation, the map is called invariant
under the group $G$, i.e. a symmetry transformation acting on the input data
$x$ does not change the output of the map. The goal of ENNs, or equivariant
learning models in general, is to design a trainable map $f$ which would
always satisfy Eq. (6). In tasks where the symmetry is known, such equivariant
models are believed to have an advantage in terms of number of parameters and
training complexity. Several studies in high-energy physics have attempted to
use classical equivariant neural networks [4, 16, 17, 18, 19]. Our ENN model
utilizes four $\mathbb{Z}_{2}\times\mathbb{Z}_{2}$ symmetric copies for each
data point, which are fed into the input layer, followed by one equivariant
layer with 3 (2) neurons and one dense layer with 4 (4) neurons in the
symmetric (anti-symmetric) case.

Quantum Neural Networks. For QNN, we utilize the one-qubit data-reuploading
model [20] with depth four (eight) for the symmetric (anti-symmetric) case,
using the angle embedding and $3$ parameters at each depth. This choice leads
to a similar number of parameters as in the classical networks. We use the
Adam optimizer and the loss

$L_{QNN}=y(1-|\mathinner{\langle{\psi}|}O_{1}\mathinner{|{\psi}\rangle}|)^{2}+(1-y)(1-|\mathinner{\langle{\psi}|}O_{2}\mathinner{|{\psi}\rangle}|)^{2}$
(7)

for any choice of two orthogonal operators $O_{1}$ and $O_{2}$ (see Ref. [21]
for more details.). In this paper, we use

$O_{1}=\dfrac{1}{4}\begin{pmatrix}[r]1&1&1&-1\\\ 1&1&1&-1\\\ 1&1&1&-1\\\
-1&-1&-1&1\end{pmatrix},\quad\quad
O_{2}=\dfrac{1}{4}\begin{pmatrix}[r]1&-1&-1&-1\\\ -1&1&1&1\\\ -1&1&1&1\\\
-1&1&1&1\end{pmatrix}.$ (8)

Equivariant Quantum Neural Networks. In EQNN models symmetry transformations
acting on the embedding space of input features are realized as finite-
dimensional unitary transformations $U_{g}$, $g\in G$. Consider the simplest
case where one trainable operator $U(\theta,x)$ acts on a state
$\mathinner{|{\psi}\rangle}$: $U(\theta,x)\mathinner{|{\psi}\rangle}$. If for
a symmetry transformation $U_{g}$, the condition

$U(\theta,x)\,U_{g}\mathinner{|{\psi}\rangle}=U_{g}\,U(\theta,x)\mathinner{|{\psi}\rangle},$
(9)

is satisfied, then the operator $U$ is equivariant, i.e., the equivariant gate
should commute with the symmetry. In general, the $U_{g}$ operators on the two
sides of Eq. (9) do not necessarily have to be in the same representation, but
are often assumed so for simplicity. The output of a QNN is the measurement of
the expectation value of the state with respect to some observable $O$. If the
gates are equivariant and we apply some symmetry transformation $U_{g}$, then
this is equivalent to measuring the observable $U_{g}^{\dagger}OU_{g}$. Hence
if $O$ commutes with the symmetry $U_{g}$, the model as a whole would be
invariant under $U_{g}$, which is the case in our symmetric example. Otherwise
the model is equivariant, as in our anti-symmetric example.

Our EQNN uses the two-qubit quantum circuit depicted in Fig. 2. The two
$R_{Z}$ gates embed $x_{1}$ and $x_{2}$, respectively. The $R_{X}$ gates share
the same parameter ($\theta_{1}$) and the $R_{ZZ}$ gate uses another parameter
($\theta_{2}$). The invariant model (for the symmetric case) uses the same
observable ${O}$ for both classes in the data. In the anti-symmetric case we
use two different observables ${O}_{1}$ and ${O}_{2}$ that correspond to each
label. They transform into one another under reflection $g_{r}$, i.e.,
$U^{\dagger}_{g_{r}}{O}_{1}U_{g_{r}}={O}_{2}$. In the symmetric case we use
binary cross-entropy loss, assuming the true label $y$ is either $0$ or $1$,

$L_{EQNN}=y\log\Big{(}\mathinner{\langle{\psi}|}O\mathinner{|{\psi}\rangle}\Big{)}+(1-y)\log\Big{(}1-\mathinner{\langle{\psi}|}O\mathinner{|{\psi}\rangle}\Big{)}\,.$
(10)

In the anti-symmetric case we used the same loss as in QNN where $O_{1}$
($O_{2}$) is the observable corresponding to $y=1$ ($y=0$). The observables
$O$ and the reflection $U_{g_{r}}$ along $x_{1}=-x_{2}$ are defined as
follows:

$O=\dfrac{1}{4}\begin{pmatrix}1&1&1&1\\\ 1&1&1&1\\\ 1&1&1&1\\\
1&1&1&1\end{pmatrix},\quad\quad U_{g_{r}}=\begin{pmatrix}0&0&0&1\\\ 0&0&1&0\\\
0&1&0&0\\\ 1&0&0&0\end{pmatrix}.$ (11)

Figure 2: Illustration of the quantum circuit used for EQNN at depth 1. This
circuit is repeated five (ten) times with different parameters for the
symmetric (anti-symmetric) case.

##  4 Results

Fig. 3 shows receiver operating characteristic (ROC) curves for each network
with $N_{\rm train}=200$ and $N_{\rm test}=2000$ samples for the symmetric
(left) and anti-symmetric (right) dataset. As expected, networks with
equivariance structure (EQNN and ENN) improve the performance of the
corresponding networks (QNN and DNN) without the symmetry. We also observe
that quantum networks perform better than the classical analogs. In the
legends, numerical values followed by network acronyms represent the number of
parameters used for each network. For the symmetric example, EQNN uses only 10
parameters, thus for fair comparison we constructed the other networks with
${\cal O}$(10) parameters as well. For the anti-symmetric example, we use 20
parameters for the EQNN. The evolution of the accuracy during training is
shown in Fig. 4. The accuracy converges faster (after only 5 epochs) for QNN
and EQNN in comparison to their classical counterparts (10-20 epochs).

Figure 3: ROC curves for the symmetric (left) and anti-symmetric (right)
example.

Figure 4: Accuracies during training for the symmetric (left) and anti-
symmetric (right) example.

Figure 5:  AUC as a function of the number of parameters (top) for fixed
$N_{\rm train}=2000$ and $N_{\rm test}=200$, and as a function of $N_{\rm
train}$ (bottom) with a fixed number of parameters as shown in the legend. The
left (right) panels show results for the symmetric (anti-symmetric) example.

To further quantify the performance of our quantum networks, in Fig. 5 we show
AUC (Area under the ROC Curve) as a function of the number of parameters (top
panels) with a fixed size of the training data ($N_{\rm train}=200$), and as a
function of the number of training samples (bottom panels) with a fixed number
of parameters ($N_{\rm params}=20$). The left (right) panels show results for
the symmetric (anti-symmetric) dataset. As the number of parameters increases,
the performance of all networks improves. All AUC values become similar when
$N_{\rm params}\approx 20$ ($N_{\rm params}\approx 40$) for the symmetric
(anti-symmetric) case. As shown in the bottom panels, the performances of all
networks become comparable to each other for both examples, once the size of
the training data reaches $\sim 400$.

##  5 Conclusion

Our study demonstrates that EQNNs and QNNs outperform their classical
counterparts, particularly in scenarios with fewer parameters and smaller
training datasets. This highlights the potential of quantum-inspired
architectures in resource-constrained settings. The code used for this study
is publicly available at https://github.com/ZhongtianD/EQNN/tree/main.

_Acknowledgements:_ KM is supported in part by the U.S. Department of Energy
award number DE-SC0022148. KK is supported in parts by US DOE DE-SC0024407. CD
is supported in part by College of Liberal Arts and Sciences Research Fund at
the University of Kansas.

## References

  * Cohen and Welling [2016] Taco Cohen and Max Welling.  Group equivariant convolutional networks.  In Maria Florina Balcan and Kilian Q. Weinberger, editors, _Proceedings of The 33rd International Conference on Machine Learning_ , volume 48 of _Proceedings of Machine Learning Research_ , pages 2990–2999, New York, New York, USA, 20–22 Jun 2016. PMLR.  URL https://proceedings.mlr.press/v48/cohenc16.html.
  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, _Advances in Neural Information Processing Systems_ , volume 25. Curran Associates, Inc., 2012.  URL https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf.
  * Jumper et al. [2021] John M. Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zídek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A A Kohl, Andy Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David A. Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis.  Highly accurate protein structure prediction with alphafold.  _Nature_ , 596:583 – 589, 2021.  URL https://api.semanticscholar.org/CorpusID:235959867.
  * Bogatskiy et al. [2020] Alexander Bogatskiy, Brandon Anderson, Jan Offermann, Marwah Roussi, David Miller, and Risi Kondor.  Lorentz group equivariant neural network for particle physics.  In Hal Daumé III and Aarti Singh, editors, _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 992–1002. PMLR, 13–18 Jul 2020.  URL https://proceedings.mlr.press/v119/bogatskiy20a.html.
  * Nguyen et al. [2022] Quynh T. Nguyen, Louis Schatzki, Paolo Braccia, Michael Ragone, Patrick J. Coles, Frederic Sauvage, Martin Larocca, and M. Cerezo.  Theory for equivariant quantum neural networks, 2022.
  * Meyer et al. [2023] Johannes Jakob Meyer, Marian Mularski, Elies Gil-Fuster, Antonio Anna Mele, Francesco Arzani, Alissa Wilms, and Jens Eisert.  Exploiting symmetry in variational quantum machine learning.  _PRX Quantum_ , 4(1), mar 2023.  doi: 10.1103/prxquantum.4.010328.  URL https://doi.org/10.1103%2Fprxquantum.4.010328.
  * West et al. [2023] Maxwell T West, Martin Sevior, and Muhammad Usman.  Reflection equivariant quantum neural networks for enhanced image classification.  _Machine Learning: Science and Technology_ , 4(3):035027, aug 2023.  doi: 10.1088/2632-2153/acf096.  URL https://doi.org/10.1088%2F2632-2153%2Facf096.
  * Skolik et al. [2023] Andrea Skolik, Michele Cattelan, Sheir Yarkoni, Thomas Bäck, and Vedran Dunjko.  Equivariant quantum circuits for learning on weighted graphs.  _npj Quantum Information_ , 9(1):47, 2023.
  * Kim [2010] Ian-Woo Kim.  Algebraic Singularity Method for Mass Measurement with Missing Energy.  _Phys. Rev. Lett._ , 104:081601, 2010.  doi: 10.1103/PhysRevLett.104.081601.
  * Franceschini et al. [2022] Roberto Franceschini, Doojin Kim, Kyoungchul Kong, Konstantin T. Matchev, Myeonghun Park, and Prasanth Shyamsundar.  Kinematic Variables and Feature Engineering for Particle Phenomenology.  _Reviews of Modern Physics_ , 6 2022.
  * Kersting [2009] N. Kersting.  On Measuring Split-SUSY Gaugino Masses at the LHC.  _Eur. Phys. J. C_ , 63:23–32, 2009.  doi: 10.1140/epjc/s10052-009-1063-6.
  * Bisset et al. [2011] M. Bisset, R. Lu, and N. Kersting.  Improving SUSY Spectrum Determinations at the LHC with Wedgebox Technique.  _JHEP_ , 05:095, 2011.  doi: 10.1007/JHEP05(2011)095.
  * Burns et al. [2009] Michael Burns, Konstantin T. Matchev, and Myeonghun Park.  Using kinematic boundary lines for particle mass measurements and disambiguation in SUSY-like events with missing energy.  _JHEP_ , 05:094, 2009.  doi: 10.1088/1126-6708/2009/05/094.
  * Debnath et al. [2016] Dipsikha Debnath, James S. Gainer, Doojin Kim, and Konstantin T. Matchev.  Edge Detecting New Physics the Voronoi Way.  _EPL_ , 114(4):41001, 2016.  doi: 10.1209/0295-5075/114/41001.
  * Debnath et al. [2017] Dipsikha Debnath, James S. Gainer, Can Kilic, Doojin Kim, Konstantin T. Matchev, and Yuan-Pao Yang.  Detecting kinematic boundary surfaces in phase space: particle mass measurements in SUSY-like events.  _JHEP_ , 06:092, 2017.  doi: 10.1007/JHEP06(2017)092.
  * Bogatskiy et al. [2022] Alexander Bogatskiy, Timothy Hoffman, David W. Miller, and Jan T. Offermann.  Pelican: Permutation equivariant and lorentz invariant or covariant aggregator network for particle physics, 2022.
  * Hao et al. [2023] Zichun Hao, Raghav Kansal, Javier Duarte, and Nadezda Chernyavskaya.  Lorentz group equivariant autoencoders.  _Eur. Phys. J. C_ , 83(6):485, 2023.  doi: 10.1140/epjc/s10052-023-11633-5.
  * Buhmann et al. [2023] Erik Buhmann, Gregor Kasieczka, and Jesse Thaler.  EPiC-GAN: Equivariant point cloud generation for particle jets.  _SciPost Phys._ , 15(4):130, 2023.  doi: 10.21468/SciPostPhys.15.4.130.
  * Batatia et al. [2023] Ilyes Batatia, Mario Geiger, Jose Munoz, Tess Smidt, Lior Silberman, and Christoph Ortner.  A general framework for equivariant neural networks on reductive lie groups, 2023.
  * Pérez-Salinas et al. [2020] Adriá n Pérez-Salinas, Alba Cervera-Lierta, Elies Gil-Fuster, and José I. Latorre.  Data re-uploading for a universal quantum classifier.  _Quantum_ , 4:226, feb 2020.  doi: 10.22331/q-2020-02-06-226.  URL https://doi.org/10.22331%2Fq-2020-02-06-226.
  * Ahmed [2019] Shahnawaz Ahmed.  Data-reuploading classifier.  https://pennylane.ai/qml/demos/tutorial_data_reuploading_classifier, 2019\.