# Causal Fine-Tuning and Effect Calibration of Non-Causal Predictive Models

Carlos Fernández-Loría*
Hong Kong University of Science and Technology
<EMAIL_ADDRESS>

Yanfang Hou
Hong Kong University of Science and Technology
<EMAIL_ADDRESS>

Foster Provost
New York University
<EMAIL_ADDRESS>

Jennifer Hill
New York University
<EMAIL_ADDRESS>
###### Abstract

This paper proposes techniques to enhance the performance of non-causal models
for causal inference using data from randomized experiments. In domains like
advertising, customer retention, and precision medicine, non-causal models
that predict outcomes under no intervention are often used to score
individuals and rank them according to the expected effectiveness of an
intervention (e.g, an ad, a retention incentive, a nudge). However, these
scores may not perfectly correspond to intervention effects due to the
inherent non-causal nature of the models. To address this limitation, we
propose causal fine-tuning and effect calibration, two techniques that
leverage experimental data to refine the output of non-causal models for
different causal tasks, including effect estimation, effect ordering, and
effect classification. They are underpinned by two key advantages. First, they
can effectively integrate the predictive capabilities of general non-causal
models with the requirements of a causal task in a specific context, allowing
decision makers to support diverse causal applications with a “foundational”
scoring model. Second, through simulations and an empirical example, we
demonstrate that they can outperform the alternative of building a causal-
effect model from scratch, particularly when the available experimental data
is limited and the non-causal scores already capture substantial information
about the relative sizes of causal effects. Overall, this research underscores
the practical advantages of combining experimental data with non-causal models
to support causal applications.

##  1 Introduction

Machine learning (ML) models are widely used to guide personalized
interventions in various domains, including targeted advertising, online
recommendations, customer retention, and precision medicine. These models are
primarily used to assign scores to individuals based on their expected
responsiveness to an intervention. However, it is important to note that these
scores often do not directly estimate the causal effect of the intervention
itself. Instead, they represent a quantity that is expected to correlate with
the effect.

Consider, for example, a model predicting a high likelihood that an individual
will enjoy some specific content (e.g., a movie). While recommending that
content could have a greater impact on the individual’s behavior than on
someone in the broader population, the model is not explicitly designed to
estimate the effect of the recommendation. There are numerous other examples
of models that provide scores that do not precisely align with the causal
effect of interest. These include models that estimate effects on surrogate or
proxy outcomes, as well as models that may not accurately reflect the effects
due to confounding bias. In this paper, we collectively refer to all these
types of models as _base scoring models_ : models that provide scores that
correlate with the causal effect of interest but are not the causal effects
themselves.

Base scoring models are often used in settings where conducting randomized
experiments or collecting data on the primary outcome of interest is
challenging or impractical, making it difficult to directly model the desired
causal effect. However, there are additional practical reasons why these
models can be effective in guiding personalized interventions.

First, base scoring models could be useful to inform various types of
interventions instead of just one. For example, a prediction that an
individual is likely to buy some specific product can be useful not only for
recommending that product but also for making decisions about pricing or
suggesting other related products. As a result, firms may benefit from
focusing their resources (data, infrastructure, personnel) on the development
of a single base scoring model that could potentially inform multiple types of
intervention decisions.

Furthermore, even in scenarios where obtaining appropriate experimental data
is feasible, the available data for learning base scoring models can be
considerably larger both vertically and horizontally. Vertically, these models
can often be trained using data generated from the day-to-day operations of
the business, resulting in a much larger training sample size. Horizontally,
the individual-level features used to construct base scoring models may offer
a level of granularity that surpasses what is available in A/B testing
platforms. For example, a recommender system might leverage a user’s entire
preference history to make predictions, a level of detail that may not be
accessible in A/B testing platforms (see, e.g., Fernández-Loría et al. 2023).

The primary drawback, however, is that the scores may not be a good
approximation of the responsiveness to the intervention. In response to this
concern, recent studies have put forth evaluation frameworks that use
experimental data to evaluate the effectiveness of scoring models in causal
tasks. Typical tasks include ranking individuals based on their effect sizes
or making the most effective intervention decisions (Yadlowsky et al. 2021,
Schuler et al. 2018). However, what remains unexplored is how experimental
data can be leveraged to improve the performance of a base scoring model for
such tasks, which is the main focus of our study.

We present two techniques to enhance the performance of base scoring models
for causal inference using data from randomized experiments: effect
calibration and causal fine-tuning. These techniques unlock the potential of
using base scoring models as “foundational” models to support causal
applications in diverse contexts. By leveraging experimental data, we can
tailor an existing base scoring model to suit specific causal inference tasks.
It works as follows.

We begin with a base scoring model that produces scores expected to correlate
with the causal effect of a particular treatment (intervention) on a specific
outcome. Such scores could correspond to the propensity to buy a product in
advertising or patient risk in precision medicine. In such contexts, we may
anticipate a correlation between the scores and the effect of an ad on
conversion (Stitelman et al. 2011) or the impact of a medical intervention on
a patient (Kent et al. 2020).

Alongside the base scoring model, there is experimental data, capturing the
specific context to which we want to fine-tune the base scoring model. Here,
the context encompasses the treatment under consideration, the outcome of
interest, and the population in which the experiment was conducted. Thus, for
each individual in the experimental dataset, we observe the treatment
assignment, the outcome, the base score, and optionally, other features.

The first technique, effect calibration, uses the experimental data to
estimate a scaling factor and a single shift that can be applied to the base
scores, improving effect estimation. This technique was originally introduced
by Leng and Dimmery (2024), but their study focused exclusively on its
application to causal effect models and did not explore its potential for base
scoring models. Our work, to the best of our knowledge, is the first to
demonstrate the potential benefits of effect calibration for any type of
model, including those that do not explicitly estimate causal effects.

We find that calibrated base scores can estimate the sign and magnitude of
effects significantly better than conventional causal-effect modeling,
particularly when the available experimental data is limited. However, the
uniform transformation applied to all base scores limits the incorporation of
individual-specific corrections. As a result, its effectiveness diminishes
with larger experimental data, and it does not improve the ability to rank
individuals based on effect magnitude.

We introduce a second technique, causal fine-tuning, to address these issues.
Causal fine-tuning algorithms leverage experimental data to learn a fine-tuner
tailored to a causal task. We focus on three causal tasks: estimating
individual effects (effect estimation), ranking individuals by effect
magnitude (effect ordering), and identifying those benefiting from the
intervention (effect classification). The fine-tuner applies corrections to
the base scores in order to enhance their utility for the intended causal
task, so fine-tuned scores are base scores after being refined by a fine-
tuner.

Our study introduces three algorithms for causal fine-tuning, each designed to
address one of the causal tasks: effect estimation, effect ordering, and
effect classification. For effect estimation, the corrections aim to reduce
mean squared error. For effect ordering, the corrections aim to improve the
Area Under the Uplift Curve (AUUC), a metric often used in uplift modeling to
evaluate effect ordering. Lastly, the corrections in effect classification aim
to increase the expected outcome of a treatment assignment policy, which is
analogous to decreasing the misclassification rate weighted by the absolute
value of the effects being classified.

These algorithms are based on tree-induction machine learning. For each
algorithm, we introduce a splitting criterion and leaf-level corrections that
are designed to enhance performance on a specific causal task. Therefore, the
methodological contribution centers around defining evaluation criteria for
causal performance that can be integrated into standard tree-based machine
learning algorithms for fine-tuning. It is worth emphasizing that we
implemented all our algorithms as subclasses of tree-based algorithms already
available in scikit-learn.

To evaluate the effectiveness of causal fine-tuning and effect calibration, we
conducted simulations where we compared these techniques with the effect
estimates of a causal tree learned from the same data used for fine-tuning and
effect calibration. The results demonstrate that causal fine-tuning generally
outperforms learning a causal-effect model from the ground up. However, when
the experimental data is small, fine-tuning can negatively impact the
performance of the base scoring model. In such cases, effect calibration
without fine-tuning leads to the best performance. We also include an
empirical example in the context of advertising that corroborates these
findings.

In conclusion, our research highlights the practical benefits of integrating
experimental data with non-causal models to better support diverse causal
applications.

##  2 Problem Setup

We employ the causal scoring framework introduced by Fernández-Loría and Loría
(2022) to define causal tasks. Causal scoring entails the estimation of a
quantity $\theta$—a causal score—that is informative of a causal quantity
$\beta$. For example, $\theta$ could be the probability of a purchase under no
intervention, while $\beta$ could refer to the Conditional Average Treatment
Effect (CATE) of a discount on the purchase probability. Both $\theta$ and
$\beta$ vary with a feature vector $X$.

In our study, we define $\beta$ as the CATE of a treatment $T\in\\{0,1\\}$ on
an outcome of interest $Y\in\mathbb{R}$. Let $Y^{0}\in\mathbb{R}$ and
$Y^{1}\in\mathbb{R}$ be the potential outcomes when untreated and treated.
Then:

$\displaystyle\beta$ $\displaystyle=\mathbb{E}[Y^{1}-Y^{0}\mid X],$ (1)
$\displaystyle Y$ $\displaystyle=TY^{1}+(1-T)Y^{0}.$ (2)

The causal interpretation of $\theta$ is the relationship between $\theta$ and
$\beta$ that we aim to derive. We consider three distinct causal
interpretations, each corresponding to a causal task:

  * •

Effect Estimation: $\theta=\beta$. The causal scores are intended to represent
the causal quantity itself. This is the most commonly adopted interpretation
in the field of causal inference.

  * •

Effect Ordering: Larger $\theta\Leftrightarrow$ Larger $\beta$. The causal
scores can be used to establish the relative ordering of the causal quantity
among individuals in the population.

  * •

Effect Classification: $\theta>\tilde{\tau}\Leftrightarrow\beta>\tau$. The
causal scores can be combined with a threshold $\tilde{\tau}$ to classify
individuals into the “high-effect” ($\beta>\tau$) and “low-effect”
($\beta\leq\tau$) classes.

Fernández-Loría and Loría (2022) discuss data-generating processes under which
these causal interpretations hold for different types of quantities that could
be defined as the causal scores. However, our work differs in focus because we
are interested in assessing the extent to which a scoring model is helpful to
achieve our desired objective. Therefore, we redefine the causal
interpretations as causal tasks with associated performance measures, thereby
allowing us to quantitatively evaluate the performance of a model in
fulfilling these tasks.

###  2.1 Causal Tasks

We use _Mean Squared Error (MSE)_ to define performance at effect estimation:

$\text{MSE}(\theta)=\mathbb{E}\left[\left(\beta-\theta\right)^{2}\right].$ (3)

Next, we use the _Area Under the Uplift Curve (AUUC)_ (Devriendt et al. 2020)
to define performance at effect ordering. The AUUC is a metric derived from
the uplift curve, which shows the incremental impact of treating a certain
percentage of individuals with the highest scores. A higher AUUC implies a
better ranking of individuals based on causal effects.

To construct the uplift curve, individuals are sorted in descending order
based on $\theta$. Each point on the curve represents the incremental causal
impact when the top $q$ fraction of individuals are treated. The AUUC is the
weighted sum of these points, where the weights correspond to the $q$ values.
The formula for calculating the AUUC is given by:

$\displaystyle\text{AUUC}(\theta)$ $\displaystyle=\int_{0}^{1}q\cdot V(q)dq$
(4) $\displaystyle V(q)$ $\displaystyle=\mathbb{E}\left[\beta|F(\theta)\geq
1-q\right],$ (5)

where $F$ is the cumulative distribution function (CDF) of $\theta$.

Finally, we approach the assessment of effect classification as the evaluation
of a treatment assignment policy. In this context, $\tau$ represents the cost
associated with providing treatment to an individual, and $a(\theta)$
represents the treatment action taken based on the causal score $\theta$. We
define performance at effect classification with the _expected policy outcome_
, denoted as $\pi$:

$\displaystyle\pi(\theta)=\mathbb{E}[Y^{a(\theta)}-a(\theta)\tau],$ (6)
$\displaystyle a(\theta)=\textbf{1}\\{\theta>\tilde{\tau}\\},$ (7)

where $a(\theta)$ corresponds to a policy that treats individuals with a
causal score above $\tilde{\tau}$.

As shown in Appendix A, ranking policies based on $\pi$ is equivalent to
ranking classifiers based on their _effect-weighted misclassification (EWM)_ :

$\displaystyle\text{EWM}(\theta)=\mathbb{E}\left[\left|\beta-\tau\right|\cdot\textbf{1}\left\\{\beta>\tau\neq\theta>\tilde{\tau}\right\\}\right].$
(8)

###  2.2 Causal Models

Our study focuses on settings where we have data on a randomized experiment
with the following variables: the treatment assignment $T\in\\{0,1\\}$, the
outcome of interest $Y\in\mathbb{R}$, and a feature vector $X$ for each
individual. In this context, we could approach any of the causal tasks by
estimating the following causal score:

$\displaystyle\theta$ $\displaystyle=\mathbb{E}[Y\mid T=1,X]-\mathbb{E}[Y\mid
T=0,X].$ (9)

Assuming the experiment was properly designed and conducted, Equation (9) is
mathematically equivalent to Equation (1). The causal score in Equation (9)
can be estimated with metalearners based on conventional machine learning
methods (Künzel et al. 2019, Nie and Wager 2021). Alternatively, there are
also machine learning methods specifically designed for estimating causal
effects (Dorie et al. 2019). These methods are designed for effect estimation
but could also be used for effect ordering and effect classification.

If the goal of the causal estimation is effect classification, classification
methods can be employed to learn causal scores that can classify individuals
based on the sign of their causal effects (Zhang et al. 2012, Zhao et al.
2012). Methods designed for this purpose are also referred to as treatment
assignment learners (A-Learners) (Fernández-Loría et al. 2023). These methods
leverage the insight that estimating a treatment assignment regime can be
viewed as a classification problem, where each example is weighted based on
its outcome. For instance, the EWM in Equation (8) can be minimized when
$\tau=0$ using the following causal scores:111This formulation assumes that
the outcome has been transformed to ensure positive expectations.

$\begin{split}\theta&=\frac{\mathbb{E}[Y\mid T=1,X]}{\mathbb{E}[Y\mid
T=1,X]+\mathbb{E}[Y\mid T=0,X]}.\end{split}$ (10)

Optimal classifications are achieved by setting $\tilde{\tau}=0.5$. These
causal scores can be estimated with a classification algorithm by defining $T$
as the target variable and $y_{i}/\mathbb{P}[T=t_{i}]$ as the cost of
misclassifying observation $i$. The primary advantage of this approach is that
classification algorithms can better navigate the bias-variance trade-off
because accurate estimation of the expectations is not necessary for effective
classification (Fernández-Loría et al. 2023).

###  2.3 Base Scoring Models

The formulation of causal scores in Equation (9) and Equation (10) faces a
significant limitation: its performance tends to suffer when the experimental
data is limited in size.

Before discussing what we could do to address this limitation, let’s consider
what would be the alternative in cases where conducting randomized experiments
is unfeasible or prohibitively expensive (i.e., there is no experimental
data). In such scenarios, a common approach for effect ordering and
classification is to use predictions of a quantity that is expected to
strongly correlate with the causal effect of interest. For instance, when
targeting marketing interventions, predictions of the probability of purchase
are often employed to decide who to target (Radcliffe and Surry 2011). Risk-
based modeling is frequently employed for customer retention strategies
(Ascarza et al. 2018) and precision medicine applications (Kent et al. 2020).
Another common strategy in customer retention and medicine is modeling the
effects on short-term outcomes as an alternative means to estimate the effects
on long-term outcomes (Yang et al. 2023).

We collectively refer to these as _base scoring models_ : models that provide
scores that are presumed to correlate with causal effects but are not the
causal effects themselves.

The effectiveness of base scoring models compared to causal models, such as
those in Equations (9) and (10), depends on two key factors: the signal-to-
noise ratio, and the strength of the relationship between the base scores and
the causal effects (Fernández-Loría and Provost 2022).

The signal-to-noise ratio becomes significant when one quantity is
considerably easier to predict than the other. For instance, if the available
data for training the causal model is limited compared to the data available
for training the base scoring model, accurately predicting causal effects
becomes more challenging (i.e., the signal-to-noise ratio is smaller), which
may lead us to prefer a base scoring model. Additionally, a base scoring model
might be favored if the signal is stronger, meaning that the features are more
strongly correlated with base scores than with causal effects.

The second factor relates to the bias present in the base scores, arising from
modeling a quantity that does not correspond exactly to the causal effect. The
impact of this bias is mitigated when there is a strong relationship between
the base scores and the causal effects. A strong correlation can arise when
the bias is relatively small compared to the magnitude of the effect, or more
commonly, when the bias provides information about the size of the effect
(Fernández-Loría and Provost 2022). The latter case implies that the bias in a
base scoring model can potentially be helpful for effect ordering or effect
classification purposes.

To sum up, base scoring models could be a better alternative than causal
models if they have a larger signal-to-noise ratio that outweighs the bias
from not estimating the causal effect of interest.

###  2.4 Effect Calibration

We first consider the technique of effect calibration as a means of improving
a base scoring model for causal inference, as introduced by Leng et al.
(2024). This method was designed to calibrate effect estimates by aligning
their sign and magnitude with model-free benchmarks. While the original study
only considered the calibration of CATE models, their method can also be
applied to base scoring models without requiring any adjustment.

However, it’s important to note that this effect calibration method is not
suitable for improving effect ordering, as it applies the same monotonic
transformation to all scores. Nevertheless, it can be a valuable enhancement
for effect estimation and potentially effect classification. Effect
calibration is a complementary technique to causal fine-tuning, which is
described below.

###  2.5 Causal Fine-Tuning

In addition to effect calibration, we propose causal fine-tuning to improve
the performance of a base scoring model. The method is motivated by the
recognition that not all bias in the base scores is inherently detrimental.
Bias in the base scores can mitigate errors due to variance, thereby improving
the performance of the scores for effect ordering (Fernández-Loría and Loría
2022) and effect classification (Fernández-Loría and Provost 2022, 2019). For
example, if the base scores corresponding to positive (negative) effects
exhibit a tendency towards positive (negative) bias, such bias could prove
beneficial for effect classification. Therefore, we use causal modeling to
enhance the base scores by specifically correcting for bias in cases where it
negatively impacts the performance of the base scores for some causal task.

Causal fine-tuning consists of learning a fine-tuner, denoted as $\delta$,
which can be combined with a base score $\theta^{b}$ to generate a causal
score:

$\theta=\theta^{b}-\delta(X)$ (11)

With this approach, the focus shifts from estimating the causal quantity
$\beta$ to instead estimating a “correction” $\delta$ that leads to an
improvement in a specific causal task, as defined by Equations (3), (4), and
(6). The appropriate correction depends on the particular causal task at hand.
For example, while it may generally be desirable to correct for large bias in
the case of effect estimation, such bias could be advantageous for effect
classification or effect ordering. We further discuss how suitable corrections
may vary based on the causal task in Section 3, where we propose multiple
algorithms for causal fine-tuning.

To learn $\delta$, we require data from a randomized experiment that includes
the following variables for each individual: the treatment assignment $T$, the
outcome of interest $Y$, the base score $\theta^{b}$, and optionally, a
feature vector $X^{e}$. In this setup, the feature vector that accounts for
variations in $\beta$ and $\theta$ is defined as $X=(X^{e},\theta^{b})$.

Another notable aspect of causal fine-tuning, as presented here, is its
applicability to _any_ type of base scoring model, without requiring
modifications to the model or knowledge of its estimation process. Causal
fine-tuning can be applied even when the base scores are generated using a
different set of features from those available in the experimental data, as
long as the base scores for each individual are known.

This situation can arise when base scores are derived from millions of
features, but such information is not recorded by A/B testing platforms. For
instance, in targeted advertising, the ad exposure selection process relies on
a multitude of high-dimensional features at the auction level, which are
continuously updated over time. However, due to substantial engineering and
storage costs, advertising platforms often do not log all of this data (Gordon
et al. 2023), making it impossible to collect A/B test data with exactly the
same features used to make advertising decisions based on base scores.

Despite the beneficial flexibility of causal fine-tuning as presented here, it
is conceivable that method-specific causal fine-tuning, such as updating the
weights of a neural network employed for producing the base scores, could
potentially lead to better results. We defer the exploration of this idea to
future research.

###  2.6 Other Related Work

Other studies have proposed the use of base scores to enhance causal modeling.
For instance, Peysakhovich and Lada (2016) propose incorporating confounded
estimates of causal effects from observational data as an additional feature
in causal models estimated using data from a randomized experiment. In a
different context, Athey et al. (2023) explore various targeting strategies to
encourage students to renew their financial-aid applications through the use
of “nudges.” These strategies involve both CATE estimation and a base scoring
model that predicts baseline outcomes, which represent outcomes in the absence
of any intervention. The study reveals that targeting based on intermediate
baseline outcomes yields the most effective results. Building upon this
finding, the authors propose hybrid approaches that aim to combine the
strengths of the base scoring model and the causal model. These approaches
incorporate base scores as an additional feature and leverage the specific
relationship between baseline outcomes and causal effects during the learning
process of the CATE model.

The primary distinction between these studies and ours lies in their approach
of incorporating base scores as an additional feature in the causal modeling,
rather than leveraging causal modeling to improve the base scores themselves.

##  3 Algorithms for Causal Fine-Tuning

We propose several algorithms to learn the fine-tuner $\delta$ in Equation
(11). Each algorithm is tailored to address a specific causal task, including
effect estimation, effect classification, and effect ordering. These
algorithms require experimental data as outlined in Section 2.5. All of the
algorithms are based on tree induction techniques.

###  3.1 Effect Estimation

The first causal fine-tuning algorithm we propose, named the Effect Estimation
(EE) algorithm, is designed to improve base scores for effect estimation. As
this algorithm uses tree induction, the resulting fine-tuner is represented as
a tree structure, with each leaf node corresponding to a distinct partition of
the feature space.

Let’s begin by considering the estimation of $\delta$ for individuals whose
feature vectors belong to a specific partition of the feature space (leaf),
denoted as $\Omega$. In this scenario, the fine-tuner applies the same
correction to all individuals within $\Omega$. By using the definition of
effect estimation performance in Equation (3) and the formulation of fine-
tuned scores in Equation (11), the objective of the EE algorithm is to
minimize the following expression:

$\displaystyle\text{MSE}\left(\delta;\Omega\right)=\mathbb{E}\left[(\beta-\theta^{b}+\delta)^{2}\mid
X\in\Omega\right].$ (12)

The minimizer is the mean bias of the base score with respect to the CATE,
given by:

$\delta_{EE}=\mathbb{E}\left[\theta^{b}-\beta\mid X\in\Omega\right]$ (13)

Given a sample of data $\mathcal{D}$ drawn from the population represented by
$\Omega$, we estimate $\delta_{EE}$ as:

$\displaystyle\hat{\delta}(\mathcal{D})$
$\displaystyle=\hat{\mu}_{b}-(\hat{\mu}_{y}(1)-\hat{\mu}_{y}(0)),$ (14)
$\displaystyle\hat{\mu}_{y}(t)$
$\displaystyle=\frac{1}{N(t)}\sum_{i\in\mathcal{D}:t_{i}=t}y_{i},$
$\displaystyle\hat{\mu}_{b}$
$\displaystyle=\frac{1}{N(1)+N(0)}\sum_{i\in\mathcal{D}}\theta^{b}_{i}.$

Each observation $i$ in the sample $\mathcal{D}$ has a treatment assignment
$t_{i}$, an outcome $y_{i}$, and a base score $\theta^{b}_{i}$. $N(t)$ denotes
the number of observations in the sample $\mathcal{D}$ with treatment $T=t$,
$\hat{\mu}_{y}(t)$ is the average outcome in $\mathcal{D}$ for $T=t$, and
$\hat{\mu}_{b}$ is the average base score in $\mathcal{D}$.

The EE algorithm uses tree induction to recursively partition the data based
on its features. When creating new partitions, the algorithm considers all
possible single-split partitions of the feature space and aims to select the
split that is best suited to reduce MSE. Let’s denote the feature space of the
parent node as $\Omega_{P}$ and its corresponding sample as $\mathcal{D}_{P}$.
A candidate split $\ell$ divides $\Omega_{P}$ into two feature spaces,
$\Omega_{L}^{\ell}$ and $\Omega_{R}^{\ell}$, resulting in subsamples
$\mathcal{D}_{L}^{\ell}$ and $\mathcal{D}_{R}^{\ell}$. The algorithm’s
objective is to choose the candidate split that maximizes the following
criterion:

$\displaystyle\text{MSE}\left(\hat{\delta}(\mathcal{D}_{P});\Omega_{P}\right)-\frac{|\mathcal{D}_{L}^{\ell}|}{|\mathcal{D}_{P}|}\cdot\text{MSE}\left(\hat{\delta}(\mathcal{D}_{L}^{\ell});\Omega_{L}^{\ell}\right)-\frac{|\mathcal{D}_{R}^{\ell}|}{|\mathcal{D}_{P}|}\cdot\text{MSE}\left(\hat{\delta}(\mathcal{D}_{R}^{\ell});\Omega_{R}^{\ell}\right),$
(15)

where $|\mathcal{D}|$ represents the size of sample $\mathcal{D}$.

The challenge with estimating Equation (15) is that it depends on $\beta$,
which is not observed. To address this issue, we adopt a similar approach to
the one introduced in Athey and Imbens (2016) and aim to minimize a modified
version of the MSE. As in Athey and Imbens (2016), the modification does not
affect how candidate splits are ranked. Here is the modification:

$\widetilde{\text{MSE}}\left(\delta;\Omega\right)=\mathbb{E}\left[(\beta-\theta^{b}+\delta)^{2}-(\theta^{b}-\beta)^{2}|X\in\Omega\right]=\delta^{2}-2\delta\cdot\delta_{EE}.$
(16)

We estimate $\delta_{EE}$ with $\delta$, leading to the following estimate of
$\widetilde{\text{MSE}}$:

$\widehat{\text{MSE}}\left(\delta\right)=-\delta^{2}.$ (17)

The EE algorithm selects the candidate splits that maximize this criterion:

$\displaystyle\widehat{\text{MSE}}\left(\hat{\delta}(\mathcal{D}_{P})\right)-\frac{|\mathcal{D}_{L}^{\ell}|}{|\mathcal{D}_{P}|}\cdot\widehat{\text{MSE}}\left(\hat{\delta}(\mathcal{D}_{L}^{\ell})\right)-\frac{|\mathcal{D}_{R}^{\ell}|}{|\mathcal{D}_{P}|}\cdot\widehat{\text{MSE}}\left(\hat{\delta}(\mathcal{D}_{R}^{\ell})\right).$
(18)

An important aspect to note is the significant advantage of applying effect
calibration to the base scores prior to training the EE fine-tuner. This is
due to the potential disparity in scale between the base scores and the
effects. Properly scaling the base scores becomes crucial in this scenario. As
a result, we highly recommend incorporating effect calibration as an initial
step before proceeding with EE fine-tuning.

###  3.2 Effect Classification

We now present another causal fine-tuning algorithm called the Effect
Classification (EC) algorithm. Its purpose is to enhance the base scores used
for classifying individuals into two groups: those with a positive CATE and
those with a negative CATE. The effect classification task is equivalent to
estimating a treatment assignment policy where the available actions are
“treat” or “not treat.” The goal is to identify the individuals who would
benefit from treatment based on their positive CATE. Optimal actions (effect
classification labels) are defined as:

$a^{*}=\mathbf{1}[\beta>0],$ (19)

where $a^{*}=1$ if the individual’s CATE is positive, and $a^{*}=0$ otherwise.

Given the formulation of fine-tuned scores in Equation (11), we consider
effect classification based on the comparison of a base score with a decision
boundary $\delta$:

$a(\delta)=\mathbf{1}[\theta^{b}>\delta],$ (20)

The fine-tuner has a tree structure and estimates a different decision
boundary for each leaf node. We consider first the estimation of a decision
boundary for a single leaf node that represents a feature space denoted as
$\Omega$. The EC algorithm aims to maximize the expected policy outcome, which
quantifies the effect classification performance:

$\pi(\delta;\Omega)=\mathbb{E}[Y^{a(\delta)}\mid X\in\Omega]$ (21)

Given a sample $\mathcal{D}$ of unconfounded data drawn from the population
represented by $\Omega$, $\pi$ can be unbiasedly estimated as follows (proof
in Appendix B):

$\displaystyle\hat{\pi}(\delta,\mathcal{D})$
$\displaystyle=\frac{1}{|\mathcal{D}|}\sum_{i\in\mathcal{D}}\frac{\mathbf{1}[t_{i}=a_{i}]}{\mathbb{P}[T=t_{i}]}y_{i},$
(22) $\displaystyle a_{i}$ $\displaystyle=\mathbf{1}[\theta^{b}_{i}>\delta].$
(23)

The EC algorithm estimates the decision boundary for individuals in a leaf
node by solving the following optimization problem:

$\displaystyle\hat{\delta}(\mathcal{D})=\mathop{\rm
arg\,max}_{\delta}~{}\hat{\pi}(\delta,\mathcal{D})$ (24)

To solve the optimization problem, we consider the scores of all observations
within $\mathcal{D}$ as candidate decision boundaries. Each of these values
corresponds to a distinct classifier that could be implemented for individuals
in that leaf. We assess the estimated policy outcome for each potential
boundary to determine the one that yields the best result. The pseudocode for
this process of identifying the optimal boundary is outlined in Algorithm 1.

This procedure is also leveraged by the EC algorithm to determine how to
partition the feature space using tree induction. Let $\mathcal{D}_{P}$
represent the data sample of a parent node, and let $\mathcal{D}_{L}^{\ell}$
and $\mathcal{D}_{R}^{\ell}$ be the subsamples that result from applying a
candidate split $\ell$. The EC algorithm chooses the candidate splits that
maximize this criterion:

$\displaystyle\frac{|\mathcal{D}_{L}^{\ell}|}{|\mathcal{D}_{P}|}\cdot\hat{\pi}\left(\hat{\delta}(\mathcal{D}_{L}^{\ell}),\mathcal{D}_{L}^{\ell}\right)+\frac{|\mathcal{D}_{R}^{\ell}|}{|\mathcal{D}_{P}|}\cdot\hat{\pi}\left(\hat{\delta}(\mathcal{D}_{R}^{\ell}),\mathcal{D}_{R}^{\ell}\right)-\hat{\pi}\left(\hat{\delta}(\mathcal{D}_{P}),\mathcal{D}_{P}\right),$
(25)

Algorithm 1 Find Optimal Threshold

1:$\mathcal{D}$

2:Initialize $\hat{\delta}=\epsilon+\max~{}\theta^{b}_{i},i\in\mathcal{D}$
$\triangleright$ $\epsilon$ is a small positive value close to 0

3:for $i\in\mathcal{D}$ sorted by base scores $\theta^{b}$, descending do

4: $\delta=\theta^{b}_{i}-\epsilon$

5: if $\hat{\pi}(\delta,\mathcal{D})>\hat{\pi}(\hat{\delta},\mathcal{D})$ then

6: $\hat{\delta}=\delta$

7: end if

8:end for

9:Adjust $\hat{\delta}$ to minimize its absolute value without changing the
classification of any individual.

10:$\hat{\delta}$

###  3.3 Effect Ordering

The third and final causal fine-tuning algorithm we present is the Effect
Ordering (EO) algorithm. It is designed to improve causal scores for the
purpose of ranking individuals based on the magnitude of their causal effects
(effect ordering).

#### AUUC estimation.

To evaluate the effect ordering performance of scoring models, we use AUUC as
our evaluation metric, defined in Equation (4). The estimation of the AUUC
with a sample $\mathcal{D}$ involves grouping observations into $m$ subsets,
which are mutually exclusive and collectively exhaustive. These subsets are
denoted as ${\mathcal{D}_{0},\mathcal{D}_{1},...,\mathcal{D}_{m-1}}$, where
each subset represents a level. Level 0 is denoted by $\mathcal{D}_{0}$, level
1 by $\mathcal{D}_{1}$, and so on.

The rank of a level is determined by its index, such that higher-ranked levels
correspond to observations with higher scores. Level $\mathcal{D}_{k}$
encompasses all observations whose scores fall within the percentile range of
$k/m$ to $(k+1)/m$. In simpler terms, $\mathcal{D}_{k}$ contains observations
whose scores are greater than or equal to $k/m$ of all observations in
$\mathcal{D}$, but less than $(k+1)/m$ of all observations.

Formally, the set of observations in level $\mathcal{D}_{k}$ is defined as
follows:

$\displaystyle\mathcal{D}_{k}$ $\displaystyle=\left\\{\forall
i\in\mathcal{D}:\frac{k}{m}\leq\hat{F}_{i}<\frac{k+1}{m}\right\\},$ (26)
$\displaystyle\hat{F}_{i}$ $\displaystyle=\frac{1}{|\mathcal{D}|}\sum_{\forall
j\in\mathcal{D}}\mathbf{1}[\theta_{i}>\theta_{j}],$ (27)

where $\theta_{i}$ is the score assigned to observation $i$ based on the
evaluated scoring model.

Let $\theta$ be the scores for the observations in $\mathcal{D}$, and let
$r_{i}(\theta)$ be the rank of the level to which observation $i$ belongs
based on $\theta$. We can estimate the AUUC associated with $\theta$ as
follows:

$\displaystyle\widehat{\text{AUUC}}(\theta)$
$\displaystyle=\sum_{q=1}^{m}\frac{q}{m}\cdot\hat{V}(\theta,m-q),$ (28)
$\displaystyle\hat{V}(\theta,r)$
$\displaystyle=\frac{R(\theta,r,1)}{N(\theta,r,1)}-\frac{R(\theta,r,0)}{N(\theta,r,0)},$
(29) $\displaystyle N(\theta,r,t)$ $\displaystyle=\sum_{\forall
i\in\mathcal{D}:r_{i}(\theta)\geq r}\mathbf{1}[t_{i}=t],$ (30) $\displaystyle
R(\theta,r,t)$ $\displaystyle=\sum_{\forall i\in\mathcal{D}:r_{i}(\theta)\geq
r}\mathbf{1}[t_{i}=t]\cdot y_{i},$ (31)

where $N(r,t)$ is the count of observations with $T=t$ within levels of rank
$r$ or higher, and $R(r,t)$ is the sum of the outcomes for these observations.

#### Single shift.

Consider the simple approach of improving the AUUC by applying a score shift
to a subset of individuals. We begin with a sample $\mathcal{D}$ with scores
$\theta^{0}$, and we partition it into two subsets, $\mathcal{D}_{L}$ and
$\mathcal{D}_{R}$. The scores after applying the score shift $d$ to the
observations in $\mathcal{D}_{R}$ is:

$\displaystyle\theta(d)=\theta^{0}+\mathbf{1}_{R}\cdot d,$ (32)

where $\mathbf{1}_{R}=1$ for observations in $\mathcal{D}_{R}$ and 0
otherwise.

We define the quality of a shift $d$ as the change in the estimated AUUC
compared to $\theta^{0}$:

$\displaystyle\Delta_{AUUC}(d)=\widehat{AUUC}(\theta(d))-\widehat{AUUC}(\theta^{0})$
(33)

We discuss next a procedure to estimate the optimal shift $\hat{d}$:

$\displaystyle\hat{d}=\mathop{\rm arg\,max}_{d}~{}\Delta_{AUUC}(d)$ (34)

Our procedure is based on the idea that a shift $d$ can change the estimated
AUUC only if it changes the rank of at least one observation in
$\mathcal{D}_{R}$. Because the shift $d$ is applied to all observations in
$\mathcal{D}_{R}$, this implies that the rank of at least one observation in
$\mathcal{D}_{L}$ must also change. In other words, if $d$ increases
(decreases) the rank of an observation in $\mathcal{D}_{R}$, it must decrease
(increase) the rank of another observation in $\mathcal{D}_{L}$.

To find the optimal shift $\hat{d}$, we calculate the change in AUUC
associated with each possible rank change resulting from applying the same
shift to all observations in $\mathcal{D}_{R}$. We then set $\hat{d}$ as the
smallest value that leads to the rank change associated with the largest
improvement in AUUC. The details of this procedure are described below, and
the pseudo-code is presented in Algorithm 2.

We discuss first the cases where $d$ is non-negative. The procedure is
iterative with the iteration number denoted by $u$. In each iteration, a shift
$d_{u}$ is calculated. We start in iteration $u=0$ with $d_{0}=0$ and define
$\theta^{u}$ as the scores that result from applying the shift $d_{u}$:

$\displaystyle\theta^{u}=\theta(d_{u})$ (35)

In iteration $u$, observations are scored and ranked based on $\theta^{u}$. We
then set $d_{u+1}$ to be the smallest positive shift greater than $d_{u}$ that
increases the rank of an observation in $\mathcal{D}_{R}$:

$\displaystyle d_{u+1}$
$\displaystyle=\epsilon+d_{u}+\min~{}(\theta^{u}_{i}-\theta^{u}_{j}),\forall
i,j:x_{i}\in\mathcal{D}_{L},~{}x_{j}\in\mathcal{D}_{R},~{}r_{i}(\theta^{u})>r_{j}(\theta^{u}),$
(36)

where $\epsilon$ is a small positive value close to zero. Increasing the shift
from $d_{u}$ to $d_{u+1}$ results in a single observation in $\mathcal{D}_{R}$
swapping ranks with another observation in $\mathcal{D}_{L}$. Therefore, when
shift $d_{u}$ is applied to $\theta^{0}$, observations in $\mathcal{D}_{R}$
increase rank $u$ times. The procedure continues increasing $u$ until all
observations in $\mathcal{D}_{R}$ have an equal or higher rank than any in
$\mathcal{D}_{L}$.

We use this formulation to calculate the improvement associated with shift
$d_{u+1}$:

$\displaystyle\Delta_{AUUC}(d_{u+1})=\Delta_{AUUC}(d_{u})+\Delta_{u},$ (37)
$\displaystyle\Delta_{u}=\widehat{AUUC}(\theta^{u+1})-\widehat{AUUC}(\theta^{u}),$
(38)

with $\Delta_{AUUC}(d_{0})=0$.

We describe next how to efficiently calculate $\Delta_{u}$ each iteration.
Note that $\Delta_{u}$ is the change in AUUC that results from swapping the
ranks of an observation $i$ in $\Omega_{L}$ and an observation $j$ in
$\Omega_{R}$. Let $r^{u}=r_{i}(\theta^{u})$ be the rank of observation $i$
before the swap. Equation (38) is mathematically equivalent to:

$\displaystyle\Delta_{u}$
$\displaystyle=\frac{m-r^{u}}{m}\cdot\left(\frac{R(\theta^{u},r^{u},1)+t_{j}y_{j}-t_{i}y_{i}}{N(\theta^{u},r^{u},1)+t_{j}-t_{i}}-\frac{R(\theta^{u},r^{u},0)+(1-t_{j})y_{j}-(1-t_{i})y_{i}}{N(\theta^{u},r^{u},0)+(1-t_{j})-(1-t_{i})}-\hat{V}(\theta^{u},r^{u})\right),$
(39)

which does not require to recalculate the AUUC estimation using the entire
sample in each iteration.

To consider non-positive values for $d$, we use an analogous procedure to the
one described above. The difference is that we set $d_{u}$ to be the
smallest222When we say smallest, we mean smallest in terms of absolute value.
_negative_ shift to _decrease_ the rank of a single observation in
$\mathcal{D}_{R}$. The procedure keeps increasing $u$ until all the
observations in $\mathcal{D}_{R}$ have an equal or lower rank than any of the
observations in $\mathcal{D}_{L}$.

Algorithm 2 Find Optimal Shift

1:$\theta^{0}$, $\mathcal{D}_{L}$, $\mathcal{D}_{R}$

2:Initialize $\hat{d}=0$

3:Set $\theta^{u}=\theta^{0}$, $d=0$ $\triangleright$ Consider positive
shifts.

4:while $\exists
i\in\mathcal{D}_{L},j\in\mathcal{D}_{R}:r_{i}(\theta^{u})>r_{j}(\theta^{u})$
do

5: $d=\epsilon+d+\min(\theta_{i}^{u}-\theta_{j}^{u})$, $\forall
i,j:i\in\mathcal{D}_{L},j\in\mathcal{D}_{R},r_{i}(\theta^{u})>r_{j}(\theta^{u})$

6: if $\Delta_{AUUC}(d)>\Delta_{AUUC}(\hat{d})$ then

7: $\hat{d}=d$

8: end if

9: $\theta^{u}=\theta(d)$

10:end while

11:Set $\theta^{u}=\theta^{0}$, $d=0$ $\triangleright$ Consider negative
shifts.

12:while $\exists
i\in\mathcal{D}_{R},j\in\mathcal{D}_{L}:r_{i}(\theta^{u})>r_{j}(\theta^{u})$
do

13: $d=\epsilon+d+\min(\theta^{u}_{i}-\theta^{u}_{j})$, $\forall
i,j:i\in\mathcal{D}_{R},j\in\mathcal{D}_{L},r_{i}(\theta^{u})>r_{j}(\theta^{u})$

14: if $\Delta_{AUUC}(-d)>\Delta_{AUUC}(\hat{d})$ then

15: $\hat{d}=-d$

16: end if

17: $\theta^{u}=\theta(-d)$

18:end while

19:$\hat{d}$, $\Delta_{AUUC}(\hat{d})$

#### Tree-based method.

The preceding subsection presented an algorithm for computing the optimal
shift $\hat{d}$ for a given partitioning of the data. The EO algorithm
considers all possible single-split partitions of the feature space and
selects the one that maximizes this criterion:

$\displaystyle\Delta_{AUUC}(\hat{d}_{\ell}),$ (40)

where $\hat{d}_{\ell}$ is the optimal shift for a split $\ell$ that partitions
the feature space into $\Omega_{L}^{\ell}$ and $\Omega_{R}^{\ell}$, resulting
in the sample subsets $\mathcal{D}_{L}^{\ell}$ and $\mathcal{D}_{R}^{\ell}$.
The outcome of this procedure is a single-split tree. The pseudo-code is
provided in Algorithm 3.

To consider multiple splits, we adopt a concept similar to boosting, which
involves learning an ensemble of weak models to create a stronger one. The EO
algorithm leverages this idea by learning an ensemble of single-split trees to
produce an effect ordering fine-tuner. Specifically, the EO Algorithm consists
of applying Algorithm 3 iteratively, so that a new single-split tree is
learned on each iteration based on base scores modified by the previously
learned tree. The single-split trees are then combined to create the fine-
tuner. The pseudo-code is presented in Algorithm 4.

Formally, we define the fine-tuner learned by the EO algorithm as follows. Let
$\Omega_{L}^{i}$ and $\Omega_{R}^{i}$ represent the partitions of the feature
space corresponding to the $i$-th tree, and let $\hat{d}_{i}$ be the estimated
optimal shift for that tree. Given the formulation of fine-tuned scores in
Equation (11), if the EO algorithm estimated $N_{trees}$ trees, then the
estimated fine-tuner is:

$\delta(X)=-\sum_{i=1}^{N_{trees}}\mathbf{1}[X\in\Omega_{R}^{i}]\cdot\hat{d}_{i}.$
(41)

Algorithm 3 Estimate Single-Split Tree

1:$\theta^{0}$, $\mathcal{D}$

2:$\Delta^{*}_{AUUC}=0$, $\hat{d}=0$, $\hat{\Omega}_{R}=\emptyset$

3:for each possible single split $\ell$ of the feature space defined by $X$
do:

4: Determine feature space partitions $\Omega_{L}^{\ell}$ and
$\Omega_{R}^{\ell}$ based on $\ell$.

5: Use $\Omega_{L}^{\ell}$ and $\Omega_{R}^{\ell}$ on sample $\mathcal{D}$ to
obtain subsets $\mathcal{D}_{L}^{\ell}$ and $\mathcal{D}_{R}^{\ell}$.

6: $\hat{d}_{\ell},\Delta=\textsc{Find Optimal
Shift}(\theta^{0},\mathcal{D}_{L}^{\ell},\mathcal{D}_{R}^{\ell})$

7: if $\Delta>\Delta^{*}_{AUUC}$ then

8: $\hat{d}=\hat{d}_{\ell}$, $\hat{\Omega}_{R}=\Omega_{R}^{\ell}$,
$\Delta^{*}_{AUUC}=\Delta$

9: end if

10:end for

11:Create a function $\delta$ that receives $X$ as an input and returns
$\hat{d}$ if $X\in\hat{\Omega}_{R}$ and $0$ otherwise.

12:$\delta$

Algorithm 4 EO Algorithm

1:$\mathcal{D}$

2:$\theta^{0}=\theta^{b}$

3:for i=1 to $N_{trees}$ do

4: $\hat{\delta}_{i}=\textsc{Estimate Single-Split
Tree}(\theta^{0},\mathcal{D})$

5: $\theta^{0}=\theta^{0}+\hat{\delta}_{i}(X)$

6:end for

7:Create function
$\hat{\delta}(X)=-\sum_{i=1}^{N_{trees}}\hat{\delta}_{i}(X)$.

8:$\hat{\delta}$

#### Computational Efficiency.

The computation time of the EO algorithm is highly dependent on the choice of
the number of levels $m$ used to divide the sample for estimating the AUUC.
During the shift search process detailed in Algorithm 2, the rank of each
observation in $\mathcal{D}_{R}$ can change up to $m-1$ times, resulting in up
to $m\cdot|\mathcal{D}_{R}|$ shift values to be considered. Considering that
the shift search is performed for every possible split, we recommend setting
$m$ to a value much smaller than $N$. For our experiments, we set $m=10$.
Additionally, computation time is sensitive to factors that can significantly
increase the number of candidate splits, such as the number of available
features and the presence of continuous features. To mitigate this, we suggest
discretizing continuous features to reduce computation time. In our
experiments, we only consider splits that have at least 40% of the sample,
which helps reduce the number of candidate splits under consideration. Lastly,
computation time can also be impacted by the number of single-split trees
estimated by the EO algorithm. In our experiments, we limit the number of
trees to 10.

To reduce the search space for shifts in Algorithm 2, we also employ a
bucketing technique. The first step in this technique is to group the
observations with adjacent scores into percentile groups, using the same
approach outlined in Equation (26) for defining levels. When the data is
divided into subsets $\mathcal{D}_{L}$ and $\mathcal{D}_{R}$, each percentile
group is further divided into individual buckets, allowing for up to two
buckets per percentile group. However, if all individuals within a percentile
group are assigned to a single subset of the data, only one bucket is created
for that group.

The search for the optimal shift is performed at the bucket level rather than
the individual level, resulting in a more efficient process. Bucket scores are
determined by taking the average score of the individuals within each bucket.
As a result, the shift update described in Equation (36) is based on comparing
bucket scores instead of individual scores. This adjustment means that the
indices $i$ and $j$ now pertain to bucket indices rather than individual
indices, significantly reducing the number of candidate shifts to be
considered. Similarly, the improvement in AUUC is computed at the bucket
level. When two buckets swap ranks, all observations within those buckets
undergo a rank swap as well. Specifically, any observation within a bucket
moving down one rank will shift down one rank, while any observation within a
bucket moving up one rank will shift up one rank. This adjustment only affects
the formulation of Equation (39).

It is important to highlight that the EO algorithm without the bucketing
strategy, which involves searching for optimal shifts at the individual level,
corresponds to the case where each percentile group consists of only one
observation, so that all buckets contain a single data point.

##  4 Simulation

In this section, we conduct a simulation study to address the following
questions:

  * •

How do effect calibration and causal fine-tuning of base scoring models
compare to conventional causal modeling approaches?

  * •

What is the gain (if any) from using fine-tuning methods designed for specific
causal tasks?

We focus on the specific scenario where the base scores are predictions of
baseline outcomes, representing the outcomes in the absence of any
intervention. There are several reasons why we narrow our focus to this
scenario. First, the use of such base scores to make intervention decisions is
common across various applications, including customer retention (Ascarza et
al. 2018), targeted advertising (Stitelman et al. 2011), nudging (Athey et al.
2023), precision medicine (Kent et al. 2020), and recommender systems (Gao et
al. 2024).

Second, a key advantage of causal fine-tuning is its potential to leverage a
“foundational” base scoring model for various causal applications involving
diverse types of interventions. Estimating the baseline outcome, which
represents what would happen without any intervention, can potentially provide
valuable information for many intervention decisions. So, we expect effect
calibration and causal fine-tuning to be particularly valuable when applied to
this type of base scores.

Lastly, in scenarios where extensive experimental data is lacking, it is
common practice to rely on baseline outcome predictions for decision making.
It is precisely in these settings that we anticipate our proposed techniques
to outperform conventional causal modeling. When the available experimental
data is not sufficiently large, accurate estimation of the causal quantity of
interest becomes challenging.333The definition of a “sufficiently large”
sample can vary significantly across contexts. In some cases, even hundreds of
millions of observations may be insufficient (Fernández-Loría et al. 2023).
Effect calibration and causal fine-tuning of base scoring models offer a
promising alternative in such situations. Common scenarios where extensive
experimental data may be lacking include cases with new interventions, like
introducing a new product in recommender systems or implementing a new
retention incentive to address churn, as well as cases where the intervention
carries inherent risks, such as in the field of medicine. In these cases,
predictions of baseline outcomes are often used to inform decisions, making
calibration and fine-tuning valuable for leveraging the limited experimental
data that may be available.

###  4.1 Data-Generating Process

Based on the experimental setup described in Section 2.5, we need to simulate
the following variables for each individual in the experimental data: their
observed outcome $Y$, their treatment assignment $T$, their base score
$\theta^{b}$, and their feature values $X^{e}$.

First, we assume the experiment was properly conducted, so the treatment
assignment is random and independent of all other variables:

$\displaystyle T\sim\text{Bern}(0.5).$ (42)

Let $X^{b}=\\{X_{1},...,X_{w}\\}$ be a set of $w$ features used to calculate
the base score $\theta^{b}$. Base scores are estimates of the expected
baseline outcome $Y^{0}$. We assume that the resources and data available for
estimating the base score are extensive, so the estimation of the target value
is highly accurate:

$\theta^{b}=\mathbb{E}[Y^{0}\mid X^{b}]$ (43)

We assume all features in $X^{b}$ are binary and independent:

$\displaystyle X_{j}\sim\text{Bern}(0.5),\forall X_{j}\in X^{b}$ (44)

We assume $X^{e}$ is a subset of size $w_{e}\leq w$ of the features in
$X^{b}$:

$X^{e}=\\{X_{j}\in X^{b}:j\leq w_{e}\\}$ (45)

We define the potential outcome under treatment $Y^{1}$ as the sum of the
baseline outcome $Y^{0}$ and the treatment effect $C$:

$Y^{1}=Y^{0}+C$ (46)

We assume $Y^{0}$ and $C$ can be expressed as linear functions of $X^{b}$:

$\displaystyle Y^{0}=\alpha_{0}+\sum_{\forall X_{j}\in
X^{b}}\alpha_{j}X_{j}+\epsilon^{y},$ (47) $\displaystyle
C=\beta_{0}+\sum_{\forall X_{j}\in X^{b}}\beta_{j}X_{j}+\epsilon^{c}.$ (48)

Here, $\epsilon^{y}$ and $\epsilon^{c}$ represent idiosyncratic variations in
the individual’s baseline outcome and causal effect, respectively. We model
the correlation between base scores and causal effects by giving a
multivariate normal distribution to $\beta_{j}$ and $\alpha_{j}$:

$\displaystyle\begin{pmatrix}\alpha_{j}\\\ \beta_{j}\end{pmatrix}$
$\displaystyle\sim\mathcal{N}\begin{bmatrix}\begin{pmatrix}0\\\ 0\\\
\end{pmatrix}\\!\\!,&\begin{pmatrix}\sigma^{2}_{\alpha}&\rho\sigma_{\alpha}\sigma_{\beta}\\\
\rho\sigma_{\alpha}\sigma_{\beta}&\sigma^{2}_{\beta}\end{pmatrix}\end{bmatrix},$
(49)

where $\rho\in[-1,1]$ is used to model the direction and strength of the
correlation. The parameters $\sigma_{\alpha}$ and $\sigma_{\beta}$ are used to
control the degree of variation in baseline outcomes and causal effects,
respectively, that can be explained by the features. We also use a normal
distribution to model idiosyncratic variation:

$\displaystyle\begin{pmatrix}\epsilon^{y}\\\ \epsilon^{c}\end{pmatrix}$
$\displaystyle\sim\mathcal{N}\begin{bmatrix}\begin{pmatrix}0\\\ 0\\\
\end{pmatrix}\\!\\!,&\begin{pmatrix}\sigma_{y}^{2}&\rho\sigma_{y}\sigma_{c}\\\
\rho\sigma_{y}\sigma_{c}&\sigma_{c}^{2}\end{pmatrix}\end{bmatrix}.$ (50)

The parameters $\sigma_{y}$ and $\sigma_{c}$ are used to control the degree of
variation in baseline outcomes and causal effects, respectively, that cannot
be explained by the features.

###  4.2 Experimental Setup

Our simulation study compares the performance of different approaches for
causal scoring. The approaches we consider are:

  * •

Base Scores (BS): Base scores as defined in Equation (43), without calibration
or fine-tuning.

  * •

Base Scores (BS_CAL): Base scores with effect calibration but no fine-tuning.

  * •

Causal Tree (CT): We use a causal tree (Athey and Imbens 2016) learned from
experimental data to model causal effects, without including the base score as
a feature. The features used by this causal tree are all in $X^{e}$. The
estimated causal score corresponds to Equation (9). We apply effect
calibration to the resulting effect estimates.

  * •

Causal Tree (CT-BS): The same as CT but including the base score as a feature.
Therefore, the features used by this causal tree are $\\{X^{e},\theta^{b}\\}$.
Includes effect calibration.

  * •

Effect Estimation Algorithm (EE): Causal fine-tuning based on the EE
algorithm. Includes effect calibration before and after the fine-tuning.

  * •

Effect Ordering Algorithm (EO): Causal fine-tuning based on the EO algorithm.
Includes effect calibration after the fine-tuning.

  * •

Effect Classification Algorithm (EC): Causal fine-tuning based on the EC
algorithm. Includes effect calibration after the fine-tuning.

We choose causal trees as the baseline for causal-effect modeling because they
are a well-known machine learning method for estimating causal effects, and
like our causal fine-tuning methods, they are learned with a tree-based
algorithm.

For effect calibration, we employ the method proposed by Leng and Dimmery
(2024), which entails estimating a scaling factor and a single shift to be
applied to all the scores. While this transformation of the scores cannot
improve effect ordering, we found that it often improves the effect estimation
and effect classification of fine-tuned scores. In the case of EE, we also
apply calibration to the base scores before conducting the effect estimation
fine-tuning. This step is desirable to ensure proper scaling because base
scores and treatment effects typically have distinct scales. We do not apply
this pre-calibration to EO and EC because these methods rely only on the score
ranking rather than the absolute values of the scores.

To evaluate the performance of these approaches, we use the measures
introduced in Section 2.1: mean squared error for effect estimation, AUUC for
effect ordering, and the expected policy outcome for effect classification. We
set $\tau=\tilde{\tau}=0$ for the effect classification.

We set the following default parameters for the simulations: $w=50$,
$w_{e}=50$, $\rho=0.5$, $\sigma^{2}_{\alpha}=1$, $\sigma^{2}_{\beta}=1$,
$\sigma^{2}_{y}=12.5$ and $\sigma^{2}_{c}=12.5$. With these parameters, 50% of
the variance in baseline outcomes and causal effects can be explained by the
features in $X^{b}$.

The reported results are the averages obtained from 100 simulations. In each
simulation, we begin by sampling the dataset-level variables $\alpha_{j}$ and
$\beta_{j}$ for each feature $X_{j}$. Next, we create an experimental dataset
consisting of 32,768 observations. This dataset serves as the basis for
creating training samples of varying sizes, ranging from $128$ to $32,768$,
with the sample sizes increasing in powers of 2. These training samples are
used for the causal fine-tuning methods, the learning of the causal trees, and
the effect calibration. Finally, to evaluate the performance of the different
approaches, we generate a separate test set consisting of 50,000 observations.

###  4.3 Main Results

The benchmark results are presented in Figure 1. All percentage improvements
are in relation to the base scoring model without calibration. Several points
warrant discussion.

(a)

(b)

(c)

Figure 1: Benchmark results with effect calibration. All fine-tuning methods
perform better than the causal-effect models, and with large enough data, than
the calibrated based scores.

First, we observe a consistent improvement in performance across all three
causal tasks when incorporating the base score as an additional feature in the
causal tree (CT-BS), as compared to using the causal tree without the base
score (CT), regardless of the dataset size. This finding is noteworthy because
the base scores are calculated using the same features available to CT, which
may initially appear as redundant information for the model. However, due to
the high correlation between baseline outcomes and causal effects, this
inclusion leads to enhanced performance through a form of feature engineering.

Furthermore, it is important to highlight that the observed performance
improvement is substantial. Given the logarithmic scale of the x-axis, a
horizontal gap of size one between two lines indicates that the method
represented by the left line can achieve the same level of performance as the
method represented by the right line with only half the amount of training
data. To illustrate, if CT with a data size of 5 achieves the same level of
performance as CT-BS with a data size of 6, it means that CT requires only
half the data to achieve the same performance level as CT-BS. The figures
demonstrate that incorporating predictions of baseline outcomes as a feature
can result in improvements equivalent to doubling the available training data.

Another interesting finding relates to the amount of data required to perform
well compared to the base scores (BS, dotted line) across different causal
tasks. Both causal trees outperform the base scores in effect estimation even
with a relatively small amount of training data. However, to achieve the same
level of performance as the base scores in effect classification, at least
four times as much data is needed. When it comes to effect ordering, a
staggering amount of sixteen times more data is required for equivalent
performance. From these results, it is evident that the base scores capture
valuable information regarding how causal effects vary from one individual to
another. However, they are not reliable estimates of the actual effect sizes,
which makes them less suitable for effect estimation and classification
purposes.

This is where effect calibration can make a substantial impact. Figure 1 shows
that the calibrated base scores outperform all other methods in all causal
tasks when there is limited experimental data available. Let’s take a closer
look at why this happens.

The effect calibration process involves scaling the base scores and applying a
single shift to all of them (i.e., a linear transformation). As a result, it
cannot possibly improve effect ordering, as shown in the center chart of
Figure 1. However, scaling and shifting are important for effect estimation,
and shifting is important for effect classification.

As a result if the base scoring model is appropriate for ranking individuals
based on the magnitude of their effects, even a small amount of experimental
data used for calibration can lead to a stark improvement in effect estimation
and possibly effect classification as well. In our example, the calibrated
base scores have comparable performance in effect estimation and effect
classification to what would be achieved through conventional causal modeling
with 8 to 16 times more data.

However, as the training data size increases, the other methods eventually
perform better than the calibrated base scores because the correlation between
baseline outcomes and causal effects is not perfect. Consequently, it can be
helpful to apply distinct corrections to the base scores of different
individuals, rather than employing a uniform correction for everyone. With a
sufficiently large dataset, the fine-tuning methods are capable of identifying
individuals who require different corrections, while causal-effect models can
better capture the variability of effects across individuals.

Notably, Figure 1 shows that causal fine-tuning outperforms causal-effect
models across all causal tasks, regardless of the data size. The improvement
in performance is particularly striking for EO, which emerges as the top
performer in all causal tasks. EO’s performance with a data size of 4 is
comparable to the performance of the other methods with a data size of 8. This
implies that the other methods require approximately 16 times more data to
achieve a similar level of performance compared to EO. Furthermore, the
performance gap between EO and the other methods remains substantial even with
very large datasets.

Apart from showcasing the considerable potential of causal fine-tuning, this
result also highlights the potential for methods not specifically designed for
a particular causal task to outperform dedicated methods. EO, which is
designed for effect ordering, achieves the best performance in effect
estimation and effect classification in this particular setting.

The underlying reason is that EO’s strength in effect ordering can be
effectively harnessed for effect estimation and effect classification by
employing effect calibration. However, it is crucial to interpret these
findings with caution, as outcomes can vary substantially depending on the
specific data-generating process (as we illustrate later in Section 5). Strong
effect ordering performance does not guarantee that good effect estimation or
good effect classification can be achieved through linear effect calibration.
The presence of non-linear relationships between scores and effects can
potentially compromise the effectiveness of the calibration approach.

###  4.4 Results With a Subset of Features

Next, we delve into the results when only a subset of the features is
available in the experimental data. Specifically, we set $w_{e}=10$,
indicating that only 10 out of the 50 features used to calculate the base
scores are present in the experimental data. The results are presented in
Figure 2. We highlight two notable findings in the following discussion.

(a)

(b)

(c)

Figure 2: Results with post-calibration using a subset of features
($w^{e}=10$).

First, we observe that CT (the causal tree without base scores as a feature)
performs significantly worse than the other methods. It fails to outperform
the calibrated base scores in any of the causal tasks, regardless of the
amount of experimental data available. This result underscores the true
effectiveness of base scoring models. Typically, these models are estimated
using a considerably larger amount of data, encompassing both a greater number
of observations and more extensive feature sets. Therefore, if the target
variable (the baseline outcome in our specific example) exhibits substantial
correlation with causal effects, base scoring models are more likely to
capture heterogeneity in causal effects compared to models specifically
designed for causal estimation but trained with smaller datasets or fewer
features.

The second notable finding is that the performance gap between CT-BS (causal
tree with base scores as a feature) and the causal fine-tuning methods is
narrower in this scenario. There are a couple of major reasons why we expect
this trend to be more prevalent when there are fewer features available in the
experimental data.

First, some of the features that are no longer present in the experimental
data might have been particularly valuable for causal fine-tuning. These
features would correspond to those that are predictive of greater base scores
but not greater causal effects, or vice versa. When these features are
removed, the ability to apply different corrections for different individuals
becomes limited. This explains the sharp drop in performance of EO fine-
tuning.

Another potential reason is that a reduced set of features could actually
improve the performance of CT-BS, although in this specific case the removal
of features did not enhance CT-BS performance. In this case, CT-BS is now
trained with only 20% of the previous features. However, to some extent, the
information previously provided by the absent features is still captured by
the base scores. As a result, the machine learning algorithm used to estimate
CT-BS may focus more on the signal provided by the base scores and less on the
signal provided by the other features, potentially leading to improved
performance.

Overall, these results demonstrate the potential value of base scoring models
for causal tasks when they incorporate a larger number of features than those
available in the experimental data. However, with fewer features available in
the experimental data, it becomes harder to identify cases where higher causal
scores do not necessarily correspond to greater causal effects. As a result,
the potential for causal fine-tuning to outperform conventional causal
modeling diminishes.

##  5 Empirical Example

We present next an empirical example to illustrate the circumstances where
causal fine-tuning and effect calibration are appropriate, as well as those
where it is not. Our example is based on a benchmark dataset for uplift
modeling provided by Criteo, an advertising company (Diemert et al. 2018). The
dataset was obtained from a randomized experiment where the treatment consists
of exposure to advertising. Each entry in the dataset corresponds to a user
and includes 11 features, a treatment indicator, and two outcomes of interest:
visit and conversion. The dataset comprises a total of 13,979,592 rows, with
the treated group accounting for 85% of the sample. The visit rate is 4.70%,
the conversion rate is 0.29%, the average treatment effect on visit is 1.03%,
and the average treatment effect on conversion is 0.12%.

The example we present corresponds to a hypothetical scenario where a company
seeks to optimize its advertising efforts to maximize purchases for a new
product. Given the novelty of the product, there is limited available data
regarding individuals’ likelihood to make a purchase. The only available data
comes from a small-scale randomized experiment conducted by the company to
implement a data-driven strategy for targeting future advertisements.

We examine the use of such experimental data alongside the following base
scores:

  * •

Baseline visit predictions: The company may have a predictive model for
estimating the likelihood of individuals visiting their website in the absence
of advertising. Although this model would not be specifically designed to
predict purchases and would not have been trained on data reflecting
individuals’ behavior when targeted by advertising, it can still be valuable
in identifying individuals interested in the company’s advertised product.
Indeed, website visits have been shown to be a good proxy for conversions
(Dalessandro et al. 2015). As a result, it might be an effective tool to rank
individuals based on the potential effectiveness of the ad on them. It is
worth noting that the company would likely have ample data to train such a
model, and its application could extend beyond a single advertising
intervention.

  * •

Value metric: Companies often employ RFM (recency, frequency, and monetary)
metrics to segment and target customers with advertising (Wei et al. 2010).
For example, advertising efforts may be directed towards customers who have
made significant purchases in the past or have made recent purchases. In our
data, the feature names have been anonymized, and their values have been
randomly projected to protect individual privacy. Consequently, it is not
possible to determine which specific features correspond to RFM metrics. To
simulate an RFM-like metric, we employ the feature that exhibits the highest
correlation with visits (f9) as a surrogate metric that could potentially be
leveraged as a base score for targeting advertisements.

###  5.1 Benchmark Setup

We use 20% of the available data as a test set for evaluating and comparing
different causal scoring approaches. From the remaining data, we randomly
select 1,000,000 observations from the control group to estimate the
predictive model for baseline visits. Additionally, we randomly sample another
250,000 observations, evenly split between treated and control groups (125,000
each), to represent the data from the hypothetical small-scale randomized
experiment. This dataset serves as the basis for generating training samples
of varying sizes, ranging from 10,000 to 250,000, with the sample sizes
increasing incrementally by 30,000. To ensure reliability, we repeat this
process 100 times and report the average results.

The causal scoring approaches we examine are the same as those introduced in
Section 4.2. We denote the causal tree for causal effect estimation as CT when
base scores are not included as a feature and CT-BS when they are. We also
denote BS_CAL for calibrated base scores, and EE, EO, and EC for causal fine-
tuning methods tailored for effect estimation, effect ordering, and effect
classification, respectively. We use effect calibration for all the
approaches. In the case of EE, we also apply calibration to the base scores
before conducting the fine-tuning.

We evaluate all the causal scoring approaches using the performance metrics
defined in Section 2.1: MSE for effect estimation, AUUC for effect ordering,
and the expected policy outcome for effect classification. Because the true
CATE is unknown, we estimate the MSE at an aggregated level, using a similar
approach as described in Leng and Dimmery (2024). We divide the test set into
10 percentile bins based on the base scores without calibration. Within each
bin, we calculate the error as the difference between the model-free average
treatment effect and the average score. The MSE is obtained by averaging the
errors across the 10 bins, as shown below.

$\displaystyle\widehat{\text{MSE}}(\theta)$
$\displaystyle=\frac{1}{|\mathcal{P}|}\sum_{\mathcal{S}\in\mathcal{P}}(\mu_{y}(1,\mathcal{S})-\mu_{y}(0,\mathcal{S})-\mu_{\theta}(\mathcal{S}))^{2}$
(51) $\displaystyle\mu_{y}(t,\mathcal{S})$
$\displaystyle=\frac{1}{N(t,\mathcal{S})}\sum_{i\in\mathcal{S}:t_{i}=t}y_{i}$
(52) $\displaystyle\mu_{\theta}(\mathcal{S})$
$\displaystyle=\frac{1}{N(0,\mathcal{S})+N(1,\mathcal{S})}\sum_{i\in\mathcal{S}}\theta_{i}$
(53)

Here, $\mathcal{P}$ is the set of bins in the test data. In a given bin
$\mathcal{S}$, $N(t,\mathcal{S})$ is the number of observations with treatment
$T=t$, $\mu_{y}(t,\mathcal{S})$ is the average outcome in $\mathcal{S}$ for
$T=t$, and $\mu_{\theta}(\mathcal{S})$ is the average score. For a given
individual $i$, $\theta_{i}\in\mathbb{R}$ is their causal score,
$y_{i}\in\\{0,1\\}$ is their conversion outcome, and $t_{i}\in\\{0,1\\}$ is
their treatment assignment.

The expected policy outcome corresponds to an estimate of the expected outcome
when the top 10% of individuals with the largest scores are targeted with the
ad. This can be estimated with data from a randomized experiment as detailed
in Equation 22.

###  5.2 Baseline Visits as the Base Score

Figure 3 shows the results when using predictions of baseline visits as base
scores. The charts for AUUC and expected policy outcome show the percentage
improvements relative to the base scoring model without calibration. In the
MSE charts, the percentage improvements are shown with regard to the approach
where everyone is predicted to have the ATE for the entire population. This is
necessary because base scores and treatment effects are measured on different
scales, so base scores without calibration are generally unsuitable for direct
effect estimation.

Notably, the calibrated base scoring model, BS_CAL, shows great performance
across all causal tasks. It outperforms all other approaches in effect
ordering (center chart). In terms of effect estimation (leftmost chart),
BS_CAL performs best except when the experimental data is too small. In terms
of effect classification (rightmost chart), BS_CAL outperforms causal-effect
models and models that were causally fine-tuned with small experimental data.
However, as the sample size increases, the causally fine-tuned models exhibit
a performance advantage of up to 1%.

(a)

(b)

(c)

Figure 3: Benchmark results with predictions of baseline visits as base
scores.

In this particular case, none of the causal methods outperform the non-
calibrated base scores in terms of effect ordering. In fact, using
experimental data to “improve” the effect ordering leads to inferior results.
Fernández-Loría and Loría (2022) discuss data-generating processes where a
base scoring model can determine the causal-effect ranking without the need
for estimating causal effects. Under such conditions, causal modeling has
little to no potential to improve effect ordering and may even have a negative
impact if the base scoring model is estimated using a significantly larger
amount of data than what is available for causal modeling or causal fine-
tuning.

In contrast, effect estimation can be significantly improved by enhancing the
base scores through causal modeling. As mentioned earlier, non-calibrated
scores are not suitable for effect estimation due to scaling issues. However,
calibrated base scores can lead to substantially better estimates of causal
effects compared to directly estimating the effects using the experimental
data alone.

For example, with a dataset size of 40,000 observations, modeling
heterogeneous treatment effects can reduce the MSE by nearly 60%. However, by
employing effect calibration, we can reduce the MSE by approximately 70%.
Notably, in this dataset, calibrated base scores consistently demonstrate
comparable effect estimation performance to that of using twice as much data
for conventional causal modeling.

This remarkable improvement is achieved through a simple calibration approach
that involves scaling and shifting the base scores. This approach surpasses
causal fine-tuning overall, potentially because the base scores exhibit strong
performance in causal ranking. So, their values can be effectively corrected
through the calibration process, eliminating the need for causal fine-tuning.

Two causal fine-tuning algorithms, EE and EC, demonstrate some improvement in
effect classification when the experimental data is sufficiently large. An
interesting observation here is that while EC does not outperform EO in effect
ordering, it performs better in effect classification. The reason is that EC
is better at ranking the top 10% of individuals with larger effects, whereas
EO performs better in overall individual ranking. This highlights the
potential value of employing fine-tuning methods that are tailored for a
specific causal task. However, it is worth noting that purpose-specific
methods are not guaranteed to dominate, as we showed in our simulation study.
In this dataset, for instance, EE performs comparably to other purpose-
specific methods in tasks different from effect estimation.

###  5.3 Value Metric as the Base Score

Figure 4 shows the results when the base scores correspond to the most
relevant feature f9. This scoring approach is meant to simulate the outcome of
targeting individuals based on an RFM-like metric instead of a predictive
model. Although these scores are not as effective as the predictions of
baseline visits for targeting, they still capture substantial causal
information. These scores, when calibrated (gray line), consistently
outperform causal-effect models in terms of all causal tasks, except for
effect estimation when the experimental data is too small (leftmost chart).

(a)

(b)

(c)

Figure 4: Benchmark results with an RFM-like metric as the base scores.

Causal fine-tuning in this case can improve performance across all causal
tasks when provided with a sufficiently large experimental dataset.
Additionally, we find that purpose-specific methods consistently rank as the
top performer or the second-best performer in the specific causal tasks for
which they were designed.

##  6 Conclusion

Our primary contribution is the introduction of techniques that can enhance
the performance of non-causal models for causal inference based on
experimental data: effect calibration and causal fine-tuning. These techniques
offer two significant advantages over conventional causal modeling.

First, they are compatible with existing scoring models used for decision
making in diverse domains, such as targeted advertising, customer retention,
recommender systems, and precision medicine. Models in these domains often
generate scores that serve as proxies for the magnitude of an intervention
effect on an individual, without being explicitly designed for causal
inference. The benefits of leveraging such scoring models are numerous,
including their versatility in informing various decisions, their access to
comprehensive feature data, and the advantages derived from training them with
large datasets and ample resources. In these contexts, developing a new causal
model for each potential intervention may prove impractical.

However, these scoring models may not be optimally suited for causal inference
due to their inherent non-causal nature. This is precisely where calibration
and fine-tuning emerge as valuable solutions, addressing this limitation by
harnessing experimental data to enhance the performance of the existing
models. These techniques incorporate causal considerations that can help
decision-makers elevate the reliability and accuracy of their scoring models
without the need of a complete overhaul. Causal fine-tuning and effect
calibration thus offer a practical solution to bridge the gap between the
predictive power of non-causal models and the requirements of causal
inference.

The second advantage of these techniques is their ability to outperform
conventional causal modeling approaches. For instance, one alternative
approach is to incorporate the outputs of existing scoring models as
additional features in causal modeling. However, our study reveals that this
approach can be statistically inefficient when the experimental data is
limited and the base scores already capture substantial information about
relative effect sizes. In such scenarios, causal fine-tuning and effect
calibration will tend to outperform conventional causal modeling.

Furthermore, our study demonstrates that the benefits of causal fine-tuning
can be further enhanced when tailored methods for specific causal tasks are
employed. For example, fine-tuning for effect ordering can yield superior
results in effect ordering tasks compared to fine-tuning for effect estimation
or effect classification. We propose three algorithms for causal fine-tuning
to effectively address different causal tasks: effect estimation, effect
ordering, and effect classification. Each algorithm is specifically designed
to optimize performance in its respective causal task.

Nonetheless, multiple avenues for future research remain. First, gaining a
better understanding of the conditions under which causal fine-tuning and
effect calibration outperform conventional causal modeling could inform future
methodological developments and provide further insights into their practical
application. Additionally, exploring alternative algorithms for causal fine-
tuning holds promise. One particularly promising direction is the direct
adjustment of the base scoring model rather than adjusting its outputs as
proposed in this study. Investigating this approach could shed light on new
possibilities and avenues for improving causal inference techniques.

## APPENDIX

##  Appendix A Proof that ranking causal scores based on Equation (6) is
equivalent to ranking them based on Equation (8)

. Minimizing EWM corresponds to

$\displaystyle\mathop{\rm
arg\,min}_{\theta}\mathbb{E}[|\beta-\tau|\cdot\mathbf{1}\\{\beta>\tau\neq\theta>\hat{\tau}\\}]$
$\displaystyle=$ $\displaystyle\mathop{\rm
arg\,min}_{\theta}\mathbb{E}[(\beta-\tau)(\mathbf{1}\\{\beta>\tau\\}-\mathbf{1}\\{\theta>\hat{\tau}\\})]$
$\displaystyle=$ $\displaystyle\mathop{\rm
arg\,min}_{\theta}\mathbb{E}[(\beta-\tau)\mathbf{1}\\{\beta>\tau\\}]-\mathbb{E}[(\beta-\tau)\mathbf{1}\\{\theta>\hat{\tau}\\}]$

Because the first term does not affect how causal scores are ranked, it can be
removed:

$\displaystyle=$ $\displaystyle\mathop{\rm
arg\,max}_{\theta}\mathbb{E}[(\beta-\tau)\mathbf{1}\\{\theta>\hat{\tau}\\}]\phantom{aaaaaaaaaaaaaaaaa}$

Apply definition in Equation 7:

$\displaystyle=$ $\displaystyle\mathop{\rm
arg\,max}_{\theta}\mathbb{E}[(\beta-\tau)\alpha(\theta)]$ $\displaystyle=$
$\displaystyle\mathop{\rm
arg\,max}_{\theta}\mathbb{E}[(Y^{1}-Y^{0}-\tau)\alpha(\theta)]$
$\displaystyle=$ $\displaystyle\mathop{\rm
arg\,max}_{\theta}\mathbb{E}[(Y^{1}-Y^{0}-\tau)\alpha(\theta)]+\mathbb{E}[Y^{0}]$
$\displaystyle=$ $\displaystyle\mathop{\rm
arg\,max}_{\theta}\mathbb{E}[Y^{1}\cdot\alpha(\theta)+Y^{0}\cdot(1-\alpha(\theta))-\tau\cdot\alpha(\theta)]$
$\displaystyle=$ $\displaystyle\mathop{\rm
arg\,max}_{\theta}\mathbb{E}[Y^{\alpha(\theta)}-\alpha(\theta)\tau]\qed$

##  Appendix B Proof that Equation (22) is an unbiased estimator of Equation
(21)

.

$\displaystyle\mathbb{E}_{X,Y,T}\left[\frac{1}{n}\sum_{i=1}^{n}\mathbf{1}\\{\alpha(X)=T\\}\cdot\frac{Y}{\mathbb{P}(T)}\right]$
$\displaystyle=\mathbb{E}_{X,Y,T}\left[\mathbf{1}\\{\alpha(X)=T\\}\cdot\frac{Y}{\mathbb{P}(T)}\right]$
$\displaystyle=\sum_{\forall j\in
T}\mathbb{E}_{X,Y}\left[\mathbf{1}\\{\alpha(X)=j\\}\cdot\frac{Y}{\mathbb{P}(T=j)}~{}\Big{|}~{}T=j\right]\cdot\mathbb{P}(T=j)$
$\displaystyle=\sum_{\forall j\in
T}\mathbb{E}_{X,Y}\left[\mathbf{1}\\{\alpha(X)=j\\}\cdot Y~{}|~{}T=j\right]$

Given the unconfoundedness assumption:

$\displaystyle=\sum_{\forall j\in
T}\mathbb{E}_{X,Y^{j}}\left[\mathbf{1}\\{\alpha(X)=j\\}\cdot Y^{j}\right]$
$\displaystyle=\mathbb{E}_{X,\\{Y^{j}:j\in T\\}}\left[\sum_{\forall j\in
T}\mathbf{1}\\{\alpha(X)=j\\}\cdot Y^{j}\right]$
$\displaystyle=\mathbb{E}\left[Y^{\alpha(X)}\right]$

## Acknowledgements

This work was supported by the Research Grants Council [grant number
26500822].

## References

  * Ascarza et al. (2018) Ascarza E, Neslin SA, Netzer O, Anderson Z, Fader PS, Gupta S, Hardie BG, Lemmens A, Libai B, Neal D, et al. (2018) In pursuit of enhanced customer retention management: Review, key issues, and future directions. _Customer Needs and Solutions_ 5:65–81.
  * Athey and Imbens (2016) Athey S, Imbens G (2016) Recursive partitioning for heterogeneous causal effects. _Proceedings of the National Academy of Sciences_ 113(27):7353–7360.
  * Athey et al. (2023) Athey S, Keleher N, Spiess J (2023) Machine learning who to nudge: causal vs predictive targeting in a field experiment on student financial aid renewal. _arXiv preprint arXiv:2310.08672_ .
  * Dalessandro et al. (2015) Dalessandro B, Hook R, Perlich C, Provost F (2015) Evaluating and optimizing online advertising: Forget the click, but there are good proxies. _Big data_ 3(2):90–102.
  * Devriendt et al. (2020) Devriendt F, Van Belle J, Guns T, Verbeke W (2020) Learning to rank for uplift modeling. _IEEE Transactions on Knowledge and Data Engineering_ 34(10):4888–4904.
  * Diemert et al. (2018) Diemert E, Betlei A, Renaudin C, Amini MR (2018) A large scale benchmark for uplift modeling. _KDD_.
  * Dorie et al. (2019) Dorie V, Hill J, Shalit U, Scott M, Cervone D (2019) Automated versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competition. _Statistical Science_ 34(1):43–68.
  * Fernández-Loría and Loría (2022) Fernández-Loría C, Loría J (2022) Causal scoring: A framework for effect estimation, effect ordering, and effect classification. _arXiv preprint arXiv:2206.12532_ .
  * Fernández-Loría and Provost (2019) Fernández-Loría C, Provost F (2019) Observational vs experimental data when making automated decisions using machine learning. _SSRN preprint abstract_id:3444678_ .
  * Fernández-Loría and Provost (2022) Fernández-Loría C, Provost F (2022) Causal classification: Treatment effect estimation vs. outcome prediction. _Journal of Machine Learning Research_ 23(59):1–35.
  * Fernández-Loría et al. (2023) Fernández-Loría C, Provost F, Anderton J, Carterette B, Chandar P (2023) A comparison of methods for treatment assignment with an application to playlist generation. _Information Systems Research_ 34(2):786–803.
  * Gao et al. (2024) Gao C, Zheng Y, Wang W, Feng F, He X, Li Y (2024) Causal inference in recommender systems: A survey and future directions. _ACM Transactions on Information Systems_ 42(4):1–32.
  * Gordon et al. (2023) Gordon BR, Moakler R, Zettelmeyer F (2023) Close enough? a large-scale exploration of non-experimental approaches to advertising measurement. _Marketing Science_ 42(4):768–793.
  * Kent et al. (2020) Kent DM, Paulus JK, Van Klaveren D, D’Agostino R, Goodman S, Hayward R, Ioannidis JP, Patrick-Lake B, Morton S, Pencina M, et al. (2020) The predictive approaches to treatment effect heterogeneity (path) statement. _Annals of internal medicine_ 172(1):35–45.
  * Künzel et al. (2019) Künzel SR, Sekhon JS, Bickel PJ, Yu B (2019) Metalearners for estimating heterogeneous treatment effects using machine learning. _Proceedings of the national academy of sciences_ 116(10):4156–4165.
  * Leng and Dimmery (2024) Leng Y, Dimmery D (2024) Calibration of heterogeneous treatment effects in randomized experiments. _Information Systems Research_ .
  * Nie and Wager (2021) Nie X, Wager S (2021) Quasi-oracle estimation of heterogeneous treatment effects. _Biometrika_ 108(2):299–319.
  * Peysakhovich and Lada (2016) Peysakhovich A, Lada A (2016) Combining observational and experimental data to find heterogeneous treatment effects. _arXiv preprint arXiv:1611.02385_ .
  * Radcliffe and Surry (2011) Radcliffe NJ, Surry PD (2011) Real-world uplift modelling with significance-based uplift trees. _White Paper TR-2011-1, Stochastic Solutions_ 1–33.
  * Schuler et al. (2018) Schuler A, Baiocchi M, Tibshirani R, Shah N (2018) A comparison of methods for model selection when estimating individual treatment effects. _arXiv preprint arXiv:1804.05146_ .
  * Stitelman et al. (2011) Stitelman O, Dalessandro B, Perlich C, Provost F (2011) Estimating the effect of online display advertising on browser conversion. _Data Mining and Audience Intelligence for Advertising (ADKDD 2011)_ 8\.
  * Wei et al. (2010) Wei JT, Lin SY, Wu HH (2010) A review of the application of rfm model. _African journal of business management_ 4(19):4199.
  * Yadlowsky et al. (2021) Yadlowsky S, Fleming S, Shah N, Brunskill E, Wager S (2021) Evaluating treatment prioritization rules via rank-weighted average treatment effects. _arXiv preprint arXiv:2111.07966_ .
  * Yang et al. (2023) Yang J, Eckles D, Dhillon P, Aral S (2023) Targeting for long-term outcomes. _Management Science_ .
  * Zhang et al. (2012) Zhang B, Tsiatis AA, Davidian M, Zhang M, Laber E (2012) Estimating optimal treatment regimes from a classification perspective. _Stat_ 1(1):103–114.
  * Zhao et al. (2012) Zhao Y, Zeng D, Rush AJ, Kosorok MR (2012) Estimating individualized treatment rules using outcome weighted learning. _Journal of the American Statistical Association_ 107(499):1106–1118.