# Splash! Robustifying Donor Pools for Policy Studies

Jared Amani Greathouse<EMAIL_ADDRESS>
Department of Public Management and Policy
Georgia State University
55 Park Pl NE, Atlanta, GA 30303, USA Mani Bayani<EMAIL_ADDRESS>
Economist, Keystone Strategy
New York, New York, USA Jason Coupet<EMAIL_ADDRESS>
Department of Public Management and Policy
Georgia State University
55 Park Pl NE, Atlanta, GA 30303, USA

###### Abstract

Policy researchers using synthetic control methods typically choose a donor
pool in part by using policy domain expertise so the untreated units are most
like the treated unit in the pre-intervention period. This potentially leaves
estimation open to biases, especially when researchers have many potential
donors. We compare how functional principal component analysis synthetic
control, forward-selection, and the original synthetic control method select
donors. To do this, we use Gaussian Process simulations as well as policy case
studies from West German Reunification, a hotel moratorium in Barcelona, and a
sugar-sweetened beverage tax in San Francisco. We then summarize the
implications for policy research and provide avenues for future work.

##  1 Introduction

Twenty years ago, synthetic control methods (SC/SCM) were developed by [4] for
causal inference in public policy/economic settings as an extension of the
difference-in-differences method (DiD). DiD and other methods commonly make
some variant of the parallel trends assumption for imputing the
counterfactual. Parallel trends posits the average of the untreated units is a
good proxy for the counterfactual of the treated unit. However, this
assumption may be hard to defend in may empirical settings [37]. Thus, SCM
posits a modification to DiD: that a weighted average of certain untreated
unit (donors) is a better proxy for the counterfactual than an average of all
of them. The original synthetic control method (OSC) does this via convex hull
restrictions, which prevents extrapolation bias and encourages a sparse set of
donors getting weight [3, 1]. This makes SCM to be more transparent in
quantifying which units contribute to the counterfactual. For this reason and
others, SCM has grown in use as a tool for policy analysts [22, 3, 16, 18, 38,
14, 17, 15, 47].

###  1.1 Problem Statement

This study is intended to aid policy analysts who are using SCM in the
selection of a donor pool. The validity of SCM is predicated on the donor pool
(or the set of untreated units) used. The donor pool should ideally be as
similar as possible to the treated unit in the pre-intervention period. These
units should also not have undergone similar shocks/interventions. This way,
policy analysts can be more confident that any deviation from the observed
series is the result of the policy change instead of other confounders. But
while the not treated condition can be fairly straightforward, composing a
donor pool that is meaningfully similar to the treated unit, particularly in a
policy context, usually involves some subjective researchers decisions. For
example, suppose a school institutes a new mathematics curriculum: should
donors be restricted based on socioeconomic status– if so, what should the
threshold be (i.e., including only communities with median annual incomes
greater than $70,000 a year)? If so, what in practical terms makes this cutoff
so different from $80,000 a year? Should we only use public schools? The same
district? Should we only include schools in the same state, bordering states,
or even districts in bordering counties from adjacent states? What if a mix of
donor types (i.e., states to regions) made better sense as suggested by [46]?
Should we impose size restrictions, and if so, what would it even mean
empirically to be similarly sized? In the West Germany case for example, [3]
note that interpolation can occur “especially if the donor pool contains units
with characteristics that are very different from those of the [treated]
unit”, and thus opted to restrict their donor pool to 16 similar nations to
West Germany.

In addition to questions about similarity, SCMs might be theoretically invalid
when we have $N>>T$ [5, 10, 46, 39] because they may cause interpolation
biases. [6] note that the risk of interpolation bias is high when $N>>T$
because the SCM may fit to idiosyncratic values of the donor time series.
Interpolation is also possible when we have outlier units or substantial
volatility of the time series. Particularly in a modern world where policy
analysts increasingly leverage large $N$ datasets with more granularity than
countries or states, researchers more frequently encounter settings where
interpolation is a much bigger threat, given the levels of dimensionality [54,
8, 28, 25, 55, 20, 22].111[8] have $\mathcal{T}_{0}=9$ to $J=32$, [54] have
$\mathcal{T}_{0}=22$ to $J=278$, [20] have $\mathcal{T}_{0}=11$ to $J=150$,
[28] have $\mathcal{T}_{0}=10$ to $J=327$, [55] have $\mathcal{T}_{0}=15$ to
$100<J<500$, [25] have $\mathcal{T}_{0}=18$ to $J=64$, and [22] have
$\mathcal{T}_{0}=20$ to $J=345$. We do not suggest that these analyses are
invalid, rather, we only highlight recent cases of analysts encountering $N>T$
settings.

For a visual example of a setting where we have noisy outcomes and potentially
irrelevant donors, consider Fig. 1 derived from the data of [12], who study
the causal impact of Barcelona’s hotel moratorium on the daily price of hotel
rooms throughout the city. The authors have 82 hotel markets across Europe to
compare Barcelona to.

Figure 1: Barcelona, 82 City Donors

We plot the weekly average of hotel room prices. Even though we have a long
pre-intervention time series, it is still clear that many units have prices
far higher or lower than Barcelona. Clearly, we have a complex and noisy time
series, with some units exhibiting considerable variance in their weekly
prices compared to Mediterranean cities or Barcelona itself. This is a setting
where interpolation is possible. By comparison, the original case studies
which used SCM– or the panel data approach (PDA), SCM’s cousin– had much fewer
donors [4, 2, 3, 30].

Whatever the potential source of interpolation bias, the common way of
preventing it is to “judiciously” select a donor pool [1]. Thus, a central
question for researchers using SCM is “Which units belong in the donor pool in
the first place?”. Specifically, [6] advise analysts to “trim” their donors to
be similar to the values of the treated unit in the pre-intervention period–
and this is where our complications begin. While in practice donor trimming
based on outcome/covariate differences is very common [48, 42, 34, 31, 26],
little formalized guidance for donor trimming exists in the policy context.
Thus, trimming is typically based on largely qualitative logic, as different
authors will ultimately have different standards on what exactly constitutes
‘too different’ for many given metrics.222Methods such as trimming via
standard deviation distances have been proposed by [39], but the author offers
no theoretical justification for this metric, and it is unclear how such a
procedure would be valid for larger datasets.

However, what if simple trimming is not straightforward: we might ask how to
trim down the donor pool from Fig. 1. What restrictions upon the predictors or
other criteria should we use to get a more feasible donor pool– do
restrictions on predictors like climate, unemployment, or geographic factors
matter? In [12], the authors have 82 donors, but only a small subset of them
are cities located on the Mediterranean Sea. Should we use only Mediterranean
donors, or is using cities in France and Spain sufficient? Unfortunately, we
cannot even try to answer these questions because we do not know which donors
are which– Booking.com anonymized them to preserve differential privacy.
Furthermore, no covariates about the donors were provided. Here, we simply
can’t make such judgements about what the “right” set of donors is a priori,
as there’s no predictors to other metrics to start from. Given these
constraints (in this example and in many real policy datasets), we argue that
leaving the selection of the donor pool to subjective judgment or mere human
intuition can compromise the robustness of the results.

When machine learning experts extended the SCM toolkit, they did so in part to
address problems from high-dimensions, missing data, noise, and corrupted
data. Some work even addresses interpolation biases [39, 32], where LASSO or
nearest neighbor estimators are used as bias-correction techniques. [10]
develop a robust SCM, employing singular value decomposition and Bayesian
methods to deal with problems such as lack of covariates, missing data, and
noisy outcome series, while generating confidence intervals. Later, this
approach was extended to a tensor/multiple outcome framework [9].

Algorithm 1 Overview of Donor Selection

1:Standard Counterfactual Methods in Policy (SCM, DID)

  * •

Find a treatment of interest.

  * •

Collect a set of outcomes (and possibly covariates) for each unit.

  * •

Drop other treated units/units which underwent shocks to their outcomes.

  * •

Trim donors using covariate similarity/domain expertise.

  * •

Estimate causal effects via the chosen method.

2:Algorithmic Approaches

  * •

Find a treatment of interest.

  * •

Collect a set of outcomes for each unit.

  * •

Use `fsPDA` or `fPCA-SYNTH` to select the most relevant donors.

  * •

Estimate counterfactuals via constrained OLS or another approach.

Even though machine-learning methods have been quintessential in developing
new/different objective functions and optimizations for SCM, a curiosity is
that not much work exists on formalized methods for donor trimming as a
concept.333Work does exist on covariate selection, for example: One extension
by [49] employs LASSO to select a sparse set of optimal predictors of the
outcome from a large set of irrelevant ones. Statistical learning aside, we
also have methods proposed by [21] to mitigate biases from which covariates to
use/pick. Our paper addresses this gap for public policy analysis.

In particular, from the title of our paper, we delve further into how one
might further ‘robustify’ an SCM estimator. The term comes from [7] who
emphasize the value of “first find[ing] a low-dimensional embedding of the
data before fitting a prediction model”. [7] use principal component analysis
(PCA) to estimate the low-dimensional embedding of their donors, and then use
Ridge regression to impute the counterfactual. We extend the idea of
‘robustification’ when applied to donor trimming: we begin by finding the low-
rank structure functional data representation of the pre-intervention
outcomes. We then select the most comparable donors to a treated unit. We
conclude by fitting a prediction model on the low-rank representation of these
most comparable donors. The sketch above highlights how our algorithmic
methods to select donors differs from simple use of intuition.

Specifically, we compare two algorithms: one method we use is functional
principal component analysis (`fPCA`) and the other is the forward selection
panel data approach (`fsPDA`), both recently presented by [11, 46].444In the
first draft of this paper, we considered the pairwise deviation minimization
method proposed by [55]. However, this method has no formalized stopping
mechanism, which would complicate comparisons. We leave this, and other
methods, to future work. The former is a 5 step method which first estimates
the low-rank structure of the pre-intervention time-series and clusters over
them to trim down the donor pool, while the latter uses a simple OLS algorithm
to pre-process the donors. Both approaches are completely based on pre-
intervention outcome trends instead of matching on many predictors, thereby
removing one potential source of subjectivity to used to compare the units.
Additionally, they are valid in cases where we have many more donors than pre-
intervention time periods. Thus, they can alleviate the taxing step of needing
to trim down a massive donor pool to a set of relevant units. Our paper is
motivated by the following empirical applications:

  * –

We revisit and update [3], studying German Reunification.

  * –

We revisit [12] which studied the effect of Barcelona’s 2015 hotel moratorium
on city-wide prices of hotel rooms. How to select the most relevant pool of
control units?

  * –

Causal impact of sugar sweetened beverage taxes on total employment,
revisiting [38].

We first compare both trimming methods to OSC in synthetic studies. Then, we
implement them in the aforementioned empirical case studies. We presume some
familiarity with the SCM and PDA approaches: for a detailed review, see [29,
24] or [50, 1].

Before we continue, we introduce notation. Denote scalars, vectors, and
matrices as $x$, $\mathbf{x}$, and $\mathbf{X}$. Let $\mathbb{R}$ denote the
real number space. Let $\|\cdot\|_{1}$ and $\|\cdot\|_{2}$ denote the vector
norms $L_{1}$ and $L_{2}$ respectively. We are concerned with three sets of
units:
$\mathcal{N}\coloneqq\bigcup\\{\widetilde{\mathcal{N}}_{0},\mathcal{N}_{0},\mathcal{N}_{1}\\}$.
Power set $\mathcal{N}$ is the universe of units, indexed by $j$. Subsets
$\mathcal{N}_{0}$ and $\mathcal{N}_{1}$ are (un)treated units, defined
respectively as $\mathcal{N}_{0}\coloneqq\\{I+1,\ldots,J\\}$ and
$\mathcal{N}_{1}\coloneqq\\{0,\ldots,I\\}$.
$\widetilde{\mathcal{N}}_{0}\subset\mathcal{N}_{0}$ is the set of units
selected by either `fsPDA` or `fPCA-SYNTH`. We observe a time series for each
unit indexed by $t\in\left(1,T\right)\cap\mathbb{N}$. Partition our time
series into sets $\mathcal{T}\coloneqq\mathcal{T}_{0}\cup\mathcal{T}_{1}$ with
their own cardinalities, where $\mathcal{T}_{0}\coloneqq\\{1\ldots T_{0}\\}$
is pre-intervention periods and $\mathcal{T}_{1}\coloneqq\\{T_{0}+1\ldots
T\\}$ denotes post-intervention periods. We also observe outcomes
$\mathbf{Y}\coloneqq\left(y_{jt}\right)_{j\in\mathcal{N},t\in\mathcal{T}}\in\mathbb{R}^{\left(J+1\right)\times
T}$. Treated unit outcomes are $\mathbf{y}_{0}\coloneqq[y_{0t}\ldots
y_{0T}]^{\top}$ and the outcomes for donors are a $T\times J$ matrix
$\mathbf{Y}_{\mathcal{N}_{0}}\coloneqq[y_{1t}\ldots y_{JT}]^{\top}$. An
indicator $d\in\\{0,1\\}$ is the treatment. We assume a linear factor model as
a DGP, where $y^{0}_{jt}$ and $y^{1}_{jt}$ represent the status of the units
being untreated or treated.

######  Assumption 1 (Potential Outcomes).

As described by [44]

$\mathbf{Y}\coloneqq\left.\begin{cases}y^{0}_{jt}=\bm{\mu}_{j}\bm{\delta}_{t}+\bm{\theta}_{t}\mathbf{Z}_{j}+\bm{\epsilon}_{jt},\quad
d=0\\\ y^{1}_{jt}=\tau_{jt}+y^{0}_{jt},\quad d=1\\\ \end{cases}\right\\}.$

Here, $y^{0}_{jt}$ represents the DGP for all units prior to treatment. If the
unit is untreated, we presume the pre-intervention factor model holds. Also,
this model prohibits spillover effects of the treatment. Here
$\bm{\mu}_{j}=\bigr{[}\mu_{j1},\ldots\mu_{jr}\bigr{]}^{\top}$ and
$\bm{\delta}_{t}=\bigr{[}\delta_{1t},\ldots\delta_{rt}\bigr{]}^{\top}$ are
$r\times 1$ factor loadings and common factors, $\bm{\theta}_{t}$ are $1\times
k$ coefficients for $k\times 1$ $\mathbf{Z}_{j}$ predictor effects, and
$\bm{\epsilon}_{jt}$ represents Gaussian noise [23, Chapter 8]. The
fundamental difficulty of causal inference is that we observe
$y_{jt}=d_{jt}y_{jt}^{1}+(1-d_{jt})y_{jt}^{0}$, where we cannot simultaneously
observe a unit being treated and untreated– thus, our goal is to impute the
counterfactual $\hat{y}_{jt}$. Our treatment effect is simply
$\tau_{jt}\coloneqq y_{jt}-\hat{y}_{jt}$, where we are typically interested in
$\\{\tau_{0T_{0+1}},\ldots,\tau_{0T_{1}}\\}$. We also make two additional
assumptions

######  Assumption 2 (Existence of Weights).

$\exists^{\geq
2}\bm{\omega}^{\ast}\,\text{s.t.}\,\bm{\omega}^{\ast}\mathbf{y}_{\mathcal{N}_{0}}^{\prime}\approx\mathbf{y}_{0t\in\mathcal{T}_{0}}$

Namely, we presume a $J\times 1$ vector of unit weights
$\bm{\omega}^{\ast}=\left(w_{1}\ldots w_{J}\right)$ will generate a SC,
$\hat{y}_{jt}\coloneqq\bm{\omega}^{\ast}\mathbf{y}_{\mathcal{N}_{0}}^{\prime}$,
such that the counterfactual approximates the observed trajectory of the
treated unit in the pre-intervention period. These weights should also match
on the common factors which drive the factor loadings from Assumption 1.

######  Assumption 3 (Factor Conditions).

There are also factor conditions on variation

$\frac{1}{T_{0}}\sum_{t\in\mathcal{T}_{0}}\bm{\delta}^{\top}_{t}\bm{\delta}_{t}\rightarrow\bm{\Omega}\succ
0$ (1)

and conditions on alpha mixing

$\delta(n)=\sup_{A,B}\left|\mathbb{P}(\mathbf{\delta}_{t}\in
A\cap\mathbf{\delta}_{t+n}\in B)-\mathbb{P}(\mathbf{\delta}_{t}\in
A)\mathbb{P}(\mathbf{\delta}_{t+n}\in B)\right|$ (2)

Note that the factor loadings embedded in the donor outcomes are a good proxy
to match on the pre-intervention time series of the treated unit [for more on
this point, see [29]. However, because we do not observe these loadings, we
use the time variant outcomes of donors to match on these loadings. We presume
these common factors are alpha mixing, or that there is weak correlation
between them such that severe serial correlation won’t affect our results, and
that there is enough variation in the common factors to match on the
unobserved factor loadings. The trimming methods are meant to select some
optimal subset of donors, $\widetilde{\mathcal{N}}_{0}\subset\mathcal{N}_{0}$,
such that we maximize the similarities of the donor pool to the treated unit,
thereby automating the trimming process. Our causal estimand and diagnostic of
interest are:

$\displaystyle\text{ATT}=\frac{1}{\mathcal{T}_{1}}\sum_{j\in\mathcal{N}_{1}}\tau_{jt}$
$\displaystyle\text{RMSE}=\sqrt{\frac{1}{\mathcal{T}_{0}}\sum_{t\in\mathcal{T}_{0}}\left(\tau_{jt}\right)^{2}}.$
(3)

or, the post-intervention average treatment effect on the treated and the pre-
intervention root-mean squared prediction error.

##  2 Donor Trimming and Counterfactual Estimation

First, we review the trimming algorithms. Then, we review the SC and PDA
approaches to estimation at a broad level.

###  2.1 Forward Selection and Functional PCA

Forward selection was recently studied by [46, 36] as a viable method of donor
trimming for causal inference. Here is a short sketch of it:

Algorithm 2 Forward Selection

1:Initialize $\widetilde{\mathcal{N}}_{0}=\emptyset$.

2:Initialize $r$ as the iteration index.

3:Regress $y_{0t\in\mathcal{T}_{0}}$ upon each $y_{jt\in\mathcal{T}_{0}}$

4:Store $\widetilde{\mathcal{N}}_{0}\coloneqq
j_{r}\leftarrow\underset{j\in\mathcal{N}\setminus\widetilde{\mathcal{N}}_{0}}{\operatorname{arg\,max}}\,R^{2}(\mathcal{N}_{r-1}\cup\\{\hat{j}\\})$.

5:Update $\widetilde{\mathcal{N}}_{0}=\mathcal{N}_{r-1}\cup{\hat{j_{r}}}$

6:Repeat until $r>R$.

where $R$ is the stopping time hyperparameter to be tuned. [46] used a
modified BIC [51] to estimate this hyperparameter:

$\hat{R}=\operatorname*{arg\,min}_{r\in\mathbb{N}}\left\\{\log\left(\hat{\sigma}_{\widetilde{\mathcal{N}}_{0}}^{2}+\log\log
N\cdot r\bigl{(}\textrm{log }T_{1}\bigr{)}/T_{1}\right)\right\\}.$ (4)

In this method, we basically regress the outcome variable of the treated unit
on the outcome variable of all donor members over the pre-intervention time
period. We then store the $R^{2}$ statistics from each estimation, and retain
the unit in $\widetilde{\mathcal{N}}_{0}$ with maximum $R^{2}$. We then repeat
this process again, until there’s no significant improvement in the
prediction.

Aside from linear regression or Euclidean distance as advocated by [6], one
viable method to extract a donor pool is $k$-means clustering [41]. $k$-means
clustering finds cluster centers indexed by $i$
$\\{\mathbf{C}=c_{1},c_{2},\ldots,c_{k}\\}$, defined as

$c_{k}=\frac{1}{{\left|{y_{i}}\right|}}\mathop{\sum}\limits_{{y\in Y_{i}}}y$
(5)

such that ${\bigcup}_{i=1}^{k}y_{j}=\mathbf{Y}_{t\in\mathcal{T}_{0}}$ and
$y_{i}\cap y_{j}=\emptyset$. Suppose $\mathbf{G}=[g_{tk}]_{N\times C}$ is an
indicator matrix, where $g_{tk}\in\\{0,1\\}$ denoting, $\forall j\in N$ if
$y_{t}$ belongs in a given cluster. The solution to k-means clustering
minimizes the within-cluster variance

$\operatorname*{minimize}_{\mathbf{C},\mathbf{G}}\|\mathbf{Y}-\mathbf{C}\mathbf{G}^{\top}\|_{2}^{2}$
(6)

Note the maximum number of clusters allowed is a hyperparameter in the K-means
algorithm. To avoid manually choosing of the number of clusters, we use
Silhouette statistics to ascertain the optimal number of clusters [43],
computed via

$\displaystyle s(i)=\begin{cases}1-\frac{b(i)}{w(i)}&\text{if }b(i)<w(i)\\\
0&\text{if }b(i)=w(i)\\\ \frac{w(i)}{b(i)}-1&\text{if }b(i)>w(i)\end{cases}$
(7)

After the clustering, we keep the cluster with the treated unit. The new donor
pool is $\widetilde{\mathcal{N}}_{0}=C\setminus\\{c_{i}\mid j=0\\}$, or all of
the units in the $i$th cluster except for the treated unit. Some papers in
recent years have adopted this exact approach to trim donors [35, 45].

However, clustering on raw data in policy or economic settings can be very
misleading, particularly when dealing with $N>>T$ or $T>>N$ settings. The
reason for this is that there may be outlier units (as a whole unit or at
specific points in time), noise, and many irrelevant units– what’s more,
standard clustering over the raw data might detect incidental similarities
between units (and cause overfitting). We are interested in the broader
temporal correlations between the treated unit and donor units [40], and do
not wish to fit to noise. So, to address this problem, the first step of
`fPCA-SYNTH` is to reduce the data’s dimensionality via functional Principal
Component Analysis (fPCA) before clustering.

One of the main limitations of standard PCA as used in [7] is that “the data
points on each curve are assumed to be independent of each other, but in
reality it is known that any point on a continuous time-series is correlated
with the data points that precede and follow that point” [53]. Thus, a main
advantage of fPCA over standard PCA is that accounts for temporal correlation
within the data points [52] via functional basis splines while reducing the
data dimensionality. Hence, the functional PCA scores preserve the temporal
correlation among unit observations. Once we have the low-dimensional
representation of the outcomes matrix, we may now k-means cluster to extract
our donor pool. We would like to highlight some differences between these
algorithms: forward selection is a greedy parametric algorithm in that it
seeks to maximize the $R^{2}$ statistic to predict the treated pre-
intervention series, and has no method of discarding donors once they’ve been
selected. Thus while forward selection performs regularization, it may suffer
from overfitting biases as discussed in [36], who solves the overfitting
probem by using difference-in-differences. In contrast, `fPCA-SYNTH` is non-
parametric in nature. Because the clustering is done over the fPCA scores,
there is less of a concern for overfitting.

###  2.2 Estimating Synthetic Weights

Broadly, SCMs and PDAs follow a minimization of the treated unit and the pre-
intervention donor matrix

$\underbrace{\mathbf{y}_{0}}_{\mathcal{T}_{0}\times
1}\coloneqq\begin{bmatrix}y_{01}\\\ y_{02}\\\ \vdots\\\
y_{0T_{0}}\end{bmatrix},\underbrace{\mathbf{Y}_{\mathcal{N}_{0}}}_{\mathcal{T}_{0}\times
J}\coloneqq\begin{bmatrix}y_{11}&y_{21}&\dots\\\ \vdots&\ddots&\\\
y_{1T_{0}}&&y_{JT_{0}}\end{bmatrix}$ (8)

The SC from any selected pool is denoted as
$\hat{y}_{jt}\coloneqq\bm{\omega}^{\ast}\mathbf{y}_{\widetilde{\mathcal{N}}_{0}}^{\prime}$.
OSC solves

$\displaystyle\underset{\omega}{\operatorname{arg\,min}}$
$\displaystyle\quad||\mathbf{y}_{0}-\mathbf{Y}_{\mathcal{N}_{0}}\omega_{j}||_{2}^{2}$
s.t.
$\displaystyle\quad\Delta^{J-1}=\left\\{\bm{\omega}:\omega_{j}\in\mathbb{I}\,\text{and}\,\|\bm{\omega}\|_{1}=1\right\\}$

where the weights lie on the simplex $\Delta^{J-1}$ and are constrained to be
within $\mathbb{I}\coloneqq[0,1]\subset\mathbb{R}$. Practically, constraints
force the weights to not allow predictions outside of the donor pool the user
supplies. Oftentimes for OSC, analysts use additional predictors to improve
pre-intervention fit, where a matrix of importance weights are computed

$\hat{\mathbf{V}}=\underset{V\in\mathcal{V}}{\text{arg
min}}\left(\mathbf{y}_{0}-\mathbf{Y}_{\mathcal{N}_{0}}{\hat{\mathbf{W}}}(\mathbf{V})\right)^{\prime}\left(\mathbf{y}_{0}-\mathbf{Y}_{\mathcal{N}_{0}}{\
\hat{\mathbf{W}}}(\mathbf{V})\right)$ (9)

However, the addition of covariates does not by itself mitigate interpolation
biases in the large $N$ setting [5], or if there are irrelevant donors/noisy
outcomes.

With `fPCA-SYNTH`, the donors are already correlated with the treated unit
over time via the clustering. However, we still wish to find a low-rank
representation of the donor-pool to estimate the counterfactual with.555See
Appendix A for more detail on low-rank estimation as well as additional
technical details. To do this, we use Robust PCA (RPCA) scores. RPCA scores
are obtained via the solution to the following optimization —s— L, S∥L∥_* +
λ∥S∥_1 Y = L \+ S. where $\mathbf{L}$ and $\mathbf{S}$ are signal and noise
matrices of the pre-intervention time-series. $\mathbf{L}$ and $\mathbf{S}$
are computed via the Alternating Direction Method of Multipliers with the
augmented Lagrangian, where the former matrix emphasizes the pre-intervention
temporal patterns and the latter shrinks the influence of the $\mathbf{S}$
matrix with each iteration. We then use $\mathbf{L}$ (again in the pre-
intervention period) to estimate the unit weights for the SC, as this matrix
represents the patterns of the donors in the pre-intervention period. Since
the units are already correlated, we no longer need to use the convex hull
restriction from OSC. RPCA-SYNTH solves the weights using convex cone
restrictions ω——y_0-L_~N_0ω_j——^2_2 ω:ω∈R_≥0 ∀ j ∈~N_0. We expect the
estimation of `fPCA-SYNTH` to be less sensitive to outliers and noise compared
to OSC, since it does not formally partition the observations using low-rank
techniques. Forward Selection PDA estimates the counterfactual as follows
using OLS. ω——y_0-Y_~N_0ω_j——^2_2 ω:ω∈R ∀ j ∈~N_0. It uses its optimal donors
as obtained from its donor trimming algorithm.

##  3 Synthetic Studies

The goal of the simulations and empirical analyses are to illustrate the
utility of algorithmic donor trimming. Thus, while we also study the treatment
effect and pre-treatment fit, our simulations have the additional demand of
picking the optimal donor pool. We meet this challenge with Gaussian Processes
(GP). GPs define a distribution over functions. More formally, GPs are
collections of random variables $\mathbf{x}\in\mathbb{R}^{d}$ where a finite
set $p(f(\mathbf{x}_{1}),\ldots,f(\mathbf{x}_{N}))$ is derived from a joint
Gaussian distribution
$p(\mathbf{f}\lvert\mathbf{x})=\mathcal{N}(\mathbf{f}\lvert\bm{\mu},\mathbf{K})$.
A GP is completely specified by its mean function $m(x)$ (defining its time
trend) and covariance function $k(x,x^{\prime})$ (defining temporal
shape/smoothness). $x$ and $x^{\prime}$ are points in the input space, denoted
as $f(x)\sim\mathcal{GP}(m(x),k(x,x^{\prime}))$. GPs are versatile approaches
to generating complex panel data distributions which might involve non-linear
relationships and non-stationary patterns. The final set of processes/donor
pools looks like

$\displaystyle\begin{bmatrix}GP(x_{1})\\\ \vdots\\\ GP(x_{n})\end{bmatrix}\sim
N\left(\ \begin{bmatrix}m(x_{1})\\\ \vdots\\\ m(x_{n})\end{bmatrix}\ ,\
\begin{bmatrix}k(x_{1},x_{1})&\cdots&k(x_{1},x_{m})\\\ \vdots&\cdots&\vdots\\\
k(x_{m},x_{1})&\cdots&k(x_{m},x_{m})\end{bmatrix}\ \right).$ (10)

Figure 2: Graph of Units

Kernel functions are used to define the covariance function in GPs. The first
is the Radial Basis Function (RBF) Kernel:
$k(x,x^{\prime})=\text{exp}\left(-||x-x^{\prime}||^{2}\mathbin{/}2l^{2}\right)$,
to define the mean. The second is the Constant Kernel which scales the overall
amplitude of the function $k(x,x^{\prime})=\kappa$. We use the Exponential
Sine Squared Kernel to model the periodicity in the time series:
$k(x,x^{\prime})=\text{exp}\left(-2*\text{sin}^{2}\left(\left[\pi||x-x^{\prime}||\mathbin{/}p\right]\mathbin{/}l^{2}\right)\right)$.
The white kernel models the white noise, denoted as: $\sigma_{n}^{2}\text{ if
}x\equiv x^{\prime},0\mbox{ otherwise.}$

We generate a synthetic high-dimensional dataset of 160 units. We generate two
donor pools, each with 80 units per pool. In the treated pool, we select one
unit to be treated. Fig. 2 shows the treated pool of units (the blue pool) and
a set of much noisier, irrelevant donors (the grey pool). We generate these
data across 40 time periods, with the hypothetical intervention occurring at
$t=30$. For simplicity, we do not assign a treatment effect, or an ATT of 0 in
this case. We report the pre and post-intervention Root-Mean Squared Errors
for each algorithm, and discuss the donors selected by each approach.

Figure 3: Results of Synthetic Studies

The fPCA method finds 112 units to be the most relevant to the treated unit.
The SC is constructed of only 5 units (all of which are from the relevant data
generating process). The forward selection method performs much worse,
selecting 23 donors, 9 of which are from the unrelated donor pool. OSC selects
11 units, 5 of which are irrelevant. fsPDA and the original synthetic control
method suggest ATTs of 1.7 and -1.4 respectively. `fPCA-SYNTH` has an ATT of
0.093 (the lowest in absolute value), even though it fits the worst of the
methods(1.9 vs 1.4 and 0 for OSC and fsPDA, respectively).

Following [10], we may use the post-intervention RMSE as a metric for
forecasting quality. Precisely, we take the ratio of the pre and post RMSE for
each method

$Ratio=\frac{\sum_{t=T_{0}+1}^{T}\left(y_{jt}-\hat{y}_{jt}\right)^{2}}{\sum_{t=1}^{T_{0}}\left(y_{jt}-\hat{y}_{jt}\right)^{2}}$
(11)

The ratio for `fPCA-SYNTH` is 0.92, for OSC is 3, and for forward selection is
566. Of these, fsPDA has the highest ratio because it clearly overfits the
preintervention period, so as a result we have in poor out of sample
performance. OSC has the next highest (also having the second best pre-
intervention fit) ratio, suggesting that it also overfits to the pre-
intervention period. The simulation results highlight the benefits of
preprocessing and denoising: in situations where there are linear trends (as
we use here), noise, and many donors to choose from, the OSC won’t match on
the units which are most representative of the target unit’s data generating
process. Having methods which are robust to outliers, trendy time series, and
noise are of use to policymakers because they offer safeguards against
overfitting biases which can draw the wrong conclusions.

##  4 Empirical Application

###  4.1 West Germany

First we revisit the West Germany example by [3], who study how West Germany
reunifying with its Eastern counterpart affected the GDP per capita of West
Germany. Before we continue though, we must lay some essential ground rules
for this exercise: this study will not be a perfect replication of [3] simply
because the exact outcome they use (as far as we are aware) is not
available.[3] use “GDP per Capita (PPP, 2002 USD)” from OECD National Account
statistics. As far as we know, the only data OECD keeps now for GDP per Capita
is for 2011 dollars. In absence of the original dataset with all of the
excluded donors, we use the newest data available from the Madison Project
[13].

The authors begin with 23 OECD nations, discarding Luxembourg and Iceland due
to their small and ‘peculiar’ economies. They also discard Turkey due to small
GDP per Capita relative to the other OECD nations. Finally, the authors
discard Ireland, Finland, Sweden, and Canada as they all experienced
structural shocks to their economies in the 1990s, finally limiting their
donor pool to 16 OECD nations. [3] write, however, their results are robust to
the inclusion of Ireland, Finland, Sweden, and Canada. So, we replicate these
analyses using the original 16 donors, while also adding in the 4 nations [3]
excluded (we even add Turkey), giving us 21 donors.666Note that our results,
with the exception of using all 23 donors, are also robust to including either
Iceland or Luxembourg (in addition to Turkey). We do not include predictors in
the OSC estimator (the original paper used seven predictors).

As before, we first estimate the counterfactual using `fPCA-SYNTH` and OSC (we
omit the forward selection algorithm for clarity). Fig. 4 plots the observed
trajectory of West Germany, and we overlay the four counterfactuals we
estimate. In the plot, “OECD Donors” refers to the original 16 donors used by
[3]. The “Full OECD Donors” means Ireland, Finland, Sweden, Canada, and Turkey
were included.

Pool | Method | NOR | FR | AUSTR | UK | JP | NL | AUS | PORT | TURK | US | GRC
---|---|---|---|---|---|---|---|---|---|---|---|---
OECD Donors | PCASC | 0.277 |  |  |  |  | 0.557 | 0.122 |  |  |  |
OECD Donors | CVX | 0.089 | 0.12 | 0.282 | 0.051 | 0.167 | 0.115 | 0.039 | 0.078 |  |  | 0.058
Full OECD Donors | PCASC | 0.177 |  | 0.142 | 0.029 |  | 0.366 |  |  |  | 0.195 |
Full OECD Donors | CVX | 0.091 | 0.002 | 0.294 |  | 0.2 | 0.091 |  |  | 0.17 |  | 0.152
Table 1: Germany Weights

We can see that the weights selected by `fPCA-SYNTH` are Australia,
Netherlands and Norway [all of which were also selected into the donor pool in
[11]]. When we include all of the donors, it weighs Austria, Netherlands,
Norway, United Kingdom, and United States. Notice that all of these with the
exception of the United States were also included as donors in [11]. The
imputed counterfactual is also robust to the exclusion of the United States.
OSC paints a different picture: substantively, the results also suggest that
the GDP per capita would otherwise be higher, however, the effect sizes are
double those of `fPCA-SYNTH`. Furthermore, the OSC results are sensitive to
the inclusion of the other irrelevant donors, even matching on Turkey (which
[3] note has a particularly low GDP per Capita). While it’s true that `fPCA-
SYNTH` includes the United States, RPCA algorithm used by `fPCA-SYNTH` guards
against the drastic changes in effect sizes.

Figure 4: Germany Replication

The results, even in medium dimensions, illustrate the utility of a donor
trimming algorithm. OSC is sensitive to changes in the donor pool, and `fPCA-
SYNTH` is less so, even getting the same effect size while including the
United States, which was not included in the cluster in [11].

###  4.2 Barcelona

In July 2015, the city of Barcelona, Spain implemented a moratorium for the
building of new hotels, ostensibly because over-tourism had deleterious social
and economic impacts to the surrounding environment. [12] study the
counterfactual of Barcelona’s average prices for hotel rooms, using daily pre-
intervention data from January 2011 to June 30th 2015 to construct the
counterfactual. Here, we estimate the causal impacts of Barcelona’s hotel
moratorium. It is tempting to follow [3] and restrict our donors to
Mediterranean cities. However, Barcelona’s average price is almost always
higher than the average of Mediterranean donors, so it is possible (if not
likely) that Barcelona could be better approximated by some non-Mediterranean
donors. In other words, it is an arbitrary assumption we make that only
Mediterranean cities would be the ideal donor set- given how cities like
London and others are in the donor pool, we elect to use the universe of units
instead of simply limiting our units to the Mediterranean.

Our outcome is the mean price of hotel rooms in each city, normalized to 100.
Here, $j=0$ is Barcelona and $\mathcal{N}_{0}=82$ represents the universe of
donors. While data from [12] are in daily format, we use the average weekly
price as our outcome. The observation window is from week 1 of January 2011 to
week 31 (August 7th) 2017.

Figure 5: Barcelona, Real and Placebo

After applying fPCA to all pre-intervention weeks, the cluster for Barcelona
retained 14 donors, or $\widetilde{\mathcal{N}}_{0}=13$. Of this subset, 9
donors, or 64%, were Mediterranean cities. For the weights, 4 donor units were
given weight. Of these, 3 were from the Mediterranean Sea donor pool. For
`fs`, 11 donors were chosen. Of those 11, 8 were Mediterranean. For OSC, 3 of
the 7 donors were Mediterranean. Notice how these methods, particularly the
PCA and fs method, appeal very nicely to intuition. Since [12] already
hypothesized Mediterranean Donors were more likely to be similar to Barcelona,
the fact fPCA and forward-selection both choose donor pools with majority
Mediterranean donors reassures us the methods are picking up on donors most
related to Barcelona. Equally, the fact we get this result without explicitly
adjusting for other predictors of room prices is doubly reassuring for the
validity of both methods. In contrast, while OSC has good fit, most of its
donors are not Mediterranean. Because SCM is justified on a latent variable
model, the counterfactual must also match on unobserved confounding, instead
of simply fitting to the pre-intervention period or covariate balance. Since
we would imagine that at least many of the weighed units would be
Mediterranean, this suggests the algorithmic approaches are more theoretically
justified than OSC in setting where there are many irrelevant units. Note,
that all methods used here agree in terms of the direction of the average
treatment effects on the treated: the moratorium seems to have raised the
prices of hotel rooms, on average.

Method | Intervention | ATT_per | ATT | RMSE | PostRMSE | Ratio
---|---|---|---|---|---|---
CVX | Main Intervention | 13.045 | 15.906 | 3.355 | 24.583 | 7.327
CVX | Placebo Intervention | 3.131 | 3.549 | 1.973 | 17.291 | 8.764
PCASC | Main Intervention | 11.372 | 12.45 | 5.295 | 16.84 | 3.18
PCASC | Placebo Intervention | 1.225 | 1.371 | 3.346 | 10.184 | 3.044
fsPDA | Main Intervention | 9.137 | 10.047 | 2.614 | 14.081 | 5.387
fsPDA | Placebo Intervention | -5.49 | -6.546 | 1.441 | 11.789 | 8.181
Table 2: Mediterranean Study

Another important point when evaluating the estimators is robustness to
changes in the intervention date, to see how well the SC tracks the observed
series. In this case, we change the intervention date to week 27 of 2011, a
full 4 years before the intervention. Here, `fPCA-SYNTH` assigns two weights
of the 19 units it selects as optimal, one of which is Mediterranean (OSC gets
the same unit). OSC’s weights in this case are actually 57% Mediterranean.
Noticeably, the forward selection method has the most drastic change in the
ATT, with the sign flipping completely, and departing far from the observed
time series, meeting it only at the end of the time series. By comparison,
`fPCA-SYNTH`’s Post to Pre RMSE ratio changes the least compared to the other
two methods. Its synthetic Barcelona tracks the observed Barcelona until the
intervention date, even though the effect size differs from the first
intervention date. The fact that its predictive capacity in terms of Post/Pre
Ratio changes so little, being so close to the start of the time series, while
retaining a similar proportion of theoretically relevant donors suggests that
it is more robust to changes in the model than the OSC method.

###  4.3 Sugar Sweetened Beverage Tax

Globally, sugar sweetened beverage (SSB) taxes, like other forms of excise
taxes, have become popular economic policies which are intended to increase
public health benefits without simply banning consumption of certain goods.
Industry argues SSB taxes will have negative effects on employment, since now
fewere people will be able to work in their warehouses/industries that are
predicated on SSB revenue. So, the causal evaluation of SSB taxes on
consumption of SSBs and secondary economic impacts has become popular among
economists and policy analysts [e.g., [33]]. [38] study the causal impact of
San Francisco’s SSB tax on various forms of employment, finding a very small
impact on employment levels. We replicate their analyses using the estimators
presented so far.

The authors limit their donor pool to 62 urban counties around the United
States, using SCM to evaluate the causal impact. They also have 62 months of
pre-intervention data, with the intervention beginning on January 1, 2018.
Precisely, our outcome is the total number of people employed in each
city/county equivalent. The full dataset in [38] consists of 93 donors. So, we
keep all 93 donors. Another key deviation from our analysis is that unlike
[38] who use many covariate predictors, we use none.

Figure 6: Results of SSB Study Method | Intervention | ATT_per | ATT | RMSE | PostRMSE | Ratio
---|---|---|---|---|---|---
CVX | Main Intervention | 4.345 | 32.625 | 17.867 | 37.213 | 2.083
CVX | Placebo Intervention | 27.376 | 203.821 | 159.161 | 204.755 | 1.286
PCASC | Main Intervention | 0.065 | 0.485 | 2.67 | 2.604 | 0.975
PCASC | Placebo Intervention | -0.27 | -2.016 | 2.693 | 3.295 | 1.224
fsPDA | Main Intervention | -0.813 | -5.647 | 1.39 | 7.061 | 5.08
fsPDA | Placebo Intervention | -0.805 | -5.591 | 1.651 | 8.043 | 4.872
Table 3: SSB Study

The weights assigned by `fPCA-SYNTH` are: ’Riverside County’: 0.096, ’San
Benito County’: 4.264, ’Denver County’: 0.049, ’Hillsborough County’: 0.138,
’St. Tammany Parish’: 1.924, ’Kings County’: 0.171, ’Wake County’: 0.122,
’Hays County’: 1.005, ’Kendall County’: 2.187. For context, Riverside and San
Benito counties are located in Combined Statistical Areas in California. Wake,
Denver, Hillsborough, and Kings counties are home to Raleigh/Durham, Denver,
Tampa, and Seattle, all major major American cities. For forward selection,
the OLS coefficients are ’Travis County’: 8690.0, ’St. Tammany Parish’: 0.788,
’Hartford County’: 2.57, ’Baltimore city’: -0.715, ’Charles County’: 0.71,
’Placer County’: 3.42, ’District of Columbia’: 0.829, ’Essex County’: -0.231.

Regarding the OSC, we’d like to highlight a mistake we made during the writing
of this paper: at first, we’d mistakenly coded the intervention as being in
February 2018 instead of January. The pre-intervention RMSE for OSC in this
case was 163.725 and the post-intervention error was 208.755. In other words,
the line did not fit the pre-intervention period at all. When we corrected
this, the pre-RMSE is now 17.867 and post-error is 37.213. While a marked
improvement from the specification with the miscoded intervention, we
highlight, as others have, how the OSC can be quite sensitive to small tweaks
in the model, suggesting its estimates are less robust. Most notably, this
also suggests the covariates the authors included originally (7 predictors, 5
county specific and 2 industry specific) can be quintessential to estimating
counterfactuals with OSC. For a more realistic in-time placebo test, we report
a 6 month placebo test, where we move the intervention to June of 2017 instead
of 2018.

As we can see, all of the other methods (especially `fPCA-SYNTH`) are quite
robust to a longer backdate in the treatment, whereas OSC is not. Note that
this does not mean the original analyses are wrong or incorrect. Our methods
still obtain the main result, that the effect of the SSB tax on employment was
quite small. We only echo the comments of others [11, 10] who remark that that
the need for covariates to obtain convergence and good fit is a limitation of
the original method. Particularly for empirical applications, this need for
auxillary predictors can sometimes not be at all trivial. Thus, public policy
analysts might prefer methods which both automate the selection of a donor
pool while obtaining similar effects.

Now we turn to the estimates with the actual intervention date. `fsPDA`
detects a -0.813% effect, and `fPCA-SYNTH` finding a 0.065% average impact on
employment. The fact that these methods (without predictors) reach
qualitatively identical conclusions of the original paper, while at the same
time selecting mostly large metropolitan areas as donors suggests that in many
cases (such as Washington DC or San Benito County), donor trimming methods or
additional advances to the standard SCM may be of use to policy analysts for
their robust properties, particularly when there are multiple donors and
little covariate information to use for the standard SCM.

##  5 Discussion and Conclusion

In this study, we delve into the significance of algorithmic methods for
selecting the optimal donor pool for SCM. The challenge of accurately
identifying an appropriate control group stands at the forefront of causal
inference, given that the control group’s validity is the basis for the causal
studies. The importance of donor pool selection is amplified within the SCM
context due to the classic model’s vulnerability to interpolation bias,
particularly when we have many donors and/or trendy/noisy outcome time-series.
Policy analysts oftentimes employ domain expertise or Euclidean distance
measures to select donors: our research underscores the necessity of adopting
modern machine learning approaches for this task. Put simply, theory is not a
substitute for formal algorithmic analysis: if we think that most of the
relevant donors should come from a given pool of units, then evidence from
unsupervised algorithms should suggest this.

Particularly as the use of big data to solve policy problems that require
causal inference grows [27], so too should the causal methodologies used to do
analysis. Particularly when analysts are faced with many potential donors,
algorithmic selection of a donor pool is useful to researchers who use SCMs.
We think our results should leave policy analysts and empirical researchers
with a few major takeaways. First, researchers can use these methods (and
others such as dynamic time-warping) when the research context provides lots
of degrees of freedom with regard to donor pool selection, or if a particular
set of SCM results is sensitive to the composition of the donor pool. So, if
leave-one-out analyses are not robust, one viable method would be to use
clustering or another method to extract the optimal donors and re-run original
SCM analyses. Algorithmic donor trimming reduces the role of arbitrary
research decisions.

Our work has limitations, too. Because policy research increasingly
substitutes synthetic control for DiD designs [19], having a better
understanding of the exact conditions under which an algorithm selects a valid
set of donors is useful. In this paper, we use Gaussian Processes and three
empirical cases– however, formal finite sample analyses of the asymptotic
properties of k-means clustering or similar algorithms may expand this
untapped area of the literature. Given recent finite sample studies of the
$k$-means algorithms and SCMs [57, 58, 56], more formalized asymptotic study
of algorithmic donor trimming methods may be warranted.

## References

  * [1] Alberto Abadie  “Using Synthetic Controls: Feasibility, Data Requirements, and Methodological Aspects”  In _Journal of Economic Literature_ 59.2, 2021, pp. 391–425  DOI: 10.1257/jel.20191450
  * [2] Alberto Abadie, Alexis Diamond and Jens Hainmueller  “Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program”  In _Journal of the American Statistical Association_ 105.490 Taylor & Francis, 2010, pp. 493–505  DOI: 10.1198/jasa.2009.ap08746
  * [3] Alberto Abadie, Alexis Diamond and Jens Hainmueller  “Comparative Politics and the Synthetic Control Method”  In _American Journal of Political Science_ 59.2, 2015, pp. 495–510  DOI: 10.1111/ajps.12116
  * [4] Alberto Abadie and Javier Gardeazabal  “The Economic Costs of Conflict: A Case Study of the Basque Country”  In _American Economic Review_ 93.1, 2003, pp. 113–132  DOI: 10.1257/000282803321455188
  * [5] Alberto Abadie and Jérémy L’Hour  “A Penalized Synthetic Control Estimator for Disaggregated Data”  In _Journal of the American Statistical Association_ 116.536 Taylor & Francis, 2021, pp. 1817–1834  DOI: 10.1080/01621459.2021.1971535
  * [6] Alberto Abadie and Jaume Vives-i-Bastida  “Synthetic Controls in Action”  arXiv, 2022  DOI: 10.48550/ARXIV.2203.06279
  * [7] Anish Agarwal, Devavrat Shah, Dennis Shen and Dogyoon Song  “On Robustness of Principal Component Regression”  In _Journal of the American Statistical Association_ 116.536 Taylor & Francis, 2021, pp. 1731–1745  DOI: 10.1080/01621459.2021.1928513
  * [8] José M Alonso and Rhys Andrews  “Government-Created Nonprofit Organizations and Public Service Turnaround: Evidence from a Synthetic Control Approach”  In _Journal of Public Administration Research and Theory_ 31.2, 2020, pp. 346–362  DOI: 10.1093/jopart/muaa035
  * [9] Muhammad Amjad, Vishal Misra, Devavrat Shah and Dennis Shen  “mrsc: Multi-Dimensional Robust Synthetic Control”  In _Proc. ACM Meas. Anal. Comput. Syst._ 3.2 New York, NY, USA: Association for Computing Machinery, 2019  DOI: 10.1145/3341617.3326152
  * [10] Muhammad Amjad, Devavrat Shah and Dennis Shen  “Robust synthetic control”  In _The Journal of Machine Learning Research_ 19.1 JMLR. org, 2018, pp. 802–852
  * [11] Mani Bayani  “Robust PCA Synthetic Control”  arXiv, working paper, 2021  DOI: 10.48550/ARXIV.2108.12542
  * [12] Germà Bel, Stephan Joseph and Ferran A. Mazaira-Font  “The effects of moratoriums on hotel building: an anti-tourism measure, or rather protection for local incumbents?”  In _Applied Economics Letters_ 29.14 Routledge, 2022, pp. 1319–1324  DOI: 10.1080/13504851.2021.1927958
  * [13] Jutta Bolt and Jan Luiten Van Zanden  “Maddison style estimates of the evolution of the world economy. A new 2020 update”  In _Maddison-Project Working Paper WP-15, University of Groningen, Groningen, The Netherlands_ , 2020
  * [14] Bernardo García Bulle, Dennis Shen, Devavrat Shah and Anette E. Hosoi  “Public health implications of opening National Football League stadiums during the COVID-19 pandemic”  In _Proceedings of the National Academy of Sciences_ 119.14, 2022, pp. e2114226119  DOI: 10.1073/pnas.2114226119
  * [15] Lindsey Rose Bullinger  “Child support and the Affordable Care Act’s Medicaid expansions”  In _Journal of Policy Analysis and Management_ 40.1 Wiley Online Library, 2021, pp. 42–77  DOI: 10.1002/pam.22238
  * [16] Gabriela Calderón, Gustavo Robles, Alberto Díaz-Cayeros and Beatriz Magaloni  “The Beheading of Criminal Organizations and the Dynamics of Violence in Mexico”  In _Journal of Conflict Resolution_ 59.8, 2015, pp. 1455–1485  DOI: 10.1177/0022002715587053
  * [17] Ezra Cohn et al.  “The effect of a proof-of-vaccination requirement, incentive payments, and employer-based mandates on COVID-19 vaccination rates in New York City: a synthetic-control analysis”  In _The Lancet Public Health_ 7.9, 2022, pp. e754–e762  DOI: 10.1016/S2468-2667(22)00196-7
  * [18] Matthew A Cole, Robert JR Elliott and Bowen Liu  “The impact of the Wuhan Covid-19 lockdown on air pollution and health: a machine learning and augmented synthetic control approach”  In _Environmental and Resource Economics_ 76.4 Springer, 2020, pp. 553–580  DOI: 10.1007/s10640-020-00483-4
  * [19] Scott Cunningham  “Has Diff in Diff Lowered the Price on Synth?”, 2023  URL: https://causalinf.substack.com/p/has-diff-in-diff-lowered-the-price
  * [20] Dhaval Dave, Drew McNichols and Joseph J Sabia  “Political violence, risk aversion, and population health: Evidence from the US Capitol riot”  In _Journal of population economics_ 35.4 Springer, 2022, pp. 1345–1384  DOI: 10.1007/s00148-022-00914-0
  * [21] Bruno Ferman, Cristine Pinto and Vitor Possebom  “Cherry Picking with Synthetic Controls”  In _Journal of Policy Analysis and Management_ 39.2, 2020, pp. 510–532  DOI: 10.1002/pam.22206
  * [22] Marco Francesconi and Jonathan James  “Alcohol Price Floors and Externalities: The Case of Fatal Road Crashes”  In _Journal of Policy Analysis and Management_ 41.4, 2022, pp. 1118–1156  DOI: 10.1002/pam.22414
  * [23] Christophe Gaillac and Jérémy L’Hour  “Machine Learning for Economists”, 2023
  * [24] Javier Gardeazabal and Ainhoa Vega-Bayo  “An Empirical Comparison Between the Synthetic Control Method and HSIAO et al.’s Panel Data Approach to Program Evaluation”  In _Journal of Applied Econometrics_ 32.5, 2017, pp. 983–1002  DOI: 10.1002/jae.2557
  * [25] Stuart Gietel-Basten, Xuehui Han and Yuan Cheng  “Assessing the impact of the “one-child policy” in China: A synthetic control approach”  In _PLOS ONE_ 14.11 Public Library of Science, 2019, pp. 1–17  DOI: 10.1371/journal.pone.0220170
  * [26] Kevin Grier, Towhid Mahmood and Benjamin Powell  “Anti-sweatshop activism and the safety-employment tradeoff: Evidence from Bangladesh’s Rana Plaza disaster”  In _Journal of Economic Behavior & Organization_ 208, 2023, pp. 174–190  DOI: https://doi.org/10.1016/j.jebo.2023.02.007
  * [27] Justin Grimmer  “We are all social scientists now: How big data, machine learning, and causal inference work together”  In _PS: Political Science & Politics_ 48.1 Cambridge University Press, 2015, pp. 80–83
  * [28] Peiming He, Jiaming Zhang and Litai Chen  “Time is money: Impact of China-Europe Railway Express on the export of laptop products from Chongqing to Europe”  In _Transport Policy_ 125, 2022, pp. 312–322  DOI: 10.1016/j.tranpol.2022.06.010
  * [29] Alex Hollingsworth and Coady Wing  “Tactics for design and inference in synthetic control studies: An applied example using high-dimensional data”  In _working paper_ , 2022  DOI: 10.31235/osf.io/fc9xt
  * [30] Cheng Hsiao, H. Steve Ching and Shui Ki Wan  “A PANEL DATA APPROACH FOR PROGRAM EVALUATION: MEASURING THE BENEFITS OF POLITICAL AND ECONOMIC INTEGRATION OF HONG KONG WITH MAINLAND CHINA”  In _Journal of Applied Econometrics_ 27.5, 2012, pp. 705–740  DOI: 10.1002/jae.1230
  * [31] Sydney Kahmann, Erin Hartman, Jorja Leap and P. Brantingham  “Impact evaluation of the LAPD community safety partnership”  In _The Annals of Applied Statistics_ 16.2 Institute of Mathematical Statistics, 2022, pp. 1215–1235  DOI: 10.1214/21-AOAS1543
  * [32] Maxwell Kellogg, Magne Mogstad, Guillaume A. Pouliot and Alexander Torgovitsky  “Combining Matching and Synthetic Control to Tradeoff Biases From Extrapolation and Interpolation”  In _Journal of the American Statistical Association_ 116.536 Taylor & Francis, 2021, pp. 1804–1816  DOI: 10.1080/01621459.2021.1979562
  * [33] Sungjin Kim, Clarence Lee and Sachin Gupta  “Bayesian Synthetic Control Methods”  In _Journal of Marketing Research_ 57.5, 2020, pp. 831–852  DOI: 10.1177/0022243720936230
  * [34] Michał Marcin Kobierecki and Michał Pierzgalski  “Sports Mega-Events and Economic Growth: A Synthetic Control Approach”  In _Journal of Sports Economics_ 23.5, 2022, pp. 567–597  DOI: 10.1177/15270025211071029
  * [35] Jongkwan Lee  “The Role of a University in Cluster Formation: Evidence from a National Institute of Science and Technology in Korea”  In _Regional Science and Urban Economics_ 86, 2021, pp. 103617  DOI: 10.1016/j.regsciurbeco.2020.103617
  * [36] Kathleen T. Li  “A Novel Forward Difference-in-Differences Method”, 2022  DOI: 10.2139/ssrn.4213705
  * [37] Kathleen T. Li and Christophe Van den Bulte  “Augmented Difference-in-Differences”  In _Marketing Science_ 0.0, 2023, pp. null  DOI: 10.1287/mksc.2022.1406
  * [38] Samantha Marinello, Julien Leider and Lisa M. Powell  “Employment impacts of the San Francisco sugar-sweetened beverage tax 2 years after implementation”  In _PLOS ONE_ 16.6 Public Library of Science, 2021, pp. 1–16  DOI: 10.1371/journal.pone.0252094
  * [39] Robert McClelland and Livia Mucciolo  “An update on the synthetic control method as a tool to understand state policy”  In _Tax Policy Center_ , 2022
  * [40] Mamta Mittal, Lalit M. Goyal, Duraisamy Jude Hemanth and Jasleen K. Sethi  “Clustering approaches for high-dimensional databases: A review”  In _WIREs Data Mining and Knowledge Discovery_ 9.3, 2019, pp. e1300  DOI: 0.1002/widm.1300
  * [41] P Olukanmi, F Nelwamondo and Tshilidzi Marwala  “Rethinking k-means clustering in the age of massive datasets: a constant-time approach”  In _Neural Computing and Applications_ 32 Springer, 2020, pp. 15445–15467
  * [42] Linan Peng and Justin T. Callais  “The authoritarian trade-off: A synthetic control analysis of development and social coercion in the Xinjiang Uyghur Autonomous Region”  In _Contemporary Economic Policy_ 41.2, 2023, pp. 370–387  DOI: 10.1111/coep.12597
  * [43] Peter J. Rousseeuw  “Silhouettes: A graphical aid to the interpretation and validation of cluster analysis”  In _Journal of Computational and Applied Mathematics_ 20, 1987, pp. 53–65  DOI: 10.1016/0377-0427(87)90125-7
  * [44] Donald B. Rubin  “[On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9.] Comment: Neyman (1923) and Causal Inference in Experiments and Observational Studies”  In _Statistical Science_ 5.4 Institute of Mathematical Statistics, 1990, pp. 472–480  DOI: 10.1214/ss/1177012032
  * [45] Niklas Rudholm, Yujiao Li and Kenneth Carling  “How does big-box retail entry affect labor productivity in durable goods retailing? A synthetic control approach”  In _The Annals of Regional Science_ 69.1 Springer, 2022, pp. 89–117  DOI: 10.1007/s00168-022-01110-4
  * [46] Zhentao Shi and Jingyi Huang  “Forward-selected panel data approach for program evaluation”  In _Journal of Econometrics_ 234.2, 2023, pp. 512–535  DOI: 10.1016/j.jeconom.2021.04.009
  * [47] Junxiu Sun, Feng Wang, Haitao Yin and Bing Zhang  “Money talks: the environmental impact of China’s green credit policy”  In _Journal of Policy Analysis and Management_ 38.3 Wiley Online Library, 2019, pp. 653–680  DOI: 10.1002/pam.22137
  * [48] Michael Thom  “Can additional funding improve mental health outcomes? Evidence from a synthetic control analysis of California’s millionaire tax”  In _PLOS ONE_ 17.7 Public Library of Science, 2022, pp. 1–17  DOI: 10.1371/journal.pone.0271063
  * [49] Jaume Vives-i-Bastida  “Predictor Selection for Synthetic Controls”  arXiv, 2023  DOI: 10.48550/ARXIV.2203.11576
  * [50] Shui-Ki Wan, Yimeng Xie and Cheng Hsiao  “Panel data approach vs synthetic control method”  In _Economics Letters_ 164, 2018, pp. 121–123  DOI: 10.1016/j.econlet.2018.01.019
  * [51] Hansheng Wang, Bo Li and Chenlei Leng  “Shrinkage Tuning Parameter Selection with a Diverging number of Parameters”  In _Journal of the Royal Statistical Society Series B: Statistical Methodology_ 71.3, 2009, pp. 671–683  DOI: 10.1111/j.1467-9868.2008.00693.x
  * [52] John Warmenhoven et al.  “PCA of waveforms and functional PCA: A primer for biomechanics”  In _Journal of Biomechanics_ 116, 2021, pp. 110106  DOI: https://doi.org/10.1016/j.jbiomech.2020.110106
  * [53] John Warmenhoven et al.  “Considerations for the use of functional principal components analysis in sports biomechanics: examples from on-water rowing”  In _Sports Biomechanics_ 18.3 Routledge, 2019, pp. 317–341  DOI: 10.1080/14763141.2017.1392594
  * [54] Xiaoxuan Yang  “Does city lockdown prevent the spread of COVID-19? New evidence from the synthetic control method”  In _Global Health Research and Policy_ 6.1 Springer, 2021, pp. 1–14  DOI: 10.1186/s41256-021-00204-4
  * [55] Lamont Yu, Trang Tran and Wang-Sheng Lee  “Revitalising the Silk Road: Evidence from Railway Infrastructure Investments in Northwest China”  IZA Discussion Paper, 2022
  * [56] Jakob Zeitler, Athanasios Vlontzos and Ciaran M. Gilligan-Lee  “Non-parametric identifiability and sensitivity analysis of synthetic control models”, 2023  arXiv:2301.07656 [stat.ME]
  * [57] Tonglin Zhang  “Asymptotics for The $k$-means”, 2022  arXiv:2211.10015 [stat.ML]
  * [58] Xiaomeng Zhang, Wendun Wang and Xinyu Zhang  “Asymptotic Properties of the Synthetic Control Method”, 2022  DOI: 10.48550/arXiv.2211.12095