# Can You Rely on Your Model Evaluation? Improving Model Evaluation with
Synthetic Test Data

Boris van Breugel
University of Cambridge
<EMAIL_ADDRESS>&Nabeel Seedat∗
University of Cambridge
<EMAIL_ADDRESS>Fergus Imrie
UCLA
<EMAIL_ADDRESS>&Mihaela van der Schaar
University of Cambridge
<EMAIL_ADDRESS>Equal Contribution

###### Abstract

Evaluating the performance of machine learning models on diverse and
underrepresented subgroups is essential for ensuring fairness and reliability
in real-world applications. However, accurately assessing model performance
becomes challenging due to two main issues: (1) a scarcity of test data,
especially for small subgroups, and (2) possible distributional shifts in the
model’s deployment setting, which may not align with the available test data.
In this work, we introduce 3S Testing, a deep generative modeling framework to
facilitate model evaluation by generating synthetic test sets for small
subgroups and simulating distributional shifts. Our experiments demonstrate
that 3S Testing outperforms traditional baselines—including real test data
alone—in estimating model performance on minority subgroups and under
plausible distributional shifts. In addition, 3S offers intervals around its
performance estimates, exhibiting superior coverage of the ground truth
compared to existing approaches. Overall, these results raise the question of
whether we need a paradigm shift away from limited real test data towards
synthetic test data.

###  1 Introduction

Motivation. Machine learning (ML) models are increasingly deployed in high-
stakes and safety-critical areas, e.g. medicine or finance—settings that
demand reliable and measurable performance [1]. Failure to rigorously test
systems could result in models at best failing unpredictably and at worst
leading to silent failures. Regrettably, such failures of ML are all too
common [2, 3, 4, 5, 6, 7, 8, 9]. Many mature industries involve standardized
processes to evaluate performance under various testing and operating
conditions [10]. For instance, automobiles use wind tunnels and crash tests to
assess specific components, whilst electronic component data sheets outline
conditions where reliable operation is guaranteed. Unfortunately, current
evaluation approaches of supervised ML models do not have the same level of
detail and rigor.

The prevailing testing approach in ML is to evaluate only using average
prediction performance on a held-out test set. This can hide undesirable
performance differences on a more granular level, e.g. for small subgroups [2,
11, 12, 4, 5], low-density regions [13, 14, 15], and individuals [16, 17, 18].
Standard ML testing also ignores distributional shifts. In an ever-evolving
world where ML models are employed across borders, failing to anticipate
shifts between train and deployment data can lead to overestimated real-world
performance [6, 7, 8, 19, 20].

_However, real test data alone does not always suffice for more detailed model
evaluation._ Indeed, testing can be done on a granular level by evaluating on
individual subgroups, and in theory, shifts could be tested using e.g.
rejection or importance sampling of real test data. The main challenge is that
_insufficient amounts of test data_ cause _inaccurate_ performance estimates
[21]. In Sec. 3 we will further explore why this is the case, but for now let
us give an example.

###### Example 1

Consider the real example of estimating model performance on the Adult
dataset, looking at race and age variables—see Fig. 1 and experimental details
in Appendix B. There are limited samples of the older Black subgroup, leading
to significantly erroneous performance estimates compared to an oracle.
Similarly, if we tried to engineer a distribution shift towards increased age
using rejection sampling, the scarcity of data would yield equally imprecise
estimates. Such imprecise performance estimates could mislead us into drawing
false conclusions about our model’s capabilities. In Sec. 5.1, we empirically
demonstrate how synthetic data can rectify this shortfall.

Figure 1: Limitation of testing with real data (Adult dataset): Insufficient
real test samples result in imprecise performance estimates when (i) testing
underrepresented subgroups (e.g. Black) and (ii) testing effect of a shift
(e.g increased age)

Aim. Our goal is to build a model evaluation framework with synthetic data
that allows engineers, auditors, business stakeholders, or policy and
compliance teams to understand better when they can rely on the predictions of
their trained ML models and where they can improve the model further. We
desire the following properties for our evaluation framework:
(P1) Granular evaluation: accurately evaluate model performance on a granular
level, even for regions with few test samples.
(P2) Distributional shifts: accurately assess how distribution shifts affect
model performance.

Our primary focus is tabular data. Not only are many high-stakes applications
predominately tabular, such as credit scoring and medical forecasting [22,
23], but the ubiquity of tabular data in real-world applications also presents
opportunities for broad impact. To put it in perspective, nearly 79% of data
scientists work with tabular data on a daily basis, dwarfing the 14% who work
with modalities such as images [24].

Moreover, tabular data presents us with interpretable feature identifiers,
such as ethnicity or age, instrumental in defining minority groups or shifts.
This contrasts with other modalities, where raw data is not interpretable and
external (tabular) metadata is needed.

Contributions. ① _Conceptually_ , we show why real test data may not suffice
for _model evaluation_ on subgroups and distribution shifts and how synthetic
data can help (Sec. 3). ② _Technically_ , we propose the framework 3S-Testing
(s.f. Synthetic data for Subgroup and Shift Testing) (Sec. 4). 3S uses
conditional deep generative models to create synthetic test sets, addressing
both (P1) and (P2). 3S also accounts for possible errors in the generative
process itself, providing uncertainty estimates for its predictions via a deep
generative ensemble (DGE) [25]. ③ _Empirically_ , we show synthetic test data
provides a more accurate estimate of the true model performance on small
subgroups compared to baselines, including real test data (Sec. 5.1.1), with
prediction intervals providing good coverage of the real value (Sec. 5.1.2).
We further demonstrate how 3S can generate data with shifts which better
estimate model performance on shifted data, compared to real data or
baselines. 3S accommodates both minimal user input (Sec. 5.2.1) or some prior
knowledge of the target domain (Sec. 5.2.2).

###  2 Related Work

This paper primarily engages with the literature on model testing and
benchmarking, synthetic data, and data-centric AI—see Appendix A for an
extended discussion.

Model evaluation. ML models are mostly evaluated on hold-out datasets,
providing a measure of aggregate performance [26]. Such aggregate measures do
not account for under-performance on specific subgroups [2] or assess
performance under data shifts [7, 27].

The ML community has tried to remedy these issues by creating better benchmark
datasets: either manual corruptions like Imagenet-C [28] or by collecting
additional real data such as the Wilds benchmark [8]. Benchmark datasets are
labor-intensive to collect and evaluation is limited to specific benchmark
tasks, hence this approach is not flexible for _any_ dataset or task. The
second approach is model behavioral testing of specified properties, e.g. see
Checklist [29] or HateCheck [30]. Behavioral testing is also labor-intensive,
requiring humans to create or validate the tests. In contrast to both
paradigms, 3S generates synthetic test sets for varying tasks and datasets.

Challenges of model evaluation. 3S aims to mitigate the challenges of model
evaluation with limited real test sets, particularly estimating performance
for small subgroups or under distributional shifts. We are not the first to
address this issue. $\blacksquare$ Subgroups: Model-based metrics (MBM [21])
model the conditional distribution of the predictive model score to enable
subgroup performance estimates.
$\blacksquare$ Distribution shift: Prior works aim to predict model
performance in a shifted target domain using (1) Average samples above a
threshold confidence (ATC [31]), (2) difference of confidences (DOC [32]), (3)
Importance Re-weighting (IM [33]). A fundamental difference to 3S is that they
assume access to _unlabeled data_ from the target domain, which is unavailable
in many settings, e.g. when studying potential effects of unseen or future
shifts. We note that work on robustness to distributional shifts is not
directly related, as the goal is to learn a model robust to the shift, rather
than reliably estimating performance of an already-trained model under a
shift.

Synthetic data. Improvements in deep generative models have spurred the
development of synthetic data for different uses [34], including privacy (i.e.
to enable data sharing, 35, 36), fairness [37, 38], and improving downstream
models [39, 40, 41, 42]. 3S provides a completely _different_ and unexplored
use of synthetic data: improving _testing and evaluation_ of ML models.
Simulated (CGI-based) and synthetic images have been used previously in
computer vision (CV) applications for a variety of purposes — often to _train_
more robust models [43, 44]. These CV-based methods require additional
metadata like lighting, shape, or texture [45, 46], which may not be available
in practice. Additionally, beyond the practical differences between
modalities, the CV methods differ significantly from 3S in terms of (i) aim,
(ii) approach, and (iii) amount of data—see Table 4, Appendix A.

###  3 Why Synthetic Test Data Can Improve Evaluation

####  3.1 Why Real Data Fails

Notation. Let $\mathcal{X}$ and $\mathcal{Y}$ be the feature and label space,
respectively. The random variable $\tilde{X}=(X,Y)$ is defined on this space,
with distribution $p(X,Y)$. We assume access to a trained black-box prediction
model $f:\mathcal{X}\rightarrow\mathcal{Y}$ and test dataset
$\mathcal{D}_{test,f}=\\{x_{i},y_{i}\\}_{i=1}^{N_{test,f}}\overset{iid}{\sim}~{}p(X,Y)$.
Importantly, we do _not_ assume access to the training data of the predictive
models, $\mathcal{D}_{train,f}$. Lastly, let
$M:\mathcal{Y}\times\mathcal{Y}\rightarrow\mathbb{R}$ be a performance metric.

Real data does not suffice for estimating granular performance (P1). In
evaluating performance of $f$ on subgroups, we assume that a subgroup
$\mathcal{S}\subset\mathcal{X}$ is given. The usual approach to assess
subgroup performance is simply restricting the test set $\mathcal{D}_{test,f}$
to the subspace $\mathcal{S}$:

$A(f;\mathcal{D}_{test,f},\mathcal{S})=\frac{1}{\mathcal{D}_{test,f}\cap\mathcal{S}}\sum_{(x,y)\in\mathcal{D}_{test,f}\cap\mathcal{S}}M(f(x),y).$
(1)

This is an unbiased and consistent estimate of the true performance, i.e. for
increasing $|\mathcal{D}_{test,f}|$ this converges to the true performance
$A^{*}(f;p,\mathcal{S})=\mathbb{E}[M(f(X),Y)|(X,Y)\in\mathcal{S}]$.

However, what happens when $|\mathcal{D}_{test,f}\cap\mathcal{S}|$ is small?
The variance $\mathbb{V}_{\mathcal{D}_{test,f}\sim
p}A(f,\mathcal{D}_{test,f};\mathcal{S})$ will be large. In other words, the
expected error of our performance estimates becomes large.

###### Example 2

To highlight how this affects Eq. 1, let us assume the very simple setting in
which $p(Y|X)=p(Y)$. Despite $Y$ being independent of $X$, Eq. 1 is not—the
smaller $\mathcal{S}$, the higher the expected error in our estimate. In
particular, for $|\mathcal{S}|\rightarrow 0$ (w.r.t. $p(X)$), almost certainly
$|\mathcal{D}_{test,f}\cap\mathcal{S}|=\emptyset$, making Eq. 1 meaningless.

As a result, we find that the smaller our subgroup $\mathcal{S}$, the harder
it becomes to measure model $f$. At the same time, ML models have been known
to perform less consistently on small subgroups [2, 3, 4, 5], hence being able
to measure performance on these groups would be most useful. Finally, by
definition, minorities are more likely to form these small subgroups, and they
are the most vulnerable to historical bias and resulting ML unfairness. In
other words, model evaluation is most inaccurate on the groups that are most
vulnerable and for which the model _itself_ is most unreliable.

Real test data fails for distributional shift (P2). If we do not take into
account shifts between the test and deployment distribution, trivially the
test performance will be a poor measure for real-world performance—often
leading to overestimated performance [6, 7, 8]. Nonetheless, even if we do
plan to consider shifts in our evaluation, for example by using importance
weighting or rejection sampling based on our shift knowledge, real test data
will give poor estimates. The reason is the same as before; in the regions
that we oversample or overweight, there may be few data points, leading to
high variance and noisy estimates. As expected, problems are most pervasive
for large shifts, because these require higher reweighting or oversampling of
individual points.

####  3.2 Why Generative Models Can Help

(a) Original data

(b) With synthetic data

Figure 2: Illustration why synthetic data can give more accurate estimates
using the same real samples. Assume we want to evaluate $f$ (decision
boundary$=$dashed line), which aims to discriminate between $Y=1$ (green
stars) and $Y=0$ (red circles). Due to the low number of samples for $Y=1$,
evaluating $f$ using the test set alone (Eq. 1) has a high variance. On the
other hand, a generative model can learn the manifold from
$\mathcal{D}_{test,f}$, and generate additional data for $Y=1$ by only
learning the offset (b, green triangles). This can reduce variance of the
estimated performance of $f$.

We have seen there are two problems with real test data. Firstly, the more
granular a metric, the higher the noise in Eq. 1—even if the distribution is
well-behaved like in Example 2. Secondly, we desire a way to emulate shifts,
and simple reweighing or sampling of real data again leads to noisy estimates.

Generative models can provide a solution to both problems. As we will detail
in the next section, instead of using $\mathcal{D}_{test,f}$, we use a
generative model $G$ trained on $\mathcal{D}_{test,f}$ to create a large
synthetic dataset $\mathcal{D}_{syn}$ for evaluating $f$. We can induce shifts
in the learnt distribution, thereby solving problem 2. It also solves the
first problem. A generative model aims to approximate $p(X,Y)$, which we can
regard as effectively interpolating $p(Y|X)$ between real data points to
generate more data within $\mathcal{S}$.

It may seem counterintuitive that $|A^{*}-A(f;\mathcal{D}_{syn},\mathcal{S})|$
would ever be lower than $|A^{*}-A(f;\mathcal{D}_{test,f},\mathcal{S})|$,
after all $G$ is trained on $|\mathcal{D}_{test,f}|$ and there is no new
information added to the system. However, even though the generative process
may be imperfect, we will see that the noise of the generative process can be
significantly lower than the noisy real estimates (Eq. 1). Secondly, a
generative model can learn implicit data representations [39], i.e. learn
relationships within the data (e.g. low-dimensional manifolds) from the entire
dataset and transfer this knowledge to small $\mathcal{S}$. We give a toy
example in Fig. 2(b). This motivates modeling the full data distribution
$p(X,Y)$, not just $p(Y|X)$.

Of course, synthetic data cannot always help model evaluation, and may in fact
induce noise due to an imperfect $G$. Through the inclusion of uncertainty
estimates, we promote trustworthiness of results (Sec 4.1), and when we
combine synthetic data with real data, we observe almost consistent benefits
(see Sec. 5). In Section 6 we include limitations.

###  4 Synthetic Data for Subgroup and Shift Testing

####  4.1 Using Deep Generative Models for Synthetic Test Data

We reiterate that our goal is to generate test datasets that provide insight
into model performance on a granular level (P1) and for shifted distributions
(P2). We propose using synthetic data for testing purposes, which we refer to
as _3S_ -testing. This has the following workflow (Fig. 3): (1) train a
(conditional) generative model on the real test set, (2) generate synthetic
data conditionally on the subgroup or shift specification, and (3) evaluate
model performance on the generated data, $A(f;\mathcal{D}_{syn},\mathcal{S})$.
This procedure is flexible w.r.t. the generative model, but a conditional
generative model is most suitable since it allows precise generation
conditioned on subgroup or shift information out-of-the-box. Throughout this
paper we use CTGAN [47] as the generative model—see Appendix C for other
generative model results and more details on the generative training process.

(a) (P1) Reliable granular evaluation

(b) (P2) Sensitivity to distributional shifts

Figure 3:  3S is a framework for evaluating model performance using synthetic
data generators. It has three phases: training the generative model,
generating synthetic data, and model evaluation. Firstly, 3S enables (P1)
_granular evaluation_ when there is (i) limited real test data in small
subgroups, by (ii) generating synthetic data conditional on subgroup
information $X_{c}$, thereby (iii) permitting more reliable model evaluation
even on small subgroups. Secondly, 3S enables assessment of (P2)
_distributional shifts_ when (i) the real test data does not reflect shifts,
by (ii) generating synthetic data conditional on marginal shift information of
features $X_{c}$, thereby (iii) quantifying model sensitivity to
distributional shift. Required inputs are denoted in yellow.

Estimating uncertainty. Generative models are not perfect, leading to
imperfect synthetic datasets and inaccurate 3S estimates. To provide insight
into the trustworthiness of its estimates, we quantify the uncertainty in the
3S generation process through an ensemble of generative models [25], similar
in vain to Deep Ensembles [48]. We (i) initialize and train $K$ generative
models independently, (ii) generate synthetic datasets
$\\{\mathcal{D}_{syn}^{k}\\}_{k=1}^{K}$, and (iii) evaluate model $f$ on each.
The final estimates are assumed Gaussian, with statistics given by the sample
mean and variance:

$\hat{\mu}(A)=\frac{1}{K}\sum_{k=1}^{K}A(f;\mathcal{D}_{syn}^{k},\mathcal{S})\text{
and
}\hat{\sigma}^{2}(A)=\frac{1}{K-1}\sum_{k=1}^{K}(A(f;\mathcal{D}_{syn}^{k},\mathcal{S})-\hat{\mu}(A))^{2},$
(2)

which can be directly used for constructing a prediction interval. In Sec.
5.1.2, we show this provides high empirical coverage of the true value
compared to alternatives.

Defining subgroups. The actual definition of subgroups is flexible. Examples
include a specific category of one feature (e.g. female), intersectional
subgroups [49] (e.g. black, female), slices from continuous variables (e.g.
over 75 years old), particular points of interest (e.g. people similar to
patient X), and outlier groups. In Appendix E, we elaborate on some of these
further.

####  4.2 Generating Synthetic Test Sets with Shifts

Distributional shifts between training and test sets are not unusual in
practice [7, 50, 51] and have been shown to degrade model performance [6, 8,
15, 52]. Unfortunately, often there may be no or insufficient data available
from the shifted target domain.

Defining shifts. In some cases, there is prior knowledge to define shifts. For
example, covariate shift [53, 54] focuses on a changing covariate distribution
$p(X)$, but a constant label distribution $p(Y|X)$ conditional on the
features. Label (prior probability) shift [55, 54] is defined vice versa, with
fixed $p(X|Y)$ and changing $p(Y)$.111Concept drifts are beyond the scope of
this work.

Generalizing this slightly, we assume only the marginal of some variables
changes, while the distribution of the other variables conditional on these
variables does not. Specifically, let $c\subset\\{1,...,|\tilde{X}|\\}$ denote
the indices of the features or targets in $\tilde{X}$ of which the marginal
distribution may shift. Equivalent to the covariate and label shift
literature, we assume the distribution $p(\tilde{X}_{\bar{c}}|\tilde{X}_{c})$
remains fixed ($\bar{c}$ denoting the complement of $c$).222This reduces to
label shift—$p(X|Y)$ constant but $p(Y)$ changed)—and covariate shift—$p(Y|X)$
constant but $p(X)$ changed)—for $\tilde{X}_{c}=Y$ and $\tilde{X}_{c}=X$,
respectively. Let us denote the marginal’s shifted distribution by
$p^{s}(\tilde{X}_{c})$ with $s$ the shift parameterisation, with
$p^{0}(\tilde{X}_{c})$ having generated the original data. The full shifted
distribution is $p(\tilde{X}_{\bar{c}}|\tilde{X}_{c})p^{s}(\tilde{X}_{c})$.

Example: single marginal shift. Without further knowledge, we study the
simplest such shifts first: only a single $\tilde{X}_{i}$’s marginal is
shifted. Letting $p^{0}(\tilde{X}_{i})$ denote the original marginal, we
define a family of shifts $p^{s}(\tilde{X}_{i})$ with $s\in\mathbb{R}$ the
shift magnitude. To illustrate, we choose a mean shift for continuous
variables, $p^{s}(\tilde{X}_{i})=p^{0}(\tilde{X}_{i}-s)$, and a logistic shift
for any binary variable, $\text{logit }p^{s}(\tilde{X}_{i}=1)=\text{logit
}(p^{s}(\tilde{X}_{i}))-s$.333We consider any categorical variable with $m$
classes using $m$ different shifts of the individual probabilities, scaling
the other probabilities appropriately. As before, we assume $p(\tilde{X}_{\neg
i}|\tilde{X}_{i})$ remains constant. This can be repeated for all $i$ and
multiple $s$ to characterize the sensitivity of the model performance to
distributional shifts. The actual shift can be achieved using any conditional
generative model, with the condition given by $\tilde{X}_{i}$.

Incorporating prior knowledge on shift. In many scenarios, we may want to make
stronger assumptions about the types of shift to consider. Let us give two use
cases. First, we may acquire high-level statistics of some variables in the
target domain—e.g. we may know that the age in the target domain approximately
follows a normal distribution $\mathcal{N}(50,10)$. In other cases, we may
actually acquire data in the target domain for some basic variables (e.g. age
and gender), but not all variables. In both cases, we can explicitly use this
knowledge for sampling the shifted variables $\tilde{X}_{c}$, and subsequently
generating $\tilde{X}_{\bar{c}}|\tilde{X}_{c}$—e.g. sample (case 1) age from
$N(50,10)$ or (case 2) (age, gender) from the target dataset. Variables
$\tilde{X}_{\bar{c}}|\tilde{X}_{c}$ are generated using the original generator
$G$, trained on $\mathcal{D}_{test,f}$.

Characterizing sensitivity to shifts. This gives the following recipe for
testing models under shift. Given some conditional generative model $G$, we
(i) train $G$ to approximate $p(X_{\bar{c}}|X_{c})$, (2) choose a shifted
distribution $p^{s}(\tilde{X}_{c})$—e.g. a marginal mean shift of the original
$p^{0}(X_{c})$ (Section 5.2.1), or drawing $x_{c}$ samples from a secondary
dataset (Section 5.2.2); (3) draw samples $x_{c}$, and subsequently use $G$ to
generate the rest of the variables $X_{\bar{c}}$ conditional on these drawn
samples—together giving$\mathcal{D}_{syn}^{s}$; and (4) evaluate downstream
models; (5) Repeat (2-4) for different shifts (e.g. shift magnitudes $s$) to
characterize the sensitivity of the model to distributional shifts.

More general shifts. Evidently, marginal shifts can be generalised. We can
consider a family of shifts $\mathcal{T}$ and test how a model would behave
under different shifts in the family. Let $\mathcal{P}$ be the space of
distributions defined on $\mathcal{\tilde{X}}$. We test models on data from
$T(p)(\tilde{X})$, for all $T\in\mathcal{T}$, with
$T:\mathcal{P}\rightarrow\mathcal{P}$. For example, for single marginal shifts
this corresponds to $T^{s}(p)(\tilde{X})=p^{s}(\tilde{X}_{i})p(\tilde{X}_{\neg
i}|X_{i})$. The general recipe for testing models under general shifts then
becomes as follows. Let $G$ be some generative model, we (1) Train generator
$G$ on $\mathcal{D}_{train,G}$ to fit $p(X)$; (2) Define family of possible
shifts $\mathcal{T}$, either with or without background knowledge; Denote
shift with magnitude $s$ by $T^{s}$; (3) Set $s$ and generate data
$\mathcal{D}_{syn}^{s}$ from $T^{s}(p)$; (4) Evaluate model on
$\mathcal{D}_{syn}^{s}$; (5) Repeat steps 2-4 for different families of shifts
and magnitudes $s$.

###  5 Use Cases of 3S Testing

We now demonstrate how 3S satisfies (P1) Granular evaluation and (P2)
Distributional shifts. We re-iterate that the aim throughout is to estimate
the true prediction performance of the model $f$ as closely as possible. We
tune and select the generative model based on Maximum Mean Discrepancy [56],
see Appendix C. We describe the experimental details, baselines, and datasets
for each experiment further in Appendix B 444Code for use cases found at:
https://github.com/seedatnabeel/3S-Testing or
https://github.com/vanderschaarlab/3S-Testing.

####  5.1 (P1) Granular Evaluation

#####  5.1.1 Correctness of Subgroup Performance Estimates

Goal. This experiment assesses the value of synthetic data when evaluating
model performance on minority subgroups. The challenge with small subgroups is
that the conventional paradigm of using a hold-out evaluation set might result
in high variance estimates due to the small sample size.

Datasets. We use the following five real-world medical and finance datasets:
Adult [57], Covid-19 cases in Brazil [58], Support [59], Bank [60], and Drug
[61]. These datasets have varying characteristics, from sample size to number
of features. They also possess representational imbalance and biases,
pertinent to 3S [4, 5]: ① _Minority subgroups_ : we evaluate the following
groups which differ in proportional representation - _Adult_ : Race; _Covid_ :
Ethnicity; _Bank_ : Employment; _Support_ : Race; _Drug_ : Ethnicity. ②
_Intersectional subgroups_ : we evaluate intersectional subgroups [49] (e.g.
black males or young females)— see Appendix F, intersectional model
performance matrix.

Set-up. We evaluate the estimates of subgroup performance for trained model
$f$ using different evaluation sets. We consider two baselines: (1)
$\mathcal{D}_{test,f}$: a typical hold-out test dataset and (2) Model-based
metrics (MBM) [21]. MBM uses a bootstrapping approach for obtaining multiple
test sets. We compare the baselines to 3S testing datasets, which generate
data to balance the subgroup samples: (i) _3S_ ($\mathcal{D}_{syn}$):
synthetic data generated by $G$, which is trained on $\mathcal{D}_{test,f}$
and (ii) _3S +_ ($\mathcal{D}_{syn}\cup\mathcal{D}_{test,f}$): test data
_augmented_ with the synthetic dataset.

For some subgroup $\mathcal{S}$, each test set gives an estimated model
performance $A(f;\mathcal{D}_{\cdot},\mathcal{S})$, which we compare to a
pseudo-oracle performance $A(f;\mathcal{D}_{oracle},\mathcal{S})$: the oracle
is the performance of $f$ evaluated on a large unseen real dataset
$\mathcal{D}_{oracle}\sim p(X,Y)$, where
$|\mathcal{D}_{oracle}|\gg|\mathcal{D}_{test,f}|$. As outlined above the
subgroups are as follows: (i) Adult: Race, (ii) Drug: Ethnicity, (iii) Covid:
Ethnicity (Region), (iv) Support: Race, (v) Bank: Employment status.

We evaluate the reliability of the different performance estimates based on
their Mean Absolute Error (MAE) relative to the Oracle predictive accuracy
estimates. We desire low MAE such that our estimates match the oracle.

Analysis. Fig. 4 illustrates across the 5 datasets that the 3S synthetic data
(red, green) closely matches estimates on the Oracle data. i.e. lower MAE vs
baselines. In particular, for small subgroups (e.g. racial minorities), 3S
provides a more accurate evaluation of model performance (i.e. with estimates
closer to the oracle) compared to a conventional hold-out dataset
($\mathcal{D}_{test,f}$) and MBM.

In addition, 3S estimates have reduced standard deviation. Thus, despite 3S
using the same (randomly drawn test set) $\mathcal{D}_{test,f}$ to train its
generator, its estimates are more robust to this randomness. The results
highlight an evaluation pitfall of the standard hold-out test set paradigm:
the estimate’s high variance w.r.t. the drawn $\mathcal{D}_{test,f}$ could
lead to potentially misleading conclusions about model performance in the
wild, since an end-user only has access to a single draw of
$\mathcal{D}_{test,f}$. e.g., we might incorrectly overestimate the true
performance of minorities. The use of synthetic data solves this.

Figure 4: Assessing the reliability of performance estimates based on mean
absolute error of the predicted performance estimate to the performance
evaluated by oracle ($\downarrow$ better). 3S better approximates true
estimates compared to both baselines. 3S\+ enjoys the best of both worlds by
combining synthetic and real data. We evaluate an RF model, with other model
classes shown in the Appendix.

Next, we move beyond single-feature minority subgroups and show that synthetic
data can also be used to evaluate performance on intersectional groups —
subgroups with even smaller sample sizes due to the intersection. 3S
performance estimates on 2-feature intersections are shown in Appendix F.
Intersectional performance matrices provide model developers more granular
insight into where they can improve their model most, as well as inform users
how a model may perform on intersections of groups (especially important to
evaluate sensitive intersectional subgroups).555N.B. low-performance estimates
by 3S only indicate poor model performance; this does not necessarily imply
that the data itself is biased for these subgroups. However, it could warrant
investigating potential data bias and how to improve the model. Appendix F
further illustrates how these intersectional performance matrices can be used
as part of model reports.

We evaluate the intersectional performance estimates of 3S and baseline
$\mathcal{D}_{test,f}$ using the MAE of the performance matrices w.r.t. the
oracle, averaged across 3 models (i.e, RF, GBDT, MLP). The error of 3S (11.90
$\pm$ 0.19) is significantly lower than $\mathcal{D}_{test,f}$ (20.29 $\pm$
0.14), hence demonstrating 3S provides more reliable intersectional estimates.

Takeaway. Synthetic data provides more accurate performance estimates on small
subgroups compared to evaluation on a standard test set. This is especially
relevant from a representational bias and fairness perspective—allowing more
accurate performance estimates on minority subgroups.

#####  5.1.2 Reliability through Confidence Intervals

Figure 5: 3S is more reliable than baselines with higher coverage, lower
widths — with less variability.

Goal. In Fig. 4, we see that all methods, including 3S, have errors in some
cases, which warrants the desire to have confidence intervals at test time. 3S
uses a deep generative ensemble to provide uncertainty estimates at test-
time—see Sec. 4.1.

Set-up. We assess coverage over 20 random splits and seeds for 3S vs the
baselines (B1) bootstrapping [62] confidence intervals for
$\mathcal{D}_{test,f}$ and (B2) MBM: which itself uses bootstrapping for the
predictive distribution. For 3S, we assess a Deep Generative Ensemble with
$K=1,5,10$ randomly initialized generative models. For each method, we take
the average estimate $\pm$ 2 standard deviations. We evaluate the intervals
based on the following two metrics defined in [63, 64, 65]: (i) Coverage =
$\mathbb{E}\left[1_{x_{i}\in\left[l_{i},r_{i}\right]}\right]$ (ii) Width =
$\mathbb{E}[|r_{i}-l_{i}|]$. Coverage measures how often the true label is in
the prediction region, while width measures how specific that prediction
region is. In the ideal case, we have high coverage with low width. See
Appendix B for more details.

Analysis. Fig. 5 shows the mean test set coverage and width averaged over the
five datasets. 3S (with K=5 and K=10) is more reliable, attaining higher
coverage rates with lower width compared to baselines. In addition, the
variability with 3S is much lower for both coverage and width. We note that
this comes at a price: computational cost scales linearly with $K$. For fair
comparison, we set $K=1$ in the rest of the paper.

Takeaway. 3S includes uncertainty estimates at test time that cover the true
value much better than baselines, allowing practitioners to decide when (not)
to trust 3S performance estimates.

####  5.2 (P2) Sensitivity to Distributional Shifts

ML models deployed in the wild often encounter data distributions differing
from the training set. We simulate distributional shifts to evaluate model
performance under different potential post-deployment conditions. We examine
two setups with varying knowledge of the potential shift.

#####  5.2.1 No Prior Information

Goal. Assume we have no prior information for the (future) model deployment
environment. In this case, we might still wish to stress test the sensitivity
for different potential operating conditions, such that a practitioner
understands model behavior under different conditions, which can guide as to
when the model can and cannot be used. We wish to simulate distribution shifts
using synthetic data and assess if it captures true performance.

Set-up. We consider shifts in the marginal of some feature $X_{i}$, keeping
$p(\tilde{X}_{\neg i}|\tilde{X}_{i})$ fixed (see Sec. 3). For instance, a
shift in the marginal $X_{i}$’s mean (see Sec. 4.2). To assess performance for
different degrees of shift, we compute three shift buckets around the mean of
the original feature distribution: large negative shift from the mean (-),
small negative/positive shift from the mean ($\pm$), and large positive shift
from the mean (+).

We define each in terms of the feature quantiles. We generate uniformly
distributed shifts (between min(feature) and max(feature)). Any shift that
shifts the mean to less than Q1 is (-) , any shift that shifts the mean to
more than Q3 is (+) and any shift in between is ($\pm$).

As before, we compare estimated accuracy w.r.t. a pseudo-oracle test set. We
compare two baselines: (i) Mean-shift (MS) and (ii) Rejection sampling (RS);
both applied to the real data.

Analysis. Table 1 shows the potential utility of synthetic data to more
accurately estimate performance for unknown distribution shifts compared to
real data alone. This is seen both with an average lower mean error of
estimates, but also across all three buckets. This implies that the synthetic
data is able to closely capture the true performance across the range of
feature shifts.

Table 1: Mean error in estimated accuracy across shift quantile buckets for
all 5 datasets. The results show 3S indeed provides more reliable estimates
even for distribution shift. $\downarrow$ is better.

| Adult | Support | Bank | Drug | SEER
---|---|---|---|---|---
| Mean | - | $\pm$ | + | Mean | - | $\pm$ | + | Mean | - | $\pm$ | + | Mean | - | $\pm$ | + | Mean | - | $\pm$ | +
3S | 2.6 | 2.2 | 1.8 | 3.9 | 2.0 | 2.6 | 2.0 | 1.1 | 5.4 | 3.7 | 3.6 | 6.7 | 5.6 | 5.7 | 4.4 | 7.8 | 2.7 | 5.5 | 3.0 | 2.0
MS | 5.9 | 5.2 | 5.6 | 6.9 | 19.3 | 22.9 | 18.5 | 15.2 | 18.6 | 18.6 | 19.9 | 17.9 | 18.5 | 19.6 | 19.0 | 16.3 | 3.3 | 2.6 | 3.9 | 3.2
RS | 15.9 | 10.5 | 17.9 | 18.6 | 25.1 | 27.8 | 24.2 | 22.3 | 18.9 | 18.9 | 20.1 | 18.3 | 20.1 | 21.7 | 21.0 | 18.2 | 20.0 | 20.3 | 23.6 | 18.6

Takeaway. Synthetic data can be used to more accurately characterize model
performance across a range of possible distributional shifts.

#####  5.2.2 Incorporating Prior Knowledge on Shift

(a) Model performance rank compared.

(b) Average accuracy vs increased prior knowledge

Figure 6: Incorporating prior knowledge of the shift. (a) $\mathcal{D}_{syn}$
matches the performance rank of the true target domain, which can help to
select the best model to use in the target domain, and (b) $\mathcal{D}_{syn}$
better estimates target domain performance compared to baselines. Performance
improves and gets closer as more prior knowledge is incorporated via added
features. Points are connected to highlight trends.

Goal. Consider the scenario where we have some knowledge of the shifted
distribution and wish to estimate target domain performance. Specifically,
here we assume we only have access to the feature marginals in the form of
high-level info from the target domain, e.g. age (mean, std) or gender
(proportions). We sample from this marginal and generate the other features
conditionally (Sec. 4.2). Set-up.

We use datasets SEER (US) [66] and CUTRACT (UK) [67], two real cancer datasets
with the same features, but with shifted distributions due to coming from
different countries. We train models $f$ and $G$ on the source domain (USA).
We then wish to estimate likely model performance in the shifted target domain
(UK). We assume access to information from $n$ features in the target domain
(features $X_{c}$), sample $X_{c}$ from this marginal, conditionally generate
$X_{\bar{c}}|X_{c}$. We estimate performance with $\mathcal{D}_{syn}$.

We use the CUTRACT dataset (Target) as the ground truth to validate our
estimate. As baselines, we use estimates on the source test set, along with
_Source Rejection Sampling (RS)_ , which achieves a distributional shift
through rejection sampling the source data using the observed target features.
We also compare to baselines which assume access to more information than 3S,
i.e. access to _full_ unlabeled data from the target domain and hence have an
advantage over 3S when predicting target domain performance. We benchmark ATC
[31], DOC [32] and IM [33]. Details on all baselines are in Appendix B. Note,
we repeat this experiment in Appendix E for _Covid-19 data_ [58], where there
is a shift between Brazil’s north and south patients.

Table 2: 3S has lower performance estimate error in target domain for
different downstream models. Rows yellow have access to more information than
3S, in the form of unlabeled data from the target domain. $\downarrow$ is
better

| mean | ada | bag | gbc | mlp | rf | knn | lr
---|---|---|---|---|---|---|---|---
3S-Testing | 0.023 | 0.051 | 0.012 | 0.030 | 0.009 | 0.015 | 0.020 | 0.029
All (Source) | 0.258 | 0.207 | 0.327 | 0.207 | 0.170 | 0.346 | 0.233 | 0.211
RS (Source) | 0.180 | 0.028 | 0.298 | 0.096 | 0.014 | 0.373 | 0.213 | 0.094
ATC [31] | 0.249 | 0.253 | 0.288 | 0.162 | 0.140 | 0.214 | 0.369 | 0.165
IM [33] | 0.215 | 0.206 | 0.278 | 0.156 | 0.126 | 0.268 | 0.131 | 0.163
DOC[32] | 0.201 | 0.207 | 0.211 | 0.162 | 0.116 | 0.223 | 0.148 | 0.161

Analysis. In Fig. 6(a), we show the model ranking of the different predictive
models based on performance estimates of the different methods. Using the
synthetic data from 3S, we determine the same model ranking as the true
ranking on the target—showcasing how 3S can be used for model selection with
distributional shifts. On the other hand, baselines provide incorrect
rankings.

Fig. 6(b) shows the average estimated performance of $f$ as a function of the
number of observed features. We see that the 3S estimates are closer to the
oracle across the board compared to baselines. Furthermore, for an increasing
number of features (i.e. increasing prior knowledge), we observe that 3S
estimates converge to the oracle. This is unsurprising: the more features we
acquire target statistics of, the better we can model the true shifted
distribution. Source RS does so too, but more slowly and with major variance
issues.

We also assess raw estimate errors in Table 2. 3S clearly has lower
performance estimate errors for the numerous downstream models. Beyond having
reduced error compared to rejection sampling, it is interesting that 3S
generally outperforms highly specialized methods (ATC, IM, DOC), which not
only have access to more information but are also developed specifically to
predict target domain performance. A likely rationale for this is that these
methods rely on probabilities and hence do not translate well to the non-
neural methods widely used in the tabular domain.

Takeaway: High-level information about potential shifts can be translated into
realistic synthetic data, to better estimate target domain model performance
and select the best model to use.

###  6 Discussion

Synthetic data for model evaluation. Accurate model evaluation is of vital
importance to ML, but this is challenging when there is _limited test data_.
We have shown in Sec. 3.1 that it is hard to accurately evaluate performance
for small subgroups (e.g. minority race groups) and to understand how models
would perform under distributional shifts using real data alone. We have
investigated the potential of synthetic data for model evaluation and found
that 3S can accurately evaluate the performance of a prediction model, even
when the generative model is trained on the same test set. A deep generative
ensemble approach can be used to quantify the uncertainty in 3S estimates,
which we have shown provides reliable coverage of the true model performance.
Furthermore, we explored synthetic test sets with shifts, which provide
practitioners with insight into how their model may perform in other
populations or future scenarios.

Model reports. We envision evaluations using synthetic data could be published
alongside models to give insight into when a model should and should not be
used—e.g. to complete model evaluation templates such as Model Cards for Model
Reporting [10]. Appendix F illustrates an example model report using 3S.

Practical considerations. We discuss and explore limitations in detail in
Appendix D. Let us highlight three practical considerations to the application
of synthetic data for testing. Firstly, evaluating the performance under
distributional shifts requires assumptions on the shift. These assumptions
affect model evaluation and require careful consideration from the end-user.
This is especially true for large shifts or scenarios where we do not have
enough training data to describe the shifted distribution well enough.
However, even if absolute estimates are inaccurate, we can still provide
insight into trends of different scenarios. Secondly, synthetic data might
have failure modes or limitations in certain settings, such as cases where
there are only a handful of samples or with many samples. Thirdly, training
and tuning a generative model is non-trivial. 3S’ DGE mechanism mitigates this
by offering an uncertainty estimate that provides insight into the error of
the generative learning process. The computational cost of such a generative
ensemble may be justified by the cost of untrustworthy model evaluation.

### Acknowledgements

Boris van Bruegel is supported by ONR-UK. Nabeel Seedat is supported by the
Cystic Fibrosis Trust. We would like to warmly thank the reviewers for their
time and useful feedback. The authors also thank Alicia Curth, Alex Chan,
Zhaozhi Qian, Daniel Jarrett, Tennison Liu and Andrew Rashbass for their
comments and feedback on an earlier manuscript.

### References

  * [1] Murtuza N Shergadwala, Himabindu Lakkaraju, and Krishnaram Kenthapadi.  A human-centric perspective on model monitoring.  In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 10, pages 173–183, 2022.
  * [2] Luke Oakden-Rayner, Jared Dunnmon, Gustavo Carneiro, and Christopher Ré.  Hidden stratification causes clinically meaningful failures in machine learning for medical imaging.  In Proceedings of the ACM Conference on Health, Inference, and Learning, pages 151–159, 2020.
  * [3] Harini Suresh and John V Guttag.  A framework for understanding unintended consequences of machine learning.  arXiv preprint arXiv:1901.10002, 2:8, 2019.
  * [4] Ángel Alexander Cabrera, Will Epperson, Fred Hohman, Minsuk Kahng, Jamie Morgenstern, and Duen Horng Chau.  FairVis: Visual analytics for discovering intersectional bias in machine learning.  In 2019 IEEE Conference on Visual Analytics Science and Technology (VAST), pages 46–56. IEEE, 2019.
  * [5] Angel Alexander Cabrera, Minsuk Kahng, Fred Hohman, Jamie Morgenstern, and Duen Horng Chau.  Discovery of intersectional bias in machine learning using automatic subgroup generation.  In ICLR Debugging Machine Learning Models Workshop, 2019.
  * [6] Oleg S Pianykh, Georg Langs, Marc Dewey, Dieter R Enzmann, Christian J Herold, Stefan O Schoenberg, and James A Brink.  Continuous learning AI in radiology: Implementation principles and early applications.  Radiology, 297(1):6–14, 2020.
  * [7] Joaquin Quinonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence.  Dataset shift in machine learning.  MIT Press, 2008.
  * [8] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang.  WILDS: A benchmark of in-the-wild distribution shifts.  In International Conference on Machine Learning, pages 5637–5664. PMLR, 2021.
  * [9] Nabeel Seedat, Fergus Imrie, and Mihaela van der Schaar.  Dc-check: A data-centric ai checklist to guide the development of reliable machine learning systems.  arXiv preprint arXiv:2211.05764, 2022.
  * [10] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford.  Datasheets for datasets.  Communications of the ACM, 64(12):86–92, 2021.
  * [11] Harini Suresh, Jen J Gong, and John V Guttag.  Learning tasks for multitask learning: Heterogenous patient populations in the ICU.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 802–810, 2018.
  * [12] Karan Goel, Albert Gu, Yixuan Li, and Christopher Re.  Model patching: Closing the subgroup performance gap with data augmentation.  In International Conference on Learning Representations, 2020.
  * [13] Suchi Saria and Adarsh Subbaswamy.  Tutorial: safe and reliable machine learning.  arXiv preprint arXiv:1904.07204, 2019.
  * [14] Alexander D’Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D Hoffman, et al.  Underspecification presents challenges for credibility in modern machine learning.  arXiv preprint arXiv:2011.03395, 2020.
  * [15] Joseph Paul Cohen, Tianshi Cao, Joseph D Viviano, Chin-Wei Huang, Michael Fralick, Marzyeh Ghassemi, Muhammad Mamdani, Russell Greiner, and Yoshua Bengio.  Problems in the deployment of machine-learned models in health care.  CMAJ, 193(35):E1391–E1394, 2021.
  * [16] Rahul C Deo.  Machine learning in medicine.  Circulation, 132(20):1920–1930, 2015.
  * [17] Neil Savage.  Better medicine through machine learning.  Communications of the ACM, 55(1):17–19, 2012.
  * [18] Milad Zafar Nezhad, Dongxiao Zhu, Najibesadat Sadati, Kai Yang, and Phillip Levi.  SUBIC: A supervised bi-clustering approach for precision medicine.  In 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 755–760. IEEE, 2017.
  * [19] Kayur Patel, James Fogarty, James A Landay, and Beverly Harrison.  Investigating statistical machine learning as a tool for software development.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 667–676, 2008.
  * [20] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do ImageNet classifiers generalize to ImageNet?  In International Conference on Machine Learning, pages 5389–5400. PMLR, 2019.
  * [21] Andrew C Miller, Leon A Gatys, Joseph Futoma, and Emily Fox.  Model-based metrics: Sample-efficient estimates of predictive model subpopulation performance.  In Machine Learning for Healthcare Conference, pages 308–336. PMLR, 2021.
  * [22] Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci.  Deep neural networks and tabular data: A survey.  arXiv preprint arXiv:2110.01889, 2021.
  * [23] Ravid Shwartz-Ziv and Amitai Armon.  Tabular data: Deep learning is not all you need.  Information Fusion, 81:84–90, 2022.
  * [24] Kaggle.  Kaggle machine learning and data science survey, 2017.
  * [25] Boris van Breugel, Zhaozhi Qian, and Mihaela van der Schaar.  Synthetic data, real errors: how (not) to publish and use synthetic data.  In International Conference on Machine Learning. PMLR, 2023.
  * [26] Peter Flach.  Performance evaluation in machine learning: the good, the bad, the ugly, and the way forward.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9808–9814, 2019.
  * [27] Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre-Alvise Rebuffi, Ira Ktena, Krishnamurthy Dj Dvijotham, and Ali Taylan Cemgil.  A fine-grained analysis on distribution shift.  In International Conference on Learning Representations, 2021.
  * [28] Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  In International Conference on Learning Representations, 2018.
  * [29] Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh.  Beyond accuracy: Behavioral testing of nlp models with checklist.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902–4912, 2020.
  * [30] Paul Röttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, Janet Pierrehumbert, et al.  Hatecheck: Functional tests for hate speech detection models.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), page 41. Association for Computational Linguistics, 2021.
  * [31] Saurabh Garg, Sivaraman Balakrishnan, Zachary Chase Lipton, Behnam Neyshabur, and Hanie Sedghi.  Leveraging unlabeled data to predict out-of-distribution performance.  In International Conference on Learning Representations, 2022.
  * [32] Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi, Trevor Darrell, and Ludwig Schmidt.  Predicting with confidence on unseen distributions.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1134–1144, 2021.
  * [33] Mayee Chen, Karan Goel, Nimit S Sohoni, Fait Poms, Kayvon Fatahalian, and Christopher Ré.  Mandoline: Model evaluation under distribution shift.  In International Conference on Machine Learning, pages 1617–1629. PMLR, 2021.
  * [34] Boris van Breugel and Mihaela van der Schaar.  Beyond privacy: Navigating the opportunities and challenges of synthetic data.  arXiv preprint arXiv:2304.03722, 2023.
  * [35] James Jordon, Jinsung Yoon, and Mihaela Van Der Schaar.  PATE-GAN: Generating synthetic data with differential privacy guarantees.  In International Conference on Learning Representations, 2018.
  * [36] Samuel A Assefa, Danial Dervovic, Mahmoud Mahfouz, Robert E Tillman, Prashant Reddy, and Manuela Veloso.  Generating synthetic data in finance: opportunities, challenges and pitfalls.  In Proceedings of the First ACM International Conference on AI in Finance, pages 1–8, 2020.
  * [37] Depeng Xu, Yongkai Wu, Shuhan Yuan, Lu Zhang, and Xintao Wu.  Achieving causal fairness through generative adversarial networks.  In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, 2019.
  * [38] Boris van Breugel, Trent Kyono, Jeroen Berrevoets, and Mihaela van der Schaar.  DECAF: Generating fair synthetic data using causally-aware generative networks.  Advances in Neural Information Processing Systems, 34:22221–22233, 2021.
  * [39] Antreas Antoniou, Amos Storkey, and Harrison Edwards.  Data augmentation generative adversarial networks.  arXiv preprint arXiv:1711.04340, 2017.
  * [40] Ayesha S Dina, AB Siddique, and D Manivannan.  Effect of balancing data using synthetic data on the performance of machine learning classifiers for intrusion detection in computer networks.  arXiv preprint arXiv:2204.00144, 2022.
  * [41] Hari Prasanna Das, Ryan Tran, Japjot Singh, Xiangyu Yue, Geoffrey Tison, Alberto Sangiovanni-Vincentelli, and Costas J Spanos.  Conditional synthetic data generation for robust machine learning applications with limited pandemic data.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11792–11800, 2022.
  * [42] Simon Bing, Andrea Dittadi, Stefan Bauer, and Patrick Schwab.  Conditional generation of medical time series for extrapolation to underrepresented populations.  PLOS Digital Health, 1(7):1–26, 07 2022.
  * [43] Qi Wang, Junyu Gao, Wei Lin, and Yuan Yuan.  Learning from synthetic data for crowd counting in the wild.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8198–8207, 2019.
  * [44] Daniel Sáez Trigueros, Li Meng, and Margaret Hartnett.  Generating photo-realistic training data to improve face recognition accuracy.  Neural Networks, 134:86–94, 2021.
  * [45] Nataniel Ruiz, Adam Kortylewski, Weichao Qiu, Cihang Xie, Sarah Adel Bargal, Alan Yuille, and Stan Sclaroff.  Simulated adversarial testing of face recognition models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4145–4155, 2022.
  * [46] Samin Khan, Buu Phan, Rick Salay, and Krzysztof Czarnecki.  Procsy: Procedural synthetic dataset generation towards influence factor studies of semantic segmentation networks.  In CVPR workshops, pages 88–96, 2019.
  * [47] Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni.  Modeling tabular data using conditional GAN.  Advances in Neural Information Processing Systems, 32, 2019.
  * [48] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  Advances in Neural Information Processing Systems, 30, 2017.
  * [49] Kimberley Crenshaw.  Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory, and antiracist politics. the university of chicago legal forum, 1989 (1), 139-167.  Chicago, IL, 1989.
  * [50] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané.  Concrete problems in AI safety.  arXiv preprint arXiv:1606.06565, 2016.
  * [51] Ruihan Wu, Chuan Guo, Yi Su, and Kilian Q Weinberger.  Online adaptation to label distribution shift.  Advances in Neural Information Processing Systems, 34:11340–11351, 2021.
  * [52] Christina X Ji, Ahmed M Alaa, and David Sontag.  Large-scale study of temporal shift in health insurance claims.  Conference on Health, Inference, and Learning (CHIL), 2023.
  * [53] Hidetoshi Shimodaira.  Improving predictive inference under covariate shift by weighting the log-likelihood function.  Journal of Statistical Planning and Inference, 90(2):227–244, 2000\.
  * [54] Jose G Moreno-Torres, Troy Raeder, Rocío Alaiz-Rodríguez, Nitesh V Chawla, and Francisco Herrera.  A unifying view on dataset shift in classification.  Pattern Recognition, 45(1):521–530, 2012.
  * [55] Marco Saerens, Patrice Latinne, and Christine Decaestecker.  Adjusting the outputs of a classifier to new a priori probabilities: a simple procedure.  Neural Computation, 14(1):21–41, 2002.
  * [56] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola.  A kernel two-sample test.  The Journal of Machine Learning Research, 13(1):723–773, 2012.
  * [57] Arthur Asuncion and David Newman.  UCI machine learning repository, 2007.
  * [58] Pedro Baqui, Ioana Bica, Valerio Marra, Ari Ercole, and Mihaela van Der Schaar.  Ethnic and regional variations in hospital mortality from COVID-19 in Brazil: a cross-sectional observational study.  The Lancet Global Health, 8(8):e1018–e1026, 2020.
  * [59] William A Knaus, Frank E Harrell, Joanne Lynn, Lee Goldman, Russell S Phillips, Alfred F Connors, Neal V Dawson, William J Fulkerson, Robert M Califf, Norman Desbiens, Peter Layde, Robert K Oye, Paul E Bellamy, Rosemarie B Hakim, and Douglas P Wagner.  The SUPPORT prognostic model: Objective estimates of survival for seriously ill hospitalized adults.  Annals of Internal Medicine, 122(3):191–203, 1995.
  * [60] Sérgio Jesus, José Pombal, Duarte Alves, André Cruz, Pedro Saleiro, Rita Ribeiro, João Gama, and Pedro Bizarro.  Turning the tables: Biased, imbalanced, dynamic tabular datasets for ML evaluation.  Advances in Neural Information Processing Systems, 35:33563–33575, 2022.
  * [61] Elaine Fehrman, Awaz K Muhammad, Evgeny M Mirkes, Vincent Egan, and Alexander N Gorban.  The five factor model of personality and evaluation of drug consumption risk.  In Data Science: Innovative Developments in Data Analysis and Clustering, pages 231–242. Springer, 2017.
  * [62] Bradley Efron.  Bootstrap methods: another look at the jackknife.  Springer, 1992.
  * [63] Benjamin Kompa, Jasper Snoek, and Andrew L Beam.  Empirical frequentist coverage of deep learning uncertainty quantification procedures.  Entropy, 23(12):1608, 2021.
  * [64] Nabeel Seedat, Jonathan Crabbé, and Mihaela van der Schaar.  Data-SUITE: Data-centric identification of in-distribution incongruous examples.  In International Conference on Machine Learning, pages 19467–19496. PMLR, 2022.
  * [65] Jiri Navratil, Matthew Arnold, and Benjamin Elder.  Uncertainty prediction for deep sequential regression using meta models.  arXiv preprint arXiv:2007.01350, 2020.
  * [66] Máire A Duggan, William F Anderson, Sean Altekruse, Lynne Penberthy, and Mark E Sherman.  The surveillance, epidemiology and end results (SEER) program and pathology: towards strengthening the critical relationship.  The American Journal of Surgical Pathology, 40(12):e94, 2016.
  * [67] Prostate Cancer UK.  Cutract.  https://prostatecanceruk.org, 2019.
  * [68] Marco Tulio Ribeiro and Scott Lundberg.  Adaptive testing and debugging of nlp models.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3253–3267, 2022.
  * [69] Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo.  “Everyone wants to do the model work, not the data work”: Data cascades in high-stakes ai.  In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–15, 2021.
  * [70] Nabeel Seedat, Jonathan Crabbé, Ioana Bica, and Mihaela van der Schaar.  Data-iq: Characterizing subgroups with heterogeneous outcomes in tabular data.  Advances in Neural Information Processing Systems, 35:23660–23674, 2022.
  * [71] Andrew Ng, Lora Aroyo, Cody Coleman, Greg Diamos, Vijay Janapa Reddi, Joaquin Vanschoren, Carole-Jean Wu, and Sharon Zhou.  NeurIPS data-centric AI workshop, 2021.
  * [72] Neoklis Polyzotis and Matei Zaharia.  What can data-centric AI learn from data and ML engineering?  arXiv preprint arXiv:2112.06439, 2021.
  * [73] Alex Chan, Ioana Bica, Alihan Hüyük, Daniel Jarrett, and Mihaela van der Schaar.  The Medkit-Learn(ing) environment: Medical decision modelling through simulation.  In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
  * [74] Brady Neal, Chin-Wei Huang, and Sunand Raghupathi.  RealCause: Realistic causal inference benchmarking.  arXiv preprint arXiv:2011.15007, 2020.
  * [75] Susan Athey, Guido W. Imbens, Jonas Metzger, and Evan Munro.  Using Wasserstein generative adversarial networks for the design of Monte Carlo simulations.  Journal of Econometrics, 2021.
  * [76] Harsh Parikh, Carlos Varjao, Louise Xu, and Eric Tchetgen Tchetgen.  Validating causal inference methods.  In International Conference on Machine Learning, pages 17346–17358. PMLR, 2022.
  * [77] Adam Kortylewski, Bernhard Egger, Andreas Schneider, Thomas Gerig, Andreas Morel-Forster, and Thomas Vetter.  Analyzing and reducing the damage of dataset bias to face recognition with synthetic data.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2019.
  * [78] Zhiheng Li and Chenliang Xu.  Discover the unknown biased attribute of an image classifier.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14970–14979, 2021.
  * [79] Daniel McDuff, Shuang Ma, Yale Song, and Ashish Kapoor.  Characterizing bias in classifiers using generative models.  Advances in Neural Information Processing Systems, 32, 2019.
  * [80] Bill Howe, Julia Stoyanovich, Haoyue Ping, Bernease Herman, and Matt Gee.  Synthetic data for social good.  arXiv preprint arXiv:1710.08874, 2017.
  * [81] Diptikalyan Saha, Aniya Aggarwal, and Sandeep Hans.  Data synthesis for testing black-box machine learning models.  In 5th Joint International Conference on Data Science & Management of Data (9th ACM IKDD CODS and 27th COMAD), pages 110–114, 2022.
  * [82] SIVEP Brazil Ministry of Health.  Ministry of Health SIVEP-Gripe public dataset.
  * [83] Bradley Efron.  Estimating the error rate of a prediction rule: improvement on cross-validation.  Journal of the American Statistical Association, 78(382):316–331, 1983.
  * [84] Thomas J DiCiccio and Bradley Efron.  Bootstrap confidence intervals.  Statistical science, 11(3):189–228, 1996.
  * [85] James Carpenter and John Bithell.  Bootstrap confidence intervals: when, which, what? A practical guide for medical statisticians.  Statistics in medicine, 19(9):1141–1164, 2000.
  * [86] Ali Borji.  Pros and cons of GAN evaluation measures.  Computer Vision and Image Understanding, 179:41–65, 2019.
  * [87] Dougal J Sutherland, Hsiao-Yu Tung, Heiko Strathmann, Soumyajit De, Aaditya Ramdas, Alexander J Smola, and Arthur Gretton.  Generative models and model criticism via optimized maximum mean discrepancy.  In International Conference on Learning Representations, 2017.
  * [88] Wacha Bounliphone, Eugene Belilovsky, Matthew B Blaschko, Ioannis Antonoglou, and Arthur Gretton.  A test of relative similarity for model selection in generative models.  In International Conference on Learning Representations, 2016.
  * [89] Kush R Varshney.  Trustworthy machine learning.  Chappaqua, NY, 2021.  Section 9.2.1.
  * [90] Ahmed Alaa, Boris Van Breugel, Evgeny S Saveliev, and Mihaela van der Schaar.  How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models.  In International Conference on Machine Learning, pages 290–306. PMLR, 2022.

## Appendix: Can You Rely on Your Model? A Case for Synthetic-Data-Based Model
Evaluation

parttoc5

###  Appendix A Extended Related Work

We present a comparison of our framework, 3S , and provide further contrast to
related work. Table 3, highlights that related methods often do not permit
automated test creation, in particular, they are often human labor-intensive
—see Table 3. These benchmark tasks are tailored to specific datasets and
tasks and hence cannot be customized and/or personalized to an end-user’s
specific task, dataset, or trained model. Finally, both benchmarking methods
and behavioral testing require additional data to be collected or created.
This contrasts 3S which only requires the original test data, already
available.

Table 3: Comparison of 3S to human-crafted testing frameworks. (i) Automated
test creation, (ii) Tests can be personalised to user dataset and use
case(e.g. distributional shift), and (iii) No additional data or info
required.

Method | Approach | Assumption | Use cases | (i) | (ii) | (iii)
---|---|---|---|---|---|---
3S Testing (Ours) | Synthetic test data |  Generative model fits the data |  (P1) Subgroup testing (P2) Distributional shift testing | ✔ | ✔ | ✔
BENCHMARK TASKS
Imagenet-C/P [28] | Create corrupted images |  Synthetic corruption reflects the real-world |  Image corruption testing | ✗ | ✗ | ✗
Wilds [8] | Collect data with real shifts |  Collected ‘wild’ data reflects sufficient use cases |  Distribution shift testing | ✗ | ✗ | ✗
MODEL BEHAVIOURAL TESTING
CheckList [29] | Human crafted test scenarios |  Know a-priori scenarios to test |  Crafted scenario tests | ✗ | ✔ | ✗
HateCheck [30] | Human crafted test scenarios |  Know a-priori scenarios to test |  Crafted scenario tests | ✗ | ✔ | ✗
AdaTest [68] | GPT-3 creates tests, human refines | Human-in-the-loop |  Weakness probing | ✔ | ✔ | ✗

Data-centric AI. The usage and assessment is vital for ML models, yet is often
overlooked as operational [69, 70]. The recent focus on data-centric AI [71,
9] aims to build systematic tools to improve the quality of data used to train
and evaluate ML models [72, 64]. 3S contributes to this nascent body of work,
specifically around the usage and generation of data to better evaluate and
test ML models

Simulators and counterfactual generators. Simulators have been used to
benchmark algorithms in settings such as causal effect estimation and
sequential decision-making [73]. The work on simulators is not directly
related, as the goal is often to test scenarios that are not available in
real-world data. For example, in [73], the goal is customization of the
decision-making policy in order to evaluate methods to better understand human
decision-making. Similarly, there has been work on generating realistic
synthetic data in the causal inference domain [74, 75, 76], but these methods
focus on benchmarking causal inference methods, do not consider distributional
shifts nor subgroup evaluation, and do not explore the value of the synthetic
data beyond realistic, ground-truth counterfactuals.

Synthetic data in computer vision. Synthetic data has been used in computer
vision both to improve model training and to test weaknesses in models. These
methods can be grouped as follows by their motivations:

  * •

Generate synthetic data for training, to reduce the reliance on collecting and
annotating large training sets— This is different from 3S as they focus on
constructing better training sets, rather than constructing better test sets
for model evaluation

  * •

Generate synthetic data to improve the model by augmenting the real dataset
with synthetic examples—*Again, this is different from 3S as they focus on
training better-performing models, rather than the evaluation of an already
trained model

  * •

Generate synthetic data to probe models on different dataset attributes. For
example, in face recognition, how the model might perform on faces with long
vs short hair. This is most similar to 3S , but there are clear differences in
both the goal for and approach to generating the data. We compare 3S to (i)
CGI- or physics-based simulators and (ii) deep generative models for probing
in computer vision.

In Table 4, we contrast both simulators and generative approaches where
synthetic data to probe models on different dataset attributes.

Table 4: Comparing 3S to computer vision synthetic data approaches. $A$ is a
performance metric (Eq.1) and $f$ the trained predictive model.
$\mathcal{D}_{syn}(x)$ denotes synthetic data is created dependent on $x$

Examples |  Data and/or generator input | Conditioning info |  Does not require pre-trained simulator/generator |  $\mathcal{D}_{syn}$ used for training or testing | Goal
---|---|---|---|---|---
3S \- Subgroup | $\mathcal{D}_{train,G}=\mathcal{D}_{test,f}$ (Any dataset) | Subgroups $\mathcal{S}$ | $\checkmark$ | Testing |  Reliable subgroup performance estimates for $f$, i.e. choose $\mathcal{D}_{syn}\sim p_{G}$ s.t. $A(f;\mathcal{D}_{syn},\mathcal{S})]\approx\mathbb{E}_{\mathcal{D}\overset{iid}{\sim}p_{R}}[A(f;\mathcal{D},\mathcal{S})]$
3S \- Shift | Shift information $T$ |  Estimate performance of $f$ under shift $T$, i.e. choose $\mathcal{D}_{syn}\sim p_{G}$ s.t. $A(f;\mathcal{D}_{syn},\mathcal{S})]\approx\mathbb{E}_{\mathcal{D}\overset{iid}{\sim}p^{s}}[A(f;\mathcal{D},\mathcal{S})]$
Computer Vision
[43] |  Video game engine (GTA5) Real-world data | Scene info $\mathcal{S}$ in virtual world | $\times$ | Training |  Improve crowd counting performance on diff. scenes by generating semi-synthetic data for training $f$, i.e. $\text{max}_{f}A(f;\mathcal{D}_{test}(\mathcal{S}))$
[44] |  $\mathcal{D}_{train,f}$ = VGGFace | Identity attributes | $\checkmark$ | Training |  Improve overall performance of facial recognition i.e. $\text{max}(A(f;\mathcal{D}_{test}))$
[77] | 3D face model | Nuisance transforms $\mathcal{N}$ | $\times$ | Testing |  Report face recognition robustness to different nuisances $\mathcal{N}$, $\mathcal{D}_{syn}(N)$ and report $A(f;\mathcal{D}_{syn}(N)),\forall N\in\mathcal{N}$
[45] | 3D face model | Simulator parameters $\rho$ | $\times$ | Testing |  Find adversarial failures for face recognition, i.e. find $\rho=\operatorname*{arg\,min}_{\rho}A(f;D_{syn}(\rho))$
[46] |  CityEngine, Unreal Engine, CARLA | Weather conditions $\mathcal{S}$ | $\times$ | Testing |  Report segmentation performance for self-driving cars under different weather conditions, i.e. $\mathcal{D}_{syn}(S)$ and report $A(f;\mathcal{D}_{syn}(S)),\forall S\in\mathcal{S}$
[78] | Pretrained StyleGAN |  Implicit attributes $S$ (e.g. age, lighting) | $\times$ | Testing |  Find attributes with poor performance, i.e. $\text{argmin}_{S}(A(f;\mathcal{D}_{syn}(S)))$
[79] |  $\mathcal{D}$ = MS-CELEB-1M | Subgroups $S$ | $\checkmark$ | Testing |  Find $S$ with poor face recognition performance, i.e. $\text{argmin}_{S}(A(f;\mathcal{D}_{syn}(S))$

Synthetic data and tabular approaches.

We contrast 3S to two works DataSynthesizer [80] and AITEST [81], which while
seemingly similar have specific differences to 3S . A side-by-side contrast is
presented in Table 5.

Data Synthesizer

We believe 3S is significantly different from DataSynthesizer, in terms of
aims, assumptions, and algorithmically.

Aim and assumptions. Data Synthesizer primarily focuses on privacy-preserving
generation of tabular synthetic data. The closest component to our work is the
extension the paper proposes around adversarial fake data generation. While
there are no experiments, the adversarial fake data consists of three areas.
We contrast them to 3S .

The major difference is Data Synthesizer assumes access to full knowledge
about the shift/distributional change. In contrast, 3S operates in a different
setting - (1) No prior knowledge on the shift and (2) high-level partial
knowledge about the shift through observing some variables in the target
domain.

  1. 1.

Edit the distribution: this assumes the user knows exactly the shift [Full
knowledge of the shift]. 3S covers two different settings: (1) No prior
knowledge on the shift, where only minimal assumptions on means of variables
allow us to create characteristic curves like in Section 5.2. and (2)
Incorporating prior knowledge, in which some features are observed from the
shifted distribution and we use these to generate the full data from the
shifted distribution, like in Section 5.2.2. Consequently, the difference is
that 3S tackles the no and partial information settings, whereas Data
Synthesizer tackles the full info setting of editing the distribution.

  2. 2.

Preconfigured pathological distributions — this requires full and exact
knowledge about the shift, which differs from 3S of partial knowledge and no
prior knowledge settings.

  3. 3.

Injecting missing data, extreme values — either such an approach is possible
to incorporate in 3S . We see these ideas as complementary.

Algorithmic.

The authors propose three methods, one with random features, one with
independent features, and one with correlated features. Due to the absence of
correlation in the first two, these reduce the data utility. Let us thus focus
on the third method, that does include correlation. This approach uses
Bayesian Networks and is only applicable to discrete data, hence needing to
discretize continuous variables. This loses utility when a coarse
discretization is chosen, while a fine discretization is often intractable and
data-inefficient due to the ordinal information being lost, e.g. results for
$age=31$ and $age=32$ will generally be similar—exactly the reason why the
independent approach was also introduced. Bayesian Nets are also limited in
other ways, e.g. results can be influenced by the feature generation order
deviating from the real data generation process’ ordering, as indicated by the
authors of DataSynthesizer in Figures 5 and 6.

AITEST

We contrast 3S to AITEST in terms of aims and assumptions, algorithm, and use
cases.

Aims and assumptions. AITEST has a significantly different aim and method
compared to 3S . As mentioned by the reviewer, AITEST can test for adversarial
robustness by generating realistic data with user-defined constraints, but
this is different from our work that aims to generate synthetic test data for
granular evaluation and distributional shifts.

Additionally, the assumptions on user input are quite different: AITEST
enables users to define constraints on features and associations between
features, whereas 3S requires information in terms of which subgroups to test
or shifts to generate. We do see possibilities to combine both frameworks,
e.g. through including constraints similar to the ones AITEST uses within the
3S method, or using fairness as a downstream task.

We have taken a step in this direction and added fairness as an additional
experiment and have included this experiment in the new Appendix D.4.

Algorithmic AITEST requires a decision tree surrogate of the black-box model,
whilst 3S does not need to model the black-box predictive model. AITEST
defines data constraints by fitting different distributions to the features
and using statistical testing to select the correct distribution. The
dependencies are then captured by a DAG. 3S does not require predefined
constraints and dependencies, but aims to learn these implicitly with the
generative model.

Use cases

  * •

Group fairness: AITEST aims to probe if a model does have a group fairness
issue or not. The goal of 3S is different — even if models don’t have group
bias issues, with 3S we desire reliable performance metric estimates (accuracy
or even fairness) which are similar to the oracle estimates on small and
intersectional subgroups for which we have limited real test data.

  * •

Adversarial robustness: AITEST does this by generating more inputs in the
neighborhood of a specific sample and seeing if they behave the same. In
reality, this is analogous to group-wise testing with $n=1$, a very specific
type of group testing. In contrast, with 3S we explore multiple definitions of
groups from specific sensitive attributes, to intersectional groups, to points
of interest (i.e. $n=1$), to high- and low density regions.

  * •

AITEST does not account for distribution shift, unlike 3S which looks at
distribution shift with no prior knowledge and high-level knowledge.

Table 5: Comparing 3S to other tabular approaches of generating synthetic test
data. $f$ is the trained predictive model and we abbreviate
$A(f;\mathcal{D})=A(f;\mathcal{D},\Omega)$ for evaluating $f$ over all of
$\mathcal{D}$ (Eq. 1). (i) used for evaluating subgroups, (ii) used for
evaluating shifts, (iii) does not require discretization of continuous
features, (iv) does not require modeling black-box $f$

Examples | Inputs | (i) | (ii) | (iii) | (iv) | Generator Type | Goal
---|---|---|---|---|---|---|---
3S |  $\mathcal{D}_{train,G}=\mathcal{D}_{test,f}$ (Any dataset) Subgroups: $S$, Shifts: No/Partial knowledge | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | GAN |  Reliable subgroup performance estimates for $f$, i.e. choose $\mathcal{D}_{syn}\sim p_{G}$ s.t. $A(f;\mathcal{D}_{syn},\mathcal{S})]\approx\mathbb{E}_{\mathcal{D}\overset{iid}{\sim}p_{R}}[A(f;\mathcal{D},\mathcal{S})]$
Estimate performance of $f$ under shift $T$, i.e. choose
$\mathcal{D}_{syn}\sim p_{G}$ s.t.
$A(f;\mathcal{D}_{syn},\mathcal{S})]\approx\mathbb{E}_{\mathcal{D}\overset{iid}{\sim}p^{s}}[A(f;\mathcal{D},\mathcal{S})]$
DataSynthesizer [80] |  Privacy-sensitive $\mathcal{D}$ Full knowledge of shift | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | Bayesian network |  Generate private data, extensions for generating pathological data through (i) manual editing, (ii) inserting extreme values/missingness, and (iii) .
AITEST [81] |  $\mathcal{D}_{train,G}=\mathcal{D}_{train,f}$, Constraints and dependencies (DAG) | $\checkmark$ | $\times$ | $\times/\checkmark$ | $\times$ |  Sample features sequentially following DAG |  Subgroup performance, e.g. fairness between two sensitive groups ($S_{1},S_{2}$), i.e. $\frac{A(f;\mathcal{D}_{Syn},S_{1})}{A(f;\mathcal{D}_{Syn},S_{2})}$

###  Appendix B Experimental Details

This appendix includes details on the experiments, including (i) the datasets,
and (ii) the different settings of the experiments, including the
implementation of baselines.

####  B.1 Datasets

Here we describe the real-world datasets used in greater detail.

ADULT Dataset The ADULT dataset [57] has 32,561 instances with a total of 13
attributes capturing demographic (age, gender, race), personal (marital
status) and financial (income) features amongst others. The classification
task predicts whether a person earns over $50K or not. We encode the features
(e.g. race, sex, gender, etc.) and a summary can be found in Table 6.

Note that there is an imbalance across certain features, such as across
different race groups, which is what we evaluate.

Table 6: Summary of features for the Adult Dataset [57] Feature | Values/Range
---|---
Age | $17-90$
education-num | $1-16$
marital-status | $0,1$
relationship | $0,1,2,3,4$
race | $0,1,2,3,4$
sex | $0,1$
capital-gain | $0,1$
capital-loss | $0,1$
hours-per-week | $1-99$
country | $0,1$
employment-type | $0,1,2,3$
salary | $0,1$

Covid-19 Dataset The Covid-19 dataset [58] consists of Covid patients from
Brazil. The dataset is publicly available and based on SIVEP-Gripe data [82].
The dataset consists of 6882 patients from Brazil recorded between February
27-May 4 2020. The dataset captures risk factors including comorbidities,
symptoms, and demographic characteristics. There is a mortality label from
Covid-19 making it a binary classification task. A summary of the
characteristics of the covariates can be found in Table 7.

Table 7: Summary of features for the Covid-19 Dataset [58]

Feature | Range
---|---
Sex | 0 (Female), 1(Male)
Age | $1-104$
Fever | $0,1$
Cough | $0,1$
Sore throad | $0,1$
Shortness of breath | $0,1$
Respiratory discomfort | $0,1$
SPO2 | $0-1$
Diharea | $0,1$
Vomitting | $0,1$
Cardiovascular | $0,1$
Asthma | $0,1$
Diabetes | $0,1$
Pulmonary | $0,1$
Immunosuppresion | $0,1$
Obesity | $0,1$
Liver | $0,1$
Neurologic | $0,1$
Branca (Region) | $0,1$
Preta (Region) | $0,1$
Amarela (Region) | $0,1$
Parda (Region) | $0,1$
Indigena (Region) | $0,1$

SEER Dataset The SEER dataset is a publicly available dataset consisting of
240,486 patients enrolled in the American SEER program [66]. The dataset
consists of features used to characterize prostate cancer, including age, PSA
(severity score), Gleason score, clinical stage, and treatments. A summary of
the covariates can be found in Table 8. The classification task is to predict
patient mortality, which is a binary label.

The dataset is highly imbalanced, where $~{}94\%$ of patients survive. Hence,
we extract a balanced subset of 20,000 patients (i.e. 10,000 with label=0 and
10,000 with label=1).

Table 8: Summary of features for the SEER [66] and CUTRACT [67] datasets.
Note: the range of age starts slightly lower for SEER (37-95) compared to
CUTRACT (44-95).

Feature | Range
---|---
Age | $37-95$
PSA | $0-98$
Comorbidities | $0,1,2,\geq 3$
Treatment | Hormone Therapy (PHT), Radical Therapy - RDx (RT-RDx),
Radical Therapy -Sx (RT-Sx), CM
Grade | $1,2,3,4,5$
Stage | $1,2,3,4$
Primary Gleason | $1,2,3,4,5$
Secondary Gleason | $1,2,3,4,5$

CUTRACT Dataset The CUTRACT dataset is a private dataset consisting of 10,086
patients enrolled in the British Prostate Cancer UK program [67]. It includes
the same features as SEER and also uses mortality as the label, see Table 8.

The dataset is highly imbalanced in its labels, hence we choose to extract a
balanced subset of 2,000 patients (i.e. 1000 with label=0 and 1000 with
label=1).

####  B.2 Experiments

For specifics on how $G$ is evaluated, tuned, and selected, please see
Appendix C.1.

#####  B.2.1 Experiment 5.1: Subgroups

In this experiment, we evaluate the performance estimates on different
subgroups based on the mean absolute error compared to the estimates of
subgroup performance using the oracle dataset. In order, to represent
potential variation of selecting different test sets, we repeat the experiment
10 times, where we sample a different test set in each run. That being said,
we keep the proportions in each dataset fixed such that
$\\{\mathcal{D}_{train,f}$,$\mathcal{D}_{test,f}$,$\mathcal{D}_{oracle}\\}$ =
$\\{8.4k,2.1k,19.6k\\}$. Given that minority subgroups have few samples, in
this experiment, we generate $n$ samples for each subgroup, where $n$ is the
size of the largest subgroup in $\mathcal{D}_{test,f}$. This allows us to
“balance” the evaluation dataset.

We provide more details on the subgroups below for each dataset:

Table 9: Details on dataset specific subgroups

Dataset | Subgroup | Specifics
---|---|---
Adult | Race | " White", "Amer-Indian-Eskimo", "Asian-Pac-Islander", "Black", " Other"
Drug | Ethnicity | ’White’, ’Other’, ’Mixed-White/Black’, ’Asian’, ’Mixed-White/Asian’, ’Black’, ’Mixed-Black/Asian’
Covid | Region | ’Branca’, ’Preta’, ’Amarela’, ’Parda’, ’Indigena’
Support | Race | ’white’,’black’, ’asian’, ’hispanic’
Bank | Employment status (Anon) | ’CA’, ’CB’, ’CC’, ’CD’, ’CE’, ’CF’, ’CG’

In producing the intersectional performance matrix, we slice the data for
these intersections. However, as we slice the data into finer intersections,
the intersectional groups naturally become smaller. Hence, to ensure we have
reliable estimates, we set a cut-off wherein we only evaluate performance for
intersectional groups where there are 100 or more samples. In computing the
mean absolute error, we do not include the corresponding intersections for
which there were insufficient samples.

For Section 5.1.2, we use bootstrapping for the naive $\mathcal{D}_{test,f}$
baseline. Bootstrapping is a prevalent method for providing uncertainty [83,
62, 84, 85, 21]. In our case, we sample a dataset $\mathcal{D}_{test,f}^{k}$
the same size as $\mathcal{D}_{test,f}$, uniformly with replacement from
$\mathcal{D}_{test,f}$. Model $f$ is evaluated on each
$\mathcal{D}_{test,f}^{k}$, and the mean and standard deviation across
$\mathcal{D}_{test,f}^{k}$ is used to construct confidence intervals. For all
methods, intervals are chosen as the mean $\pm$ 2 standard deviations.

#####  B.2.2 Experiments 5.2 Distributional shifts

No prior knowledge: characterizing sensitivity across operating ranges. In
this experiment, we assess performance for different degrees of shift, we
compute three shift buckets around the mean of the original feature
distribution: large negative shift from the mean (-), small negative/positive
shift from the mean ($\pm$), and large positive shift from the mean (+).

We define each in terms of the feature quantiles. We generate uniformly
distributed shifts (between min(feature) and max(feature)). Any shift that
shifts the mean to less than Q1 is (-) , any shift that shifts the mean to
more than Q3 is (+) and any shift in between is ($\pm$). The Oracle target is
created using rejection sampling of the oracle source data, see Section B.3.

Prior Knowledge. In this experiment, we assume we observe some of the features
in the target domain, i.e. we observe the empirical marginal distribution of
$X_{c}$. This empirical marginal distribution is used to sample from, and
conditioned on when generating the other features. The Source RS target is
created using rejection sampling of the test data, see Section B.3.

####  B.3 Rejection sampling for creating shifted datasets

In experiments 5.2 and 5.3, we use rejection sampling for the oracle and
source baselines, respectively. Let us briefly explain how this is achieved.

Let $\mathcal{D}$ be some dataset with distribution $p^{0}(X)$ that can be
split into parts $p(X_{\bar{c}}|X_{c})$ and $p(X_{c})$. As noted in Section 3,
we assume the latter changes (inducing a shift in distribution), while the
former is fixed. We denote the shifted marginal distribution as $p^{s}(X_{c})$
and the full distribution as $p^{s}(X)=p^{s}(X_{c})p(X_{\bar{c}}|X_{c})$.

In experiment 5.2, we desire a ground-truth target dataset for a given shift.
We do not have data from $p^{s}(X)$, however we can use rejection sampling to
_create_ such dataset, which we denote by $\mathcal{D}^{s}$. Since we do not
know $p^{0}(X)$ either, we sample from the empirical distribution, i.e. from
data $\mathcal{D}$ itself, which will converge to the true distribution when
$|\mathcal{D}|$ becomes large. To approximate $p^{0}(X_{c})$, we train a
simple KDE model and $p^{s}(X_{c})$ is defined by shifting this distribution
(see Section 4.2). This gives the following algorithm:

Input Source dataset $\mathcal{D}$, shift $T$ and desired shifted set size
$n_{s}$

Fit density model $\hat{p}^{0}(X_{c})$ to
$\\{\bf{x}_{c}|\bf{x}\in\mathcal{D}\\}$

$\hat{p}^{s}(X_{c})\leftarrow T(\hat{p}^{0})(X_{c})$

$M\leftarrow\max_{\bf{x}_{c}\in\mathcal{D}}\frac{\hat{p}^{s}(\bf{x}_{c})}{\hat{p}^{0}(\bf{x}_{c})}$

$\mathcal{D}^{s}\leftarrow\\{\\}$

while $|\mathcal{D}^{s}|<n_{s}$ do

Sample $\bf{x}$ from $\mathcal{D}$ uniformly

Sample $u\sim U(0,1)$

if $\frac{\hat{p}^{s}(\bf{x}_{c})}{\hat{p}^{0}(\bf{x}_{c})}>Mu$ then
$\mathcal{D}^{s}\leftarrow\mathcal{D}^{s}\cup\\{\bf{x}\\}$

end if

end while

return $\mathcal{D}^{s}$

Algorithm 1 Rejection sampling from source dataset $\mathcal{D}$, given a
predefined marginal shift $T$ and desired test set size.

In Experiment 5.2 we run the above with $\mathcal{D}$ an oracle test set.
Since the oracle test set is very large, it covers $p^{0}$ relatively well.
This allows us to approximate $p^{0}(X_{c})$, and also means that draws from
the empirical distribution are distributed approximately like the true
underlying distribution.

In experiment 5.3 we use a similar set-up for creating baseline _Source (RS)_
based on $\mathcal{D}_{test,f}$ alone, and to have a fair comparison we use
rejection sampling to weigh the points.666Effectively, this reduces to an
importance weighted estimate of the performance. In this case, however, the
distribution $p^{0}(X_{c})$, and in turn $p^{s}(X_{c})$, cannot be
approximated accurately. In addition, we may have very little data such that
the same points need to be included many times (in regions with large
$p^{s}(X_{c})/p^{0}(X_{c})$). As a result, although we see that _Source (RS)_
performs better than unshifted _Source (all)_ , it is a poor evaluation
approach.

###  Appendix C Generative Model Choice

Any generative model can be used to produce the synthetic test data, but some
models may be better or worse than others. 3S uses CTGAN [47] since this model
is designed specifically for tabular data and has shown good performance. In
this section, we explain how it is tuned and include a comparison to other
generative models.

####  C.1 Assessing the quality of generative model G in 3S.

Approaches to model selection and quality assessment of generative models
often measure the distance between the generated and the true distributions
[86]. In 3S , we use Maximum Mean Discrepancy (MMD) [56], a popular choice for
synthetic data quality [87, 88].

MMD performs a statistical test on distributions $P^{r}$ (Real) and $P^{g}$
(Generated), measuring the difference of their expected function values, with
a lower MMD implying $P^{g}$ is closer to $P^{r}$.

We use MMD in our auto-tuning and model selection step, comparing the
generated data to a held-out test set, with $G$ selected as the model with the
lowest MMD. This step also serves to ensure that the data generated by 3S is
indeed close to the real-world reference dataset of interest. Specifically,
hyper-parameters of 3S when training $G$ are tuned via a Tree-structured
Parzen Estimator. We search over the number of epochs of training [100, 200,
300, 500], learning rate [2e-4, 2e-5, 2e-6], and embedding dimension
[64,128,256]. For all methods, we have a small hyper-parameter validation set
with a size of 10% of the training dataset. Our objective is based on MMD
minimization.

That said, of course, alternative widely used metrics such as Inverse KL-
Divergence or the Jensen-Shannon divergence could also be used as metrics of
assessment.

####  C.2 Influence of model choice.

Any generative model can be used as the core of 3S . For efficiency, a
conditional generative model is highly desirable; this allows direct
conditioning on subgroup or shift information, and not e.g. post-generation
rejection sampling. Furthermore, some generative models may provide more or
less realistic data. Here we compare 3S estimates provided by CTGAN, vs
estimates given by TVAE and Normalizing Flows.

We assess these different base models for $G$ on the race subgroup task from
Sec. 5.1. Using 3S can assess the generative models based on MMD, but for
completeness we also show inverse KL-divergence and Jensen-Shannon Divergence
(JSD), where the metrics are computed vs a held-out validation dataset.

We show in Table 10, that the better quality metric does indeed translate into
better performance when we use the synthetic data for model evaluation. We
find specifically that CTGAN outperforms the other approaches, serving as
validation for our selection.

Additionally, the results highlight that for practical application, one could
evaluate the quality of $G$ first using metrics such as MMD, inverse KLD, or
JSD, as a proxy for how well the generative model should perform.

We assume for the purposes of this experiment that the three classes of models
are trained with the same optimization hyperparameters (epochs=200, learning
rate=2e-4).

Table 10: Assessing the influence of model choice for $G$ and illustrating how
our quality assessment metrics in 3S can be used to select the best model
which indeed will provide the best performance. We see that indeed CTGAN
performs best in this case.

Base $G$ | MMD $\downarrow$ | Inverse KLD $\uparrow$ | JSD $\downarrow$ | Subgroup (%) | Mean Absolute Error % $\downarrow$
---|---|---|---|---|---
CTGAN | 0.0014 | 0.995 | 0.03 | #1 (86%) | 0.28 $\pm$ 0.24
#2 (9%) | 17.64 $\pm$ 0.29
#3 (3%) | 2.96 $\pm$ 1.02
#4 (1%) | 1.14 $\pm$ 0.62
|  |  |  | #5 (1%) | 1.03 $\pm$ 0.85
NF | 0.0034 | 0.970 | 0.09 | #1 (86%) | 16.25 $\pm$ 0.53
#2 (9%) | 26.35 $\pm$ 1.07
#3 (3%) | 20.50 $\pm$ 3.31
#4 (1%) | 27.14 $\pm$ 0.97
|  |  |  | #5 (1%) | 26.04 $\pm$ 2.87
TVAE | 0.4557 | 0.4987 | 0.505 | #1 (86%) | 25.93 $\pm$ 1.34
#2 (9%) | 35.40 $\pm$ 0.83
#3 (3%) | 24.05 $\pm$ 1.12
#4 (1%) | 33.0 $\pm$ 0.31
|  |  |  | #5 (1%) | 37.69 $\pm$ 0.61

Why do small changes in metrics, lead to large MAE differences? One might
wonder that small changes in for example JSD lead to large MAE. For instance,
CTGAN to NF of 0.03 to 0.09. We assess the sensitivity of performance
estimates to small changes in divergence measures.

To do so, we conduct an experiment where we synthetically increase the JSD of
the synthetic dataset through corruption. We then compute the MAE performance
estimate on the synthetic dataset compared to the Oracle for different JSD
values.

The result in Figure 7 shows that even small changes in the JSD can
significantly harm the MAE. This provides an explanation of why a method with
seemingly relatively similar JSD (or divergence metric), with only a minor
difference, may perform very differently as measured by MAE. This motivates
why we select the generative model, for example with the lowest metric despite
them looking similar.

Figure 7: Small changes in JSD lead to large changes in MAE vs oracle.

###  Appendix D Limitations and Failure Cases

####  D.1 Limitations

We summarise 3S’s main limitations (where applicable referencing limitations
already outlined in the paper).

  1. 1.

Subgroups in real test set are large and 3S is unnecessary. When the aim is
subgroup evaluation and there is sufficient real data for this subgroup, the
real data estimate on $\mathcal{D}_{test,f}$ will be sufficiently accurate and
there is no need to use 3S given the computational overhead. See*Failure Case
1 for new experiments and further discussion. Note, even for very large
datasets, there can be very sparse regions or small subgroups for which using
3S is still beneficial. Also note that this limitation mostly applies to
subgroup evaluation and less to Generating Synthetic Data with Shifts, because
performance estimates for possible shifts is less trivial using real data
alone (e.g. require reweighting or resampling of test data) and we show (Sec.
5.2, Table 1 and Fig. 6(b)) that 3S beats real data baselines consistently.

  2. 2.

Possible errors in the generative process. Errors in the generative process
can affect downstream evaluation. This is especially relevant for groups or
regions with few samples, as it is more likely a generative model does not fit
the distribution perfectly here. By replacing the generative model by a deep
generative ensemble [25], 3S provides insight into its own generative
uncertainty. When there is very little data and 3S’s uncertainty bounds are
too large, practitioners should consider gathering additional data. See
Failure Case 2 below.

  3. 3.

Not enough knowledge or data to generate realistic shifts. The success of
modelling data with distributional shifts relies on the validity of the
distributional shift’s assumptions. This is true for 3S as much as for
supervised works on distributional shifts (e.g. see [89]). In Sec. 5.2.2 we
show how a lack of shift knowledge may affect evaluation. We aimed to generate
data to test a model on UK patients, using mostly data from US patients. We do
not define the shift explicitly—we only assume that we observe a small number
(1 to 4) of features $X_{c}$ (hence the conditional distribution of the rest
of the features conditional on these features is the same across countries).
In Fig. 6b, we show the assumption does not hold when we only see one
feature—this lack of shift knowledge is a failure case. When we add more
features, the assumption is more likely to approximately hold and 3S converges
to a better estimate. We reiterate that invalid shift assumptions are a
limitation of any distributional shift method, e.g. the rejection sampling
(RS) baseline using only real data is worse than 3S overall.

  4. 4.

Computational cost. The computational cost and complexity of using generative
models is always higher than using real test data directly. For typical
tabular datasets, the computational requirements are often very manageable:
under 5 min for most datasets and under 40 min for the largest dataset Bank
(Table 11). The reported times correspond to the dataset sizes in the paper.
We would like to emphasise that the cost at deployment time for a poorly
evaluated model can be unexpectedly high, which warrants 3S’s higher
evaluation cost. Additionally, pre-implemented generative libraries (e.g.
Patki 2016, Qian 2023) can accelerate the generative process, automating
generator training with minimal user input.

Table 11: Training time for generator on an NVIDIA GeForce RTX3080 for main paper settings. Dataset | Adult | Covid | Drug | Support | Bank
---|---|---|---|---|---
Tuning time (min) | 4.5 | 4 | 1.5 | 2 | 38

####  D.2 Failure cases

For limitations 1 and 2 mentioned above, we include two new experiments
highlighting failure cases on two extreme settings:

  1. 1.

3S is similarly accurate to real data evaluation when the real test data is
large.  With sufficiently large real data, 3S provides no improvement despite
higher complexity. We can determine sufficient size by estimating the variance
$Var(A)$ of the performance metric $A(f;D,S)$ w.r.t. the random variable
denoting the test data $D$. With only access to one real dataset $D_{test}$
however, we can approximate $Var(A)$ via bootstrapping (Fig. 5). If $Var(A)$
falls below a small threshold $\alpha$ (e.g. 0.05), practitioners may for
example decide to trust their real data estimate and not use 3S, as further
evaluation improvements are unlikely. In our Bank dataset experiment (500k
examples), we vary age and credit risk thresholds, retaining samples above
each cut-off to shrink the test set (see Figure 9). Note that for large
datasets but very small subgroups, the $D_{test,f}$ estimate still has a high
variance, (reflected in the large bootstrapped $Var(A)$ ), hence this should
urge a practitioner to use 3S.

  2. 2.

Large uncertainty for very small test sets. At the other extreme, when there
are too few samples the uncertainty of 3S (quantified through a DGE [25]) can
become too large. In Figure 8 in which we reduce subgroups to fewer than 10
samples in the test data. We train $G$ in 3S on the overall test set (which
includes the small subgroup) with $n_{samples}$. Despite good performance
versus an oracle, the uncertainty intervals from 3S’s ensemble span 0.1-0.2.
These wide intervals make the 3S estimates unreliable and less useful, and
would urge a practitioner to consider gathering additional data. The key
takeaway: With extremely sparse subgroups, the large uncertainties signal that
more data should be gathered before relying on 3S’s uncertain estimates.

(a) Covid

(b) Drug

(c) Support

Figure 8: Limitation Analysis: Extremely small sample size

(a) Credit risk vs Oracle

(b) Age vs Oracle

Figure 9: Limitation analysis: Large sample size

###  Appendix E Additional Experiments

This appendix includes a number of additional results. First, we include
additional results for the main paper’s experiments, using different
downstream models, predictors, metrics, and baselines. Second, we discuss
other types of subgroup definitions, including subgroups based on points of
interest and density of regions. Third, we include results for 3S when we have
access to the training set of predictor model $f$; since this dataset is
usually larger, it can help train a more accurate generative model.

####  E.1 Experimental additions to main paper results

#####  E.1.1 Sanity check

(a) $\mathcal{D}_{test,f}$

(b) $\mathcal{D}_{oracle}$

(c) 3S

Figure 10: Qualitative assessment of synthetic data. T-SNE on the Adult
dataset comparing the real test data $\mathcal{D}_{test,f}$, oracle data
$\mathcal{D}_{oracle}$ and 3S data $\mathcal{D}_{syn}$. We find that 3S
generates synthetic test data that covers the oracle dataset well, despite
only having access to $\mathcal{D}_{test,f}$ during training. The data,
evaluating subgroups where blue (Race 1 - majority) and orange (Race 5 -
minority)

We perform a sanity check on the data generated by 3S as compared to
$\mathcal{D}_{oracle}$ and $D_{test}$. The results are visualized in Figure 10
and show that 3S generates synthetic test data that covers the oracle dataset
well, even on the minority subgroup in blue, capturing the distribution
appropriately.

#####  E.1.2 Subgroup performance evaluation: more metrics and downstream
models

Motivation. The performance of the model, on subgroups, is likely influenced
by the class of downstream predictive model $f$. We aim to assess the
performance of the granular subgroups for a broader class of downstream models
$f$. In addition, to accuracy, we assess F1-score performance estimates as
another showcase of downstream performance estimates.

Setup. This experiment evaluates the mean performance difference for (i) 3S ,
(ii) 3S +, and (iii) $\mathcal{D}_{test,f}$. We follow the same setup as the
granular subgroup experiment in Section 5.1. We increase the predictive models
beyond RF, MLP, and GBDT and further include SVM, AdaBoost, Bagging
Classifier, and Logistic Regression.

Analysis. Figures 11 and 12 illustrate that 3S , when evaluated with more
models, still better approximates the true performance on minority subgroups,
compared to test data alone. This is in terms of mean absolute performance
difference between predicted performance and performance evaluated by the
oracle.

See next page for results figures

(a) Random Forest

(b) Gradient Boosting

(c) MLP

(d) Adaboost

(e) Logistic regression

(f) Decision tree

Figure 11: Performance estimate error wrt an Oracle: F1-score

(a) Random Forest

(b) Gradient Boosting

(c) MLP

(d) Adaboost

(e) Logistic regression

(f) Decision tree

Figure 12: Performance estimate error wrt an Oracle: Accuracy

#####  E.1.3 Subgroup worst-case performance evaluation

Motivation. When estimating sub-group performance, of course we want to have
as low error as possible on average (i.e. low mean performance difference).
That said, average performance glosses over the worst-case scenario. We desire
that the worst-case mean performance difference is also low. This is to ensure
that, by chance, the performance estimates are not wildly inaccurate. This
scenario is particularly relevant, as by chance the testing data could either
over- or under-estimate model performance, leading us to draw incorrect
conclusions.

Setup. This experiment evaluates the worst-case mean performance difference
for (i) 3S , (ii) 3S +, and (iii) $\mathcal{D}_{test,f}$. We follow the same
setup as the granular subgroup experiment in Section 5.1.

Analysis. Figures 13 and 14 illustrates that 3S and the augmented 3S \+ have a
lower worst-case performance compared to evaluation with real test data. This
further shows that, by chance, evaluation with real data can severely over- or
under-estimate performance, leading to incorrect conclusions about the model’s
abilities. 3S ’s lower worst-case error, means even in the worst scenario,
that 3S ’s estimates are still closer to true performance.

See next page for results figures

(a) Random Forest

(b) Gradient Boosting

(c) MLP

(d) Adaboost

(e) Logistic regression

(f) Decision tree

Figure 13: Worst case, Performance estimate error wrt an Oracle: F1 Score

(a) Random Forest

(b) Gradient Boosting

(c) MLP

(d) Adaboost

(e) Logistic regression

(f) Decision tree

Figure 14: Worst case, Performance estimate error wrt an Oracle: Accuracy

#####  E.1.4 Fairness metrics: subgroup evaluation

Motivation. We have primarily studied reliable estimation of model performance
on different subgroups. This is easily generalized to estimate fairness
metrics of ML models on specific subgroups. This can provide further insight
into the use of synthetic data for model testing.

Setup. This experiment evaluates the mean performance difference for (i) 3S ,
(ii) 3S +, and (iii) $\mathcal{D}_{test,f}$. We follow the same setup as the
granular subgroup experiment in Section 5.1. We evaluate an RF model. We
assess the following fairness metrics: (i) Disparate Impact (DI) ratio
(demographic parity ratio) and (ii) Equalized-Odds (EO) ratio. When estimating
these metrics for each subgroup (e.g. race group), we then condition on sex as
the sensitive attribute.

The DI ratio is: the ratio between the smallest and the largest group-level
selection rate $E[f(X)|A=a]$ , across all values of the sensitive feature(s)
$a\in A$.

The EO ratio is the smaller of two metrics between TPR ratio (smallest and
largest of $P[f(X)=1|A=a,Y=1]$ , across all values of the sensitive
feature(s)) and FPR ratio (similar but defined for $P[f(X)=1|A=a,Y=0]$), ,
across all values of the sensitive feature(s) $a\in A$.

Analysis. Figures 15 and 16 illustrate that 3S ’s performance on both fairness
metrics, better approximates the true oracle metric on minority subgroups,
compared to test data alone. This is in terms of mean absolute performance
difference between predicted performance and performance evaluated by the
oracle.

Note, we also assess the worst-case scenario as well, as done previously.

Figures 17 and 18 illustrate that 3S and the augmented 3S \+ have a lower
worst-case estimated difference compared to evaluation with real test data.
This further shows that, by chance, evaluation with real data can over- or
under-estimate fairness, leading to incorrect conclusions about the model’s
abilities. 3S ’s lower worst-case error, means even in the worst scenario,
that 3S ’s estimates are still closer to the true fairness metric.

See next page for results figures

(a) Random Forest

(b) Gradient Boosting

(c) MLP

(d) Adaboost

(e) Logistic regression

(f) Decision tree

Figure 15: Fairness (in terms of EO) estimate error

(a) Random Forest

(b) Gradient Boosting

(c) MLP

(d) Adaboost

(e) Logistic regression

(f) Decision tree

Figure 16: DI estimate error

(a) Random Forest

(b) Gradient Boosting

(c) MLP

(d) Adaboost

(e) Logistic regression

(f) Decision tree

Figure 17: Worst case, EO estimate error

(a) Random Forest

(b) Gradient Boosting

(c) MLP

(d) Adaboost

(e) Logistic regression

(f) Decision tree

Figure 18: Worst case, DI estimate error

#####  E.1.5 Intersectional subgroups: performance matrix

Motivation. We perform a deep-dive of the intersectional performance matrices
generated by 3S , simply, using test data alone and the oracle.

Analysis. We first see in Table 12 that, in general, the intersectional matrix
3S has a much _lower_ error when estimating performance for intersectional
subgroups. i.e. this is more similar to the oracle, compared to using
$\mathcal{D}_{test,f}$. Note that we set the minimum number of samples
required for validation $=100$ samples. This induces sparseness, of course,
but is necessary in order to prevent evaluation on too few data points.
However, in cases where $\mathcal{D}_{test,f}$ does not have data for the
intersection (i.e. $n<100$), we do not consider these NaN blocks as part of
our calculation; in fact, this makes it easier for $\mathcal{D}_{test,f}$.

The rationale is evident when evaluating the intersectional performance
matrices for each group. We present the following findings.

  * •

3S ’s insights are correct: the underperforming subgroups, as noted by 3S ,
match the oracle. Therefore, it serves as further validation.

  * •

$\mathcal{D}_{test,f}$ is very sparse after cut-offs: the 100 sample cut-off
highlights the key challenge of evaluation on a test set. We may not have
sufficient samples for each intersection to perform an evaluation.

Table 12: Adult: Intersectional performance matrix difference vs the oracle Model | 3S | $\mathcal{D}_{test,f}$
---|---|---
Average | 0.13 $\pm$ 0.005 | 0.21 $\pm$ 0.002
RF | 0.133 | 0.211
GBDT | 0.128 | 0.207
MLP | 0.128 | 0.207
SVN | 0.138 | 0.209
AdaBoost | 0.126 | 0.206
Bagging | 0.138 | 0.211
LR | 0.126 | 0.207

#####  E.1.6 Incorporating prior knowledge on shift: Covid-19

Motivation. We have the ability to assess distributional shift where we have
some knowledge of the shifted distribution. Specifically, here we assume we
only observe a few of the features, from the target domain.

Setup. Our setup is similar to Section 5.2.2, however on a different dataset -
i.e. Covid-19. There are known distributional differences between the north
and south of Brazil. For example, different prevalence of respiratory issues,
sex proportions, obesity rates, etc. Hence we train the predictive model on
patients from the South (larger population) and seek to evaluate potential
performance on patients from the North. We take the largest sub-regions for
each.

To validate our estimate, we use the actual northern dataset (Target) as
ground-truth. Our baselines are as in Section 5.2.2. Since the features are
primarily binary, we parameterize the distributions as binomial with a
probability of prevalence for their features. We can then sample from this
distribution.

Analysis. Fig. 19 shows the average estimated performance of $f$, as a
function of the number of features observed from the target dataset. We see
that the 3S estimates are closer to the oracle across the board compared to
baselines. Furthermore, for increasing numbers of features (i.e. increasing
prior knowledge), we observe that 3S estimates converge to the oracle.

Figure 19: $\mathcal{D}_{syn}$ is better able to approximate performance in
the target domain compared to baselines and that performance improves as more
prior knowledge is incorporated via added features. Points are connected to
highlight trends.

Raw performance. We also illustrate raw estimate errors for different
downstream models in Table 13, which illustrates that 3S outperforms real data
alone, rejection sampling, as well as, specialized methods ATC, IM and DOC for
OOD prediction with lower downstream performance estimate errors.

Table 13: 3S has lower performance estimate error in target domain for different downstream models. Rows yellow have access to more information than 3S, in the form of unlabeled data from the target domain. $\downarrow$ is better | mean | ada | bag | gbc | mlp | rf | knn | lr | knn | dt
---|---|---|---|---|---|---|---|---|---|---
3S-Testing | 0.0446 | 0.0112 | 0.0962 | 0.0205 | 0.1777 | 0.0278 | 0.0100 | 0.0105 | 0.0100 | 0.0100
All (Source) | 0.2262 | 0.1029 | 0.3890 | 0.0929 | 0.2276 | 0.3732 | 0.1774 | 0.0521 | 0.1774 | 0.3942
RS (Source) | 0.1421 | 0.1842 | 0.1781 | 0.1303 | 0.2954 | 0.1613 | 0.0497 | 0.1138 | 0.0497 | 0.0244
ATC [31] | 0.1898 | 0.0236 | 0.2948 | 0.0542 | 0.0331 | 0.2264 | 0.3985 | 0.0638 | 0.3985 | 0.4198
IM [33] | 0.1782 | 0.1030 | 0.3638 | 0.0330 | 0.1563 | 0.2828 | 0.0555 | 0.0140 | 0.0555 | 0.4170
DOC [32] | 0.1634 | 0.1019 | 0.2646 | 0.0304 | 0.1557 | 0.2215 | 0.1186 | 0.0010 | 0.1186 | 0.4060

#####  E.1.7 Sensitivity to shifts on additional features

Motivation. In the main paper, we have characterized the sensitivity across
operating ranges for two features in two datasets (Adult and SEER). Ideally, a
practitioner would like to understand the sensitivity to all features in the
dataset. We now conduct this assessment on the Adult dataset, producing model
sensitivity curves for all features.

Analysis. We include the model sensitivity curves for all features as part of
the example model report in Appendix F, see Figures 25, (a)-(t).

####  E.2 Other definitions of subgroups

We illustrate how the definition of subgroup allows two other interesting use
cases for granular evaluation: (i) subgroups defined using a point of interest
(i.e. how would a model perform on patients that look like X), and (ii)
subgroups defined in terms of local density (i.e. how would a model perform on
unlikely patients).

#####  E.2.1 Subgroups relating to points of interest

Motivation. End-users may also be interested in knowing how a model performs
on some point of interest $x^{*}$. For example, a clinician may have access to
a number of models and may need to decide for the specific patient in front of
them what the model’s potential predictive performance might be for the
specific patient. This is relevant because global performance metrics may hide
that models underperform for specific samples.

We can often assess model performance on samples similar to the sample of
interest, i.e. “neighborhood performance”. A challenge in reality is that we
often only have access to a held-out test dataset (i.e. $\mathcal{D}_{test}$),
rather than the entire population (i.e. $\mathcal{D}_{oracle}$). How then can
we quantify performance?

Usually, it is not possible to assess local performance using test data alone,
since there may be very few samples that are similar enough—with “similar”
defined in terms of some distance metric, i.e.
$\mathcal{S}=\\{x\in\mathcal{X}|d(x,x^{*})<\epsilon\\}$, with distance metric
$d$ and some small distance $\epsilon\geq 0$. As before, we can instead
generate synthetic data in the region $\mathcal{S}$ and compute the
performance on this set instead. This reduces the dependence on the small
number of samples, in turn reducing variance in the estimated performance.

Set-up. We compare two approaches: (i) find nearest-neighbor points in
$\mathcal{D}_{test}$)—which might suffer from limited similar samples—, or
(ii) use 3S and generate synthetic samples $\mathcal{D}_{syn}$ in some
neighborhood of $x^{*}$. Similarly as before, we assess these two methods by
comparing estimates with a pseudo-ground truth that uses nearest neighbors on
a much larger hold-out set, $\mathcal{D}_{oracle}$. Again, we compare (1)
_mean absolute performance difference_ and (2) _worst-case performance
difference_ between a specific evaluation set and the oracle dataset. We
average across 10 randomly queried points $x^{*}$ and use $k=10$ nearest
neighbors.

Analysis. The results in Table 14 show that 3S has a much lower neighborhood
performance gap, both average and worst case for $x^{*}$ across models, when
compared to the assessment using $\mathcal{D}_{test}$. The rationale is that
by using synthetic data, we can generate more examples $\mathcal{D}_{syn}$
that closely resemble $x^{*}$, whereas with $\mathcal{D}_{test}$ we might be
limited in the similar samples that can be queried - hence resulting in the
higher variance estimates and poorer overall performance.

Table 14: Comparing two types of query methods to evaluate performance on
points of interest $x^{*}$, which illustrates that 3S closer approximates an
oracle both on average and worst case.

Model | Mean performance difference $\downarrow$ | Worst-case performance difference $\downarrow$
---|---|---
|  $\mathcal{D}_{syn}$ (3S ) | $\mathcal{D}_{test,f}$ |  $\mathcal{D}_{syn}$ (3S ) | $\mathcal{D}_{test,f}$
MLP | 0.083 | 0.15 | 0.482 | 0.60
RF | 0.086 | 0.18 | 0.256 | 0.50
GBDT | 0.093 | 0.18 | 0.50 | 0.50

Takeaway. 3S ’s synthetic data can more robustly estimate performance on
individual points of interest based on the samples generated in the
neighborhood of $x^{*}$.

#####  E.2.2 Subgroups as high- and low-density regions

Motivation Models often perform worse on outliers and low-density regions due
to the scarcity of data during training. We generate insight into this by
defining subgroups in terms of density.

Methodology.

We would like to partition the points into sets of most likely to least likely
w.r.t. density. We use the notion of $\alpha$-support from [90], namely:

$\text{Supp}^{\alpha}(p)=\underset{S\subseteq\tilde{\mathcal{X}}}{\arg\min}\
V(S)\ s.t.\ p(\tilde{X}\in S)=\alpha,$ (3)

with $V$ some volume measure (e.g. Lebesgue). In other words, $\alpha$-support
$\text{Supp}^{\alpha}(p)$ denotes the smallest possible space to contain
$\tilde{X}$ with probability $\alpha$—which can be interpreted as the $\alpha$
most likely points. Subsequently, we can take a sequence of quantiles,
$(q_{i})_{i=0}^{k}\in[0,1],$ with $q_{i}=\frac{i}{k}$ and look at the sequence
of support sets, $(\text{Supp}^{q_{i}}(p))_{i=0}^{k}$.

The $\alpha$-support itself always contains the regions with the highest
density. To actually partition the points into sets from likely to unlikely,
we instead look at the difference sets. That is, let us define sets
$S_{i}=\text{Supp}^{q_{i}}(p)\backslash\text{Supp}^{q_{i-1}}(p)$ for
$i=1,...,k$.

In fact, we do not know $p$ exactly and even if we did, it is usually
intractable to find an exact expression for the $\alpha-$support. Instead, we
compute the $\alpha$-support in the generative model’s latent space, and not
the original space. The latent distribution is usually chosen as a
$d_{z}$-dimensional standard Gaussian, which has two advantages: (i) the
distribution is continuous—cf. the original space, in which there may exist a
lower-dimensional manifold on which all data falls; and (ii) the
$\alpha$-support is a simple sphere, with the radius given by
$\text{CDF}^{-1}_{\chi^{2}_{d_{z}}}(\alpha)$.

Figure 20: Performance on likely vs unlikely data, where error bars are over
10 runs

Set-up. We train a generative model
$G:\mathcal{Z}\rightarrow\tilde{\mathcal{X}}$ as before, where the input
$\mathbf{z}$ is $d_{z}$-dimensional Gaussian noise. During generation we save
inputs $\\{z_{j}\\}_{j=1}^{kN}$ and generate corresponding data
$\mathcal{D}_{syn}=\\{\tilde{x}_{j}\\}_{j=1}^{kN}$. Sets for likely and
unlikely points are defined using $\alpha$-support in latent space
$\mathcal{Z}$, for quantiles $\mathbf{q}=(q_{i})_{i=1}^{k}$—see Section E.2.2.
Specifically, let the index sets for each quantile set be given by
$I_{i}=\\{j|||z_{j}||_{2}<\text{Quantile}(\\{||z_{j}||_{2}\\},q_{i})\\}$,
giving synthetic sets
$\mathcal{D}_{syn}^{q_{i}}=\\{\tilde{x}_{j}\in\mathcal{D}_{syn}|j\in I_{i}\\}$
for all $i$. We then evaluate the predictive model $f$ on each
$\mathcal{D}_{syn}^{q_{i}}$ and plot results w.r.t. $\mathbf{q}$. We use the
SEER dataset.

Analysis. Fig. 20 shows that for the tail quantiles (unlikely) data that the
model performance is indeed highly variable and prone to possible poor
performance. This is in contrast to the likely data, which has much more
stable performance.

Takeaway. 3S ’s synthetic data can be used to quantify model performance on
unlikely and likely data.

####  E.3 Improving $G$ when training data of $f$ is available

Motivation. In previous experiments, we have assumed that we have access to
only $\mathcal{D}_{test,f}$ for training $G$. In some scenarios, we have
access to $\mathcal{D}_{train,f}$. For example, the model developer has access
to $\mathcal{D}_{train,f}$. Since the training dataset is often larger than
the testing dataset, we could use this data to train $G$. Evidently, this
results in some bias: we are now using synthetic data generated using $G$,
which is trained on $\mathcal{D}_{train,f}$, to evaluate a predictive model
that is also trained on $\mathcal{D}_{train,f}$. However, here we show that
this bias is outweighed by the improved quality of $G$.

Setup. This experiment evaluates the mean performance difference for (i) 3S ,
(ii) 3S +, and (iii) $\mathcal{D}_{test,f}$. We follow the same setup as the
granular subgroup experiment in Section 5.1, with the only difference being
that $\mathcal{D}_{train,G}=\mathcal{D}_{train,f}$. We assess the utility of
this setup for granular subgroup evaluation.

Analysis. Table 15 illustrates the performance similar to that of the main
paper. We find that 3S mostly provides a more accurate evaluation of model
performance (i.e. with estimates closer to the oracle) compared to a
conventional hold-out dataset. This is especially true for the smaller
subgroups for which synthetic data is indeed necessary. Furthermore, 3S has a
lower worst-case error compared to $\mathcal{D}_{test,f}$. The results with 3S
\+ also illustrate the benefit of augmenting real data with synthetic data,
leading to lower performance differences.

Table 15: Mean absolute performance difference between predicted performance
and performance evaluated by the oracle, where $G$ is trained on
$\mathcal{D}_{train,f}$. 3S better approximates true performance on minority
subgroups, compared to test data alone.

Model | Subgroup (%) | Mean performance diff. $\downarrow$ | Worst-case performance diff. $\downarrow$
---|---|---|---
|  | 3S |  3S + | $\mathcal{D}_{test,f}$ | 3S |  3S + | $\mathcal{D}_{test,f}$
RF | #1 (86%) | 6.05 $\pm$ 0.72 | 2.33 $\pm$ 0.42 | 9.16 $\pm$ 4.48 | 7.08 | 2.84 | 11.65
#2 (9%) | 4.16 $\pm$ 0.33 | 4.29 $\pm$ 0.47 | 5.85 $\pm$ 3.18 | 4.44 | 4.82 | 8.94
#3 (3%) | 2.15 $\pm$ 1.04 | 1.88 $\pm$ 0.8 | 13.59 $\pm$ 5.34 | 3.41 | 3.03 | 19.73
#4 (1%) | 4.54 $\pm$ 0.33 | 4.54 $\pm$ 0.34 | 7.1 $\pm$ 3.93 | 4.89 | 4.96 | 13.23
| #5 (1%) | 3.15 $\pm$ 0.77 | 3.06 $\pm$ 0.77 | 8.72 $\pm$ 0.0 | 4.2 | 4.12 | 8.72
GBDT | #1 (86%) | 7.37 $\pm$ 1.09 | 3.25 $\pm$ 0.32 | 1.19 $\pm$ 0.81 | 9.22 | 3.6 | 2.42
#2 (9%) | 3.44 $\pm$ 0.52 | 3.37 $\pm$ 0.54 | 2.56 $\pm$ 1.08 | 3.98 | 4.02 | 4.49
#3 (3%) | 2.6 $\pm$ 1.53 | 2.46 $\pm$ 1.38 | 4.53 $\pm$ 1.04 | 4.29 | 3.97 | 5.73
#4 (1%) | 0.76 $\pm$ 0.58 | 0.76 $\pm$ 0.57 | 6.51 $\pm$ 5.07 | 1.31 | 1.34 | 14.53
| #5 (1%) | 2.23 $\pm$ 0.82 | 2.15 $\pm$ 0.85 | 7.73 $\pm$ 3.33 | 3.47 | 3.45 | 9.4
MLP | #1 (86%) | 6.72 $\pm$ 1.2 | 3.0 $\pm$ 0.46 | 1.01 $\pm$ 0.62 | 8.55 | 3.5 | 1.92
#2 (9%) | 4.32 $\pm$ 0.46 | 4.13 $\pm$ 0.54 | 2.45 $\pm$ 1.46 | 4.91 | 4.87 | 4.41
#3 (3%) | 1.98 $\pm$ 1.1 | 1.88 $\pm$ 0.99 | 4.06 $\pm$ 0.8 | 3.58 | 3.33 | 4.69
#4 (1%) | 2.57 $\pm$ 0.5 | 2.49 $\pm$ 0.53 | 7.44 $\pm$ 3.17 | 3.37 | 3.36 | 12.42
| #5 (1%) | 2.17 $\pm$ 0.62 | 2.13 $\pm$ 0.64 | 4.3 $\pm$ 3.66 | 3.24 | 3.22 | 8.72
SVM | #1 (86%) | 6.74 $\pm$ 0.58 | 3.0 $\pm$ 0.49 | 1.26 $\pm$ 0.93 | 7.37 | 3.6 | 2.46
#2 (9%) | 5.62 $\pm$ 0.14 | 5.31 $\pm$ 0.4 | 3.63 $\pm$ 1.25 | 5.83 | 5.81 | 5.63
#3 (3%) | 1.09 $\pm$ 0.73 | 1.04 $\pm$ 0.73 | 2.51 $\pm$ 1.09 | 2.43 | 2.33 | 3.88
#4 (1%) | 1.08 $\pm$ 0.4 | 0.97 $\pm$ 0.39 | 11.62 $\pm$ 4.31 | 1.51 | 1.44 | 16.01
| #5 (1%) | 4.22 $\pm$ 0.6 | 4.17 $\pm$ 0.61 | 3.1 $\pm$ 2.99 | 5.2 | 5.18 | 6.71
AdaBoost | #1 (86%) | 6.89 $\pm$ 0.96 | 3.06 $\pm$ 0.31 | 1.11 $\pm$ 0.71 | 8.61 | 3.44 | 2.14
#2 (9%) | 3.91 $\pm$ 0.3 | 3.76 $\pm$ 0.44 | 2.31 $\pm$ 1.86 | 4.24 | 4.31 | 5.15
#3 (3%) | 2.13 $\pm$ 1.43 | 2.01 $\pm$ 1.3 | 4.59 $\pm$ 2.48 | 4.22 | 4.0 | 7.79
#4 (1%) | 1.24 $\pm$ 0.65 | 1.16 $\pm$ 0.69 | 8.66 $\pm$ 6.46 | 1.99 | 1.97 | 19.89
| #5 (1%) | 3.42 $\pm$ 0.71 | 3.36 $\pm$ 0.73 | 5.17 $\pm$ 3.55 | 4.47 | 4.44 | 8.05
Bagging | #1 (86%) | 5.81 $\pm$ 1.02 | 2.65 $\pm$ 0.53 | 10.01 $\pm$ 4.49 | 7.03 | 3.61 | 12.83
#2 (9%) | 4.89 $\pm$ 0.68 | 5.06 $\pm$ 0.44 | 7.65 $\pm$ 3.22 | 5.97 | 5.54 | 10.69
#3 (3%) | 3.72 $\pm$ 1.65 | 3.37 $\pm$ 1.64 | 8.5 $\pm$ 4.77 | 5.98 | 5.67 | 14.55
#4 (1%) | 7.01 $\pm$ 0.28 | 7.0 $\pm$ 0.3 | 6.16 $\pm$ 3.51 | 7.46 | 7.48 | 11.21
| #5 (1%) | 5.06 $\pm$ 1.09 | 5.0 $\pm$ 1.09 | 4.41 $\pm$ 4.07 | 6.59 | 6.55 | 9.4
LR | #1 (86%) | 6.51 $\pm$ 0.63 | 2.97 $\pm$ 0.3 | 1.09 $\pm$ 0.41 | 7.57 | 3.38 | 1.81
#2 (9%) | 4.11 $\pm$ 0.32 | 3.9 $\pm$ 0.51 | 3.33 $\pm$ 0.98 | 4.52 | 4.52 | 4.57
#3 (3%) | 1.15 $\pm$ 0.73 | 1.11 $\pm$ 0.64 | 4.6 $\pm$ 2.72 | 2.28 | 2.08 | 7.61
#4 (1%) | 1.69 $\pm$ 0.59 | 1.62 $\pm$ 0.62 | 7.16 $\pm$ 4.27 | 2.57 | 2.55 | 13.48
| #5 (1%) | 4.08 $\pm$ 0.76 | 4.02 $\pm$ 0.78 | 4.76 $\pm$ 3.21 | 5.44 | 5.41 | 7.38

###  Appendix F Example Model Report

Below we present an example of the type of model report that could be produced
when evaluating models using 3S testing.

Dataset: Adult [57].

Intersectional model performance matrix: diagnosing at a granular level.

Figure 21: Intersectional performance matrix for the RF model, which diagnoses
underperforming 2-feature subgroups (darker implies underperformance).

INSIGHT: Model underperformance on self-employed who work more than 80 hours a
week. (bottom right arrow)

Model sensitivity curves: helping to understand performance trends for shifts
across the operating range:

(a) Age: Performance decreases as mean age increases.

(b) Country of origin: performance increases as the proportion of non-US
individuals increases

(c) Number of years of education. Performance decreases as mean number of
education years increases.

(d) Employment type (government): performance increases as the proportion of
government employed individuals increases

(e) Employment type (private): performance increases as the proportion of
private employed individuals increases

(f) Employment type (self-employed): performance is consistent even as
proportion of self-employed individuals increases

Figure 22: Model sensitivity curves for different features, illustrating the
relationships/model performance across the operating range.

(a) Employment type (without pay): performance is consistent even as
proportion of without-pay individuals increases

(b) Marital status (couple): performance decreases as the proportion of
married individuals increases

(c) Race (American-Indian-Eskimo): performance remains consistent even as the
proportion of American-Indian-Eskimo individuals increases

(d) Race (Asian-Pacific-Islander): performance remains consistent even as the
proportion of Asian-Pacific-Islander individuals increases

(e) Race (Black): performance increases as the proportion of black individuals
increases

(f) Race (Other): performance increases as the proportion of other individuals
increases

Figure 23: Model sensitivity curves for different features, illustrating the
relationships/model performance across the operating range.

(a) Race (White): performance decreases as the proportion of white individuals
increases

(b) Relationship (Husband): performance decreases as the proportion of
individuals classed as husbands increases

(c) Relationship (Not-in-family): performance decreases as the proportion of
individuals classed as Not-in-family increases

(d) Relationship (other-relative): performance remains consistent as the
proportion of individuals classed as other-relative increases

(e) Relationship (Own-child): performance remains consistent as the proportion
of individuals classed as Own-child increases

(f) Relationship (Unmarried): performance increases as the proportion of
individuals classed as Unmarried increases

Figure 24: Model sensitivity curves for different features, illustrating the
relationships/model performance across the operating range.

(a) Relationship (Wife): performance decreases as the proportion of
individuals classed as Wife increases

(b) Sex (Female): performance increases as the proportion of females increases

Figure 25: Model sensitivity curves for different features, illustrating the
relationships/model performance across the operating range.