1Department of Biomedical Data Science, Stanford, California, USA
2 Department of Epidemiology and Population Health, Stanford, California, USA
3Center for Biomedical Informatics Research, Stanford, California, USA
4Division of Hospital Medicine, Stanford, California, USA
5 Clinical Excellence Research Center, Stanford, California, USA

# Avoiding Biased Clinical Machine Learning Model Performance Estimates in the
Presence of Label Selection

Conor K. Corbin1,3 Michael Baiocchi2 Jonathan H. Chen  MD  PhD3,4,5

When evaluating the performance of clinical machine learning models, one must
consider the deployment population. When the population of patients with
observed labels is only a subset of the deployment population (label
selection), standard model performance estimates on the observed population
may be misleading. In this study we describe three classes of label selection
and simulate five causally distinct scenarios to assess how particular
selection mechanisms bias a suite of commonly reported binary machine learning
model performance metrics. Simulations reveal that when selection is affected
by observed features, naive estimates of model discrimination may be
misleading. When selection is affected by labels, naive estimates of
calibration fail to reflect reality. We borrow traditional weighting
estimators from causal inference literature and find that when selection
probabilities are properly specified, they recover full population estimates.
We then tackle the real-world task of monitoring the performance of deployed
machine learning models whose interactions with clinicians feed-back and
affect the selection mechanism of the labels. We train three machine learning
models to flag low-yield laboratory diagnostics, and simulate their intended
consequence of reducing wasteful laboratory utilization. We find that naive
estimates of AUROC on the observed population undershoot actual performance by
up to 20%. Such a disparity could be large enough to lead to the wrongful
termination of a successful clinical decision support tool. We propose an
altered deployment procedure, one that combines injected randomization with
traditional weighted estimates, and find it recovers true model performance.

## Introduction

Increased access to real-world data has accelerated the use of machine
learning (ML) models for precision medicine. ML practitioners tasked with
evaluating the performance of these models must consider the population of
patients on which inference will be performed — the target population. If a
held out test set used to evaluate a clinical ML model’s performance
represents only a subset of examples in the target population, model
performance metrics estimated using the test set may differ from the model’s
performance in production.

One mechanism that drives these situations is label selection (censoring) [1,
2]. Consider the time-to-event problem framed as a binary task of predicting
all cause 1-year mortality using electronic medical record (EMR) data [3].
Without linking to external sources, for example the social security death
index, class labels for patients (examples) that lack encounters one year
after prediction time may not be observed. An ML practitioner tasked with
evaluating this model may find it convenient to only include examples with
observed labels in their analysis. However in deployment, this model will
necessarily perform inference on examples whose labels go unobserved. If the
model performs preferentially for examples more likely to be observed (a
plausible scenario given the model will see more of this data during training)
then estimates of model performance on the test set will overshoot the model’s
real-world performance in deployment. Label selection is not a phenomena
strictly secluded to time-to-event tasks. Consider the ML task of flagging
low-yield laboratory test results to mitigate wasteful laboratory utilization;
and conversely the task of suggesting additional testing when results are less
certain [4, 5, 6, 7]. If a test set only includes the subset of patients for
whom laboratory tests were ordered (examples with observed labels) yet the
model is deployed on a more general population, performance estimates on the
test set may fail to capture true performance in production.

Label selection can occur after successful model development and retrospective
validation. Deployed clinical prediction models exhibit feed-back loops and
affect prospectively collected data [8]. One way in which a deployed model can
induce feed-back is through label selection. Consider a laboratory prediction
task similar to the above designed to alert clinicians of laboratory orders
with highly predictable results — this time with the sole purpose of helping
clinicians avoid ordering wasteful diagnostics. Once deployed, and assuming
some adherence, labels of laboratory orders the model was most sure of will be
go unobserved, as the tests will not be ordered. Without taking the selection
mechanism into consideration, an ML practitioner tasked with monitoring the
performance of an already deployed model might incorrectly conclude that it
has experienced a degradation in predictive acuity, and perhaps needlessly
remove it from production. In reality what the ML practitioner is noticing is
not a degradation in performance but rather the model’s intended effect — an
enrichment of ambiguous cases and removal of those in which test results are
mostly certain.

Though selection mechanisms and their implications on estimates of average
treatment effects have been widely studied in causal inference literature [9,
10, 11], label selection in the context of evaluating and monitoring the
performance of clinical ML models has received less attention. In one related
study, Powers et al. discuss the evaluation of clinical ML models when the
distribution of features in a source population varies from that of a target
population [12]. In ML literature this phenomena is called covariate shift
[13, 14]. Label selection that depends only on observed features (selection at
random) is an instance of covariate shift. We expand upon this work by
simulating over several classes of label selection, revealing in which cases
ML performance metrics are biased. In another related work, the impact of
selection mechanisms on model fitting is assessed [15]. While the author does
discuss the implications of selection on estimates of model risk, we expand
upon their work by addressing its impact on a suite of common ML performance
metrics, and discuss when certain kinds of selection mechanisms bias some
metrics and not others. Further, we study label selection as it relates to the
real-world use case of monitoring the performance of ML models whose
deployments feed-back and affect the selection mechanism of prospectively
collected data.

In this study we describe three classes of label selection: selection
completely at random, selection at random, and selection not at random,
borrowing terminology from Little & Rubin [16]. Adhering to this
categorization, we simulate five causally distinct scenarios to assess how
different label selection mechanisms impact a suite of commonly tracked binary
ML performance metrics, measuring both discrimination and calibration. In each
scenario we assess the ability of traditional weighting estimators from causal
inference literature to recover true model performance [17, 18]. We then study
the effect of label selection on a real-world task. We train three ML models
using EMR data to predict the results of stand-alone laboratory tests,
intended to reduce wasteful laboratory utilization. We evaluate the
performance of these models retrospectively, and then simulate their
deployment and feed-back on new data (label selection). We measure
inconsistencies between observed and actual performance, and then propose a
simple correction method that couples injected randomization into the clinical
decision support alerting mechanism with traditional weighting estimators.
Simulations reveal that this approach corrects disparities between observed
and actual performance using only the observed data, without the need to
assume away unmeasured confounding.

## Methods

### Simulation study

Here we study the effect of label selection on common binary ML model
performance metrics. We discuss three broad categories of selection
mechanisms: selection completely at random, selection at random, and selection
not at random. We simulate five causally distinct scenarios. In each scenario
we estimate model performance in terms of both discrimination and calibration
using a suite of commonly tracked performance metrics. Discrimination is
tracked with estimates of specificity, sensitivity, accuracy, positive
predictive value (PPV), area under the receiver operating characteristics
curve (AUROC) and area under the precision/PPV recall curve (AUPRC). Threshold
dependent metrics are estimated with the threshold set to $t=0.5$. Calibration
is estimated with calibration plots; that is, we bin predictions into $n=5$
equal width buckets and compare the average predicted probability within the
bucket to the actual prevalence of the outcome. We track all performance
metrics across the full and observed populations for all scenarios. We then
assess the ability of traditional weighting estimators from causal inference
literature to recover discrepancies in model performance between full and
observed populations.

#### Data generating process

In each scenario we sample a dataset $\\{x^{i},y^{i},s^{i}\\}_{i=0}^{n}$ of
size $n=10,000$ from a joint distribution $P(X,Y,S)$ that we specify. Each
example is a tuple of features vector $x^{i}$, binary class label $y^{i}$, and
binary selection variable $s^{i}$ indicating whether the class label of the
example is observed. For all three scenarios, $X$ and $Y$ are sampled
according to the following data generating process where $\sigma(x)$ is the
sigmoid function.

$\displaystyle X_{1},X_{2}$ $\displaystyle\sim Uniform(\alpha,\beta)$ (1)
$\displaystyle Y$ $\displaystyle\sim
Bernoulli(p=\sigma(\omega_{1}x_{1}+\omega_{2}x_{2}+\gamma))\quad\text{where}\quad\sigma(x)=\frac{1}{1+e^{-x}}$
(2)

We allow a hypothetical clinical ML model $h(x):X\rightarrow Y$ to perfectly
specify the mapping from $X$ to $Y$ by assigning it the following functional
form: $h(x):=\sigma(\omega_{1}x_{1}+\omega_{2}x_{2}+\gamma)$. For all
simulated scenarios, we set $\alpha,\beta:=2$, $\omega_{1},\omega_{2}:=1$, and
$\gamma:=0$. Note that although we have a perfectly specified model,
irreducible error induced by stochastic sampling of $Y$ exists leading to
imperfect discrimination acuity [19]. Our model however is perfectly
calibrated. Our data generating process induces a decision boundary on the
line described by $\omega_{1}X_{1}+\omega_{2}X_{2}+\gamma=0$.

#### Selection completely at random

In Scenario 1 we simulate an environment where class labels are selected
completely at random. Concretely, our selection indicator $S$ is independent
of both $X$ and $Y$. We incorporate this into our data generating process by
sampling $S$ from a Bernoulli distribution with a constant parameter set to
$\pi_{1}:=0.5$.

$\displaystyle S_{scenario1}\sim Bernoulli(p=\pi_{1})$ (3)

#### Selection at random

In Scenario 2 and Scenario 3 we simulate environments where selection is
affected by the observed features. In Scenario 2 (select hard) selection is
more likely on examples difficult to predict. We induce this behavior by
sampling $S$ such that it takes a value of 1 more frequently when the
example’s feature vector resides closer to the model’s decision boundary. We
sample $S$ according to the feature vector’s distance to the decision
boundary, and apply an exponential function such that the probability of
observing the data is 1 when the point resides on the decision boundary and
trends towards zero as the distance increases. In Scenario 3 (select easy) we
simulate the inverse phenomena. We sample $S$ such that it takes the value one
with higher probability when the example’s feature vector is further from the
decision boundary. We similarly induce this by setting the parameter of the
Bernoulli $S$ is sampled from such that the probability of selection is $1$
when the feature vector resides at the edge of the feature space furthest from
the decision boundary and trends towards zero as it gets closer. We make this
concrete below.

$\displaystyle S_{scenario_{2}}$ $\displaystyle\sim
Bernoulli(p=exp(\frac{-|\omega_{1}x_{1}+\omega_{2}x_{2}+\gamma|}{\sqrt{\omega_{1}^{2}+\omega_{2}^{2}}}))$
(4) $\displaystyle S_{scenario_{3}}$ $\displaystyle\sim
Bernoulli(p=exp(\frac{|\omega_{1}x_{1}+\omega_{2}x_{2}+\gamma|}{\sqrt{\omega_{1}^{2}+\omega_{2}^{2}}}-\delta))\quad\text{where}\quad\delta=max(\frac{|\omega_{1}x_{1}+\omega_{2}x_{2}+\gamma|}{\sqrt{\omega_{1}^{2}+\omega_{2}^{2}}})$
(5)

#### Selection not at random

Here we simulate environments where selection is affected by the label itself.
In Scenario 4 (select negative) selection is more likely when $y=0$. In
Scenario 5 (select positive) selection is more likely when $y=1$. We induce
this by sampling $S$ according to the distribution defined below. In Scenario
4 we set $\pi_{1}:=0.5$ and $\pi_{2}:=1$. In Scenario 5 we set $\pi_{1}:=1$
and $\pi_{2}:=0.5$.

$\displaystyle S_{scenario4,5}\sim Bernoulli(p=\pi_{1}y+\pi_{2}(1-y))$ (6)

#### Weighted model performance estimates

Across each scenario we track model performance in the full and observed
populations using a suite of common binary ML performance metrics, assessing
both discrimination and calibration acuity. We then borrow traditional
weighting techniques from causal inference literature to define weighted
performance estimates, and assess their ability to recover disparities in
observed vs full populations. We construct weighted estimates of each
performance metric by assigning weight to each example in the observed
population equal to the inverse probability of observing it. Note we know the
probability of observing each example because we explicitly define it. In
practice, this probability is instead estimated. This re-weighing procedure is
commonly conducted in causal inference literature to correct for selection
bias in average treatment effect estimates, and is known as inverse
probability weighting (IPW) [17]. The IPW estimate of an arbitrary performance
metric $m(h(x^{i}),y^{i})$ that maps a single example’s prediction and label
to a score is defined below. IPW estimates of AUROC and AUPRC (both undefined
for a single example) are generated by building up an IPW ROC and PR curve and
performing numerical integration.

$\displaystyle\widehat{Metric}_{IPW}$ $\displaystyle=$
$\displaystyle\frac{1}{W}\sum_{i=1}^{N}\frac{m(h(x^{i}),y^{i})}{P(s^{i}=1)}\quad\text{where}\quad
W=\sum_{i=1}^{N}\frac{1}{P(s^{i}=1)}$ (7)

### Real world application: monitoring model performance in production

We now move to the real-world problem of monitoring clinical ML models
hypothetically deployed within the EMR’s production environment. Consider the
task of developing and deploying ML models that use EMR data to flag low-yield
laboratory diagnostics. Many laboratory tests ordered within the hospital are
ordered even though their results are highly predictable. Flagging these tests
at order time has the potential to cut down on wasteful ordering behavior —
saving patients from burdensome blood draws and hospital operational costs.
Here we describe our methods for training, retrospectively evaluating, and
simulating the deployment of three clinical ML models tasked with flagging
laboratory orders with predictable results. In our simulated deployment, we
study the effect of model feed-back loops, and propose a method to recover
true performance on the full deployment population.

#### Data source and cohort extraction

We used the STAnford Research Repository (STARR) to extract de-identified
patient medical records [20]. STARR contains electronic health record data
collected from over 2.4 million unique patients spanning 2009-2021 who have
visited Stanford Hospital (academic medical center in Palo Alto, California),
ValleyCare hospital (community hospital in Pleasanton, California) and
Stanford University Healthcare Alliance affiliated ambulatory clinics. Using
this repository we extracted cohorts of patients for whom three stand-alone
laboratory procedures (Hematocrit, Troponin I, Sodium) had been ordered
between the years 2015 and 2020. Ten thousand laboratory procedures per year
were randomly sampled from the total dataset to yield the final cohort. Train,
validation, and test sets were constructed by year (training set 2015-2018,
validation set 2019, test set 2020). Prediction time was defined as the
timestamp associated with the order of the laboratory procedure.

#### Feature representation

For each task (laboratory test), a timeline of medical events was constructed
from structured electronic medical record data available before prediction
time. A mix of categorical and numerical data elements were included as events
in the feature set. Categorical features included diagnosis (ICD 10) codes on
a patient’s problem list, procedure and medication orders, and demographic
variables including race and sex. Numerical features included prior laboratory
results and the patient’s age at prediction time. Numerical features were
discretized into tokens based on the percentile values they assumed in the
training set distribution, such that they could be embedded alongside the
already token like categorical features. All diagnosis codes prior to
prediction time were included in the patient’s constructed timeline. Procedure
and medication orders placed within 28 days of prediction time were included,
as were laboratory results made available within 14 days of prediction time.
Each timeline was then constructed as a series of days, and features taking
place on the same day were grouped together. Demographic variables that lacked
association with a particular day were included on the patient’s timeline as
if they were made available in the final day.

#### Model description, training procedure and evaluation

Sequences of patient days were embedded into a $512$ dimensional
representation, and fed into a gated recurrent unit layer with a hidden
representation size of $256$. The final hidden representation was input into a
classification layer parameterized by a feed-forward neural network consisting
of two linear layers, ReLU activations, and dropout to output a single score
in $\mathbb{R}$. Dropout was set to 0.2 across all tasks. Models were trained
end-to-end with a binary cross entropy loss function using Adam optimization
with learning rate set to $10^{-4}$ and weight decay set to $10^{-5}$. The max
number of epochs was set to 200, a learning rate scheduler reduced the
learning rate by a power of ten after every 50 epochs, and early stopping was
used to terminate model training if validation set AUROC failed to improve
after 25 epochs. The model with the best validation AUROC across epochs was
selected for each task, and evaluated on the test set. We measure each model’s
discrimination acuity by estimating the ROC curve, and calibration acuity with
calibration plots.

#### Simulating model deployment and feed-back loop

We simulate the deployment of all three models into the EMR system and their
resulting feed-back mechanisms (label selection) using predictions and labels
on the test sets. We model a production use case where upon order entry an
alert is displayed to clinicians suggesting not to order the laboratory test
if the probability of an abnormal lab is above threshold $p_{t}$ or below
$1-p_{t}$, consistent with the hypothesized use case from existing literature
[4]. We track performance in terms of AUROC.

#### Correcting model performance estimates in production under label
selection

Assuming complete adherence to model suggestions, the probability of observing
the labels of laboratory tests with predicted probabilities above $p_{t}$ or
below $1-p_{t}$ is zero. In practice, clinician adherence to model suggestions
will not be 100%. One could imagine estimating the adherence probability
(fraction of alerts accepted) and use a function of its inverse
$\frac{1}{1-p_{a}}$ to weigh examples where alerts were shown. The problem
with this approach however is that clinician adherence would likely depend on
patient characteristics, and likely some not recorded in the EMR — leaving our
selection probabilities misspecified. Instead we propose injecting
randomization into the alert triggering procedure; that is, we propose a
production pipeline where an alert will only trigger if 1) the predicted
probability of the laboratory test is above $p_{t}$ or below $1-p_{t}$, and 2)
the result of a single sample from a Bernoulli distribution with parameter
$p_{withhold}$ is zero. This procedure ensures that the resulting selection
mechanism is at random — only dependent on the output of the model (a function
of observed features) and a random number generator. Further, we know the
selection probability of each example because we explicitly define it. In our
simulations, we ablate examples in our test set where an alert would
hypothetically trigger irrespective of clinician adherence. We track actual,
observed, and weighted estimates. Weighted estimates are computed by re-
weighing each of the alert-triggering examples by the inverse of the alert
withholding probability $\frac{1}{p_{withhold}}$. Weights for non-alert-
triggering examples are set to 1. In our simulations we perform two parameter
sweeps. First we sweep $p_{t}$ from 0.99 to 0.51 at $p_{withold}$ set to 0.05.
Next we sweep $p_{withold}$ from 0.99 to 0.01 at $p_{t}$ set to 0.9. We
simulate each parameter setting 1000 times and estimate actual, observed and
weighted AUROC, reporting mean values and intervals between the 2.5 and 97.5
percentiles.

## Results

### Simulation study

Here we show the results of our simulation study, highlighting how different
selection mechanisms bias particular binary ML performance metrics. We discuss
metrics that measure discrimination and calibration separately — reporting
full, observed and weighted estimates across all five scenarios.

#### Evaluating model discrimination under label selection

In Table 1 we we show the how metrics that assess discrimination acuity are
impacted by varying selection mechanism. Instances when estimates on the
observed population differ from full population estimates are bolded. In
Scenario 1 (selection completely at random) label selection was independent of
both the features and labels. Our simulations reveal no difference between
actual, observed, and weighted performance estimates across all six measures,
save for a reduction in power due to smaller sample size in the observed
population.

In Scenario 2 and Scenario 3 (selection at random), simulations reveal
differences between performance estimates on the full and observed
populations. In Scenario 2 (select hard), estimates using only the observed
population undershoot actual performance across all six measures. In Scenario
3 (select easy), estimates on the observed population overshoot true
performance. Weighted estimates of all performance measures successfully
recover true performance using only the observed data.

In Scenario 4 and Scenario 5 (selection not at random) simulations reveal
variation between observed and actual estimates of PPV and AUPRC, but no
variation in estimates of sensitivity, specificity, accuracy, or AUROC. We
note that because accuracy is a weighted average of sensitivity and
specificity (weighted by class prevalence) this metric would vary in
situations where sensitivity and specificity took on different values. In
Scenario 4 (select negative) both PPV and AUPRC estimates in the observed
population are lower than estimates on the full population. In Scenario 5
(select positive) PPV and AUPRC estimates are higher than in the full
population. Note that baseline performance in terms of PPV and AUPRC depend on
the prevalence of the positive class, which vary in the observed and full
population in both scenarios. Though estimates of PPV and AUPRC vary between
populations, estimates relative to the baseline performance remains stable. In
both scenarios, weighted estimates on the observed population recover actual
performance on the full population.

Table 1: Discrimination performance metrics across selection scenarios: Scenario 1 (selection completely at random), Scenario 2 (select hard), Scenario 3 (select easy), Scenario 4 (select negative), Scenario 5 (select positive). Metric | Estimator | Scenario 1 | Scenario 2 | Scenario 3 | Scenario 4 | Scenario 5
---|---|---|---|---|---|---
Sensitivity | Actual | 0.75 [0.74, 0.77] | 0.75 [0.74, 0.77] | 0.75 [0.74, 0.77] | 0.75 [0.74, 0.77] | 0.75 [0.74, 0.77]
Observed | 0.75 [0.74, 0.77] | 0.63 [0.6, 0.65] | 0.85 [0.82, 0.87] | 0.75 [0.74, 0.77] | 0.75 [0.74, 0.77]
Weighted | 0.75 [0.74, 0.77] | 0.75 [0.72, 0.79] | 0.75 [0.72, 0.79] | 0.75 [0.74, 0.77] | 0.75 [0.74, 0.77]
Specificity | Actual | 0.75 [0.74, 0.77] | 0.75 [0.74, 0.77] | 0.75 [0.74, 0.77] | 0.75 [0.74, 0.77] | 0.75 [0.74, 0.77]
Observed | 0.75 [0.74, 0.77] | 0.63 [0.6, 0.65] | 0.85 [0.82, 0.87] | 0.75 [0.74, 0.77] | 0.75 [0.74, 0.77]
Weighted | 0.75 [0.74, 0.77] | 0.75 [0.72, 0.79] | 0.75 [0.72, 0.79] | 0.75 [0.74, 0.77] | 0.75 [0.74, 0.77]
PPV | Actual | 0.75 [0.74, 0.77] | 0.75 [0.74, 0.77] | 0.75 [0.74, 0.77] | 0.75 [0.74, 0.77] | 0.75 [0.74, 0.77]
Observed | 0.75 [0.74, 0.77] | 0.63 [0.6, 0.65] | 0.85 [0.82, 0.87] | 0.61 [0.59, 0.62] | 0.86 [0.85, 0.87]
Weighted | 0.75 [0.74, 0.77] | 0.75 [0.71, 0.79] | 0.75 [0.72, 0.79] | 0.75 [0.74, 0.77] | 0.75 [0.74, 0.77]
Accuracy | Actual | 0.75 [0.75, 0.76] | 0.75 [0.75, 0.76] | 0.75 [0.75, 0.76] | 0.75 [0.75, 0.76] | 0.75 [0.75, 0.76]
Observed | 0.75 [0.74, 0.77] | 0.63 [0.61, 0.64] | 0.85 [0.83, 0.86] | 0.75 [0.74, 0.76] | 0.75 [0.74, 0.76]
Weighted | 0.75 [0.74, 0.77] | 0.75 [0.73, 0.78] | 0.75 [0.73, 0.78] | 0.75 [0.74, 0.76] | 0.75 [0.74, 0.76]
AUROC | Actual | 0.84 [0.83, 0.84] | 0.84 [0.83, 0.84] | 0.84 [0.83, 0.84] | 0.84 [0.83, 0.84] | 0.84 [0.83, 0.84]
Observed | 0.84 [0.82, 0.85] | 0.68 [0.66, 0.7] | 0.91 [0.9, 0.93] | 0.84 [0.83, 0.84] | 0.84 [0.83, 0.84]
Weighted | 0.84 [0.82, 0.85] | 0.84 [0.8, 0.87] | 0.84 [0.82, 0.85] | 0.84 [0.83, 0.84] | 0.84 [0.83, 0.84]
AUPRC | Actual | 0.83 [0.82, 0.84] | 0.83 [0.82, 0.84] | 0.83 [0.82, 0.84] | 0.83 [0.82, 0.84] | 0.83 [0.82, 0.84]
Observed | 0.83 [0.82, 0.85] | 0.68 [0.65, 0.71] | 0.91 [0.89, 0.92] | 0.73 [0.71, 0.74] | 0.9 [0.9, 0.91]
Weighted | 0.83 [0.82, 0.85] | 0.83 [0.76, 0.88] | 0.83 [0.81, 0.85] | 0.83 [0.82, 0.84] | 0.83 [0.82, 0.84]

#### Evaluating model calibration under label selection

In Figure 1 we show model calibration estimates in the form of calibration
plots. Because our model is perfectly specified, it is perfectly calibrated.
In Figure 1 we omit the calibration curve on the full population, and focus on
calibration estimates on the observed population (both with and without
weighting). In Scenario 1 (selection completely at random) calibration
estimates remain stable across populations estimates, save for a reduction in
power due to smaller sample size in the observed population.

In Scenario 2 and Scenario 3 (selection at random) simulations also reveal no
change in calibration estimates across populations, except for reductions in
power. Distinct from Scenario 1, power is reduced differentially across each
of the binned calibration estimates. In Scenario 2 (select hard), we see a
reduction in power in bins where the model is most confident (further from 0.5
predicted probability). In Scenario 3 (select easy), power is reduced in
binned calibration estimates where the predicted probability is closer to 0.5.

In Scenario 4 and Scenario 5 (selection not at random), calibration estimates
differ when computed on the observed and full populations. In Scenario 4
(select negative), the observed calibration plot suggests that the model is
over predicting risk even though in reality the model is perfectly calibrated.
In Scenario 5 (select positive) the observed calibration plot suggests the
model is under predicting risk. In both scenarios, weighted estimates recover
the true calibration curve.

Figure 1: Model calibration estimates across selection scenarios

### Real world application: monitoring model performance in production

Here we present results of training, evaluating, and simulating the deployment
of three clinical ML models intended to reduce unnecessary laboratory
utilization. We begin this section by presenting model performance in the
standard way on a held out retrospective test set. We then shift to presenting
simulated results where model deployment induces label selection, and
highlight differences in estimates using observed and actual deployment
populations. We conclude the section by evaluating whether our proposed
deployment workflow corrects disparities between observed and full
populations.

#### Retrospective evaluation of stand-alone laboratory prediction models

We trained clinical ML models to predict the results of three stand-alone
laboratory procedures: hematorcrit, troponin I and sodium. Each cohort
contained a total of 60,000 randomly sampled laboratory procedures stratified
by year between 2015 and 2020. Training sets included 40,000 laboratory tests
between 2015 and 2018. Validation sets contained 10,000 laboratory tests in
ordered in 2019. Test sets contained 10,000 laboratory tests ordered in 2020.
Prevalence of the outcome in each test set was 0.79 [0.78, 0.80], 0.57 [0.57,
0.58] and 0.27 [0.26, 0.28] for hematocrit, troponin I and sodium tasks
respectively. Performance on the test set for each of the three tasks in terms
of both discrimination (ROC curve) and calibration (calibration plots) are
shown in Figure 2.

Figure 2: Model performance on held out test sets

#### Simulating model deployment

In Figure 3 we show the results of our simulated model deployment. In the top
row, we show actual, observed and weighted AUROC estimates over a range of
model alert trigger thresholds $p_{t}$. Recall we trigger an alert upon order
entry suggesting not to order the laboratory test if 1) the probability
estimate of the model is greater than $p_{t}$ or below $1-p_{t}$ and 2) a
single sample from a Bernoulli with parameter $p_{withhold}$ is zero. We sweep
values of $p_{t}$ from 0.99 to 0.51. Simulations reveal across all three
models that observed performance initially decays, reaches a minima, and then
increases towards actual performance as $p_{t}$ decreases from 0.99 to 0.51.
The initial decline in observed performance occurs as observations further
from the decision boundary are differentially selected, as in Scenario 2
(select hard). After the minima is reached, observed model performance then
trends back to actual performance — as the pool of examples failing to trigger
an alert become small enough to be outweighed by observations where an alert
would have triggered had we not injected random alert withholding probability
$p_{withhold}$, here set to 0.05. Importantly our weighted estimates of AUROC
successfully recover actual performance across all values of $p_{t}$.

In the second row of Figure 3, we show actual, observed and weighted model
performance estimates as a function of our randomly injected alert withholding
probability $p_{withhold}$. Here we set $p_{t}$ (varied in row above) to 0.9.
We sweep values of $p_{withhold}$ ranging from 0.99 to 0.01. Our simulations
reveal that as $p_{withhold}$ trends toward zero (leading to more alerts and
thus more missing labels), AUROC on the observed population decays further
from AUROC on the actual deployment population. Importantly, weighted
estimates successfully recover actual performance across all values of
$p_{withhold}$. It should be noted that the variance of weighted AUROC
estimates increases as $p_{withhold}$ shrinks to zero, which in turn explodes
our weights $\frac{1}{p_{withold}}$. This is a known criticism of IPW
estimates, where the precision of the estimators decrease with larger weights
[17].

## Discussion

In this study we assess how different types of label selection yield biased
clinical ML model performance estimates. We simulate five causally distinct
scenarios, and probe how they impact a suite of typically reported ML
performance metrics. We categorize our scenarios into three broad classes: 1)
selection completely at random, where selection is independent of both
features and labels, 2) selection at random, where selection is affected by
features, and 3) selection not at random, where selection is affected by the
labels themselves. Our simulations reveal that when selection is completely at
random, model performance estimates are unaffected, save for a reductions in
power. When selection is at random, performance metrics that measure a model’s
discrimination acuity (sensitivity, specificity, positive predictive value,
accuracy, AUROC, AUPRC) can be biased if naively estimated on the observed
population. If selection occurs such that examples harder to predict are
preferentially selected (Scenario 2), discrimination metrics may underestimate
true performance. If selection occurs such that easier examples are
preferentially selected (Scenario 3), discrimination metrics may overestimate
true performance. While selection at random can yield biased estimates of
model discrimination acuity, estimates of model calibration on the observed
population remain consistent. Power to estimate calibration in regions of the
feature space with more missing labels is reduced. When selection is only
affected by observed features, labels are independent of the selection
mechanism when conditioned on features, implying $p(y\mid x)=p(y\mid x,s=1)$.

When selection occurs not at random, performance metrics on the observed
population that condition on the class label (sensitivity, specificity, AUROC)
remain consistent while calibration estimates veer from truth. This is
revealed in Scenario 4 and Scenario 5, which result in estimates of
sensitivity, specificity, and AUROC that do not differ between observed and
full populations, while calibration plots indicate over and under estimation
of risk. In our simulations we keep our model (which is perfectly specified)
fixed. If a model had instead been fit to the observed data, calibration plots
may imply well calibrated models when in reality they would be under and over
estimating risk respectively. An important note, discrimination metrics
sensitive to the prior $p(y)$ like PPV (precision), AUPRC, and accuracy may
not reflect their true population values when selection is not at random. In
our simulations, we see a change in PPV and AUPRC. Accuracy, effectively a
weighted average of sensitivity and specificity weighted by class prevalence,
remained stable only because in our case sensitivity and specificity were
equal at our cutoff probability threshold of 0.5. In our simulations,
traditional weighting estimators borrowed from causal inference literature
effectively corrected performance estimates, recovering the full population
parameters. These estimators rely on well specified estimates of the selection
probabilities. In our simulations these were known, in practice they must be
estimated. Our simulations reveal how weighted machine learning model
performance estimators can recover full population parameters in the presence
of missing labels.

Figure 3: Actual, observed and weighted AUROC as a function deployment
simulation parameters

We then tackle the real-world task of monitoring the performance of deployed
ML models whose interaction with clinicians feed-back and induce label
selection. We train three ML models using real-world electronic medical record
data that at order time predict whether stand-alone laboratory tests will
result outside their normal range. We hypothesize a deployment use case in
which upon order entry alerts are triggered in the electronic medical record
suggesting not to order the tests. Alerts are triggered only if the
probability estimate of the model is above threshold $p_{t}$ or below
$1-p_{t}$. We then propose a slight adjustment to this production pipeline to
enable consistent estimates of performance on the full deployment population.
We propose injecting randomization into the alert triggering mechanism, such
that predicted probabilities above $p_{t}$ or below $1-p_{t}$ only trigger an
alert if the result of a single sample from a Bernoulli distribution
parameterized with $p=p_{withhold}$ is zero. By injecting randomization, and
treating overridden alerts as missing, we ensure the selection mechanism is at
random: only dependent on model predictions (a function of observed features)
and a random number generator. Further, we know the selection probability for
each alert triggering example because we explicitly define it: $p_{withhold}$.
In deployment, labels of examples easiest to predict (furthest from the
decision boundary) are more likely to go missing, resembling Scenario 2
(select hard) from our simulations. Our simulations reveal how naive model
performance monitoring using the observed data leads to a large (up to 20%)
reduction in perceived discrimination acuity — perhaps large enough to cause
an uninformed data science team to wrongfully scrap the project. ML
practitioners equipped with such a workflow can reliably monitor model
performance even when deployment itself feeds back to induce label selection.

## Conclusion

We simulate five causally distinct selection scenarios that affect the
estimation of common binary ML performance metrics. When selection is at
random, naive estimates of discrimination acuity may be biased while
calibration estimates remain intact. When selection is not at random, naive
estimates of calibration can be biased while discrimination metrics that
condition on class labels (sensitivity, specificity, AUROC) remain intact. We
study the real-world use case of monitoring deployed ML models under feed-back
mechanisms that induce label selection. Naive estimates of performance lead to
substantial (up to 20%) reduction in perceived AUROC. We propose a deployment
workflow that couples randomization and weighted estimates of performance and
find that it consistently recovers true performance on the deployment
population.

##### Acknowledgements

This research used data provided by STARR, STAnford medicine Research data
Repository,” a clinical data warehouse containing de-identified Epic data from
Stanford Health Care (SHC), the University Healthcare Alliance (UHA) and
Packard Children’s Health Alliance (PCHA) clinics and other auxiliary data
from Hospital applications such as radiology PACS. The STARR platform is
developed and operated by the Stanford Medicine Research IT team and is made
possible by Stanford School of Medicine Research Office. The content is solely
the responsibility of the authors and does not necessarily represent the
official views of the NIH or Stanford Health Care.

## References

  * [1] Leung KM, Elashoff RM, Afifi AA.  Censoring issues in survival analysis.  Annual review of public health. 1997;18(1):83-104.
  * [2] Wang P, Li Y, Reddy CK.  Machine learning for survival analysis: A survey.  ACM Computing Surveys (CSUR). 2019;51(6):1-36.
  * [3] Avati A, Jung K, Harman S, Downing L, Ng A, Shah NH.  Improving palliative care with deep learning.  BMC medical informatics and decision making. 2018;18(4):55-64.
  * [4] Xu S, Hom J, Balasubramanian S, Schroeder LF, Najafi N, Roy S, et al.  Prevalence and predictability of low-yield inpatient laboratory diagnostic tests.  JAMA network open. 2019;2(9):e1910967-7.
  * [5] Luo Y, Szolovits P, Dighe AS, Baron JM.  Using machine learning to predict laboratory test results.  American journal of clinical pathology. 2016;145(6):778-88.
  * [6] Lidbury BA, Richardson AM, Badrick T.  Assessment of machine-learning techniques on large pathology data sets to address assay redundancy in routine liver function test profiles.  Diagnosis. 2015;2(1):41-51.
  * [7] Chen JH, Podchiyska T, Altman RB.  OrderRex: clinical order decision support and outcome predictions by data-mining electronic medical records.  Journal of the American Medical Informatics Association. 2016;23(2):339-48.
  * [8] Lenert MC, Matheny ME, Walsh CG.  Prognostic models will be victims of their own success, unless….  Journal of the American Medical Informatics Association. 2019;26(12):1645-50.
  * [9] Frandsen BR.  Treatment effects with censoring and endogeneity.  Journal of the American Statistical Association. 2015;110(512):1745-52.
  * [10] Wu MC, Bailey KR.  Estimation and comparison of changes in the presence of informative right censoring: conditional linear model.  Biometrics. 1989:939-55.
  * [11] Ozenne BMH, Scheike TH, Stærk L, Gerds TA.  On the estimation of average treatment effects with right-censored time to event outcome and competing risks.  Biometrical Journal. 2020;62(3):751-63.
  * [12] Powers S, McGuire V, Bernstein L, Canchola AJ, Whittemore AS.  Evaluating disease prediction models using a cohort whose covariate distribution differs from that of the target population.  Statistical methods in medical research. 2019;28(1):309-20.
  * [13] Sugiyama M, Krauledat M, Müller KR.  Covariate shift adaptation by importance weighted cross validation.  Journal of Machine Learning Research. 2007;8(5).
  * [14] Quinonero-Candela J, Sugiyama M, Schwaighofer A, Lawrence ND.  Dataset shift in machine learning.  Mit Press; 2008.
  * [15] Zadrozny B.  Learning and evaluating classifiers under sample selection bias.  In: Proceedings of the twenty-first international conference on Machine learning; 2004. p. 114.
  * [16] Vink G. Roderick J. Little and Donald B. Rubin: Statistical Analysis with Missing Data. Springer; 2022.
  * [17] Hernán MA, Hernández-Díaz S, Robins JM.  A structural approach to selection bias.  Epidemiology. 2004:615-25.
  * [18] Haneuse S, Schildcrout J, Crane P, Sonnen J, Breitner J, Larson E.  Adjustment for selection bias in observational studies with application to the analysis of autopsy data.  Neuroepidemiology. 2009;32(3):229-39.
  * [19] James G, Hastie T.  Generalizations of the bias/variance decomposition for prediction error.  Dept Statistics, Stanford Univ, Stanford, CA, Tech Rep. 1997.
  * [20] Datta S, Posada J, Olson G, Li W, O’Reilly C, Balraj D, et al.  A new paradigm for accelerating clinical data science at Stanford Medicine.  arXiv preprint arXiv:200310534. 2020.


# Temporal Analysis of Transaction Ego Networks with Different Labels on
Ethereum

Baoying Huang1, Jieli Liu2, Jiajing Wu14, Quanzhong Li1, Hao Lin3  1School of
Computer Science and Engineering, Sun Yat-sen University, Guangzhou 510006,
China
2School of Software Engineering, Sun Yat-sen University, Zhuhai 519082, China
3Merchants Union Consumer Finance Company Limited, Shenzhen 518000, China
4Correponding Author<EMAIL_ADDRESS>
###### Abstract

Due to the widespread use of smart contracts, Ethereum has become the second-
largest blockchain platform after Bitcoin. Many different types of Ethereum
accounts (ICO, Mining, Gambling, etc.) also have quite active trading
activities on Ethereum. Studying the transaction records of these specific
Ethereum accounts is very important for understanding their particular
transaction characteristics, and further labeling the pseudonymous accounts.
However, traditional methods are generally based on static and global
transaction networks to conduct research, ignoring useful information about
dynamic changes. Our work chooses six kinds of important account labels, and
builds ego networks for each kind of Ethereum account. We focus on the
interaction between the target node and neighbor nodes with temporal analysis.
Experiments show that there is a significant difference between various types
of accounts in terms of several network features, helping us better understand
their transaction patterns. To the best of our knowledge, this is the first
work to analyze the dynamic characteristics of Ethereum labeled accounts from
the perspective of transaction ego networks.

##  I INTRODUCTION

After the development of more than a decade, blockchain technology has become
a buzzword and brought world-shaking innovations to human life. Generally
speaking, blockchains are append-only distributed ledgers originated from
Bitcoin [1]. Transactions on blockchain platforms are publicly verifiable
based on open data architecture. Ethereum [2] is the largest public blockchain
enabling Turing-complete smart contracts, which are computer programs that can
automatically be executed in the Ethereum Virtual Machine (EVM). Since the
outbreak of COVID-19, investors have found that Ethereum is suitable as a
short-term safe-haven [3].

Therefore, increasing research efforts have been devoted to the analysis of
the publicly accessible Ethereum transaction records. A considerable part of
existing work modeled the transaction data into networks, which are a general
language in describing interacting systems[4][5] [6]. In the Ethereum
transaction network, an Ethereum account is regarded as a node, and the edges
between two nodes present the interactions between the two accounts with
attributes such as transaction value, timestamp and so on.

Recent research [7] [8] [9] have analyzed the entire Ethereum transaction
network from a static perspective for understanding the whole Ethereum
ecosystem. They have built several static networks using transaction data, and
analyzed network properties such as density and clustering coefficients,
comparing them with social networks and the Web. However, the daily new
transaction volume of Ethereum is quite large, and static analysis leads to
the loss of insights from a dynamic evolution perspective. Consequently,
several recent studies [10] [11] [12] provided insights into the temporal and
evolution properties of the entire Ethereum transaction network. But there is
few studies investigating the microscopic structure of a kind of nodes in the
Ethereum transaction network, which can give explanations for the different
behaviors of accounts. Analyzing the Ethereum transaction network from a
perspective of various types of nodes can provide us with a new understanding
of their microscopic network structure, which also helps to label the
pseudonymous accounts in Ethereum. In this paper, we are interested in
studying the temporal changes of multiple transaction characteristics based on
the ego network of each labeled account. The ego network consists of the
central node “EGO” and the nodes directly connected to EGO (these are called
“ALTER”), as well as the connections between the EGO and ALTER nodes, ALTER
and ALTER nodes [13]. Ego network is widely used in social network analysis
and helps to find many patterns [14] [15]. We mainly select six typical
Ethereum account labels, including four normal labels (ICO, Mining, Gambling,
Exchange) and two illegal labels (Ponzi and Phish), which are collected from
Etherscan’s label word cloud111https://cn.etherscan.com/labelcloud. A normal
account has no illegal activities such as deceiving users. While an illegal
account is the opposite. Then, we collect the required transaction records
from a blockchain data platform and build multiple ego networks. After
analyzing how the transaction features change over time for each type of node,
we discover the different features between various types of accounts and
better understand their roles on Ethereum.

The remainder of this paper is structured as follows. Section II introduces
the process of collecting transaction data and building the ego network of
various accounts. The analysis of network properties will be given in Section
III. Finally, we will give a brief conclusion in Section IV.

##  II DATASET AND EXPERIMENT SETUP

In this part, we firstly introduce the source and processing of data, and then
explain the definition of ego network constructed by using transaction data.

###  II-A Data Collection

Etherscan is one of the most widely used Ethereum browsers, recording lots of
account labels that are reliably provided by multiple users. A lot of previous
research work on Ethereum also used the labeled accounts on Etherscan [16]
[17]. We choose six types of the most common and important accounts on
Ethereum as the targets. Especially, according to the crime report about
cryptocurrency [18], ponzi schemes and phish scams have defrauded millions of
dollars. Now we will introduce the definition of these accounts in detail.

  * $\bullet$

ICO: The accounts used to receive crowdfunding for the initial offering of the
tokens

  * $\bullet$

Mining: The accounts of miners who participate in block mining

  * $\bullet$

Gambling: The accounts of the gambling platforms

  * $\bullet$

Exchange: The accounts of exchanges that allow users to sell or buy
cryptocurrencies

  * $\bullet$

Ponzi: The accounts related to the manipulator of ponzi schemes

  * $\bullet$

Phish: The accounts related to the manipulator of phish scams

In addition, we collect all target labeled accounts as well as their complete
$Ether$222$Ether$ is the official cryptocurrency of Ethereum. transaction
records. In order to construct the ego network, we also collect the
transactions between their first-order neighbors. These transaction records
are extracted from the latest Ethereum dataset on
XBlock333http://xblock.pro/xblock-eth.html, which is a blockchain data
platform. Therefore, a total of 2584 accounts and nearly 38 million
transaction records are collected.

###  II-B Construction of Ego Networks

Let $G=(V,E)$ represents the ego network of an Ethereum account, where $V$
represents the set of nodes, and $E$ is the set of edges. $V$ contains the
central node “EGO” and its neighbor nodes “ALTER”. The transactions between
EGO and ALTER nodes, ALTER and ALTER form the edge set $E$. Particularly, an
edge can be represented as $e=<u,v,w,t>$, indicating that at time $t$, account
$u$ transfers Ether of amount $w$ to account $v$. Consequently, the ego
network in our work is a weighted directed network.

Fig. 1 shows an example of an ego network. Moreover, we construct an ego
network for each labeled account, classify the ego networks of the same label
into one category, and take the average of various statistics as the final
feature value of the label accounts. The details of each feature will be shown
in the next section.

Figure 1: An example Of ego network.

##  III THE ANALYSIS OF NETWORK PROPERTIES

First, we focus on the local clustering coefficient of each labeled account to
determine whether the nodes in the network are closely connected. Then we
analyze several transaction characteristics of labeled accounts. Finally, we
obtain the unique transaction features of each labeled account, which also
explain the role of the corresponding label in the Ethereum ecosystem.

###  III-A Local Clustering Coefficient

The local clustering coefficient is a measure of the degree to which the
neighbors of the central node in the graph tend to cluster together. Let
$G_{i}=(V,E)$ represents the ego network whose ego is node $i$. Here the edge
of node $u$ pointing to node $v$ is written as $e_{uv}$. Then $N_{i}$
represents the neighbor nodes of node $i$ and $k_{i}=|N_{i}|$. Hence the local
clustering coefficient of node $i$ is:

$\tau_{i}=\frac{|\\{e_{uv}:u,v\in N_{i},e_{uv}\in E\\}|}{k_{i}(k_{i}-1)}.$ (1)

Then we calculate the average local clustering coefficient of each type of ego
network, and the results are shown in Table I. According to the table, the
probability of transactions occurring between the neighbors of ICO accounts is
about 18$\%$, followed by Mining accounts and Exchange accounts. It tells us
there is a higher probability that investors of ICO accounts will have
transactions with each other. The coefficient of Gambling accounts is the
smallest, which means the users participating in the gambling are generally
not connected. Besides, the local clustering coefficient of Phish accounts is
larger than that of Ponzi accounts, indicating that the neighbors of Phish
accounts are also more likely to be connected. We speculate that there are
fixed fraud partners in phishing activities.

TABLE I: The local clustering coefficient of labeled egos Label | ICO | Mining | Gambling | Exchange | Ponzi | Phish
---|---|---|---|---|---|---
$\tau$ | 0.1812 | 0.1354 | 0.0242 | 0.1324 | 0.0379 | 0.1226

###  III-B Ego’s Life Cycle and Time Window

For an Ethereum account $u$ with $M$ transactions, we define its life cycle as
$Life_{u}=t_{M}-t_{1}$, where $t_{M}$ represents the time when the last
transaction occurred and $t_{1}$ represents the time when the first
transaction occurred. We count the life cycles of each kind of account and
record its median and maximum values in Table II. It can be concluded that the
life cycle of illegal accounts is much shorter than that of normal accounts,
and their life cycle is less than 20 days. This is because the criminals
usually transfer the illegal gains out of the accounts as soon as possible and
no longer use the original accounts to enhance anonymity.

TABLE II: The life cycle of labeled ego (days) Label | ICO | Mining | Gambling | Exchange | Ponzi | Phish
---|---|---|---|---|---|---
Median | 419.92 | 646.24 | 292.59 | 603.06 | 19.99 | 15.76
Mean | 1700.18 | 2150.09 | 1527.35 | 2132.30 | 1642.22 | 1529.09

Afterwards, in order to observe the ego network of each type of label from a
dynamic perspective, we refer to the previous work [12] and consider two types
of time windows, the sliding window and the incremental window. Firstly, we
take $1/5$ of the life cycle of each type of labeled account as a fixed time
span $T$, which is also the initial length of the two time windows. The
sliding distance of the sliding window is $T$ each time, while the size of the
incremental window increases by $T$. Next, we use different time windows to
extract transaction details of the egos, and obtain different characteristics
in five phases.

###  III-C The number of transactions between EGOs and ALTERs

In order to study the transaction activities of different types of accounts,
we compare the evolution in the number of transactions between EGO and ALTER
nodes during their life cycles in the incremental window mode. It can be seen
from Fig. 2 that the number of transactions of illegal accounts is less than
that of the normal accounts. In addition, the number of transactions on
Exchange, Mining and Gambling accounts is relatively large. Because Exchange
accounts are an intermediary on Ethereum, they generate a lot of transactions
with users to buy and sell tokens. According to the description on Etherscan,
we can know that Mining accounts are mining pools composed of many miners. So
they usually return bonuses to miners multiple times. Additionally, Gambling
accounts representing a kind of entertainment activity, also attract lots of
users. Significantly, the curves of ICO accounts and Ponzi accounts are
relatively similar, which shows that they have certain commonalities in the
trading pattern. We presume that it is one of the reasons why it’s difficult
for users to distinguish whether the investment project is a scam or not.

Figure 2: Changes of the number of transactions at each phase

For the purpose of further exploring the roles of the labeled accounts on
Ethereum, the changes in the ratio of two kinds of transactions (in/out) in
the incremental window mode are also recorded. An in-transaction is an
interaction in which the EGO receives Ether, and an out-transaction is an
interaction in which the EGO transfers Ether to another account.

TABLE III: Changes in the proportion of the number of in/out transactions | Phase | P0 | P1 | P2 | P3 | P4
---|---|---|---|---|---|---
ICO | in | 92.36$\%$ | 90.83$\%$ | 62.84$\%$ | 62.92$\%$ | 61.24$\%$
out | 7.64$\%$ | 9.17$\%$ | 37.16$\%$ | 37.08$\%$ | 38.76$\%$
Mining | in | 0.19$\%$ | 0.14$\%$ | 0.17$\%$ | 0.17$\%$ | 0.16$\%$
out | 99.81$\%$ | 99.86$\%$ | 99.83$\%$ | 99.83$\%$ | 99.84$\%$
Gambling | in | 55.73$\%$ | 60.13$\%$ | 61.45$\%$ | 61.31$\%$ | 61.27$\%$
out | 44.27$\%$ | 39.87$\%$ | 38.55$\%$ | 38.69$\%$ | 38.73$\%$
Exchange | in | 43.83$\%$ | 44.18$\%$ | 41.34$\%$ | 33.25$\%$ | 26.67$\%$
out | 56.17$\%$ | 55.82$\%$ | 58.66$\%$ | 66.75$\%$ | 73.33$\%$
Ponzi | in | 29.04$\%$ | 28.28$\%$ | 28.38$\%$ | 28.98$\%$ | 30.58$\%$
out | 70.96$\%$ | 71.72$\%$ | 71.62$\%$ | 71.02$\%$ | 69.42$\%$
Phish | in | 61.15$\%$ | 61.82$\%$ | 61.46$\%$ | 61$\%$ | 57.78$\%$
out | 38.85$\%$ | 38.18$\%$ | 38.54$\%$ | 39$\%$ | 42.22$\%$

(a) ICO (b) Mining (c) Gambling (d) Exchange

(e) Ponzi (f) Phish

Figure 3: Changes of the transaction amount at each phase

The results in Table III show that only Mining, Exchange, and Ponzi accounts
have a higher proportion of out-transactions than in-transactions. We suppose
that the reasons for this phenomenon are as follows. Firstly, the rewards
obtained from mining blocks will be distributed to users who invest in miners,
which is a long-term stable mechanism. Secondly, when an Exchange account is
used as the initiator of transactions, it means that users are buying Ether
from this platform. Generally speaking, investors will continue to purchase
Ether multiple times and then sell them all at the highest price point. And
thus Exchange accounts have more in-transactions than out-transactions.
Thirdly, a Ponzi account intends to act like a normal project at first to
attract more people. Then, a small part of the investment of latecomers will
also be used as bonuses to return to the initial users several times.

Moreover, the in-transaction ratio of ICO accounts is relatively large at the
initial phase of token issuance, and the later transaction volume gradually
decreases. The other two kinds of accounts with a higher proportion of in-
transactions are Gambling and Phish. A gambling activity is a game where the
dealer makes a profit, so the number of users who win money is less than the
total number of participants. Phish accounts are fraudulent, so they deceive
multiple users to obtain Ether, and then transfer out all the illegal gains at
the final phase.

###  III-D The amount of transactions between EGO and ALTER

Besides the number of transactions, the transaction amount is also an
extremely important feature. Therefore, we observe the transfer amount of
Ethereum accounts at each phase in the sliding window mode, and analyze the
difference and essence. It can be seen from Fig. 3 that ICO accounts initially
receive a large amount of investment and then gradually transfer it to the
token-related accounts. Interestingly, the total amount transferred out from
Mining accounts is always greater than the total amount transferred in,
because most of the balance in the accounts comes from mining rewards. These
rewards are not be recorded in the transactions, but they can be queried
through the Ethereum browser. In addition, when no player participates in
gambling games, Gambling accounts will transfer the balance out, indicating
that Gambling accounts are essentially a profitable entity. In general, only
the transfer-in and transfer-out amount at all phases of Exchange accounts are
basically the same, which also reflects the role of Exchange accounts as an
intermediary for the sale of tokens. Finally, we find that both types of
illegal accounts have reaped huge profits, which means it’s urgent to identify
and combat fraudulent accounts on Ethereum. It is worth noting that Phish
accounts usually transfer out a large amount of Ether in the final phase.
While Ponzi accounts transfer Ether out in batches, and some Ether will be
returned to investors.

##  IV CONCLUSIONS AND FUTURE WORK

With the rapid development of Ethereum, a large number of accounts on Ethereum
have involved in various business roles. For exploring the transaction
behavior of these accounts and obtaining a more comprehensive description of
their roles in the Ethereum ecosystem, we used transaction records to
construct multiple ego networks with transaction amount and time information
for each type of account. By analyzing the dynamic changes of the interaction
characteristics of EGO (labeled accounts) and ALTER (first-order neighbor
accounts), we found that the trading of Mining accounts and Exchange accounts
is more stable. In addition, the two illegal accounts, Ponzi and Phish, also
have obvious abnormalities in the ratio of the amount of in/out transactions.
The dynamic changes of transaction characteristics also enabled us to label
unknown accounts. Most of all, we understood the evolutionary phases of scams,
which can provide some help in identifying fraudulent activities on Ethereum.

In the future, we will continue to expand our research by investigating more
network properties in the dynamic transaction network, such as transaction
objects and network motifs. We expect the corresponding results to tell us
whether there is a specific transaction rule and help us discover hidden
communities in the network. Furthermore, we will consider more account labels,
and more dynamic changes in complex network characteristics.

## Acknowledgment

The work described in this paper is supported by the National Key R&D Program
of China (2020YFB1006005), the National Natural Science Foundation of China
(61973325) and the Natural Science Foundations of Guangdong Province
(2021A1515011661)

## References

  * [1] S. Nakamoto, “Bitcoin: A peer-to-peer electronic cash system,” https://bitcoin.org/bitcoin.pdf, 2008.
  * [2] G. Wood, “Ethereum: A secure decentralised generalised transaction ledger,” https://ethereum.github.io/yellowpaper/paper.pdf, 2014.
  * [3] E. Mnif, A. Jarboui, and K. Mouakhar, “How the cryptocurrency market has performed during COVID 19? A multifractal analysis,” _Finance Research Letters_ , vol. 36, p. 101647, 2020.
  * [4] F. Victor and B. K. Lüders, “Measuring Ethereum-based ERC20 token networks,” in _Proceedings of the International Conference on Financial Cryptography and Data Security_. Frigate Bay, St. Kitts and Nevis: Springer, 2019, pp. 113–129.
  * [5] J. Wu, J. Liu, W. Chen, H. Huang, Z. Zheng, and Y. Zhang, “Detecting mixing services via mining Bitcoin transaction network with hybrid motifs,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , 2021, to be published, doi: 10.1109/TSMC.2021.3049278.
  * [6] Y. Huang, J. Wu, W. Ren, C. K. Tse, and Z. Zheng, “Sequential restorations of complex networks after cascading failures,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , vol. 51, no. 1, pp. 400–411, 2021.
  * [7] D. Lin, J. Wu, Q. Yuan, and Z. Zheng, “Modeling and understanding Ethereum transaction records via a complex network approach,” _IEEE Transactions on Circuits and Systems II: Express Briefs_ , vol. 67, no. 11, pp. 2737–2741, 2020\.
  * [8] J. Wu, J. Liu, Y. Zhao, and Z. Zheng, “Analysis of cryptocurrency transactions from a network perspective: An overview,” _Journal of Network and Computer Applications_ , vol. 190, p. 103139, 2021.
  * [9] J. Wu, Q. Yuan, D. Lin, W. You, W. Chen, C. Chen, and Z. Zheng, “Who are the phishers? Phishing scam detection on Ethereum via network embedding,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , vol. 52, no. 2, pp. 1156–1166, 2022.
  * [10] J. Geng, Y. Li, L. Fang, and P. Chen, “VW-DBG: A dynamically evolving Bitcoin transaction network model,” _IEEE Transactions on Network Science and Engineering_ , 2021, to be published, doi:10.1109/TNSE.2021.3116243.
  * [11] D. Lin, J. Chen, J. Wu, and Z. Zheng, “Evolution of Ethereum transaction relationships: Toward understanding global driving factors from microscopic patterns,” _IEEE Transactions on Computational Social Systems_ , 2021, to be published, doi:10.1109/TCSS.2021.3093384.
  * [12] L. Zhao, S. Sen Gupta, A. Khan, and R. Luo, “Temporal analysis of the entire Ethereum blockchain network,” in _Proceedings of the Web Conference 2021_. Ljubljana, Slovenia: ACM, 2021, pp. 2258–2269.
  * [13] F. Odella, “Using ego-network in surveys: Methodological and empirical research issues,” in _Proceedings of the International Conference on Network Science_. Bloomington, Indiana: Springer, 2006.
  * [14] J. Leskovec, L. Backstrom, R. Kumar, and A. Tomkins, “Microscopic evolution of social networks,” in _Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_. Las Vegas, Nevada, USA: ACM, 2008, pp. 462–470.
  * [15] V. Arnaboldi, M. Conti, A. Passarella, and F. Pezzoni, “Analysis of ego network structure in online social networks,” in _Proceedings of the 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing_. Amsterdam, Netherlands: IEEE, 2012, pp. 31–40.
  * [16] Q. Yuan, B. Huang, J. Zhang, J. Wu, H. Zhang, and X. Zhang, “Detecting phishing scams on Ethereum based on transaction records,” in _Proceedings of the 2020 IEEE International Symposium on Circuits and Systems (ISCAS)_. Seville, Spain: IEEE, 2020, pp. 1–5.
  * [17] W. Chen, X. Guo, Z. Chen, Z. Zheng, and Y. Lu, “Phishing scam detection on Ethereum: Towards financial security for Blockchain ecosystem,” in _Proceedings of the International Joint Conference on Artificial Intelligence_. Yokohama, Japan: IJCAI, 2020, pp. 4506–4512.
  * [18] Chainalysis, “The Chainalysis 2021 crypto crime report,” https://go.chainalysis.com/2021-Crypto-Crime-Report.html, 2021.