# Curating Naturally Adversarial Datasets for Trustworthy AI in Healthcare

Sydney Pugh1, Ivan Ruchkin2, Insup Lee1, James Weimer3

###### Abstract

Deep learning models have shown promising predictive accuracy for time-series
healthcare applications. However, ensuring the robustness of these models is
vital for building trustworthy AI systems. Existing research predominantly
focuses on robustness to _synthetic adversarial examples_ , crafted by adding
imperceptible perturbations to clean input data. However, these synthetic
adversarial examples do not accurately reflect the most challenging real-world
scenarios, especially in the context of healthcare data. Consequently,
robustness to synthetic adversarial examples may not necessarily translate to
robustness against _naturally occurring adversarial examples_ , which is
highly desirable for trustworthy AI. We propose a method to curate datasets
comprised of natural adversarial examples to evaluate model robustness. The
method relies on probabilistic labels obtained from automated weakly-
supervised labeling that combines noisy and cheap-to-obtain labeling
heuristics. Based on these labels, our method adversarially orders the input
data and uses this ordering to construct a sequence of increasingly
adversarial datasets. Our evaluation on six medical case studies and three
non-medical case studies demonstrates the efficacy and statistical validity of
our approach to generating naturally adversarial datasets.

## Introduction

Deep learning models have shown significant potential in analyzing medical
time-series data to facilitate tasks such as disease diagnosis, anomaly
detection, and intervention planning (Faust et al. 2018; Morid, Sheng, and
Dunbar 2023). However, deep learning models have also been shown to be brittle
and non-robust (Szegedy et al. 2013; Alcorn et al. 2019). A typical approach
to analyzing the robustness of such models is to evaluate their performance
against adversarial examples — perturbed inputs that cause misclassification
while being nearly imperceptible to humans.

Much work has been done to evaluate model robustness to adversarial examples,
specifically focusing on _synthetic_ adversarial examples. These examples are
generated by intentionally adding small perturbations to _labeled_ input data.
However, synthetic adversarial examples for medical time-series data are
problematic for two primary reasons. Firstly, they generally do not resemble
adversarial examples that would be encountered in the real world (Koh et al.
2021). Simply adding random perturbations to medical data usually yields
invalid and unrealistic examples. Specialized perturbations can mitigate this,
but defining them is a complex task that requires significant domain
expertise. Furthermore, defining perturbations to cover all clinically
plausible real-world adversarial attacks is infeasible. Moreover, improving
robustness to synthetic adversarial examples does not necessarily improve
robustness to natural adversarial examples.

Secondly, the lack of labeled medical time-series datasets presents a
challenge in the generation of synthetic adversarial examples. Such datasets
are time-consuming and expensive to create. A common way to do so is to
collect data via an observational study (Bonafide et al. 2017) and then have
domain experts manually label the data. However, such a study is a major
commitment of effort and time.

Instead, we focus on _natural_ adversarial examples — real patient examples
that are difficult to classify (Hendrycks et al. 2021). Natural adversarial
examples capture the inherent variations and uncertainties present in real-
world medical data that effectively deceive models. Therefore, evaluating the
robustness of medical deep learning models against such examples becomes
crucial to ensuring model robustness in the real world. We aim to identify
natural adversarial examples in _unlabeled_ medical time-series data so that
we can produce naturally adversarial _datasets_.

Robustness is commonly evaluated by examining a model’s accuracy on a labeled
adversarial dataset. While fully-supervised labels are expensive to procure,
obtaining weak labels is a cheap and quick alternative. However, assessing
accuracy with respect to weak labels becomes challenging due to the
uncertainty surrounding the correctness of the weak labels. In our experience,
weak labels for adversarial examples are prone to significant inaccuracies due
to the inherent difficulty in classifying these examples. Hence a robust model
would tend to disagree with the weak labels of natural adversarial examples.
Therefore, we aim to curate _adversarially ordered_ datasets, wherein the
proportion and severity of adversarial examples gradually increase per
dataset, correspondingly leading to a diminishing accuracy of each dataset’s
weak labels. In practice, a robust model would be expected to show decreasing
accuracy with respect to our weak labels on adversarially ordered datasets.

This paper introduces a weakly-supervised method for curating adversarially
ordered datasets containing natural adversarial examples. A key step of our
method is to probabilistically label data using _weakly-supervised data
labeling_. This process leverages _labeling functions_ , which assign labels
to subsets of the data. Labeling functions are noisy: their labels can be
incorrect, incomplete, or contradictory. Furthermore, labeling functions are
assumed to be conditionally independent. We start with a dataset of unlabeled
time series and a set of labeling functions. We then select a subset of
_independent labeling functions_ to be used in our method via a heuristic. The
outputs of the selected labeling functions are combined to yield a
probabilistic label, i.e., a label and corresponding confidence in it, for
each sample in the dataset.

Adversarial examples are by definition difficult to classify, hence the
_uncertainty in the label_ of an example can indicate how adversarial it is.
We identify adversarial examples as those with high label uncertainty. As
indication of label uncertainty, we will use label confidence scores.
Consequently, they will serve as the basis for adversarial ordering.

Unfortunately, label confidences do not reflect the true accuracies of the
labels: weakly-supervised data labeling techniques are generally _poorly
calibrated_. To overcome this issue, we construct _confidence intervals_ for
the probabilistic labels output by these techniques to quantify the
uncertainty of the labeling process. An important consideration not previously
considered is the _number of labeling functions_ combined to produce the
label. Informed by this circumstance, our intervals indicate, for a given
level of confidence, the range of possible confidence values in the assigned
label. Finally, using the lower bounds of confidence intervals, we order the
examples by their natural adversarialness.

We validate our method on six medical case studies and three non-medical case
studies. The medical case studies include five clinical alarm datasets derived
from a 551-hour alarm labeling effort from Children’s Hospital of Philadelphia
(CHOP), and one medical imaging dataset extracted from the Open-i dataset
(https://openi.nlm.nih.gov/faq) maintained by the National Institutes of
Health (NIH). The three non-medical case studies include weather-related
tweets, book reviews, and synthetic YouTube comments. Our evaluation
demonstrates that our approach successfully generates natural adversarial
datasets with statistically valid adversarial ordering.

In summary, this paper makes the following contributions:

  * •

An approach to building datasets with increasing amounts of natural
adversarial examples,

  * •

A method of selecting independent labeling functions to use for weakly-
supervised data labeling,

  * •

Confidence intervals to quantify uncertainty in the probabilistic labels,

  * •

An evaluation of the statistical validity of our curated adversarially ordered
natural datasets on 9 case studies.

## Related Work

#### Adversarial Examples.

Szegedy et al. discovered the existence of _adversarial examples_ : inputs
intentionally designed to cause machine learning models to predict
incorrectly. Subsequent research has predominately focused on _synthetic_
adversarial examples, which are artificial inputs specifically generated to
deceive models. A common approach for generating synthetic adversarial
examples is to apply adversarial perturbations to clean inputs. For example,
$\ell_{p}$ adversarial examples are generated by perturbing an input by some
worst-case distortion that is small in the $\ell_{p}$ sense (Vorobeychik and
Kantarcioglu 2018).The small distortion is nearly imperceptible to humans,
making it challenging to detect by inspection — but can have a significant
negative impact on the behavior of a model. Unfortunately, $\ell_{p}$
adversarial examples are not suitable for time series medical data because
adding such noise to such data typically yields invalid and unrealistic
examples. Adversarial spatial transformations (e.g., rotations, translations)
may also be used to perturb inputs to generate synthetic adversarial examples
(Xiao et al. 2018). However, this approach is only applicable for data with
spatial structure (like image data) and hence is not applicable for time-
series data.

An alternative approach for generating synthetic adversarial examples is to
use synthetic data generation techniques. Researchers have considered using
state-of-the-art patient simulators (Chen et al. 2016; Kushner et al. 2019)
and Generative Adversarial Networks (GANs) (Baluja and Fischer 2017; Xiao et
al. 2019; Hu and Tan 2022) to efficiently produce perceptually realistic
adversarial examples. However, both techniques have substantial limitations.
Patient simulators and GANs can struggle to fully replicate complex
physiological variations found in real medical data, resulting in unrealistic
examples. This difficulty arises due to the inherent complexity of human
physiology and the limitations of computational modeling, which require these
techniques to learn simplified models of medical scenarios. Adversarial
examples generated by these techniques can also be biased. The quality of the
adversarial examples generated by these techniques is highly dependent on the
quality of the technique’s training data. Thus, biases in the training data
(such as under-representation of specific ethnicities) can propagate into the
generated data. In contrast to these techniques, our approach consistently
produces realistic adversarial examples by sampling them from real medical
data. However, our examples may reflect the bias of the input data.

Recently, Hendrycks et al. found that clean, realistic inputs can reliably
degrade the performance of machine learning models. These inputs are referred
to as _natural_ adversarial examples, as they are examples that occur
naturally but still to lead to erroneous model predictions. In practice,
natural adversarial examples are obtained by selecting them from existing
datasets. Adversarial filtration is a popular approach to select natural
adversarial examples from an existing dataset by removing examples that are
diverse in appearance but classified easily via very predictable
classification boundaries (Sung 1996). Several works have explored filtration
by removing examples solved with simple spurious cues in the image domain
(Hendrycks et al. 2021) and natural language processing (NLP) domain
(Sakaguchi et al. 2021; Bhagavatula et al. 2019; Zellers et al. 2019; Dua et
al. 2019; Bisk et al. 2020; Hendrycks et al. 2020). Unfortunately, the
application of this specific method may not be suitable for time-series
medical data since these datasets generally lack spurious cues. However, our
approach can be considered an adversarial filtration technique that removes
examples based on the lower bound of weak label confidences. We leverage the
intuition that low label confidence implies a sample is more “naturally
adversarial”. An alternative way to measure label confidence is to
crowdsource, i.e., have multiple human labelers label an example and then
compute the disagreement amongst the labelers. Hence, adversarial filtration
via crowdsourced label uncertainty is an approach alternative to ours.

Adversarial examples typically have a different underlying data distribution
than non-adversarial (clean) examples. This suggests that it may be possible
to select natural adversarial examples from existing data by observing the
input data distribution. For example, natural adversarial examples can present
as outliers in the input data feature distribution (Aggarwal 2013). The
feature distribution can be estimated via a density estimator, and then an
outlier detection method can be used to select the natural adversarial
examples. Out-of-distribution (OOD) detectors (Ruff et al. 2021) may also be
used to identify natural adversarial examples in existing data. Adversarial
examples are similar to the clean examples from the training data distribution
but with small imperceptible perturbations, and thus can be considered OOD.
This paper focuses on selecting natural adversarial examples based on label
uncertainty rather than feature values. We leave the investigation of feature-
based approaches to be explored in future work.

#### Weakly-Supervised Data Labeling.

Recently, a quick and inexpensive way of labeling data has emerged, known as
_weakly-supervised data labeling_. Its key element is a set of quantitative
intuitions about how the data corresponds to labels. For example, a clinician
might say, “when a patient over 60 years old has had a heart rate over 120
beats for over a minute, such an alarm is a high priority.” These intuitions,
algorithmically represented as _labeling functions_ , are allowed to be
incomplete, sometimes incorrect, and contradictory. A labeling function
returns a class label or an “abstain” verdict for any input. Given a diverse
combination of many labeling functions and an unlabeled dataset, weakly-
supervised data labeling techniques produce probabilistic labels for each
sample in the dataset in the form of probability distributions over the label
space. Such a label is represented as a probability distribution over the
label space. Subsequently, for each sample, the weak label consists of the
label with the highest probability and the confidence equal to that
probability. A prominent weakly-supervised data labeling technique _Snorkel_
(Ratner et al. 2016, 2020) estimates an optimal weight for each labeling
function by using a generative graphical model and a prior on the class
balance. Our approach takes data and labeling functions as input, feeds them
into Snorkel, and builds on the resulting probabilistic labels to order the
data adversarially.

Extending the above work, adversarial data programming generates data in
addition to labeling it (Pal and Balasubramanian 2018, 2020). A GAN is used to
estimate the weight of each labeling function as well as the dependencies
between them given a set of labeling functions and an unlabeled dataset. The
weights and dependencies are used by the GAN’s generator to produce labeled
samples that come from the data distribution. Hence, one may be able to train
an adversarial data programming model to generate more examples given a
dataset comprised of natural adversarial examples. However, as mentioned
earlier, GAN generated examples can be unrealistic and biased. An intriguing
avenue for future work is imposing realistic constraints (Sridhar et al. 2023)
on generators of natural adversarial examples.

## Adversarially Ordered Natural Datasets

We start with several notational conventions. Given a set $B$, we write $|B|$
to be the set’s cardinality. Given a value $v\in\mathbb{R}$, we write $|v|$.
We are given an evaluation dataset $X\subset\mathcal{X}$ with _unknown_ true
labels $Y$.

Before formally stating the problem considered in this work, we define
adversarially ordered datasets and statistically valid adversarially ordered
datasets.

######  Definition 1 (Adversarially Ordered Natural Datasets)

Consider a sequence of natural (non-synthetic) datasets $D_{1},\ldots,D_{N}$,
where $D_{i}=(X_{i},Y_{i},\hat{Y}_{i})$ is composed of samples $X_{i}$,
corresponding unknown true labels $Y_{i}$, and corresponding known noisy
labels $\hat{Y}_{i}$. These datasets are _adversarially ordered_ if their
accuracy (non-strictly) monotonically decreases. That is,

$\textsc{acc}(Y_{1},\hat{Y}_{1})\geq\textsc{acc}(Y_{2},\hat{Y}_{2})\geq\ldots\geq\textsc{acc}(Y_{N},\hat{Y}_{N})$

where

$\textsc{acc}(Y,\hat{Y})=\frac{1}{|Y|}\sum_{i=1}^{|Y|}\mathbf{1}\left(y_{i}=\hat{y}_{i}\right)$

In short, the accuracy of the weak labels for each dataset in adversarially
ordered natural datasets does not increase. It is important to validate this
trend to verify its direction and significance. Spearman’s rank correlation
coefficient $\rho$ is a widely used measure of the strength and direction of a
monotonic relationship (Spearman 1987). The coefficient $\rho$ is a value
between -1 and 1, where close to -1 indicates a strong, monotonically
decreasing trend and close to 1 indicates a strong, monotonically increasing
trend. The coefficient is accompanied by a p-value $p^{*}$, which quantifies
the statistical significance of the observed relationship. The following
definition describes how to compute Spearman’s Rank Correlation for
adversarially ordered natural datasets.

######  Definition 2 (Spearman’s Rank Correlation)

Consider adversarially ordered natural datasets $D_{1},\ldots,D_{N}$. Let
$R=(i_{1},\ldots,i_{N})$ where $i_{n}\in\\{1,\dots,N\\}$ be the rank order of
the accuracies of the datasets (i.e.,
$\textsc{acc}(Y_{i_{1}},\hat{Y}_{i_{1}})\leq\textsc{acc}(Y_{i_{2}},\hat{Y}_{i_{2}})\leq\ldots\leq\textsc{acc}(Y_{i_{N}},\hat{Y}_{i_{N}})$).
We compute _Spearman’s rank correlation_ coefficient $\rho$ as follows:

$\rho=\frac{6\sum_{i=1}^{N}(R_{i}-i)^{2}}{N(N^{2}-1)}$

The corresponding p-value is $p^{*}=2\times\mathbf{P}(T\geq|t|)$, where $T$
follows a t-distribution with $N-2$ degrees of freedom and

$t=\rho\sqrt{\frac{N-2}{1-\rho^{2}}}$

We determine the statistical validity of adversarially ordered datasets by
checking that Spearman’s rank correlation is negative and p-value is below a
predetermined significance level. We formalize this idea in the definition
below.

######  Definition 3 (Statistically Valid Adversarial Ordering)

Consider adversarially ordered natural datasets $D_{1},\ldots,D_{N}$. Let
$\rho$ and $p^{*}$ be the Spearman’s Rank Correlation Coefficient and
corresponding p-value, computed from the weak label accuracies of the
datasets. The datasets are _statistically valid adversarially ordered_ if the
coefficient is negative (i.e., $\rho<0$) and the p-value is statistically
significant (i.e., $p^{*}\leq\gamma$ where $\gamma$ is the predetermined
adversarial ordering significance threshold).

We are now ready to state our central technical problem.

#### Problem.

Given unlabeled data $X$ and labeling functions $\Lambda$, produce
statistically valid adversarially ordered natural datasets
$D_{1},\ldots,D_{N}$.

Now we highlight the challenges of obtaining statistically valid adversarially
ordered medical datasets — and our steps to overcome them. Our main challenge
is the absence of true labels in a given dataset $X$, which makes it
impossible to directly compute the accuracies in Def. 1. To address this
challenge, we will create a _probabilistic labeler_ — an algorithm that takes
a sample and assigns it an estimated label and a confidence in that label (a
value between 0 and 1). We refer to the assigned label as $\hat{f}(x)$ and its
confidence as $\hat{g}(x)$ for any sample $x\in\mathcal{X}$. We will build
that labeler from _labeling functions_ , which encode the rules of thumb and
heuristics acquired from medical experts. A labeling function
$\lambda:\mathcal{X}\to\mathcal{Y}\cup\\{0\\}$ takes a sample
$x\in\mathcal{X}$ and either abstains (i.e., assigns label 0) or assigns one
of the classes to it. Labeling functions can contradict each other or abstain
in different combinations.

Our second challenge is that probabilistic labelers are generally
overconfident in their estimated labels (as our experience shows). That is,
confidence scores $\hat{g}(x)$ are unreliable and should not be used as an
indication of the accuracy of label $\hat{f}(x)$. Hence, our second task is to
better quantify the uncertainty in the estimated labels. For each sample $x\in
X$, we generate an interval $I$ of possible confidences in label $\hat{f}(x)$.
This interval should contain the _true_ confidence in label $\hat{f}(x)$, call
it $g(x)$, with probability of at least $1-\alpha$ where $\alpha$ is the
significance level specified by the user. The next section details our
solution to the problem of this paper.

## Dataset Design via Adversarial Ordering

This section describes our approach to design adversarially ordered natural
datasets. Figure 1 summarizes the steps of our approach. Our approach takes as
input an unlabeled dataset $X$ and a set of labeling functions $\Lambda$. The
first step of our approach selects a subset of the labeling functions to be
used in the next step, probabilistic labeling of the unlabeled dataset. Next,
we quantify the uncertainty in the probabilistic labels by constructing
confidence intervals around them. Then the confidence intervals are used in
the final step to design a sequence of progressively more adversarial
datasets.

Figure 1: Our approach for curating adversarially ordered natural datasets.

### Labeling Function Pruning

Labeling functions are a key component of weak supervision techniques,
offering a pragmatic approach for annotating data with noisy yet informative
labels. However, in practice, labeling functions often exhibit statistical
dependencies between them, (e.g., multiple functions relying on the same
features or patterns) (Ratner et al. 2016). Dependent labeling functions
result in duplicate information that can bias the outputs of weak supervision
techniques, unless accounted for in the underlying model. Most techniques
commonly assume the independence of labeling functions (conditioned on the
true label); however, they also offer the option to provide labeling function
dependencies as additional input (Ratner et al. 2016, 2019). When provided,
the dependencies are embedded into the underlying model. Unfortunately, in
many applications the number of dependencies can become too large for the
technique, especially when given many labeling functions. Hence it may be
preferred to provide a smaller set of independent labeling functions instead
of a larger set of dependent ones. We opt to use weakly-supervised data
labeling techniques under the independent labeling function assumption.
Therefore, it is crucial to identify a subset of the labeling functions
$\Lambda^{\prime}\subseteq\Lambda$ where the labeling functions are
independent of each other prior to applying these techniques.

Algorithm 1 Selecting independent labeling functions.

Input: Labeling functions $\Lambda$, unlabeled data $X$, and correlation
threshold $\delta$
Output: Independent subset of labeling functions
$\Lambda^{\prime}\subseteq\Lambda$

1: Apply labeling functions $\Lambda$ to the data $X$ to get weak labels
$\mathcal{L}_{\Lambda}(X)$.

2: Compute the correlation $c_{ij}$ between all pairs of labeling functions
$\lambda_{i},\lambda_{j}\in\Lambda$ where $i\neq j$ using
$\mathcal{L}_{\Lambda}(X)$.

3: Construct a labeling function dependency graph $G$ with labeling functions
$\lambda$ as nodes and edges between labeling functions (nodes) with
correlation $c_{ij}>\delta$.

4: Rank labeling functions in descending order by the number of maximal
cliques of $G$ they belong to, breaking ties by labeling function coverage
(higher coverage corresponds to a higher rank). Let $R$ denote this ranking.

5: Let $\Lambda^{\prime}=\Lambda$. For each labeling function $\lambda$ in the
ranking $R$, if $\lambda\in\Lambda^{\prime}$, remove all other labeling
functions with which it shares a maximal clique in $G$ from
$\Lambda^{\prime}$.

The pseudo-code for our independent labeling function selection procedure is
presented in Algorithm 1. First, the labeling functions $\Lambda$ are applied
to the dataset $X$ to obtain the weak labels. We will refer to the weak labels
for $X$ as
$\mathcal{L}_{\Lambda}(X)=(\mathcal{L}_{\lambda_{i}}(X),\ldots,\mathcal{L}_{\lambda_{|\Lambda|}}(X))$.
Next, we compute the Pearson correlation coefficients $c_{ij}$ for each pair
of labeling functions $(\lambda_{i},\lambda_{j})$ where
$\lambda_{i},\lambda_{j}\in\Lambda$ and $i\neq j$ from the weak labels, i.e.,
$c_{ij}=\textsc{Pearson}(\mathcal{L}_{\lambda_{i}}(X),\mathcal{L}_{\lambda_{j}}(X))$.
These coefficients are measures of linear correlation (dependence) between
pairs of our labeling functions.

Now we aim to rank the labeling functions from most to least independent. To
do this, we first construct a graph representation of the labeling function
dependencies (Ratner et al. 2019). Concretely, the nodes of the graph $G$ are
the labeling functions in $\Lambda$. We add an edge to $G$ for each labeling
function pair whose correlation is sufficiently large, that is, add edge
$(\lambda_{i},\lambda_{j})$ if $|c_{ij}|>\delta$ where $\delta$ is a user-
specified minimum threshold on correlation. Next, we identify maximal cliques
of labeling functions, which effectively reveal subsets of labeling functions
that tend to share similar labeling patterns (i.e., cover the same
information). Hence our intuition is that labeling functions belonging to many
cliques are more dependent. We rank the labeling functions by the number of
maximal cliques they belong to in $G$. Ties in the ranking are resolved by
labeling function coverage, that is, the proportion of samples for which a
labeling function emits a (non-abstain) label, i.e., $\frac{1}{|X|}\sum_{x\in
X}\mathbf{1}\left(\lambda(x)\neq 0\right)$. We assign higher rank to labeling
functions with higher coverage to retain a substantial proportion of weak
labels for the data $X$.

Finally, we determine a subset of independent labeling functions
$\Lambda^{\prime}\subseteq\Lambda$. Our goal is to select the smallest subset
of labeling functions that cover all the cliques. Let
$\Lambda^{\prime}=\Lambda$ to start. Then for each labeling function $\lambda$
in the ranking (in descending order), if $\lambda\in\Lambda^{\prime}$, then
remove all other labeling functions that share a maximal clique with
$\lambda$. Now we have an independent subset
$\Lambda^{\prime}\subseteq\Lambda$ for weak supervision.

### Probabilistic Labeling

In this step, we combine the weak labels produced by the labeling functions
into a single “strong” probabilistic label. This label is characterized by a
confidence score between 0 and 1, indicating the level of certainty in the
label’s accuracy. Mathematically, for each sample $x\in X$, we combine its
weak labels $\mathcal{L}_{\Lambda}(x)$ into a probabilistic strong label
$\hat{f}(x)$ with confidence $\hat{g}(x)$.

Our approach computes the strong label and corresponding confidence using a
weak supervision technique that performs a _weighted combination_ over the
weak labels. The weights $w$ has a fixed vector per class containing one
weight per labeling function. While our approach supports a variety of
weighted combination techniques, we consider two of them in this paper:

  * •

Majority vote

  * •

Generative model with an uninformed prior

These two techniques are applied in two steps:

  1. 1.

Determine the weights $w$

  2. 2.

Combine the weak labels $\mathcal{L}_{\Lambda}(x)$ using weights $w$ for each
$x\in X$

The first step learns a non-negative weight vector where each weight indicates
the relative priority of the corresponding labeling function. Majority vote, a
widely-used and straightforward method for combining multiple discrete signals
into one, assigns equal priority to each labeling function. Hence, the weight
vector for majority vote is uniform (i.e., $w_{i}^{(y)}=1$ for all
$i\in\\{1,\ldots,|\Lambda|\\}$ and $y\in\mathcal{Y}$ ). Generative models are
popular in state-of-the-art weak supervision literature (Ratner et al. 2016;
Bach et al. 2017; Pal and Balasubramanian 2018; Fu et al. 2020) These models
give higher weights to labeling functions with higher accuracies. The
accuracies are unknown a priori, so the model estimates them by observing the
agreements and disagreements of the labeling functions in the data
($\mathcal{L}_{\Lambda}(x)$) during its training phase. Furthermore, the model
can be trained with a prior that specifies the expected frequency of each
label in $\mathcal{Y}$. We assume the actual prior is unknown and we do not
have a labeled dataset on which we can estimate it, so we use an uninformed
prior which assigns equal probability to every label. The technical details on
this use of generative models can be found in (Pugh et al. 2021).

In the second step, the learned weights are used to combine the weak labels
into a single probabilistic label for each sample. Suppose now we want to
label a sample $x$ given its weak labels $\mathcal{L}_{\Lambda}(x)$ and the
weights $w$. First, we obtain label weights by adding up the weights of all
labeling functions choosing that label, i.e.,
$\sum_{i=1}^{|\Lambda|}w_{i}^{(y)}\cdot\lambda_{i}(x)$ for $y\in\mathcal{Y}$.
Then we pass the label weights through the softmax function, which is a
standard way to normalize positive real numbers into a probability
distribution. Finally, the largest normalized weight is used as the confidence
$\hat{g}(x)$ and the label corresponding to that weight becomes the estimated
label $\hat{f}(x)$.

Unfortunately, as mentioned in the previous section, the confidence scores
output by these two weak supervision techniques are typically over-confident;
that is, the confidence $\hat{g}(x)$ overestimates the accuracy of the
estimated label $\hat{f}(x)$. In other words, weak supervision techniques are
poorly calibrated (Guo et al. 2017) – the confidence scores do not reflect the
true accuracy of the label. Majority vote can be especially prone to over-
confidence because if many of the labeling functions are inaccurate, the label
prediction can be incorrect with high confidence. Generative models have an
advantage over majority vote because they can account for the inaccuracies of
labeling functions and, thus, yield more accurate confidences. However, as we
witness in practice, generative models are also often poorly calibrated.
Hence, the confidences output by weak supervision techniques should be used
with caution or disregarded entirely. In response, we propose using confidence
intervals to account for the uncertainty in these confidences.

### Confidence Intervals for Weak Label Accuracies

Our confidences in the estimated labels from the prior subsection can be
unreliable. More precisely, the confidences in the estimated labels may not
reflect the true accuracy of those estimated labels. We quantify the potential
inaccuracy in the estimated labels by providing _confidence intervals_. Our
confidence intervals contain the likely true confidences in the estimated
labels. For each sample $x\in X$, we aim to generate an interval $I$
containing the true confidence in the estimated label $\hat{f}(x)$, call it
$g(x)$, with probability of at least $1-\alpha$, i.e., $\mathbf{P}[g(x)\in
I]\geq 1-\alpha$.

We bound the unknown true confidences $g(X)$ in our estimated labels
$\hat{f}(X)$ using the Clopper-Pearson (CP) interval $[\theta_{L},\theta_{U}]$
where $\theta_{L},\theta_{U}\in[0,1]$ and $\theta_{L}<\theta_{U}$ (Clopper and
Pearson 1934). This interval bounds the true success probability $\mu$,
constructed from a sample $s\sim B(n,\mu)$ from a binomial distribution with
$n$ trials and success probability $\mu$, which holds with probability at
least $1-\alpha$, i.e., $\mathbf{P}_{s\sim
B(n,\mu)}\left[\theta_{L}(\alpha;n,s)\leq\mu\leq\theta_{U}(\alpha;n,s)\right]\geq
1-\alpha$.

Intuitively, few non-abstaining labeling functions with large weights should
yield a lower confidence than many non-abstaining labeling functions with
moderate weights. Hence, in addition to the labeling function weights, the
_number of labeling functions_ $n$ contributing to the estimation of the
labels should be considered in the calculation of confidence in $\hat{f}(x)$.
While the confidences provided by probabilistic labelers $\hat{g}(X)$ do not
take this into account (which can lead to their unreliability), we incorporate
the number of voting labeling functions into the construction of our
intervals. When a labeling function emits a (non-abstain) label for some
sample $x$, we consider it a Bernoulli trial whose outcome is a success if
that label is correct or a failure if it is incorrect. We let the number of
successes $s$ (of the $n$ trials) be the normalized probability of label
$\hat{f}(x)$ weighted by the number of non-abstaining labeling functions $n$.
Hence by definition of the CP interval, we derive the interval
$I=[\theta_{L},\theta_{U}]$ for $g(x)$ given any sample $x\in X$ and
significance level $\alpha$ as,

$\theta_{L}(\alpha;n,s)=B\left(\frac{\alpha}{2};s,\left(n-s+1\right)\right)$
$\theta_{U}(\alpha;n,s)=B\left(1-\frac{\alpha}{2};(s+1),\left(n-s\right)\right)$

where

$n(x)=\sum_{i=1}^{|\Lambda|}\mathbf{1}\left(\lambda_{i}(x)\neq 0\right)$
$s(x)=n(x)\cdot\frac{\mbox{exp}\left\\{\sum_{i=1}^{|\Lambda|}w_{i}^{\left(\hat{f}(x)\right)}\lambda_{i}(x)\right\\}}{\sum_{y\in\mathcal{Y}}\mbox{exp}\left\\{\sum_{i=1}^{|\Lambda|}w_{i}^{(y)}\lambda_{i}(x)\right\\}}$

where $B(q;a,b)$ is the $q$-th quantile from a beta distribution with shape
parameters $a$ and $b$.

### Adversarial Dataset Design

Now we create a sequence of adversarially ordered natural datasets. Our
confidence interval lower bounds indicate the smallest possible certainty in
the estimated label. So as more samples with small lower bounds are included
in a dataset, the more adversarial it gets. We arrange the samples in $X$ by
their confidence interval lower bound in descending order, i.e.,
$x_{i_{1}},x_{i_{2}},\ldots,x_{i_{|X|}}$ where
$\theta_{L}(x_{i_{1}})\geq\theta_{L}(x_{i_{2}})\geq\ldots\geq\theta_{L}(x_{i_{|X|}})$
and $i_{1},\ldots,i_{|X|}\in\\{1,\ldots,|X|\\}$. Then we produce a sequence of
datasets $D_{1},\ldots,D_{N}$ where each dataset $D_{n}$ contains the top
$\frac{i}{N}$ percent of ordered samples and their corresponding estimated
labels, that is,

$D_{n}=\\{(x_{i_{j}},\hat{f}(x_{i_{j}}))\\}\quad\mbox{for
}j\in\left\\{1,\ldots,\frac{i\cdot|X|}{N}\right\\}$

In conclusion, we have natural datasets $D_{1},\ldots,D_{N}$ that are
progressively more adversarial.

## Results

Case Study | Probabilistic | PL Conf | PL Conf | CI LB | Our
---|---|---|---|---|---
Labeler (PL) | with all LFs | with Indep. LFs | with all LFs | Approach
HR Low | Majority Vote | -0.719 (0.019) | -0.903 (0.000) | -0.863 (0.001) | -0.730 (0.017)
| Snorkel | -0.313 (0.379) | -0.730 (0.017) | -0.827 (0.003) | 0.595 (0.070)
HR High | Majority Vote | 0.818 (0.004) | -0.976 (0.000) | 0.857 (0.002) | -1.000 (0.000)
| Snorkel | 0.964 (0.000) | 0.927 (0.000) | 0.988 (0.000) | -0.806 (0.005)
RR Low | Majority Vote | -0.997 (0.000) | -0.879 (0.001) | -0.997 (0.000) | -0.891 (0.001)
| Snorkel | -0.997 (0.000) | -0.394 (0.260) | -0.997 (0.000) | -0.806 (0.005)
RR High | Majority Vote | -1.000 (0.000) | -0.988 (0.000) | -1.000 (0.000) | -0.952 (0.000)
| Snorkel | -1.000 (0.000) | -0.952 (0.000) | -1.000 (0.000) | -0.455 (0.187)
SpO2 Low | Majority Vote | -0.309 (0.385) | 0.891 (0.001) | -0.248 (0.489) | 0.188 (0.603)
| Snorkel | -0.988 (0.000) | 0.430 (0.214) | -0.455 (0.187) | -0.612 (0.060)
Cross Modal | Majority Vote | -1.000 (0.000) | -1.000 (0.000) | -0.879 (0.001) | -0.782 (0.008)
| Snorkel | -1.000 (0.000) | -0.988 (0.000) | -0.624 (0.054) | -0.806 (0.005)
Crowdsourcing | Majority Vote | -0.864 (0.001) | -0.988 (0.000) | -0.864 (0.001) | -0.976 (0.000)
| Snorkel | -0.976 (0.000) | -0.988 (0.000) | -0.864 (0.001) | -0.976 (0.000)
Recsys | Majority Vote | -1.000 (0.000) | -1.000 (0.000) | -0.321 (0.365) | -0.321 (0.365)
| Snorkel | -0.988 (0.000) | -0.988 (0.000) | -0.539 (0.108) | -0.539 (0.108)
Spam | Majority Vote | -0.985 (0.000) | -0.985 (0.000) | -0.767 (0.010) | -0.767 (0.010)
| Snorkel | -0.673 (0.033) | -0.673 (0.033) | -0.656 (0.039) | -0.656 (0.039)
Table 1: Spearman’s Rank Correlation of the adversarially ordered natural
datasets. The values reported are the correlation coefficient and
corresponding p-value in parentheses. Values in bold indicate datasets with a
statistically significant monotonically decreasing accuracy, while underlined
values indicate statistically significant increasing accuracy (i.e., a
violation of our requirements). The regular font is used for values indicating
a statistically insignificant trend in accuracy.

In this section, we evaluate our approach on nine case studies (six with
clinical relevance). Specifically, we summarize the nine case studies describe
comparative approaches, and evaluate the adversarial ordering of natural
adversarial datasets produced by our approach.

### Case Studies

We evaluate our approach on five physiological alarm suppression case studies,
one clinical text classification case study, and three non-clinical text
classification case studies.

#### Physiologic Alarm Suppression.

Alarm suppression, which involves distinguishing suppressible (uninformative
to clinical care) and non-suppressible physiologic-monitoring alarms, is the
aim of the first five datasets. Datasets of heart rate (HR) low/high,
respiratory rate (RR) low/high, and SpO2 low alarms were extracted from
(MacMurchy et al. 2017). Each alarm sample is represented as a multi-vital
sign time series data. We use the set of sixty-two clinician-designed labeling
functions developed for this example from (Pugh et al. 2022, 2021). The
labeling functions analyze the time series data to make predictions on
suppressibility, e.g., an alarm is non-suppressible if the heart rate is above
220 for longer than 10 seconds after the alarm starts, otherwise it abstains.

#### Clinical Text Classification.

The Crossmodal dataset aims to label radiography, computed tomography, and
electroencephalography images by writing labeling functions over an auxiliary
modality, namely, corresponding imaging text reports (Dunnmon et al. 2019).
The labeling functions in this example are clinician-designed, expressing
simple pattern-matching or ontology-lookup heuristics.

#### Non-Clinical Text Classification.

The Crowdsourcing, Recsys, and Spam datasets are publicly-available at
www.snorkel.org. The objective of Crowdsourcing is to label tweets pertaining
to weather expressing either a positive or negative sentiment. Recsys aims to
predict whether a user will read and like any given book or not. Finally, the
Spam dataset aims to classify spam emails.

### Comparative Approaches

We previously discussed two primary challenges of probabilistic labeling via
weak supervision techniques: (1) labeling function dependence, and (2)
unreliability of confidence scores output by probabilistic labelers. In our
evaluation, we will demonstrate why addressing these challenges is necessary
to achieve sufficient performance. Hence we define three comparative
approaches as follows. PL Conf with all LFs produces datasets by ordering
samples by the probabilistic labeler’s confidences ($\hat{g}(X)$) computed
using _all_ of the provided labeling functions (regardless of the dependencies
between them), and then selecting the top-$p$ percent of samples per dataset
(as done in our approach). PL Conf with Indep. LFs repeats the same steps as
PL Conf with all LFs but uses a set of independent labeling functions selected
by the same procedure from our approach. Lastly, CI LB w/ all LFs is an
instance of our approach that skips the labeling function pruning step, that
is, it is our approach using _all_ provided labeling functions.

### Evaluation of Natural Adversarial Ordering

Finally, we present the results of our approach applied to several case
studies. Recall that the goal of the paper is to generate a sequence of
statistically valid adversarially ordered natural datasets as per Def. 3. We
validate the progressive increase of “adversarialness” in the datasets by
analyzing the Spearman’s rank correlation (Def. 2) of the adversarially
ordered datasets.

The parameters of our approach are set as follows. Correlation between two
labeling functions greater than 0.5 indicates that the labeling functions are
dependent, i.e., correlation threshold $\delta=0.5$. We allow for a 5% chance
of the confidence intervals not containing the true confidence in our
probabilistic labels, i.e., significance level $\alpha=0.05$. Lastly, we let
the, i.e., adversarial ordering significance threshold $\gamma=0.01$.

We report the Spearman’s rank correlation coefficients and corresponding
p-values for the accuracies of the datasets produced by our approach and by
the comparative approaches in Table 1. We plot the actual accuracies of the
datasets in the Appendix. The results demonstrate that our approach yields
statistically significant adversarially ordered datasets in ten cases, and
statistically invalid datasets on all other examples. We interpret this as a
substantive outcome: our approach is applicable in a majority of the examples,
and never yields datasets with statistically valid increasing accuracy. All
the comparative approaches produce at least as many datasets with
statistically valid decreasing accuracy, but also produce datasets with
statistically significant monotonically increasing accuracy — thus violating
our requirements. If used for robustness evaluation of learning models in
practice, such datasets can cause incorrect and misleading results. Hence, we
conclude that our approach yields more reliable results than the comparative
approaches.

## Conclusion

In this paper, we proposed an approach for designing a sequence of
adversarially ordered natural datasets for the purpose of evaluating model
robustness to natural adversarial examples. Our approach identifies a set of
independent labeling functions to use for probabilistic labeling.
Probabilistic labels obtained via weak supervision techniques were used as
proxy of the unknown true labels. We quantify the uncertainty in these labels
with Clopper-Pearson confidence bounds, and construct our datasets according
to the lower bound which is a indication of how “naturally adversaria” a
sample may be. Finally, we evaluated our approach on six clinical case studies
and three others and showed that we produce natural datasets with
statistically valid adversarial ordering. Directions for future work include
(1) evaluating the robustness of real classifiers using our statistically
valid adversarially ordered natural datasets, (2) developing weakly-supervised
methods for evaluating additional properties of machine learning models, and
(3) calibrating weakly-supervised data labeling techniques.

## References

  * Aggarwal (2013) Aggarwal, C. C. 2013.  _Outlier Analysis_.  New York: Springer.  ISBN 978-1-4614-6395-5.
  * Alcorn et al. (2019) Alcorn, M. A.; Li, Q.; Gong, Z.; Wang, C.; Mai, L.; Ku, W.-S.; and Nguyen, A. 2019\.  Strike (With) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 4840–4849.  ISSN: 2575-7075.
  * Bach et al. (2017) Bach, S. H.; He, B.; Ratner, A.; and Ré, C. 2017.  Learning the structure of generative models without labeled data.  In _International Conference on Machine Learning_ , 273–282. PMLR.
  * Baluja and Fischer (2017) Baluja, S.; and Fischer, I. 2017.  Adversarial transformation networks: Learning to generate adversarial examples.  _arXiv preprint arXiv:1703.09387_.
  * Bhagavatula et al. (2019) Bhagavatula, C.; Bras, R. L.; Malaviya, C.; Sakaguchi, K.; Holtzman, A.; Rashkin, H.; Downey, D.; Yih, S. W.-t.; and Choi, Y. 2019.  Abductive commonsense reasoning.  _arXiv preprint arXiv:1908.05739_.
  * Bisk et al. (2020) Bisk, Y.; Zellers, R.; Gao, J.; Choi, Y.; et al. 2020.  Piqa: Reasoning about physical commonsense in natural language.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, 7432–7439.
  * Bonafide et al. (2017) Bonafide, C. P.; Localio, A. R.; Holmes, J. H.; Nadkarni, V. M.; Stemler, S.; MacMurchy, M.; Zander, M.; Roberts, K. E.; Lin, R.; and Keren, R. 2017.  Video Analysis of Factors Associated With Response Time to Physiologic Monitor Alarms in a Children’s Hospital.  _JAMIA Pediatrics_ , 171(1): 524–531.
  * Chen et al. (2016) Chen, S.; Sokolsky, O.; Weimer, J.; and Lee, I. 2016.  Data-driven Adaptive Safety Monitoring using Virtual Subjects in Medical Cyber-Physical Systems: A Glucose Control Case Study.  _Journal of Computer Science and Engineering_ , 75–84.
  * Clopper and Pearson (1934) Clopper, C. J.; and Pearson, E. S. 1934.  The use of confidence or fiducial limits illustrated in the case of the binomial.  _Biometrika_ , 26(4): 404–413.
  * Dua et al. (2019) Dua, D.; Wang, Y.; Dasigi, P.; Stanovsky, G.; Singh, S.; and Gardner, M. 2019.  DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs.  _arXiv preprint arXiv:1903.00161_.
  * Dunnmon et al. (2019) Dunnmon, J.; Ratner, A.; Khandwala, N.; Saab, K.; Markert, M.; Sagreiya, H.; Goldman, R.; Lee-Messer, C.; Lungren, M.; Rubin, D.; and Ré, C. 2019.  Cross-Modal Data Programming Enables Rapid Medical Machine Learning.
  * Faust et al. (2018) Faust, O.; Hagiwara, Y.; Hong, T. J.; Lih, O. S.; and Acharya, U. R. 2018.  Deep learning for healthcare applications based on physiological signals: A review.  _Computer methods and programs in biomedicine_ , 161: 1–13.
  * Fu et al. (2020) Fu, D.; Chen, M.; Sala, F.; Hooper, S.; Fatahalian, K.; and Ré, C. 2020.  Fast and three-rious: Speeding up weak supervision with triplet methods.  In _International Conference on Machine Learning_ , 3280–3291. PMLR.
  * Guo et al. (2017) Guo, C.; Pleiss, G.; Sun, Y.; and Weinberger, K. Q. 2017.  On calibration of modern neural networks.  In _Proceedings of the 34th International Conference on Machine Learning - Volume 70_ , ICML’17, 1321–1330. Sydney, NSW, Australia: JMLR.org.
  * Hendrycks et al. (2020) Hendrycks, D.; Burns, C.; Basart, S.; Critch, A.; Li, J.; Song, D.; and Steinhardt, J. 2020.  Aligning ai with shared human values.  _arXiv preprint arXiv:2008.02275_.
  * Hendrycks et al. (2021) Hendrycks, D.; Zhao, K.; Basart, S.; Steinhardt, J.; and Song, D. 2021.  Natural Adversarial Examples.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 15262–15271.
  * Hu and Tan (2022) Hu, W.; and Tan, Y. 2022.  Generating adversarial malware examples for black-box attacks based on GAN.  In _International Conference on Data Mining and Big Data_ , 409–423. Springer.
  * Koh et al. (2021) Koh, P. W.; Sagawa, S.; Marklund, H.; Xie, S. M.; Zhang, M.; Balsubramani, A.; Hu, W.; Yasunaga, M.; Phillips, R. L.; Gao, I.; Lee, T.; David, E.; Stavness, I.; Guo, W.; Earnshaw, B.; Haque, I.; Beery, S.; Leskovec, J.; Kundaje, A.; Pierson, E.; Levine, S.; Finn, C.; and Liang, P. 2021.  WILDS: A Benchmark of in-the-Wild Distribution Shifts.
  * Kushner et al. (2019) Kushner, T.; Bequette, B. W.; Cameron, F.; Forlenza, G. P.; Maahs, D. M.; and Sankaranarayanan, S. 2019.  Models, Devices, Properties, and Verification of Artificial Pancreas Systems.  In _Automated Reasoning for Systems Biology and Medicine_.
  * MacMurchy et al. (2017) MacMurchy, M.; Stemler, S.; Zander, M.; and Bonafide, C. P. 2017.  Research: Acceptability, feasibility, and cost of using video to evaluate alarm fatigue.  _Biomedical instrumentation & technology_, 51(1): 25–33.
  * Morid, Sheng, and Dunbar (2023) Morid, M. A.; Sheng, O. R. L.; and Dunbar, J. 2023.  Time Series Prediction Using Deep Learning Methods in Healthcare.  _ACM Trans. Manage. Inf. Syst._ , 14(1).
  * Pal and Balasubramanian (2018) Pal, A.; and Balasubramanian, V. N. 2018.  Adversarial data programming: Using gans to relax the bottleneck of curated labeled data.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 1556–1565.
  * Pal and Balasubramanian (2020) Pal, A.; and Balasubramanian, V. N. 2020.  Generative adversarial data programming.  _arXiv preprint arXiv:2005.00364_.
  * Pugh et al. (2022) Pugh, S.; Ruchkin, I.; Bonafide, C.; Demauro, S.; Sokolsky, O.; Lee, I.; and Weimer, J. 2022.  Evaluating Alarm Classifiers with High-Confidence Data Programming.  _ACM Trans. Comput. Healthcare_ , 3(4).
  * Pugh et al. (2021) Pugh, S.; Ruchkin, I.; Bonafide, C. P.; DeMauro, S. B.; Sokolsky, O.; Lee, I.; and Weimer, J. 2021.  High-Confidence Data Programming for Evaluating Suppression of Physiological Alarms.  In _2021 IEEE/ACM Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE)_ , 70–81. IEEE.
  * Ratner et al. (2020) Ratner, A.; Bach, S. H.; Ehrenberg, H.; Fries, J.; Wu, S.; and Ré, C. 2020.  Snorkel: Rapid training data creation with weak supervision.  _The VLDB Journal_ , 29(2-3): 709–730.
  * Ratner et al. (2019) Ratner, A.; Hancock, B.; Dunnmon, J.; Sala, F.; Pandey, S.; and Ré, C. 2019\.  Training complex models with multi-task weak supervision.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, 4763–4771.
  * Ratner et al. (2016) Ratner, A. J.; De Sa, C. M.; Wu, S.; Selsam, D.; and Ré, C. 2016.  Data programming: Creating large training sets, quickly.  _Advances in neural information processing systems_ , 29.
  * Ruff et al. (2021) Ruff, L.; Kauffmann, J. R.; Vandermeulen, R. A.; Montavon, G.; Samek, W.; Kloft, M.; Dietterich, T. G.; and Müller, K.-R. 2021.  A Unifying Review of Deep and Shallow Anomaly Detection.  _Proceedings of the IEEE_ , 109(5): 756–795.  ArXiv:2009.11732 [cs, stat].
  * Sakaguchi et al. (2021) Sakaguchi, K.; Bras, R. L.; Bhagavatula, C.; and Choi, Y. 2021.  Winogrande: An adversarial winograd schema challenge at scale.  _Communications of the ACM_ , 64(9): 99–106.
  * Spearman (1987) Spearman, C. 1987.  The proof and measurement of association between two things.  _The American journal of psychology_ , 100(3/4): 441–471.
  * Sridhar et al. (2023) Sridhar, K.; Dutta, S.; Weimer, J.; and Lee, I. 2023.  Guaranteed Conformance of Neurosymbolic Models to Natural Constraints.  In _L4DC 2023_.  ArXiv:2212.01346 [cs].
  * Sung (1996) Sung, K.-K. 1996.  Learning and example selection for object and pattern detection.
  * Szegedy et al. (2013) Szegedy, C.; Zaremba, W.; Sutskever, I.; Bruna, J.; Erhan, D.; Goodfellow, I.; and Fergus, R. 2013.  Intriguing properties of neural networks.  _arXiv preprint arXiv:1312.6199_.
  * Vorobeychik and Kantarcioglu (2018) Vorobeychik, Y.; and Kantarcioglu, M. 2018.  _Adversarial Machine Learning_.  Cham, Switzerland: Morgan & Claypool Publishers.  ISBN 978-1-68173-395-1.
  * Xiao et al. (2019) Xiao, C.; Li, B.; Zhu, J.-Y.; He, W.; Liu, M.; and Song, D. 2019.  Generating Adversarial Examples with Adversarial Networks.  arXiv:1801.02610.
  * Xiao et al. (2018) Xiao, C.; Zhu, J.-Y.; Li, B.; He, W.; Liu, M.; and Song, D. 2018.  Spatially transformed adversarial examples.  _arXiv preprint arXiv:1801.02612_.
  * Zellers et al. (2019) Zellers, R.; Holtzman, A.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019.  Hellaswag: Can a machine really finish your sentence?  _arXiv preprint arXiv:1905.07830_.

##  Appendix A Appendix

##  Appendix B Case Study Summary

The case studies used in the evaluation of our approach for curating
adversarially ordered natural datasets are summarized in Table 2.

Example | Train | Valid | Test | Num. LFs
---|---|---|---|---
HR Low | 79 | - | - | 62
HR High | 1315 | - | - | 62
RR Low | 312 | - | - | 62
RR High | 574 | - | - | 62
SpO2 Low | 3265 | - | - | 62
Crossmodal | 2630 | 376 | 378 | 18
Crowdsourcing | 187 | 50 | 50 | 103
Recsys | 796956 | 8339 | 42191 | 5
Spam | 1586 | - | 250 | 9
Table 2: Summary of our nine case studies.

##  Appendix C Implementation

In this section, we provide details on the implementation of our approach for
curating adversarially ordered natural datasets. The implementation of
probabilistic labeling via majority vote is straightforward. For probabilistic
labeling via a generative model, we use a tool called Snorkel to train a
generative model. Snorkel is the state-of-the-art tool for weak label
combination and has been applied to several applications. We use the latest
version at the time of writing, version 0.9.7 (www.snorkel.org).

For our confidence intervals, we use the confidence interval for a binomial
proportion implementation, namely the “proportion_confint” function, from the
statsmodels Python library, version 0.14.0 (www.statsmodels.org). We specify
the hyperparameters such that the Clopper-Pearson interval based on the Beta
distribution with a 5% significance level is used. Specifically,
hyperparameters “alpha” and “method” are set to 0.05 and “beta”, respectively.

Generative models do not generalize to unseen samples, so we combine the
train, validation, and test splits (without labels) as training data for the
generative model. In order to evaluate the accuracy of the noisy datasets
produced by our approach, we select the samples to be included in our datasets
from the available labeled data (i.e., train for the alarm suppression
examples and the validation and test splits combined for all other examples).

##  Appendix D Additional Results

Figure 2 shows the accuracy of the adversarially ordered natural datasets
produced by our approach and comparative approaches across our nine case
studies. For each case study, we curate ten datasets (i.e., $N=10$) and then
plot their accuracy scores. We also quantify the uncertainty in these
accuracies by placing a 90% binomial confidence interval around them. For each
dataset $D_{i}=(X_{i},Y_{i},\hat{Y}_{i})$ where $i\in\\{1,\ldots,N\\}$, the
width of the confidence interval is computed as,

$1.64\cdot\sqrt{\frac{\textsc{acc}(Y_{i},\hat{Y}_{i})\cdot\left(1-\textsc{acc}(Y_{i},\hat{Y}_{i})\right)}{\left|Y_{i}\right|}}$

where

$\textsc{acc}(Y,\hat{Y})=\frac{1}{|Y|}\sum_{i=1}^{|Y|}\mathbf{1}\left(y_{i}=\hat{y}_{i}\right)$

and $|\cdot|$ denotes the cardinality of the given set.

Figure 2: Accuracy of datasets produced by our approach and comparative
approaches. Recsys and Spam do not have dependent labeling functions hence the
lines for PL with all LFs and PL with Indep. LFs, and CI LB with all LFs and
our approach overlap.