# Heuristic Stopping Rules For Technology-Assisted Review

Eugene Yang  IR Lab, Georgetown UniversityWashingtonDCUSA
<EMAIL_ADDRESS>,  David D. Lewis  Reveal DataChicagoILUSA
<EMAIL_ADDRESS>and  Ophir Frieder  IR Lab, Georgetown
UniversityWashingtonDCUSA<EMAIL_ADDRESS>
(2021)

###### Abstract.

Technology-assisted review (TAR) refers to human-in-the-loop active learning
workflows for finding relevant documents in large collections. These workflows
often must meet a target for the proportion of relevant documents found (i.e.
recall) while also holding down costs. A variety of heuristic stopping rules
have been suggested for striking this tradeoff in particular settings, but
none have been tested against a range of recall targets and tasks. We propose
two new heuristic stopping rules, Quant and QuantCI based on model-based
estimation techniques from survey research. We compare them against a range of
proposed heuristics and find they are accurate at hitting a range of recall
targets while substantially reducing review costs.

††journalyear: 2021††copyright: acmlicensed††conference: ACM Symposium on
Document Engineering 2021; August 24–27, 2021; Limerick, Ireland††booktitle:
ACM Symposium on Document Engineering 2021 (DocEng ’21), August 24–27, 2021,
Limerick, Ireland††price: 15.00††doi: 10.1145/3469096.3469873††isbn:
978-1-4503-8596-1/21/08††ccs: Theory of computation Active learning††ccs:
Applied computing Law††ccs: Information systems Users and interactive
retrieval

##  1\. Introduction

Technology-assisted review (TAR) refers to human-in-the-loop iterative active
learning workflows for large scale document review. A major application area
is eDiscovery: review of documents in the law (civil litigation, regulatory
review, and investigations) (Baron et al., 2016; Roegiest and Cormack, 2015;
Grossman et al., 2016). Other applications include open government document
requests (Baron et al., 2020) and systematic review in medicine (Wallace et
al., 2010; Kanoulas et al., 2019, 2018, 2017).

On each iteration a TAR workflow uses a predictive model to select a batch of
documents to review (typically a few hundred) using an active learning method
such as relevance feedback (Rocchio, 1971; Lewis and Gale, 1994; Cormack and
Grossman, 2014). Those documents are reviewed by a human expert and added to
the collection of reviewed documents. The review documents are used to train
another predictive model, and the cycle repeats.

In a one-phase TAR workflow this process is iterated until some stopping
condition is met. Relevance feedback (training on top-ranked documents) is
often used as the active learning method. In a two-phase TAR workflow, the
process is stopped before completion and the final trained classifier is used
to identify a large set of documents to go to another review team (often at a
lower per-document cost) to finish the review (Yang et al., 2021).

TAR workflows typically must meet a target for the proportion of relevant
documents found (i.e. recall) while also holding down costs. The choice of
stopping point is therefore critical. Two-phase workflows often condition
stopping on a sample-based estimate of recall, with the cost of random
sampling offset by the cost savings from using a cheaper second phase review
team. One-phase reviews, on the other hand, are often used in situations, e.g.
a single investigator searching for key information, where the cost of
labeling a sample purely to support a stopping rule is untenable.

A variety of heuristic stopping rules have therefore been proposed for
stopping one-phase reviews (Li and Kanoulas, 2020; Callaghan and Müller-
Hansen, 2020; Cormack and Grossman, 2016a, b; Roegiest and Cormack, 2015;
Grossman et al., 2016). These have mostly been developed in the context of a
single TAR application or recall goal and their general applicability is
unknown.

In this paper we propose the theory of model-based estimation in survey
research (Särndal et al., 1978) provides a foundation for more flexible and
rigorous stopping rules in technology-assisted review. Our contributions are
(1) two new heuristic stopping rules, Quant and QuantCI, that adjust for task
characteristics and let stopping be conditioned on an arbitrary user-chosen
recall goal, and (2) a rigorous evaluation of our proposed rules against
heuristic stopping rules from both the academic literature and current
industry practice. We examine how the recall goal, prevalence of relevant
documents, and the difficult of the classification problem combine to affect
stopping rule behavior. We find our proposed rules are accurate at hitting a
range of recall targets while substantially reducing review costs.

##  2\. Background

We can classify stopping rules for TAR workflows into three groups: sample-
based, interventional, and heuristic. Our focus in this paper is on one-phase
reviews, so we emphasize stopping rules for those.

Sample-based stopping rules are based on drawing a random sample from a
collection of interest, having it reviewed, and using that labeled sample to
estimate the recall of review process at each potential stopping point.
Sample-based stopping rules have been proposed for both one-phase (Lewis et
al., 2021; Cormack and Grossman, 2016a; Saha et al., 2015) and two-phase
(Bagdouri et al., 2013; Webber et al., 2013) reviews. When estimated recall is
high enough, the review is stopped or, in the case of a two-phase review, a
cutoff is set on the second phase.

Sample-based stopping rules are the method of choice when a statistical
guarantee on effectiveness is needed, as in some legal contexts.
Unfortunately, the size of a sample necessary to estimate recall with a given
confidence is inversely proportional to the prevalence of relevance documents
(Lewis, 2016). For example, estimating recall with a margin of error of 0.05
and a confidence level of 95% requires a simple random sample containing
roughly 385 relevant documents. If the prevalence of relevant documents was
1%, one could need on average a sample size of 38,500 documents. This makes
sample-based rules prohibitively expensive in many settings.

Interventional stopping rules instead take the approach of modifying the TAR
process itself in support of estimating recall. One approach is to modify the
active learning algorithm to incorporate probabilistic sampling (Cormack and
Grossman, 2016b; Li and Kanoulas, 2020). This lets the selected data do triple
duty: training the selection model, estimating recall, and accomplishing
review. Another approach is to terminate conventional active learning at some
point and switch to purely random review (Callaghan and Müller-Hansen, 2020),
allowing recall to be estimated. While powerful, these methods are not a free
lunch: they reduce the effectiveness of active learning in exchange for
supporting estimation. They also cannot be used, for instance, by review
managers who are limited to the active learning methods provided by a
particular piece of commercial review software.

Heuristic stopping rules are the most generally applicable, and are the focus
of this paper. They make a decision of when to stop a TAR workflow using only
the data naturally labeled during the workflow. We can distinguish between
target-aware heuristic rules which modify their behavior based on a desired
recall target, and target-agnostic heuristic rules that attempt to optimize
some general condition on the quality of the review. We survey a wide range of
particular heuristic rules in the next section.

##  3\. Heuristic Stopping Rules

In this section we review a range of heuristic stopping rules from both the
research literature and operational practice. While our treatment is not
exhaustive, we cover the major styles of rules that have been proposed.

###  3.1. Fixed Iterations Method

Both one-phase and two-phase TAR processes are sometimes stopped simply after
a predetermined number of training iterations as a “simple, operational
stopping criteria” described by Wallace et al. (2010). Other works also
referred this rule as a pragmatic stopping rule (Callaghan and Müller-Hansen,
2020). This has the advantage of making review cost known in advance (assuming
uniform document review costs), but provides no ability to adapt to the
category prevalence, difficulty, or recall target.

###  3.2. The “2399” Method

Some early work in document review for eDiscovery used stopping rules based on
a sample-based estimate of elusion (the proportion of relevant documents in
the unreviewed population) (Roitblat, 2007). A sample of 2399 documents is
sufficient to estimate elusion with 95% confidence and a 2% margin of error.
This led to an odd fixation on the number 2399 in eDiscovery.

One result was the the “2399” heuristic method (Cormack and Grossman, 2015b,
2016b). It stops a one-phase TAR review after $2399+xR$ documents have been
reviewed, where $R$ is the number of positive documents the review has found,
and $x$ is a tuned hyperparameter. Our experiments use $x=1.2$ as proposed by
the rule’s inventors (Cormack and Grossman, 2015b, 2016b).

###  3.3. Batch Positives Method

A more adaptive approach to one-phase TAR reviews is to stop iterations when
one or more recent training batches contain only a few positive documents,
i.e., low precision. This can be given an economic rationale. If the fraction
of relevant documents in a batch is $x$, then the number of documents that
must be examined to find a single relevant document is roughly $1/x$. This is
a measure of the marginal cost to find the next relevant document. Such
marginal utility arguments can be linked to the legal concept of
proportionality in the discovery process (Hirt, 2011). Further, under the
assumption that batch precision declines in a roughly monotonic fashion after
some point in training, once the marginal cost exceeds a threshold, it never
declines, so stopping is appropriate.

###  3.4. Probability Cutoff Method

Closely related to the batch precision rule are rules that stop review when
all unreviewed documents have a score below some cutoff. In particular, assume
that the scores are in fact well-calibrated predicted probabilities. Then the
reciprocal $1/p$ of a probability score $p$ is the number of documents that
must be examined at that point to find a single relevant document. As with
batch precision, this is a measure of the marginal cost of finding the next
relevant document. However, also as with batch precision, there is no
particular connection with recall goals.

###  3.5. Knee Method

A refinement of the Batch Positive method is the Knee Method (Cormack and
Grossman, 2016a). This is based on the gain curve (Roegiest and Cormack, 2015)
for a one-phase TAR process. A gain curve plots how the number of positive
documents found increases as more documents are reviewed, typically on a per-
batch basis.

At each potential stopping point $s$, the knee method computes the ratio of
slopes in a two-segment approximation to the gain curve:

(1) $\displaystyle\rho(s)$ $\displaystyle=$
$\displaystyle\frac{Slope((0,0),(i,Rel(i)))}{Slope((i,Rel(i)),(s,Rel(s)+1)}$
$\displaystyle=$ $\displaystyle\frac{Rel(i)}{i}\frac{s-i}{Rel(s)-Rel(i)+1}$

Here $Rel(k)$ is the number of relevant documents found at rank k. The value
of $i$ is chosen such that $(i,Rel(i))$ is a “knee”, i.e., has maximum
perpendicular distance from the line segment between $(0,0)$ and $(s,Rel(s))$.

The Knee Method stops at the first $s$ such that

  * •

$\rho(s)\geq 156-min(Rel(s),150)$, and

  * •

$s\geq 1000$

The Knee Method is targeted at a recall goal of 0.7, and Cormack and Grossman
(2016a) do not discuss how it might be adapted to other recall targets.

####  3.5.1. Adapting the Knee Method to Fixed Size Batches

Cormack and Grossman (2015a) specify that values of $s$ for the Knee Method
should be based on the batch schedule used by their BMI (Baseline Model
Implementation) system111https://plg.uwaterloo.ca/~gvcormac/trecvm/. That
schedule uses a single relevant seed document (batch size $B=1$) on iteration
1222For supervised learning, 100 random documents artificially labeled as
negative examples are used on round 1 only, but these are not considered to be
part of the batch size.. Then the batch size $B_{k}$ for round $k$ is
$B_{k-1}+\lceil B_{k-1}/10\rceil$, i.e., batches grow exponentially in size.

Table 1. Transformed rounds for knee tests from exponential scheduling in BMI to fixed batch size of 200.  BMI Schedule | Batch Size 200 Schedule
---|---
| Batch | Training | Knee |  | Training | Knee
Round | Size | Set Size | Test? | Round | Set Size | Test?
|  |  |  | 5 | 801 | No
33 | 104 | 991 | No | 6 | 1001 | No
34 | 115 | 1106 | Yes | 7 | 1201 | Yes
35 | 127 | 1233 | Yes |  |  |
36 | 140 | 1373 | Yes | 8 | 1401 | Yes
37 | 154 | 1527 | Yes | 9 | 1601 | Yes
38 | 170 | 1697 | Yes | 10 | 1801 | Yes
39 | 187 | 1884 | Yes | 11 | 2001 | Yes
40 | 206 | 2090 | Yes | 12 | 2201 | Yes
41 | 227 | 2317 | Yes | 13 | 2401 | Yes
42 | 250 | 2567 | Yes | 14 | 2601 | Yes
|  |  |  | 15 | 2801 | No
43 | 275 | 2842 | Yes | 16 | 3001 | Yes

We can easily adapt the Knee Method, however, to fixed size batches. Since the
method is intended to be conservative, our goal is to never stop sooner than
the Knee Method would, but also to bound the extra cost we incur by using
fixed size batches. We assume, as in the BMI schedule, that the first round is
a seed set consisting of a single document, but then follow that with fixed
size batches.

We assume batches of size 200 as an example. Table 1 shows the BMI batch
schedule in the vicinity of the earliest potential stopping point: $s=1000$.
For batch size 200, the first fixed training set size over 1000 is 1001. Since
that is smaller than the first Knee Method batch, we conservatively wait until
training set size 1201 for the first check on the modified schedule. We then
check at training set sizes 1401, 1601, … 2601, each of which is subsequent to
at least one check that the BMI-based knee method would do. We do not check at
2801, because it would not be conservative to give an additional chance to
stop then. We do check at training set size 3001 (greater than 2842), and
subsequently at the first training set size larger than each training set size
for the BMI-based schedule.

###  3.6. Budget Method

Cormack and Grossman (2016a) also proposed the Target Method, which draws a
random sample of documents until 10 positive documents are found, hiding those
documents from the TAR process. It then stops a one-phase TAR review after the
10 positive documents are rediscovered (Cormack and Grossman, 2016a; Lewis et
al., 2021). As a sample-based method, it is out of the scope of this paper,
but Cormack and Grossman (2016a) have proposed a heuristic alternative to it:
the Budget Method.

The Budget Method is based on three observations:

  * •

For any amount of effort, a TAR process should find more positive documents
than would random sampling.

  * •

The target method on average will draw a target set of roughly $10C/R$
documents, where $C$ is the collection size and $R$ is the number of relevant
documents in the collection.

  * •

At any moment in time, the number of relevant documents $Rel(s)$ is a lower
bound on $R$, and thus $C/Rel(s)$ is an upper bound on the expected size of
the size of the sample the target method would draw.

The Budget Method stops review at the first end-of-batch training set size $s$
if

  * •

$s\geq 0.75C$, or

  * •

both $\rho(s)\geq 6$ and $s\geq 10C/Rel(s)$ are true

The first test is based on the assumption that reviewing a 75% random sample
of the collection would likely achieve a recall of at least 0.7, and reviewing
75% found by a TAR process should do even better. The second way of stopping
is based on two criteria: a simplified version of the knee test that is
assumed to be safe when $R$ is large, and a test that keeps the knee test from
taking effect until $Rel(s)$ is somewhat large. Again, it is not clear how to
adapt this method to recall goals differing 0.7.

###  3.7. CMH Heuristic Method

Recently, Callaghan and Müller-Hansen (2020) proposed an interventional method
that stops the TAR process based on a heuristic method, and then incrementally
samples random documents to both fulfill the recall target and statistically
verify the fulfillment. While the sampling phase is interventional (and thus
out of scope for this paper), the heuristic approach (CMH heuristic method) is
of interest.

At each round of active learning with end-of-batch training set size $s$, the
CMH heuristic method splits the successive rounds into two parts by a pivot
round $j$, hypothetically assumes the second part was sampled randomly, and
calculates the probability of the current recall being larger than or equal to
the target recall $p_{j}$. Specifically, let $s_{j}$ being the end-of-batch
training set size of any successive round $j$ and $\tau_{tar}$ being the
recall target,

$\displaystyle p_{j}=P\Big{(}X\leq Rel(s)-Rel(s_{j})\Big{)}$
$\displaystyle\text{where
}X\sim\text{Hypergeometric}(C-Rel(s_{j}),K_{tar},s-j)$
$\displaystyle\phantom{\text{where
}}K_{tar}=\Bigg{\lfloor}\frac{Rel(s)}{\tau_{tar}}-Rel(s_{j})+1\Bigg{\rfloor}$

The TAR process is stopped if $min(p_{j})<1-\alpha$ for some confidence level
$\alpha$. In our experiment, we follow the original proposal of the method and
set $\alpha$ to 95%.

##  4\. Stopping Based on a Recall Estimate

The heuristic stopping rules above except the CMH heuristic method either
ignore the review’s recall target, or are designed for a particular target
(e.g. 0.7 for the knee rule). A practical heuristic stopping rule should allow
an arbitrary recall target to be chosen and respond accordingly.

Stopping based on an estimate of recall is a natural approach. Labeling a
random sample purely to estimate recall is often viewed as too expensive
(Section 2). However, if recall could be estimated, even roughly, from the
training data already chosen by relevance feedback this would provide a
broadly applicable rule. We present such an approach below based on the
statistical technique of model-based estimation.

###  4.1. Model-Based Estimation

Using randomness properties of a sample to justify extrapolation to a
population (design-based estimation) (Särndal et al., 1978) is not the only
approach to estimation. An alternative, widely used in survey research, is
model-based estimation (Särndal et al., 1978). In this approach, (1) the
sampled items are used to fit a predictive model in a supervised fashion, (2)
that model is applied to unsampled items to predict a value for each, and (3)
the predicted values for unsampled items are used in producing the population
estimate. In information retrieval, similar approaches have been explored for
text quantification tasks (Lewis, 1995; Thomas et al., 1995; Gao and
Sebastiani, 2016).

In our context, we already have an appropriate predictive model: the logistic
regression model (Section 5.2) trained on relevance feedback samples to
prioritize documents for review. It outputs, for any document $i$, a predicted
probability of relevance $p_{i}$, i.e. an estimate of a 0/1 relevance label.
If $\mathcal{R}$ and $\mathcal{U}$ are the set of reviewed and unreviewed
documents respectively, then

(2) $\displaystyle\widehat{R}_{r}$ $\displaystyle=$
$\displaystyle\sum_{i\in\mathcal{R}}p_{i}$ (3) $\displaystyle\widehat{U}_{r}$
$\displaystyle=$ $\displaystyle\sum_{i\in\mathcal{U}}p_{i}$

are, respectively, the model-based estimate of the number of relevant
documents in the reviewed $\mathcal{R}$ and unreviewed documents
$\mathcal{U}$. Two plausible point estimates of recall are then:

(4) $\displaystyle\frac{R_{r}}{R_{r}+\widehat{U}_{r}}$ $\displaystyle=$
$\displaystyle\frac{R_{r}}{R_{r}+\sum_{i\in\mathcal{U}}p_{i}}$ (5)
$\displaystyle\frac{\widehat{R}_{r}}{\widehat{R}_{r}+\widehat{U}_{r}}$
$\displaystyle=$
$\displaystyle\frac{\sum_{i\in\mathcal{R}}p_{i}}{\sum_{i\in\mathcal{C}}p_{i}}$

Equation 5 may seem strange, since we know $R_{r}$ (the number of reviewed
relevant documents). However, recall is a ratio estimator, and there is an
advantage to having any systematic modeling error affecting $\widehat{U}_{r}$
to be present in both numerator and denominator. We in practice found Equation
5 to provide substantially better results, so discuss rules based only on
that.

This estimate is based on strong assumptions that rarely hold exactly. In this
case the assumption is that the probabilities from the relevance model are
well-calibrated, i.e., that if one took a large sample of documents with
predicted probability $p_{i}$, it would contain a fraction $p_{i}$ of relevant
documents (Dawid, 1982).

###  4.2. Approximating Variance

Stopping when a point estimate, even an unbiased one based on a random sample,
equalled the recall goal would miss that goal a substantial fraction of the
time. (50% of the time if sampling errors fell equally on either side of the
point estimate.) Stopping rules based on sample-based estimates address this
by instead stopping when the lower bound of a confidence interval exceeds the
recall goal (Lewis et al., 2021; Cormack and Grossman, 2016a; Bagdouri et al.,
2013). As larger samples are used, the confidence interval narrows and
stopping occurs sooner.

Producing a confidence interval requires an estimate of the variance of the
point estimate. A general method for approximating the variance of a
complicated random variable (in our case our recall estimator) is to use a
truncated Taylor series. This is variously referred to as the linearization
method, propagation of error method, delta method, and Taylor series method
(Valliant et al., 2013; Webber, 2013).

The estimate in Equation 5 is based on modeling the relevance of a document
$i$ as the outcome of a Bernoulli random variable $D_{i}\sim
Bernoulli(p_{i})$. Indeed, we can rewrite our point estimate in Equation 5 as
an approximation of the expected value of a random variable:

(6)
$\mathbb{E}\left[\frac{D_{\mathcal{R}}}{D_{\mathcal{C}}}\right]=\mathbb{E}\left[\frac{\sum_{i\in\mathcal{R}}D_{i}}{\sum_{i\in\mathcal{C}}D_{i}}\right]\approx\frac{\mathbb{E}\left[\sum_{i\in\mathcal{R}}D_{i}\right]}{\mathbb{E}\left[\sum_{i\in\mathcal{C}}D_{i}\right]}=\frac{\sum_{i\in\mathcal{R}}p_{i}}{\sum_{i\in\mathcal{C}}p_{i}}$

where $D_{\mathcal{R}}=\sum_{i\in\mathcal{R}}D_{i}$,
$D_{\mathcal{U}}=\sum_{i\in\mathcal{U}}D_{i}$, and
$D_{\mathcal{C}}=D_{\mathcal{R}}+D_{\mathcal{U}}$. (This expression also makes
clear that our point estimate uses the ratio of estimated values of random
variables to approximate the computationally awkward expected value of a ratio
of random variables.)

Approximating $D_{\mathcal{R}}/(D_{\mathcal{R}}+D_{\mathcal{U}})$ by a Taylor
series truncated to first order gives (Papanicolaou, 2009):

$\displaystyle\phantom{{}+{}}Var\left(\frac{D_{\mathcal{R}}}{D_{\mathcal{R}}+D_{\mathcal{U}}}\right)$
$\displaystyle=$ $\displaystyle\phantom{{}+{}}\mathbb{E}\left[\frac{\partial
f(D_{\mathcal{R}},D_{\mathcal{U}})}{\partial
D_{\mathcal{R}}}\right]^{2}Var(D_{\mathcal{R}})+\mathbb{E}\left[\frac{\partial
f(D_{\mathcal{R}},D_{\mathcal{U}})}{\partial
D_{\mathcal{U}}}\right]^{2}Var(D_{\mathcal{U}})$ (7)
$\displaystyle+2\mathbb{E}\left[\frac{\partial
f(D_{\mathcal{R}},D_{\mathcal{U}})}{\partial
D_{\mathcal{R}}}\right]\mathbb{E}\left[\frac{\partial
f(D_{\mathcal{R}},D_{\mathcal{U}})}{\partial
D_{\mathcal{U}}}\right]Cov(D_{\mathcal{R}},D_{\mathcal{U}})$

where

(8) $\displaystyle\frac{\partial f(D_{\mathcal{R}},D_{\mathcal{U}})}{\partial
D_{\mathcal{R}}}$
$\displaystyle=\frac{1}{D_{\mathcal{R}}+D_{\mathcal{U}}}-\frac{D_{\mathcal{R}}}{(D_{\mathcal{R}}+D_{\mathcal{U}})^{2}}$
(9) $\displaystyle\frac{\partial f(D_{\mathcal{R}},D_{\mathcal{U}})}{\partial
D_{\mathcal{U}}}$
$\displaystyle=\frac{-D_{\mathcal{R}}}{(D_{\mathcal{R}}+D_{\mathcal{U}})^{2}}$

Since the partial derivative of $D_{\mathcal{R}}$ (Equation 8) is always
positive and the partial derivative of $D_{\mathcal{U}}$ (Equation 9) is
always negative, the coefficient of the covariance
$Cov(D_{\mathcal{R}},D_{\mathcal{U}})$ is negative. By omitting the negative
terms, an upper bound on the variance is:

$\displaystyle
Var\left(\frac{D_{\mathcal{R}}}{D_{\mathcal{R}}+D_{\mathcal{U}}}\right)\leq$
$\displaystyle\phantom{{}+{}}\mathbb{E}\left[\frac{1}{(D_{\mathcal{R}}+D_{\mathcal{U}})^{2}}\right]Var(D_{\mathcal{R}})$
$\displaystyle+\mathbb{E}\left[\frac{D_{\mathcal{R}}^{2}}{(D_{\mathcal{R}}+D_{\mathcal{U}})^{4}}\right]\Big{(}Var(D_{\mathcal{R}})+Var(D_{\mathcal{U}})\Big{)}$

The logistic regression model is based on the assumption that the Bernoulli
random variables are independent. Making that assumption here as well, and
continuing to approximate the expected value of a ratio by a ratio of expected
values, we can approximate the right hand side by

$\displaystyle\frac{Var(D_{\mathcal{R}})}{(\widehat{R}_{r}+\widehat{U}_{r})^{2}}+\frac{\widehat{R}_{r}^{2}\left(Var(D_{\mathcal{R}})+Var(D_{\mathcal{U}})\right)}{(\widehat{R}_{r}+\widehat{U}_{r})^{4}}$

where

(10) $\displaystyle
Var(D_{\mathcal{S}})=Var\left(\sum_{i\in\mathcal{S}}D_{i}\right)=\sum_{i\in\mathcal{S}}Var(D_{i})=\sum_{i\in\mathcal{S}}p_{i}(1-p_{i})$

for $\mathcal{S}=\mathcal{R}$ and $\mathcal{U}$.

Finally, by assuming the recall estimator
$D_{\mathcal{R}}/(D_{\mathcal{R}}+D_{\mathcal{U}})$ is approximately normally
distributed, the 95% confidence interval of the recall is

(11) $\displaystyle\frac{\widehat{R}_{r}}{\widehat{R}_{r}+\widehat{U}_{r}}$
(12) $\displaystyle\pm$ $\displaystyle
2\sqrt{\frac{1}{(\widehat{R}_{r}+\widehat{U}_{r})^{2}}Var(D_{\mathcal{R}})+\frac{\widehat{R}_{r}^{2}}{(\widehat{R}_{r}+\widehat{U}_{r})^{4}}\Big{(}Var(D_{\mathcal{R}})+Var(D_{\mathcal{U}})\Big{)}}$

###  4.3. Stopping Rules Based on Recall Estimates

Given the above, we define two stopping rules based on the above approach of
quantifying the number of relevant documents using model-based estimation. The
Quant method stops the TAR process when the point estimate of recall (Equation
5) first reaches or exceeds the recall goal. The QuantCI method stops the TAR
process when the lower end of a 95% (2 standard deviations) confidence
interval first reaches or exceeds the recall goal. As the number of reviewed
relevant documents increases, QuantCI approaches the behavior of Quant.

##  5\. Experimental Setup

###  5.1. Dataset

We simulated recall-controlled retrieval tasks using RCV1-v2 (Lewis et al.,
2004), a widely-used text categorization collection consisting of 804,414 news
articles and 658 categories (including topics, industries, and geographical
regions).

To study tasks with a wide variety of characteristics, we selected the same 5
categories from each of 9 bins based on three ranges of class prevalence and
three ranges of difficulties from the previous studies (Yang et al., 2021).
Each selected category exhibits different characteristics, enabling the
simulation of a wide variety of tasks. The lowest and highest prevalence bins
were not used due to the wide range of values. Difficulty bins were based on
the effectiveness of a logistic regression model trained on a random 25% of
documents and evaluated by R-precision on the remaining 75%. For efficiency
purposes, we downsampled the collection to 20%.

###  5.2. Implementation

We implemented active learning based on libact (Yang et al., 2017), an open-
source, active-learning experimental framework. At round 0, one random
positive document is used as the seed document to instantiate the active
learning, and 200 documents based on relevance feedback (Rocchio, 1971;
Cormack and Grossman, 2014) are sampled and reviewed at each round after.

Supervised learning used the scikit-learn implementation of Logistic
Regression with an L2 regularization weight of 1.0. All words are used as
features and are represented by BM25-saturated term frequencies (Robertson,
2004; Yang et al., 2019).

###  5.3. Baselines

We compare Quant and QuantCI with multiple existing heuristics stopping
methods, which are all target-agnostic except the CMH heuristic method (CMH-
heuristic). These target-agnostic rules include, the Knee and Budget Methods
(Cormack and Grossman, 2016a), 2399-rule, BatchPos, MaxProb, and CorrCoef.

The Batch Positive Method (BatchPos) stops when the number of positive
documents in the batch is less than or equal to 20 (i.e., precision is less
than or equal 0.1) based on industrial convention. We tested stopping
immediately (patient=1) and waiting for three additional rounds (patient=4)
when the threshold is met.

The Probability Cutoff Method (MaxProb) stops when the maximum probability of
the unreviewed documents is less than or equal to 0.1. This threshold yields a
similar cost criterion as the BatchPos in which, in expectation, one positive
document costs nine negative documents to retrieve.

Apart from the rules discussed in Section 3, a score correlation-style
approach that stops based on the correlation coefficients between two
consecutive rounds (CorrCoef) is also tested. This approach stops when the
average correlation coefficient of the last three rounds is greater than or
equal to 0.99.

###  5.4. Evaluation

For each category, we run 10 active learning runs with different seed
documents, with a total of 450 runs. The trajectories of the runs are fixed
since the stopping methods tested do not intervene with the iterative process.

We evaluate the rules on the recall at stopping and the total cost at stopping
for the given recall target.

To quantitatively evaluate the reliability of each rule under different
targets, we report the mean square error (MSE) of the recall at stopping
instead of the reliability measure (proportion of the runs that fulfilled the
recall target at stopping) proposed by Cormack and Grossman (2016a). The
reliability measure favors stopping late since longer runs yield higher recall
and, thus, easier to fulfill the target, resulting in massive recall
overshooting for low recall targets. In an extreme case, a null stopping rule
that never stops the process would have perfect but uninformative reliability
for all recall targets. By reporting MSE, over and undershooting the target
are both penalized.

We assign an idealized cost penalty for undershoot the target recall (Lewis et
al., 2021). If the process stops before fulfilling the target, we assign a
cost penalty corresponding to the number of documents one needs to review
based on the ordering suggested by the predictive model at stopping before
reaching the target. This penalty is also called the optimal second phase
review (Yang et al., 2021). If overshooting the recall, the cost of reviewing
the excessive documents is also charged. For example, for a collection of
10,000 documents with 1,000 relevant documents and a recall target of 0.5,
suppose one has only retrieved 300 relevant documents at iteration 10
(reviewed a total of 2,001 documents). If the rank list provided by the final
predictive model requires going down to rank 2,300 among the 7,999 unreviewed
documents to retrieve the additional 200 requested relevant documents, the
idealized penalty is 2,300. And the total cost is therefore 4,301.

In the next section, we normalize the cost of each run by its optimal cost.
Since the cost varies by task characteristics and target, we report the ratio
of the cost using a specific stopping rule over the optimal cost to avoid
naturally high-cost categories diluting the values. In Table 3, the average of
such ratios over 450 runs is reported for each stopping rule.

##  6\. Results and Analysis

Both Quant and QuantCI demonstrate strong overall reliability on recall. In
Table 2, QuantCI has the lowest MSE under the low recall targets (0.1 to 0.5)
while the unadjusted version (Quant) remains strongly comparable. QuantCI is
also very competitive in high recall targets (e.g., 0.7 and 0.9), where the
Knee and Budget Methods are designed for. CMH-heuristic has an extremely small
loss under 0.9 recall target but one of the worst in the lower recalls,
indicating a vast overshooting. In the next section, we focus on the
distribution of recall at stopping points for each target-aware rule.

The budget method provides the most stable recall over the 450 runs among the
tested existing target-agnostic heuristic stopping rules in high recall
targets. As shown in Table 2, it presents the smallest loss on 0.7 and 0.9
recall targets. The knee method has a slightly larger loss than the budget
method. Both rules met their designed purpose for high-recall tasks but
overshot for lower recall targets, such as 0.3, as they are unaware of the
target.

Other methods exhibit large variations across the tasks and runs, making these
rules impractical for any recall target. The BatchPos-20-1 stops extremely
early, favoring low recall targets but still exhibiting a large loss. While
BatchPos-20-4 delays the stopping and reduces the recall loss under higher
recall targets, the loss is still large compared to other approaches. Despite
being target-agnostic, stopping rules that fail to provide a stable recall
cannot serve as the stopping rule even for a specific recall target. In the
rest of the section, we compare the target-aware methods with the Knee and
Budget Methods.

Table 2. Mean square error (MSE) of recall at stopping against different
recall targets. The last column reports the mean of all targets of each rule.

| 0.1 | 0.3 | 0.5 | 0.7 | 0.9 | Avg.
---|---|---|---|---|---|---
Knee | 0.522 | 0.279 | 0.115 | 0.032 | 0.029 | 0.195
Budget | 0.546 | 0.294 | 0.123 | 0.031 | 0.019 | 0.203
2399-Rule | 0.493 | 0.272 | 0.131 | 0.070 | 0.089 | 0.211
CorrCoef | 0.445 | 0.241 | 0.117 | 0.072 | 0.108 | 0.197
MaxProb-0.1 | 0.447 | 0.253 | 0.140 | 0.106 | 0.153 | 0.220
BatchPos-20-1 | 0.294 | 0.199 | 0.185 | 0.250 | 0.396 | 0.265
BatchPos-20-4 | 0.446 | 0.258 | 0.150 | 0.122 | 0.173 | 0.230
CMH-heuristics | 0.498 | 0.362 | 0.196 | 0.063 | 0.008 | 0.225
Quant | 0.236 | 0.164 | 0.102 | 0.072 | 0.050 | 0.124
QuantCI | 0.236 | 0.163 | 0.092 | 0.046 | 0.021 | 0.112

###  6.1. Recall at Stopping

Figure 1. Recall at stopping of recall target-aware stopping rules. The
numbers associated with the labels on the x-axis are the target recall. 50
runs (10 seeds and 5 categories) of each category bin are displayed using
boxplot conventions: the box ranges from the 25% (Q1) to 75% (Q3) quartiles of
recall with the 50 runs; the green lines and triangles are the median and mean
recall, respectively, and whiskers extend to the lesser of the most extreme
cost observed or to a distance 1.5(Q3 - Q1) from the edges of the box.
Outliers are presented as dots above and below the whiskers.

All four target-aware stopping rules adjust the stopping decision based on the
recall target and exhibit clear decreasing trends in Figure 1. Despite the
high classification effectiveness on the easy tasks (the right column of
Figure 1), it is difficult to estimate the number of relevant documents in
general. All four approaches overshoot all levels of recall targets in the
three easy category bins.

On the other hand, CMH-heuristic, on average, overshoots the target in all
nine bins, similar to what we observed in Table 2. Despite the awareness the
target, assuming parts of the active learning queried documents are random
creates the dependency to the task’s characteristics. For hard categories that
active learning often experiences difficulties to find relevant documents from
some iterations, those reviewing batches are similar to random, resulting in
better-tracked stopping recall. The assumption rarely holds from the actual
scenarios in other bins, producing larger overshooting in results.

Quant tracks the target recall closer than CMH-heuristic. Since Quant
estimates the number of unreviewed relevant document by the sum of the
predicted probabilities, the estimation is usually too big for easy categories
(a lot of documents seem positive) but too small for hard categories (most
documents seem negative), resulting in overshooting and undershooting the
recall, respectively. The medium categories (the middle column in Figure 1)
strike a balance between the two and, therefore, allow the Quant to more
accurately track the recall.

Finally, QuantCI further improves the confidence of stopping and alleviates
undershooting the target. For rare categories (the first row in Figure 1),
recall at stopping for the high recall targets increases while remaining
similar recall for the lower recall targets. This phenomenon indicates that
the standard deviation adjustment does not uniformly delay the stopping like
the patient parameter for BatchPos but delays according to the quality of the
estimation.

Despite the similarity between Quant and QuantCI in the recall at stopping,
the costs induced by each approach significantly differ. In the next section,
we discuss the total cost of applying each stopping rule.

###  6.2. Total Cost at Stopping

Table 3. Average ratio over the cost of stopping each category against the optimal cost for each recall target. Values in parentheses are the standard deviations. † and ∗ indicate the statistical difference with 95% confidence of QuantCI against the Knee Method and Quant, respectively. The significant test is conducted by paired t-test with Bonferroni correction.  | Target = 0.1 | Target = 0.3 | Target = 0.5 | Target = 0.7 | Target = 0.9
---|---|---|---|---|---
Knee | 21.52 (062.78) | 5.96 (10.07) | 3.43 (05.40) | 2.42 (03.13) |  †2.36 (01.70)
Budget | 56.45 (180.25) | 13.62 (43.61) | 5.43 (07.93) | 4.02 (06.70) | 3.35 (04.16)
2399-Rule | 22.52 (041.64) | 7.16 (09.56) | 4.35 (03.99) | 3.76 (04.30) | 3.42 (04.36)
CorrCoef | 16.64 (030.13) | 5.80 (07.06) | 3.94 (04.87) | 3.48 (06.06) | 3.64 (05.51)
MaxProb-0.1 | 11.29 (012.93) | 5.42 (04.12) | 4.93 (06.27) | 4.75 (07.61) | 3.56 (05.65)
BatchPos-20-1 | 12.44 (011.90) | 12.41 (13.94) | 13.90 (16.51) | 13.72 (18.48) | 10.34 (20.34)
BatchPos-20-4 | 12.89 (017.27) | 5.84 (04.39) | 5.36 (06.04) | 5.19 (07.40) | 3.57 (04.62)
CMH-heuristics | 27.37 (059.29) | 18.18 (30.47) | 16.18 (14.67) | 12.28 (13.21) | 17.33 (25.30)
Quant | 6.12 (009.45) | 3.47 (03.60) | 3.07 (03.71) | 2.94 (03.53) |  ∗2.81 (01.81)
QuantCI |  †6.01 (009.79) |  †∗3.01 (03.15) |  †∗2.45 (02.41) |  ∗2.38 (01.95) | 3.35 (03.57)

Among the target-agnostic rules we tested, the Knee Method provides the lowest
cost as shown in Table 3. The Budget Method, despite stops with stabler
recall, costs significantly higher than the Knee Method under all tested
recall targets (95% confidence with Bonferroni correction of 5 tests on paired
t-test). Even with 0.7 as the recall target, where the rules are designed for,
the Budget Method (in contrast to the implication from the name) costs 4.02
times more than the optimal cost compared to 2.42 times by the Knee Method.

For lower recall targets, such as 0.1, target-agnostic rules all induce large
cost overheads. Since these heuristics rules all rely on certain notions of
stability testing, the models are usually still unstable when the low recall
targets are reached, resulting in vast overshooting and excessive reviewing
cost.

Simple rules such as MaxProb, BatchPos, and 2399-Rule, while exposing higher
cost in high recall targets, yield cost lower than the Knee Method. Since the
stability tests of these simple approaches are weaker, they tend to stop the
process earlier than the more sophisticated ones, such as Knee and Budget
Methods.

The awareness of the target does not guarantee lower cost. CMH-heuristic
creates higher cost than BatchPos-20-4 (95% confidence with Bonferroni
corrections of 5 tests on paired t-test). Regardless of the decreasing trend,
CMH-heuristics still vastly overshoots the target. The average recall at
stopping CMH-heuristic rule is 0.76 with a 0.1 recall target, which is even
higher than 0.67 using BatchPos-20-4 for all targets.

On the other hand, both our proposed methods, Quant and QuantCI, tightly track
the target to provide a total cost that is closer to optimal.

For low (e.g., 0.1) to medium (e.g., 0.5) recall targets, QuantCI yields
significantly lower cost than any other tested rule except Quant with 0.1
recall target (95% confidence with Bonferroni corrections of 27 tests on
paired t-test). In Figure 1, the trend of recalls at stopping using Quant is
slightly concave or convex in some category bins, such as Rare-Hard and
Medium-Medium. Since the Quant method estimates the total number of relevant
documents based on the probabilities produced by the classifiers, the quality
of such estimations depends on the task’s difficulty. Therefore, standard
deviation adjustment provides additional protection to the stopping rule
against the instability that often appears during early iterations of active
learning. The resulting recall at stopping approaches linear in Figure 1. The
cost also reduces significantly.

However, for 0.1 recall target, the optimal stopping point (iteration just
achieved the target) often appears when the predicted probabilities are not
yet stabilized. For QauntCI, it conservatively waits until the predictions are
sufficiently confident (when the lower bound of confidence intervals are
larger than the target), usually implying overshooting the target. Therefore,
despite that the average cost ratio of QuantCI is lower than Quant under 0.1
recall target, the variance of QuantCI is larger, and the difference is not
significant.

For higher recall targets (e.g., 0.7 and 0.9), our proposed approaches perform
similarly with the Knee Method, the best existing heuristic rule for such
targets. Note that the Knee Method was designed for a 0.7 recall target. Our
proposed Quant and QuantCI produce very similar cost even in this crowded
battlefield.

When targeting 0.9 recall, the Knee Method offers a slightly lower cost ratio
than Quant and QuantCI. Again, since the Quant method relies on the underlying
classifier to estimate the total number of relevant documents, identifying the
last several relevant documents is challenging both ranking-wise (for
prioritizing the reviewing) and probabilistic-wise (for estimating its
existence). When the last several needles are still in the haystack, the
number of relevant documents in the unreviewed set is under-estimated,
resulting inflated estimated recall, and therefore, undershooting the real
recall. Furthermore, the variance of such recall estimation during those
unstable stages is undoubtedly high, driving QuantCI to stop the process
later; often, it is too late and overshoots the recall. This results in cost
overheads (3.35).

Both Quant and QuantCI outperform the widely-compared target-agnostic Knee
Method except when setting an extremely high recall target without any
intervention and additional random samples. The challenging high recall target
often requires a high reviewing cost even with optimal stopping, providing a
large incentive for using a sample-based stopping approach. The cost of the
random sample can be amortized to the longer active learning run. Next, we
investigate a specific active learning run closely.

###  6.3. A case study on stopping and cost

Figure 2. Cost and stopping round of each rule on category I24000 with seed
#7. The height of the blue and orange blocks indicate the cost and penalty if
stopping at the corresponding round. Vertical dashed lines indicate the
stopping round of the rules labeled at the right of the line. CMH-heuristic
stops beyond the range presented for all recall targets. Triangles and squares
represent the cost at stopping of random sample with 20 and 50 positive
documents, respectively.

In Figure 2, we demonstrate an active learning run on category I24000 (a Rare-
Medium category with prevalence 0.002133) with seed #7. Each target-agnostic
rule stops at the same iteration across the targets but appears at different
locations because of each graph’s different x-axis range. The optimal stopping
point is where the early stopping penalty first becomes zero.

Additional to the heuristic rules, we tested a sample-based rule that stops
when the recall estimated by the sample reaches the target, which is similar
to the Target Rule proposed by Cormack and Grossman (2016a). We evaluated both
samples with 20 and 50 relevant documents. By sampling uniformly at random
until enough relevant documents are selected, the expected sample sizes
(including the non-relevant ones) are 9,378 and 23,445, respectively. We
conducted 100 trials for each sampling size and reported the median stopping
iterations.

Despite the same increasing rate of the reviewing cost (blue blocks in Figure
2), stopping late for low recall targets with low optimal cost yields a higher
relative cost overhead. This observation also implies that it is cheaper to
stop later for a high recall target than a low one. The same observation can
be made on the left side of the optimal stopping point but with a convex
relationship instead of linear due to the nonlinearity of the learning curves.
If stopping vastly prematurely, e.g., BatchPos-20-1, the cost overhead is
excessive because of the under-trained model. For the 0.9 recall target, the
improvement of the model plateaued (the decrements of the orange block), so
stopping between iteration 25 to 70 yields a very similar total cost.

Besides leading to a lower cost, the overhead of stopping early and late
differs across recall targets as well. The shape of the penalty also differs
based on the characteristics of the task. Due to the space constraint, we
cannot demonstrate other categories. But the distinctions between the targets
already demonstrate the need to track the target.

Compared to the heuristic approaches, estimating the progress based on a
random sample is much more accurate than heuristics. In each tested recall
target except 0.9, the sampling approach either stops at the optima or off by
one iteration. However, this accuracy comes with a price on the sample, which
in total cost usually exceeds heuristic approaches. For the challenging 0.9
recall target, the sampling approaches are still much more accurate than the
heuristics but are off by more iterations. In this case, the cost of using
such sampling methods is similar or lower than the heuristic ones, providing
the incentive of reviewing the large random sample a priori.

However, routinely testing the same goal under the same random sample yields
sequential biases (Webber, 2013). The simple approach we tested here is no
exception. While not proposing this simple rule for a high recall target, this
observation motivates the usage of sample-based stopping rules. A valid
approach that avoids sequential biases is still an active research topic.

##  7\. Summary

In this work, we proposed a target-aware heuristic stopping rule Quant that
estimates the total number of relevant documents and the recall progress using
a model-based approach for one-phase technology-assisted review. We provided
the standard deviation of our recall estimator and further improved the
stopping rule based on the stability of the estimation. The Quant Method with
standard deviation adjustment QunatCI has the lowest MSE on recall at stopping
among the tested rules. It also yields the lowest total cost when targeting
low recall and competitive with the popular Knee method for high recall
targets.

## References

  * (1)
  * Bagdouri et al. (2013) Mossaab Bagdouri, William Webber, David D Lewis, and Douglas W Oard. 2013.  Towards minimizing the annotation cost of certified text classification. In _CIKM 2013_. ACM, 989–998.
  * Baron et al. (2016) J.R. Baron, R.C. Losey, and M.D. Berman. 2016.  _Perspectives on Predictive Coding: And Other Advanced Search Methods for the Legal Practitioner_.  American Bar Association, Section of Litigation.  https://books.google.com/books?id=TdJ2AQAACAAJ
  * Baron et al. (2020) Jason R Baron, Mahmoud F Sayed, and Douglas W Oard. 2020\.  Providing More Efficient Access To Government Records: A Use Case Involving Application of Machine Learning to Improve FOIA Review for the Deliberative Process Privilege.  _arXiv preprint arXiv:2011.07203_ (2020).
  * Callaghan and Müller-Hansen (2020) Max W Callaghan and Finn Müller-Hansen. 2020.  Statistical stopping criteria for automated screening in systematic reviews.  _Systematic Reviews_ 9, 1 (2020), 1–14.
  * Cormack and Grossman (2014) Gordon F. Cormack and Maura F. Grossman. 2014.  Evaluation of machine-learning protocols for technology-assisted review in electronic discovery.  _SIGIR 2014_ (2014), 153–162.  https://doi.org/10.1145/2600428.2609601.
  * Cormack and Grossman (2015a) Gordon V Cormack and Maura R Grossman. 2015a.  Autonomy and reliability of continuous active learning for technology-assisted review.  _arXiv preprint arXiv:1504.06868_ (2015).
  * Cormack and Grossman (2015b) Gordon V Cormack and Maura R Grossman. 2015b.  Waterloo (Cormack) Participation in the TREC 2015 Total Recall Track.. In _TREC_.
  * Cormack and Grossman (2016a) Gordon V. Cormack and Maura R. Grossman. 2016a.  Engineering Quality and Reliability in Technology-Assisted Review. In _SIGIR_. ACM Press, Pisa, Italy, 75–84.  https://doi.org/10.1145/2911451.2911510 00024\.
  * Cormack and Grossman (2016b) Gordon V Cormack and Maura R Grossman. 2016b.  Scalability of continuous active learning for reliable high-recall text classification. In _Proceedings of the 25th ACM international on conference on information and knowledge management_. 1039–1048.
  * Dawid (1982) A Philip Dawid. 1982\.  The well-calibrated Bayesian.  _J. Amer. Statist. Assoc._ 77, 379 (1982), 605–610.
  * Gao and Sebastiani (2016) Wei Gao and Fabrizio Sebastiani. 2016.  From classification to quantification in tweet sentiment analysis.  _Social Network Analysis and Mining_ 6, 1 (2016), 19\.
  * Grossman et al. (2016) Maura R. Grossman, Gordon V. Cormack, and Adam Roegiest. 2016\.  TREC 2016 Total Recall Track Overview.
  * Hirt (2011) Theodore C Hirt. 2011\.  Applying Proportionality Principles in Electronic Discovery-Lessons for Federal Agencies and Their Litigators.  _US Att’ys Bull._ 59 (2011), 43.
  * Kanoulas et al. (2017) Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2017\.  CLEF 2017 Technologically Assisted Reviews in Empirical Medicine Overview. In _CEUR workshop proceedings_ , Vol. 1866. 1–29.
  * Kanoulas et al. (2018) Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2018\.  CLEF 2018 Technologically Assisted Reviews in Empirical Medicine Overview.  _CEUR Workshop Proceedings_ 2125 (July 2018).  https://strathprints.strath.ac.uk/66446/
  * Kanoulas et al. (2019) Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2019\.  CLEF 2019 technology assisted reviews in empirical medicine overview. In _CEUR workshop proceedings_ , Vol. 2380.
  * Lewis (1995) David D Lewis. 1995\.  Evaluating and optimizing autonomous text classification systems. In _Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval_. 246–254.
  * Lewis (2016) David D Lewis. 2016\.  Defining and Estimating Effectiveness in Document Review.  In _Perspectives on Predictive Coding: And Other Advanced Search Methods for the Legal Practitioner_. American Bar Association, Section of Litigation.
  * Lewis and Gale (1994) David D. Lewis and William A Gale. 1994.  A sequential algorithm for training text classifiers. In _SIGIR 1994_. 3–12.
  * Lewis et al. (2021) David D Lewis, Eugene Yang, and Ophir Frieder. 2021\.  On Sample-Based Stopping Rules for Technology-Assisted Review (Under review).  (2021).
  * Lewis et al. (2004) David D. Lewis, Yiming Yang, Tony G. Rose, and Fan Li. 2004\.  RCV1: A New Benchmark Collection for Text Categorization Research.  _JMLR_ 5 (2004), 361–397.
  * Li and Kanoulas (2020) Dan Li and Evangelos Kanoulas. 2020.  When to Stop Reviewing in Technology-Assisted Reviews: Sampling from an Adaptive Distribution to Estimate Residual Relevant Documents.  _ACM Transactions on Information Systems (TOIS)_ 38, 4 (2020), 1–36.
  * Papanicolaou (2009) Alex Papanicolaou. 2009\.  Taylor approximation and the delta method.  _April_ 28 (2009), 2009.
  * Robertson (2004) Stephen Robertson. 2004\.  Understanding inverse document frequency: on theoretical arguments for IDF.  _JDoc_ 60, 5 (2004), 503–520.
  * Rocchio (1971) Joseph John Rocchio. 1971\.  Relevance feedback in information retrieval.  (1971).
  * Roegiest and Cormack (2015) Adam Roegiest and Gordon V. Cormack. 2015.  TREC 2015 Total Recall Track Overview.  (2015).
  * Roitblat (2007) Herbert L Roitblat. 2007\.  Search and information retrieval science. In _Sedona Conf. J._ , Vol. 8. HeinOnline, 225.
  * Saha et al. (2015) T. K. Saha, M. A. Hasan, C. Burgess, M. A. Habib, and J. Johnson. 2015.  Batch-mode active learning for technology-assisted review. In _2015 IEEE International Conference on Big Data (Big Data)_. 1134–1143.  00003\.
  * Särndal et al. (1978) Carl-Erik Särndal, Ib Thomsen, Jan M Hoem, DV Lindley, O Barndorff-Nielsen, and Tore Dalenius. 1978.  Design-based and model-based inference in survey sampling [with discussion and reply].  _Scandinavian Journal of Statistics_ (1978), 27–52.
  * Thomas et al. (1995) T Thomas, C Scovel, C Kruger, and J Shumate. 1995\.  Text to information: Sampling uncertainty in an example from physician/patient encounters. In _Proceedings of the Fourth Annual Symposium on Document Analysis and Information Retrieval”, Information Science Research Institute, University of Las Vegas_. 347–58.
  * Valliant et al. (2013) Richard Valliant, Jill A Dever, and Frauke Kreuter. 2013\.  _Practical tools for designing and weighting survey samples_.  Springer.
  * Wallace et al. (2010) Byron C Wallace, Thomas A Trikalinos, Joseph Lau, Carla Brodley, and Christopher H Schmid. 2010\.  Semi-automated screening of biomedical citations for systematic reviews.  _BMC bioinformatics_ 11, 1 (2010), 55.
  * Webber (2013) William Webber. 2013\.  Approximate recall confidence intervals.  _ACM Transactions on Information Systems (TOIS)_ 31, 1 (2013), 1–33.
  * Webber et al. (2013) William Webber, Mossaab Bagdouri, David D Lewis, and Douglas W Oard. 2013.  Sequential testing in classifier evaluation yields biased estimates of effectiveness. In _Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval_. 933–936.
  * Yang et al. (2019) Eugene Yang, David D Lewis, and Ophir Frieder. 2019\.  Text Retrieval Priors for Bayesian Logistic Regression. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1045–1048.
  * Yang et al. (2021) Eugene Yang, David D Lewis, and Ophir Frieder. 2021\.  On Minimizing Cost in Legal Document Review Workflows (Under review).  (2021).
  * Yang et al. (2017) Yao-Yuan Yang, Shao-Chuan Lee, Yu-An Chung, Tung-En Wu, Si-An Chen, and Hsuan-Tien Lin. 2017\.  _libact: Pool-based Active Learning in Python_.  Technical Report. National Taiwan University.  https://github.com/ntucllab/libact