# Bayesian ranking and selection with applications to field studies, economic
mobility, and forecasting

Dillon Bowen Wharton School of Business, University of Pennsylvania,
<EMAIL_ADDRESS>
Decision-making often involves ranking and selection. For example, to assemble
a team of political forecasters, we might begin by narrowing our choice set to
the candidates we are confident rank among the top 10% in forecasting ability.
Unfortunately, we do not know each candidate’s true ability but observe a
noisy estimate of it. This paper develops new Bayesian algorithms to rank and
select candidates based on noisy estimates. Using simulations based on
empirical data, we show that our algorithms often outperform frequentist
ranking and selection algorithms. Our Bayesian ranking algorithms yield
shorter rank confidence intervals while maintaining approximately correct
coverage. Our Bayesian selection algorithms select more candidates while
maintaining correct error rates. We apply our ranking and selection procedures
to field experiments, economic mobility, forecasting, and similar problems.
Finally, we implement our ranking and selection techniques in a user-friendly
Python package (Bowen, 2022) documented here: https://dsbowen-conditional-
inference.readthedocs.io/en/latest/111Code available here:
https://gitlab.com/dsbowen/selection.

In 2011, the United States Intelligence Community launched a tournament “to
dramatically enhance the accuracy, precision, and timeliness of intelligence
forecasts for a broad range of event types, through the development of
advanced techniques that elicit, weight, and combine the judgments of many
intelligence analysts” (Matheny and Rieber, 2015). The eventual winner of this
tournament was a team called the Good Judgment Project (Tetlock and Gardner,
2016; Mellers et al., 2014). In its first year, the Good Judgment Project
gathered forecasts from thousands of participants about geopolitical events
ranging from a possible civil war in Syria to Palestinian membership in the
United Nations. The Intelligence Community scored the teams based on
forecasting accuracy. If they correctly predicted the winner of the Nicaraguan
presidential election, for example, they would receive a good score.

The Good Judgment Project set out to assemble a team of “superforecasters” who
ranked in the top 2% in forecasting ability for the second year of the
competition. Of course, the Good Judgment Project did not know each
participant’s true forecasting ability. Instead, it had to select forecasters
using noisy estimates of forecasting ability based on each participant’s
performance in the tournament’s first year.

Ranking and selection problems, like those faced by the Good Judgment Project,
are ubiquitous in decision-making. Jack Welch famously fired the bottom 10% of
his workforce each year when he ran General Electric. The Organization for
Economic Cooperation and Development regularly ranks countries, for example,
by math, science, and reading scores so countries can model their education
systems after those that rank near the top (Schleicher, 2019). Economists rank
teachers by the value they add to their students’ education (Chetty et al.,
2014a, b; Hanushek, 2011) and United States neighborhoods by the economic
opportunity they afford their children (Chetty and Hendren, 2018a, b). Social
scientists run large-scale field experiments to compare many treatments or
interventions simultaneously and select the most effective for further
research and implementation (Milkman et al., 2021a).

Researchers and practitioners often perform ranking and selection using point
estimates alone. For example, the Good Judgment Project ranked forecasters
according to point estimates of their forecasting accuracy and selected those
whose point estimates fell in the top 2%. Unfortunately, this procedure has
high error rates (Gu and Koenker, 2020). Suppose that all the Good Judgment
Project’s forecasters were about equally talented. By chance, some would
appear much better than others. In a world where the variance in estimated
forecasting ability is almost entirely due to chance, the Good Judgment
Project’s false discovery rate would be $1-.02=98\%$. That is, 98% of their
“superforecasters” would not rank among the top 2% in forecasting ability.

We want statistical procedures that account for noise when ranking and
selecting candidates to address this issue. When ranking candidates, we want
rank confidence intervals (RCIs) with correct coverage. For example, an RCI
might suggest that a particular forecaster is between the 10th- and 50th-best
with 80% confidence. RCIs have correct marginal coverage if each covers its
individual candidate’s true rank with the desired confidence. RCIs have
correct simultaneous coverage if all simultaneously cover their candidate’s
true rank with the desired confidence. Recent research proposed frequentist
procedures for obtaining RCIs with correct marginal and simultaneous coverage
(Mogstad et al., 2020).

When selecting candidates, we want to select sets of candidates with correct
false discovery or family-wise error rates. For example, we might select 40
forecasters that rank in the top 10% in forecasting ability with a false
discovery rate of 5%. The selection set has a correct false discovery rate if
fewer than $0.05*40=2$ selected forecasters are not genuinely in the top 10%.
Controlling the false discovery rate allows us to claim that 95% of the
forecasters on our team are in the top 10% in forecasting ability.

Alternatively, we might want to control the family-wise error rate. For
example, we might choose 20 forecasters that rank in the top 10% with a
family-wise error rate of 5%. The selection set has a correct family-wise
error rate if there is at least a $1-.05=95\%$ chance that all 20 are
genuinely in the top 10%. Controlling the family-wise error rate allows us to
claim that we are 95% confident that all of the forecasters on our team are in
the top 10% in forecasting ability.

Given a false discovery or family-wise error rate, we usually want to select
the largest possible set of candidates. For example, General Electric probably
wants to fire as many under-performing employees as possible.

Recent research proved that a Bayesian selection procedure based on posterior
tail probabilities has correct false discovery rates (Gu and Koenker, 2020).
However, this procedure does not control family-wise error rates. Researchers
have also proposed using RCIs to perform selection (Mogstad et al., 2022). For
example, if we want to select forecasters in the top 50 with 80% confidence,
we can select those whose 80% RCI lies entirely between one and 50. Selection
based on marginal RCIs (i.e., rank confidence intervals with correct marginal
coverage) results in correct false discovery rates. Similarly, selection based
on simultaneous RCIs results in correct family-wise error rates. Therefore, we
can use Mogstad et al. (2020)’s frequentist procedures for marginal and
simultaneous RCIs to select candidates with correct false discovery and
family-wise error rates, respectively (Mogstad et al., 2022).

This paper develops new Bayesian algorithms that improve upon existing ranking
and selection methods. First, we develop new Bayesian algorithms to produce
marginal and simultaneous RCIs. Using simulations based on nine datasets, we
show that our Bayesian procedures yield substantially shorter RCIs than their
frequentist counterparts with approximately correct marginal and simultaneous
coverage.

Because we can use RCIs for selection, our Bayesian RCIs can select candidates
with correct false discovery and family-wise error rates. Additionally,
because our Bayesian RCIs are shorter than frequentist RCIs, Bayesian RCIs
select more candidates than frequentist RCIs. Still, it is possible to do
better. Indeed, our most significant theoretical contribution is a “direct”
Bayesian selection algorithm which selects more candidates than simultaneous
RCIs while maintaining correct family-wise error rates.

Our most significant practical contribution is a user-friendly statistics
package implementing our Bayesian ranking and selection algorithms (Bowen,
2022) documented here: https://dsbowen-conditional-
inference.readthedocs.io/en/latest/. We also provide template notebooks and
preconfigured virtual environments. These allow practitioners to upload a
`csv` file of their conventional estimates (e.g., ordinary least squares or
instrumental variables estimates) and click “run” to analyze their data
without downloading any software or writing a single line of code.

The rest of our paper is structured as follows. Section 1 describes
frequentist and Bayesian ranking procedures and analyzes their performance
using simulations and empirical illustrations. We find that Bayesian RCIs have
approximately correct coverage and are substantially shorter than their
frequentist counterparts. Section 2 introduces our direct Bayesian selection
algorithm. It then analyzes various selection procedures’ performance using
simulations and empirical illustrations. We find that Bayesian selection
procedures - especially our direct Bayesian selection algorithm - have correct
error rates but select many more candidates than frequentist procedures.
Finally, section 3 concludes.

##  1 Rank confidence intervals

Suppose we have $K$ candidates with values $\mu_{1},...,\mu_{K}$. For example,
$\mu_{k}$ might be forecaster $k$’s true forecasting ability. We do not
observe each candidate’s true forecasting ability but instead a noisy estimate
of it $Y_{k}$. For example, $Y_{k}$ might by forecaster $k$’s average Brier
score from the tournament’s first year. In many applications, the estimates
come from a joint normal distribution $Y\sim\mathcal{N}(\mu,\Sigma)$,
motivated by the central limit theorem. We assume the covariance matrix
$\Sigma$ is known, although, in practice, we usually replace it with a
consistent estimate. Define the rank of candidate $k$ as

$r_{k}\coloneqq 1+\sum_{j=1}^{K}\mathbf{1}\\{\mu_{k}<\mu_{j}\\}.$

A set of rank confidence intervals $R=\\{R_{1},...,R_{K}\\}$ has correct
marginal coverage at level $\alpha$ if each rank confidence interval covers
its candidate’s true rank with probability $1-\alpha$,

$Pr\\{r_{k}\in R_{k}\\}\geq 1-\alpha\quad\forall k\in\\{1,...,K\\}.$

RCIs with correct marginal coverage are marginal rank confidence intervals
(MRCIs).

We say that a set of rank confidence intervals has correct simultaneous
coverage at level $\alpha$ if all the rank confidence intervals simultaneous
cover their candidate’s true rank with probability $1-\alpha$,

$Pr\big{\\{}r_{k}\in R_{k}\quad\forall k\in\\{1,...,K\\}\big{\\}}\geq
1-\alpha.$

RCIs with correct simultaneous coverage are simultaneous rank confidence
intervals (SRCIs).

###  1.1 Frequentist and Bayesian construction

Mogstad et al. (2020) develops an impressive frequentist procedure for
constructing RCIs. To construct an MRCI for candidate $k$, we form
simultaneous confidence intervals for the pairwise differences
$\mu_{k}-\mu_{1},...,\mu_{k}-\mu_{K}$ and then count how many intervals lie
entirely above or below zero. Suppose there is one candidate $j$ for whom the
95% confidence interval around pairwise difference $\mu_{k}-\mu_{j}$ lies
entirely below zero and $K-5$ candidates for whom the confidence interval
around the pairwise difference lies entirely above zero. Then, candidate $k$
ranks between second and fifth with 95% probability.

Constructing SRCIs follows a similar procedure. We begin by forming
simultaneous confidence intervals around all pairwise differences
$\mu_{j}-\mu_{k}\quad\forall j,k$. Then, as for the MRCIs, we count how many
confidence intervals around the pairwise difference between candidate $k$ and
another candidate $\mu_{k}-\mu_{1},...,\mu_{k}-\mu_{K}$ lie entirely above or
below zero.

We now introduce new Bayesian algorithms for RCIs. Instead of treating the
values $\mu$ as fixed, we treat them as a random draw from a prior
distribution $\mu\sim\mathcal{G}$. We can use Bayesian estimators to obtain a
joint posterior distribution $\mu|Y\sim\mathcal{F}$ (James and Stein, 1992;
Stein, 1956; Cai et al., 2021; Brown and Greenshtein, 2009). Given the
posterior, the marginal coverage probability for a RCI $R_{k}$ is

$Pr\\{r_{k}\in
R_{k}|Y\\}=\int_{\mu\in\mathbb{R}^{K}}\mathbf{1}\big{\\{}1+\sum_{j=1}^{K}\mathbf{1}\\{\mu_{k}<\mu_{j}\\}\in
R_{k}\big{\\}}f(\mu|Y)d\mu,$

where $f$ is the joint posterior density. In practice, we estimate coverage
probabilities by sampling from the joint posterior distribution.

Suppose we want our RCIs to be convex. A naive algorithm might compute an MRCI
$R_{k}$ by searching over all possible convex RCIs and selecting the shortest
with correct coverage. This naive algorithm takes quadratic time for each
candidate $k$. Therefore, constructing MRCIs for all candidates using the
naive algorithm takes cubic time. Algorithm 1 presents a more efficient method
for constructing MRCIs222The results we present later further restrict RCIs to
include each candidate’s estimated rank. For example, if a particular
forecaster came in 50th place during the first year, we restrict their RCI to
include 50. This modification does not affect the algorithm’s time
complexity.. This algorithm will find the same RCI as the naive algorithm but
in linear time for each candidate (so, quadratic time for all candidates).

Algorithm 1 Bayesian marginal rank confidence intervals

Given a marginal coverage probability $\alpha$

$l^{*}_{k}\leftarrow 1$

$u^{*}_{k}\leftarrow K$

$l_{k}\leftarrow K$

$u_{k}\leftarrow K$

Estimate $Pr\\{r_{k}=j|Y\\}\quad\forall j\in\\{1,...,K\\}$ using joint
posterior samples

$P\leftarrow Pr\\{r_{k}=K\\}$

while $l_{k}>0$ do

while $P<1-\alpha$ do

if $l_{k}=1$ then

break

end if

$l_{k}\leftarrow l_{k}-1$

$P\leftarrow P+Pr\\{r_{k}=l_{k}|Y\\}$

end while

if $u_{k}-l_{k}<u^{*}_{k}-l^{*}_{k}$ then

$l^{*}_{k}\leftarrow l_{k}$

$u^{*}_{k}\leftarrow u_{k}$

end if

$u_{k}\leftarrow u_{k}-1$

$P\leftarrow P-Pr\\{r_{k}=u_{k}|Y\\}$

end while

Result. Marginal rank confidence interval $R_{k}=[l^{*}_{k},u^{*}_{k}]$

Similarly, the simultaneous coverage probability for a set of rank confidence
intervals $\\{R_{1},...,R_{K}\\}$ is

$\begin{split}Pr\big{\\{}r_{k}\in R_{k}\quad\forall
k\in\\{1,...,K\\}|Y\big{\\}}\\\
=\int_{\mu\in\mathbb{R}^{K}}\mathbf{1}\big{\\{}1+\sum_{j=1}^{K}\mathbf{1}\\{\mu_{k}<\mu_{j}\\}\in
R_{k}\quad\forall k\in\\{1,...,K\\}\big{\\}}f(\mu|Y)d\mu.\end{split}$

A naive algorithm might construct SRCIs by searching over all possible sets of
convex RCIs and selecting those that are shortest on average with correct
simultaneous coverage. However, this is an exponential time algorithm.
Algorithm 2 presents a quadratic time algorithm for SRCIs, although its RCIs
will be longer than the naive algorithm’s in expectation.

Algorithm 2 Bayesian simultaneous rank confidence intervals

Given a simultaneous coverage probability $\alpha$

$R\leftarrow\\{R_{1},...,R_{K}\\}=\\{[l_{1},u_{1}],...,[l_{K},u_{K}]\\}$ using
algorithm 1

$P\leftarrow Pr\big{\\{}r_{k}\in R_{k}\quad\forall
k\in\\{1,...,K\\}|Y\big{\\}}$

while $P<1-\alpha$ do

$m\leftarrow\arg\max_{k}Pr\\{r_{k}=l_{k}-1|Y\\}$

$n\leftarrow\arg\max_{k}Pr\\{r_{k}=u_{k}+1|Y\\}$

if $Pr\\{r_{m}=l_{m}-1|Y\\}>Pr\\{r_{n}=u_{n}+1|Y\\}$ then

$P\leftarrow Pr\\{r_{k}\in R_{k}\quad\forall k\neq m,r_{m}\in R_{m}\text{ or
}r_{m}=l_{m}-1|Y\\}$

$R_{m}\leftarrow[l_{m}-1,u_{m}]$

else

$P\leftarrow Pr\\{r_{k}\in R_{k}\quad\forall k\neq n,r_{n}\in R_{n}\text{ or
}r_{n}=u_{n}+1|Y\\}$

$R_{n}\leftarrow[l_{n},u_{n}+1]$

end if

end while

Result. Simultaneous rank confidence intervals $\\{R_{1},...,R_{K}\\}$

###  1.2 Simulations

Next, we compare Mogstad et al. (2020)’s frequentist procedures with our
Bayesian procedures for constructing RCIs using simulations based on nine
datasets. Because our simulation procedure is identical for all datasets, we
begin by describing our simulation procedure in detail for a single dataset,
then briefly describe our remaining datasets.

Superforecasters. This dataset contains estimates of forecasting accuracy from
the first year of the Good Judgment Project (Good Judgment Project, 2016).
Over 1,000 forecasters predicted more than 100 geopolitical events with
discrete or categorical outcomes (e.g., “[From 0-100%, how likely is it that
the] Spanish government generic 10-year bond yields equal or exceed 7% at any
point before 1 September 2012?”). After filtering out forecasters who
responded to fewer than half the forecasting questions, our dataset included
842 forecasters. The Good Judgment Project scored each forecast using Brier
scores (Tetlock and Gardner, 2016). We can estimate each forecaster’s ability
by averaging their Brier scores across the forecasting questions to which they
responded333Technically, Brier scores measure inaccuracy, so we use negative
Brier scores to rank candidates from most to least accurate.. So, for each
forecaster, we have a point estimate $Y_{k}$ and standard error $\sigma_{k}$.
By the central limit theorem, the point estimates are approximately normally
distributed $Y\sim\mathcal{N}(\mu,\Sigma)$ for
$\Sigma_{j,k}=\mathbf{1}\\{j=k\\}\sigma_{k}^{2}$. We aim to rank the
forecasters based on their forecasting accuracy.

For each of 1,000 simulations, we began by randomly selecting 200 forecasters
to rank. (Running 1,000 simulations using the full dataset of 842 forecasters
is computationally prohibitive.) We treated the point estimates $Y$ as the
ground truth. Therefore, the ground truth rankings are
$r_{k}=1+\sum_{j=1}^{K}\mathbf{1}\\{Y_{k}<Y_{j}\\}$. We then sampled “new”
estimates $Y^{\prime}\sim\mathcal{N}(Y,\Sigma)$. Using the new estimates
$Y^{\prime}$ and covariance matrix $\Sigma$, we constructed MRCIs and SRCIs
using the frequentist and Bayesian procedures for error rates
$\alpha=0.05,0.1,0.2$. For MRCIs, $1-\alpha$ is the desired marginal coverage
probability. For SRCIs, $1-\alpha$ is the desired simultaneous coverage
probability. Finally, we compared the RCIs in marginal and simultaneous
coverage and average length.

Movers. The Moving to Opportunity (or “Movers”) dataset estimates the causal
effect on children’s outcomes in adulthood of moving to a new commuting zone
(Chetty et al., 2016; Bergman et al., 2019). We aim to rank 50 United States
commuting zones based on how they affect the future earnings of children who
grow up in them.

Opportunity Altas. The Opportunity Atlas is similar to the Movers dataset. It
measures children’s outcomes in adulthood based on the commuting zone in which
they grew up (Chetty et al., 2018). The primary difference between the
Opportunity Altas and Movers datasets is that the Opportunity Altas dataset is
correlational, whereas the Movers dataset is causal. We aim to rank the 100
largest United States commuting zones based on the future earnings of children
who grow up in them.

PISA. In 2016, the Organization for Economic Cooperation and Development
conducted the Programme for International Student Assessment (PISA)
(Schleicher, 2019). PISA measured 15-year-olds’ reading, mathematics, and
science skills in 37 countries. We aim to rank countries based on their
students’ mathematics scores.

24-Hour Fitness. Behavioral science researchers partnered with 24-Hour Fitness
to test the effects of 53 behavioral nudges encouraging 60,000 customers to
exercise more (Milkman et al., 2021a). The treatments involved planning,
reminders, microincentives, and other interventions. The researchers defined
the treatment effects as the increase in weekly gym visits during a four-week
intervention period compared to a control condition. We aim to rank the
treatments based on their effects on weekly gym visits.

Flu study. Behavioral science researchers partnered with Penn Medicine and
Geisinger Health to test the effects of 19 text messages encouraging 50,000
patients to get a flu vaccine (Milkman et al., 2021b). The treatments involved
texting patients various reminders. The researchers measured the treatments’
effectiveness as the increase in vaccination rates compared to the control
group. We aim to rank treatments based on their effects on vaccination rates.

State representatives. Economists called state representatives’ offices 40
times in all 50 United States to estimate the probability of reaching a live
representative by state (Dube et al., 2021). We aim to rank states based on
the probability of reaching a live representative.

Diversity in academia. Business school researchers sent emails to 6,500
professors at top U.S. universities from fictional students interested in
applying to a doctoral program (Milkman et al., 2015). The fictional students’
names signalled their race (White, Black, Hispanic, Chinese, or Indian) and
gender. In their emails, the fictional students asked the professors for a
meeting. The researchers estimated how often professors agreed to meet
students in each demographic group (i.e., groups based on race and gender). We
aim to rank demographic groups based on the probability that professors agreed
to meet them.

Discriminatory sentencing. Criminology researchers studied sentencing patterns
among over 800 federal judges (Smith et al., 2021). They measured how much
longer each judge’s sentences were for Black defendants than White defendants,
controlling for type of crime committed, criminal record, and other
observables. For each simulation, we randomly selected 200 judges to analyze.
(Using the entire dataset of 800 judges is computationally prohibitive.) We
aim to rank judges based on their level of sentencing discrimination.

Our procedures for constructing RCIs will work for any Bayesian estimator.
Because parametric Bayesian estimators outperform nonparametric Bayesian
estimators for datasets with few candidates (Berger, 2013), we use a
parametric estimator (normal-prior, normal-likelihood fit using maximum
likelihood estimation) for datasets with fewer than 30 candidates. Otherwise,
we use a nonparametric estimator (Dirac-delta prior fit using expectation
maximization) (Cai et al., 2021).

Figure 1 shows that all RCIs (frequentist and Bayesian, marginal and
simultaneous) have correct marginal coverage on average across all datasets.
Additional analyses (not shown) confirm that all RCIs have correct marginal
coverage for each dataset individually.

Figure 1: Marginal coverage from 1,000 simulations averaged across all
datasets. The coverage is correct if it is above the dashed line.

The left panel of Figure 2 shows the simultaneous coverage for the frequentist
and Bayesian SRCIs on average across all datasets. Unfortunately, the Bayesian
SRCIs have incorrect simultaneous coverage. For example, the 95% Bayesian
SRCIs have only 91% simultaneous coverage. This is driven primarily by
incorrect simultaneous coverage for the Movers, State representatives, and
Discriminatory sentencing datasets, for which the 95% Bayesian SRCIs have only
87%, 85%, and 76% simultaneous coverage, respectively. (The 95% Bayesian SRCIs
have at least 93% simultaneous coverage for each other dataset individually).

Figure 2: Simultaneous and approximate simultaneous coverage from 1,000
simulations averaged across all datasets. The coverage is correct if it is
above the dashed line.

At first, these results appear to reflect poorly on the Bayesian SRCIs.
However, the results become more promising if we slightly relax the
simultaneous coverage requirement. RCIs have simultaneous coverage if they
contain every candidate’s true rank. We say that RCIs have approximate
simultaneous coverage if they contain the true rank of at least 98% of the
candidates. That is, we are willing to tolerate coverage errors for at most
one of 50 commuting zones in the Movers dataset, one of 50 states in the State
representatives dataset, and four of 200 judges in the Discriminatory
sentencing dataset.

The right panel of Figure 2 shows that Bayesian SRCIs have correct approximate
simultaneous coverage on average across all datasets. Additional analyses (not
shown) confirm that Bayesian SRCIs have correct approximate simultaneous
coverage for each dataset individually. For example, the 95% Bayesian SRCIs
for the Movers, State representatives, and Discriminatory sentencing datasets
have 99%, 98%, and 100% approximate simultaneous coverage, respectively.

Additionally, it is worth asking what the marginal coverage is when the
simultaneous coverage is incorrect. For example, the 95% Bayesian SRCIs have
only 76% simultaneous coverage for the Discriminatory sentencing dataset. What
is the average marginal coverage across the 1-0.76=24% of simulations where
the RCIs do not cover all 200 judges? Figure 3 plots the marginal coverage for
the simulations in which the 95% Bayesian SRCIs do not cover every candidate’s
rank. In this case, the 95% Bayesian SRCIs for the Movers, State
representatives, and Discriminatory sentencing datasets have 98%, 98%, and
99.5% marginal coverage, respectively. That is, when the 95% Bayesian SRCIs do
not cover every candidate’s rank, they cover nearly 49 out of 50 commuting
zones in the Mover’s dataset, 49 out of 50 states in the State representatives
dataset, and 199 out of 200 judges in the Discriminatory sentencing dataset on
average.

Figure 3: Marginal coverage in simulations where Bayesian SRCIs do not cover
the ranks of all candidates.

In sum, Bayesian SRCIs do not have correct simultaneous coverage. However,
they have correct approximate simultaneous coverage, meaning that the
$1-\alpha$ Bayesian SRCIs cover at least 98% of the candidates’ true ranks
with probability $1-\alpha$. Additionally, when they do not achieve (exact)
simultaneous coverage, Bayesian SRCIs have high marginal coverage and likely
fail to cover only a single candidate.

Having established that Bayesian RCIs have approximately correct coverage, we
now ask how long the RCIs are. Figure 4 plots the average length of the RCIs
as a proportion of the candidates in each dataset. For example, if the RCIs
were 50 ranks long on average in a dataset of 100 candidates, the average
length as a proportion of the candidates is $50/100=0.5$.

Figure 4: Average rank confidence interval length.

Bayesian RCIs are substantially shorter than their frequentist counterparts.
Bayesian SRCIs are consistently 20% shorter than frequentist SRCIs, and
Bayesian MRCIs are up to 50% shorter than frequentist MRCIs on average across
all datasets. Additional analyses (not shown) confirm that Bayesian RCIs are
shorter than their frequentist counterparts for every error rate $\alpha$ in
every dataset by similar margins.

###  1.3 Empirical illustration

In our simulations, we used the conventional estimates as the ground truth and
sampled new estimates from this distribution. We found that Bayesian RCIs are
consistently shorter than their frequentist counterparts. Here, we estimate
RCIs using the conventional estimates themselves. Figure 5 plots the average
length of the RCIs across all candidates and datasets. Consistent with our
simulation results, the Bayesian RCIs are shorter than their frequentist
counterparts.

Figure 5: Length of RCIs applied to conventional estimates.

Additionally, Figure 6 shows the 80% RCIs for the best-performing, worst-
performing, and median candidates in three datasets (the Good Judgment
Project, 24-Hour Fitness, and Opportunity Atlas datasets). These results
suggest substantial uncertainty for the Good Judgment Project and 24-Hour
Fitness datasets. For example, according to Bayesian SRCIs, we cannot be even
80% confident that the best-performing forecaster in the Good Judgment Project
was among the top 200 of 842 forecasters in the tournament. Additionally, the
median forecaster (who ranked 421st according to conventional estimates) could
be one of the best or worst forecasters. However, it is easier to identify the
worst forecasters, as shown by the short Bayesian SRCI for the forecaster who
performed worst according to conventional estimates.

Similarly, many of the Bayesian SRCIs for the 24-Hour Fitness experiment span
nearly the entire 53 behavioral treatments. By contrast, the RCIs for the
Opportunity Atlas are relatively narrow, suggesting that we can confidently
rank commuting zones by their children’s future income.

Figure 6: 80% RCIs for the Good Judgment Project, 24-Hour Fitness experiment,
and Opportunity Atlas. Note that frequentist and Bayesian models may disagree
about the point estimate of a candidate’s rank, which is why the blue and
orange (frequentist) dots do not exactly align with the green and red
(Bayesian) dots.

##  2 Selection

Selection is closely related to ranking. We often want to select a set of
candidates that we are confident rank near the top. Formally, suppose we want
to select the top $r^{*}$ candidates. Let $S\coloneqq\\{k:r_{k}\leq r^{*}\\}$
be the set of truly best candidates. Because we do not know each candidate’s
true rank, the set of candidates we select is a random set $\hat{S}$.

We say that our selection set has a correct marginal false discovery rate
(mFDR) at level $\alpha$ if the percentage of selected candidates ranked below
$r^{*}$ is less than $\alpha$,

$mFDR=\frac{\mathbf{E}[|\hat{S}\setminus
S|]}{\mathbf{E}[|\hat{S}|]}\leq\alpha.$

As the number of candidates increases, the marginal false discovery rate
approaches the usual false discovery rate (Genovese and Wasserman, 2002),

$FDR=\mathbf{E}\left[\frac{|\hat{S}\setminus S|}{|\hat{S}|}\right].$

We say that our selection set has a correct family-wise error rate at level
$\alpha$ if all the selected candidates rank above $r^{*}$ with probability
$1-\alpha$,

$FWER=Pr\\{\hat{S}\setminus S=\emptyset\\}\geq 1-\alpha.$

Controlling the false discovery or family-wise error rate, we usually prefer
to identify as many top candidates as possible. For example, the Good Judgment
Project probably prefers to identify as many superforecasters as possible.
That is, we want to maximize $|\hat{S}|$ without inflating the false discovery
or family-wise error rate.

###  2.1 Frequentist and Bayesian construction

We can easily use ranking procedures to perform selection. For example, if we
want to select forecasters in the top $r^{*}$ with 80% confidence, we can
simply select those whose 80% RCI lies entirely between one and $r^{*}$,

$\hat{S}=\\{k:R_{k}\subseteq[1,r^{*}]\\}.$

Asserting that candidate $k$ ranks in the top $r^{*}$ is a directional claim,
so one-tailed RCIs are more appropriate than two-tailed RCIs for selection.
Selection based on MRCIs leads to correct false discovery rates, whereas
selection based on SRCIs leads to correct family-wise error rates (Mogstad et
al., 2022). Because Bayesian RCIs are shorter than frequentist RCIs, we expect
that Bayesian RCIs will select more candidates than frequentist RCIs.

However, direct selection algorithms may outperform selection based on RCIs.
Mogstad et al. (2020) propose a direct frequentist selection algorithm that
controls the family-wise error rate. Their direct selection algorithm selects
more candidates than frequentist SRCIs. Unfortunately, their algorithm takes
exponential time and is infeasible in many practical applications.

In the same spirit, we consider direct Bayesian selection algorithms. We start
by expressing the false discovery rate of a selection set $\hat{S}$ given the
posterior density. The false discovery rate is,

$\mathbf{E}\left[\frac{|\hat{S}\setminus
S|}{|\hat{S}|}\Big{|}Y\right]=\frac{1}{|\hat{S}|}\sum_{k\in\hat{S}}\int_{\mu_{k}\in\mathbb{R}}\mathbf{1}\big{\\{}1+\sum_{j=1}^{K}\mathbf{1}\\{\mu_{k}<\mu_{j}\\}>r^{*}\big{\\}}f(\mu_{k}|Y)d\mu_{k}.$

As we did for Bayesian RCIs, we approximate the false discovery rate by
sampling from the joint posterior distribution.

A seminal paper on Bayesian selection provides a selection procedure
guaranteed to control the marginal false discovery rate based on posterior
tail probabilities (Gu and Koenker, 2020). Similarly, Algorithm 3 is a simple
selection procedure based on samples from the joint posterior distribution. We
call this the Bayesian FDR direct method because it is a direct Bayesian
selection method designed to control the false discovery rate. This algorithm
begins with an empty set. According to Bayesian estimates, we then select
candidates one at a time in order of how likely they are to rank above
$r^{*}$. We stop when we have reached a given false discovery rate, as
estimated by the Bayesian posterior.

Algorithm 3 Bayesian selection to control the false discovery rate

Given marginal posterior probabilities that candidate $k$ is one of the
$r^{*}$-best $Pr\\{r_{k}\leq r^{*}|Y\\}$

Given a desired false discovery rate $\alpha$

$\hat{S}\leftarrow\emptyset$

$\Omega\leftarrow\\{1,...,K\\}$

$FDR\leftarrow 0$

while $FDR<\alpha$ do

$j\leftarrow\arg\max_{k\in\Omega}Pr\\{r_{k}\leq r^{*}|Y\\}$

$FDR\leftarrow\frac{|\hat{S}|*FDR+1-Pr\\{r_{j}\leq r^{*}|Y\\}}{|\hat{S}|+1}$

if $FDR\geq\alpha$ then

break

end if

$\hat{S}\leftarrow\hat{S}\cup\\{j\\}$

$\Omega\leftarrow\Omega\setminus\\{j\\}$

end while

Result. Selected set of candidates $\hat{S}$

Similarly, the family-wise error rate of a selection set is,

$Pr\\{|\hat{S}\setminus
S|>0|Y\\}=1-\int_{\mu\in\mathbb{R}^{K}}\mathbf{1}\big{\\{}1+\sum_{j=1}^{K}\mathbf{1}\\{\mu_{k}<\mu_{j}\\}\leq
r^{*}\quad\forall k\in\hat{S}\big{\\}}f(\mu|Y)d\mu.$

Gu and Koenker (2020)’s Bayesian selection procedure was not designed to have
correct family-wise error rates. To address this issue, we designed Algorithm
4 to control the family-wise error rate when performing Bayesian selection. We
call this the Bayesian FWER direct method because we expect it to control the
family-wise error rate. This algorithm begins with the set of all candidates.
We then reject candidates one at a time in order of how likely they are to
rank below $r^{*}$ according to Bayesian estimates (more precisely, we reject
the candidate that minimally increases the estimated family-wise error rate).
We stop when we have $1-\alpha$ confidence that we have rejected all those who
rank below $r^{*}$, as estimated by the Bayesian posterior. Therefore, we can
have $1-\alpha$ confidence that all the remaining candidates rank above
$r^{*}$.

Algorithm 4 Bayesian selection to control the family-wise error rate

Given a desired family-wise error rate $\alpha$

$\hat{S}\leftarrow\\{1,...,K\\}$

$\Omega\leftarrow\emptyset$

$FWER\leftarrow 1$

while $FWER>\alpha$ do

$j\leftarrow\arg\max_{k\in\hat{S}}Pr\\{r_{k}>r^{*}|Y,\sum_{i\in\Omega}\mathbf{1}\\{r_{i}>r^{*}\\}<K-r^{*}\\}$

$\hat{S}\leftarrow\hat{S}\setminus\\{j\\}$

$\Omega\leftarrow\Omega\cup\\{j\\}$

$FWER\leftarrow 1-Pr\\{\sum_{k\in\Omega}\mathbf{1}\\{r_{k}>r^{*}\\}\geq
K-r^{*}|Y\\}$

end while

Result. Selected set of candidates $\hat{S}$

###  2.2 Simulations

So far, we have considered six selection algorithms. We have four based on
RCIs (marginal versus simultaneous and frequentist versus Bayesian) and two
direct Bayesian approaches. We now compare these algorithms using simulations
with the same datasets and sampling procedure as Section 1.2. After sampling
“new” estimates $Y^{\prime}$, we apply our selection algorithms to select
candidates at various quantiles $q$ (i.e., the top 10%, 20%, 80%, or 90%) with
error rates $\alpha=0.05,0.1,0.2$. Here, $\alpha$ is the desired false
discovery or family-wise error rate.

Figure 7 shows the average marginal false discovery rate across datasets. As
expected, every selection procedure has correct false discovery rates.
Additional analyses (not shown) suggest that every procedure also has
approximately correct false discovery rates for each quantile and dataset
444For some parameter combinations, there is too much noise to accurately
estimate the false discovery rate. For example, the estimated false discovery
rate using frequentist RCIs for the Diversity in academia dataset when
selecting the top 10% of candidates with a desired false discovery rate of 5%
is 33%. This is because it selected only six candidates in all 1,000
simulations, and two happened to be false discoveries. Clearly, frequentist
RCIs did not select enough candidates across the 1,000 simulations to
accurately estimate the false discovery rate..

Figure 8 shows the average family-wise error rate across datasets for the
three procedures that control the family-wise error rate (the frequentist and
Bayesian SRCI methods and the Bayesian FWER direct method). As expected, all
of these procedures have correct family-wise error rates on average.
Additional results (not shown) confirm that these procedures also have
approximately correct family-wise error rates for each quantile and dataset.

Figure 9 shows the average proportion of candidates selected by each procedure
across all datasets. Among the procedures that control the false discovery
rate (the frequentist and Bayesian MRCI methods and the Bayesian FDR direct
method), the Bayesian FDR direct method selects the most candidates. Notably,
both Bayesian methods select many more candidates than the frequentist method.
Among the procedures that control the family-wise error rate, the Bayesian
FWER direct method selects the most candidates by a wide margin. Additional
results (not shown) confirm that these patterns hold for each dataset.

Figure 7: Average marginal false discovery rates across datasets for 1,000
simulations. For comparability, we label the RCI methods according to the
error rate they control for. For example, selection based on frequentist SRCIs
is labeled frequentist FWER CI to highlight that selection based on SRCIs
controls the family-wise error rate (FWER). The false discovery rate is
correct if it is below the dashed line. Figure 8: Average family-wise error
rates across datasets for 1,000 simulations. The family-wise error rate is
correct if it is below the dashed line. Figure 9: Average proportion of
candidates selected across datasets for 1,000 simulations.

###  2.3 Empirical illustration

In our simulations, we used the conventional estimates as the ground truth and
sampled new estimates from this distribution. We found that Bayesian selection
procedures, especially the direct procedures, selected more candidates than
their frequentist counterparts. Here, we select candidates based on the
conventional estimates themselves. Figure 10 plots the average proportion of
selected candidates across all datasets. Consistent with our simulation
results, the Bayesian procedures select more candidates than their frequentist
counterparts. Additionally, the direct Bayesian selection procedures select
more candidates than Bayesian selection based on RCIs.

Figure 10: Average proportion of candidates selected across datasets.

For a more concrete illustration, Figure 11 plots the results of these
selection procedures for three datasets (the Good Judgment Project, 24-Hour
Fitness, and Opportunity Altas datasets) when selecting the top 10% of
candidates with a 20% error rate. For example, suppose we want to assemble a
forecasting team in which 80% of the team members are among the most accurate
10% of forecasters. Using Bayesian selection and controlling the false
discovery rate, we can select 2% of forecasters from the Good Judgment Project
as being among the most accurate 10%. Alternatively, suppose we want to
assemble a forecasting team which we are 80% confident contains only
forecasters among the most accurate 10%. Using the Bayesian FWER direct
method, we can select less than 1% of forecasters. While it might seem like
Bayesian procedures select very few forecasters, note that frequentist
procedures select none.

Figure 11: Proportion of candidates selected as being among the top 10% with a
20% error rate.

Similarly, suppose we want to select the most effective behavioral
interventions in the 24-Hour Fitness experiment for implementation or follow-
up studies. Unfortunately, there is so much noise in the 24-Hour Fitness
experiment that we cannot say which treatments are most effective when
controlling the error rate at 20%. All procedures select zero treatments as
being among the most effective 10%.

By contrast, there is much less noise in the Opportunity Atlas dataset. Using
the Bayesian FDR direct method with a false discovery rate at 20%, we select
12% of commuting zones as being in the top 10%. It may seem undesirable to say
that 12% of commuting zones are among the top 10%. However, if the selected
commuting zones are a superset of the true top 10%, then the false discovery
rate would be correct; $(0.12-0.10)/0.12=0.17<0.2$. Alternatively, using the
Bayesian FWER direct method with a family-wise error rate of 20%, we select 9%
of commuting zones as being in the top 10%. Note that this is a substantial
improvement over the Bayesian SRCI method (which selects 6%) and the
frequentist SRCI method (which selects 5%).

In sum, our selection procedures show similar results when applied directly to
conventional estimates as they did in our simulations. Bayesian procedures
select more candidates than their frequentist counterparts, with direct
Bayesian selection performing exceptionally well.

##  3 Conclusion

Decision-making often involves ranking and selection. Unfortunately, we rarely
know candidates’ true ranks and must perform ranking and selection based on
noisy estimates. This paper developed new Bayesian algorithms for ranking and
selection and showed that they outperform frequentist analogues using
simulations and empirical illustrations. Specifically, Bayesian RCIs are
shorter than frequentist RCIs and make only minimal coverage sacrifices, and
Bayesian selection algorithms select more candidates than frequentist
algorithms while controlling the error rates. We highlighted the practical
applicability of our ranking and selection methods for field experiments,
economic mobility analysis, and forecasting.

## References

  * Bowen [2022] Dillon Bowen.  Multiple inference: A python package for comparing multiple parameters.  _Journal of Open Source Software_ , 7(75):4492, 2022.  doi: 10.21105/joss.04492.  URL https://doi.org/10.21105/joss.04492.
  * Matheny and Rieber [2015] Jason Matheny and Steve Rieber.  Aggregative contingent estimation (ace).  _Intelligence Advanced Research Projects Agency_ , 2015.
  * Tetlock and Gardner [2016] Philip E Tetlock and Dan Gardner.  _Superforecasting: The art and science of prediction_.  Random House, 2016.
  * Mellers et al. [2014] Barbara Mellers, Lyle Ungar, Jonathan Baron, Jaime Ramos, Burcu Gurcay, Katrina Fincher, Sydney E Scott, Don Moore, Pavel Atanasov, Samuel A Swift, et al.  Psychological strategies for winning a geopolitical forecasting tournament.  _Psychological science_ , 25(5):1106–1115, 2014\.
  * Schleicher [2019] Andreas Schleicher.  Pisa 2018: Insights and interpretations.  _oecd Publishing_ , 2019.
  * Chetty et al. [2014a] Raj Chetty, John N Friedman, and Jonah E Rockoff.  Measuring the impacts of teachers i: Evaluating bias in teacher value-added estimates.  _American economic review_ , 104(9):2593–2632, 2014a.
  * Chetty et al. [2014b] Raj Chetty, John N Friedman, and Jonah E Rockoff.  Measuring the impacts of teachers ii: Teacher value-added and student outcomes in adulthood.  _American economic review_ , 104(9):2633–79, 2014b.
  * Hanushek [2011] Eric A Hanushek.  The economic value of higher teacher quality.  _Economics of Education review_ , 30(3):466–479, 2011.
  * Chetty and Hendren [2018a] Raj Chetty and Nathaniel Hendren.  The impacts of neighborhoods on intergenerational mobility i: Childhood exposure effects.  _The Quarterly Journal of Economics_ , 133(3):1107–1162, 2018a.
  * Chetty and Hendren [2018b] Raj Chetty and Nathaniel Hendren.  The impacts of neighborhoods on intergenerational mobility ii: County-level estimates.  _The Quarterly Journal of Economics_ , 133(3):1163–1228, 2018b.
  * Milkman et al. [2021a] Katherine L Milkman, Dena Gromet, Hung Ho, Joseph S Kay, Timothy W Lee, Pepi Pandiloski, Yeji Park, Aneesh Rai, Max Bazerman, John Beshears, et al.  Megastudies improve the impact of applied behavioural science.  _Nature_ , 600(7889):478–483, 2021a.
  * Gu and Koenker [2020] Jiaying Gu and Roger Koenker.  Invidious comparisons: Ranking and selection as compound decisions.  _arXiv preprint arXiv:2012.12550_ , 2020.
  * Mogstad et al. [2020] Magne Mogstad, Joseph P Romano, Azeem Shaikh, and Daniel Wilhelm.  Inference for ranks with applications to mobility across neighborhoods and academic achievement across countries.  Technical report, National Bureau of Economic Research, 2020.
  * Mogstad et al. [2022] Magne Mogstad, Joseph Romano, Azeem Shaikh, and Daniel Wilhelm.  Comment on “invidious comparisons: Ranking and selection as compound decisions”.  Technical report, 2022.  URL https://www.econometricsociety.org/system/files/20460-1.pdf.
  * James and Stein [1992] William James and Charles Stein.  Estimation with quadratic loss.  In _Breakthroughs in statistics_ , pages 443–460. Springer, 1992\.  doi: 10.1007/978-1-4612-0919-5˙30.
  * Stein [1956] Charles Stein.  Inadmissibility of the usual estimator for the mean of a multivariate normal distribution.  In _Proceedings of the Third Berkeley symposium on mathematical statistics and probability_ , volume 1, pages 197–206, 1956.  doi: 10.1525/9780520313880-018.
  * Cai et al. [2021] Junhui Cai, Xu Han, Ya’acov Ritov, and Linda Zhao.  Nonparametric empirical bayes estimation and testing for sparse and heteroscedastic signals.  _arXiv preprint arXiv:2106.08881_ , 2021.
  * Brown and Greenshtein [2009] Lawrence D Brown and Eitan Greenshtein.  Nonparametric empirical bayes and compound decision approaches to estimation of a high-dimensional vector of normal means.  _The Annals of Statistics_ , pages 1685–1704, 2009.
  * Good Judgment Project [2016] Good Judgment Project.  GJP Data.  Technical report, 2016.  URL https://doi.org/10.7910/DVN/BPCDH5.
  * Chetty et al. [2016] Raj Chetty, Nathaniel Hendren, and Lawrence F Katz.  The effects of exposure to better neighborhoods on children: New evidence from the moving to opportunity experiment.  _American Economic Review_ , 106(4):855–902, 2016\.
  * Bergman et al. [2019] Peter Bergman, Raj Chetty, Stefanie DeLuca, Nathaniel Hendren, Lawrence F Katz, and Christopher Palmer.  Creating moves to opportunity: Experimental evidence on barriers to neighborhood choice.  Technical report, National Bureau of Economic Research, 2019.
  * Chetty et al. [2018] Raj Chetty, John N Friedman, Nathaniel Hendren, Maggie R Jones, and Sonya R Porter.  The opportunity atlas: Mapping the childhood roots of social mobility.  Technical report, National Bureau of Economic Research, 2018.
  * Milkman et al. [2021b] Katherine L Milkman, Mitesh S Patel, Linnea Gandhi, Heather N Graci, Dena M Gromet, Hung Ho, Joseph S Kay, Timothy W Lee, Modupe Akinola, John Beshears, et al.  A megastudy of text-based nudges encouraging patients to get vaccinated at an upcoming doctor’s appointment.  _Proceedings of the National Academy of Sciences_ , 118(20), 2021b.
  * Dube et al. [2021] Oeindrila Dube, Sendhil Mullainathan, and Devin G Pope.  A note on the level of customer support by state governments: A mystery-shopping approach.  Technical report, National Bureau of Economic Research, 2021.
  * Milkman et al. [2015] Katherine L Milkman, Modupe Akinola, and Dolly Chugh.  What happens before? a field experiment exploring how pay and representation differentially shape bias on the pathway into organizations.  _Journal of Applied Psychology_ , 100(6):1678, 2015.
  * Smith et al. [2021] Christian M Smith, Nicholas Goldrosen, Maria-Veronica Ciocanel, Rebecca Santorella, Chad M Topaz, and Shilad Sen.  Racial disparities in criminal sentencing vary considerably across federal judges.  _SocArXiv (July 2021)_ , 2021.
  * Berger [2013] James O Berger.  _Statistical decision theory and Bayesian analysis_.  Springer Science & Business Media, 2013.
  * Genovese and Wasserman [2002] Christopher Genovese and Larry Wasserman.  Operating characteristics and extensions of the false discovery rate procedure.  _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , 64(3):499–517, 2002.