[148]. Since $\mathsf{Ent}(f^{\star})=\tau$, the previous equality implies
that $\gamma^{\star}=\frac{1}{\pi}e^{2\tau-1}$. It follows that
$f^{\star}(x)=\frac{1}{\sqrt{2e^{2\tau-1}}}\exp\left(-\pi\frac{(x-m)^{2}}{e^{2\tau-1}}\right)$
which is the Gaussian density with mean $m$ and variance
$\frac{1}{2\pi}e^{2\tau-1}$. ∎

##### Shift in mean with increase in $\alpha$.

We consider the effect of the parameter $\alpha$ on the mean of the output
density. We consider an instance
$\mathcal{I}_{1}=(\Omega,{f_{\mathcal{D}}},\ell,\alpha,\tau)$ where
$\Omega=\mathbb{R}$, ${f_{\mathcal{D}}}$ is the normal density
$N(\mu,\sigma^{2})$, $\ell(v,x)=(x-v)^{2}$,
$\tau=\mathsf{Ent}({f_{\mathcal{D}}})=\ln(\sigma\sqrt{2\pi e})$ and
$\alpha=1$. We know from the proof of Theorem 10.2 that the output density
$f^{\star}$ is the same as ${f_{\mathcal{D}}}$ and hence, has mean $0$.

Now consider an instance $\mathcal{I}_{2}$, that corresponds to the
disadvantaged group and has the same parameters as that of $\mathcal{I}_{1}$,
except that the parameter $\alpha$ is larger than $1$. Let
$f^{\star}_{\alpha}$ denote the corresponding output density. We know from
Theorem 6.16 that the output density is proportional to
$e^{-I_{\alpha}(x)/\gamma^{\star}_{\alpha}}$, where $\gamma^{\star}_{\alpha}$
is the optimal dual variable for the entropy constraint and Lemma 10.1 shows
that $I_{\alpha}(x)$ is given by:

$I_{\alpha}(x)=(\alpha-1){\sigma^{2}}\left((w^{2}+1)\Phi(w)+w\phi(w)\right)+{\sigma^{2}}\left(w^{2}+1\right),$

where $w=(x-\mu)/\sigma$. For sake of brevity, let $g(w)$ denote
$(w^{2}+1)\Phi(w)+w\phi(w)$, and assume w.l.o.g. that $\mu=0$ and $\sigma=1,$
i.e., the input density is standard normal. Thus, $I_{\alpha}(x)$ can be
written as $(\alpha-1)g(x)+(x^{2}+1).$ When $\alpha$ is large, the first term
here dominates the second term as long as $\Phi(x)\gg 1/\alpha$. Since
$\Phi(x)\geq 1/2$ for all $x\geq 0$, $I_{\alpha}(x)$ is relatively large for
all $x\geq 0$, and hence, $f^{\star}_{\alpha}(x)$ goes to 0 for
$x\in[0,\infty)$ as $\alpha$ increases. In fact, $g(x)$ is much larger than
$1/\alpha$ when $x$ is larger than $-\Omega(\sqrt{\ln(\alpha)})$ and, hence,
$f^{\star}_{\alpha}$ should be small for such values of $x$ as well.
Therefore, we expect the mean of the output density $f^{\star}_{\alpha}$ to go
to $-\infty$ as $\alpha$ goes to $\infty$. Further, the mean of
$f^{\star}_{\alpha}$ should decrease at a logarithmic rate with respect to
$\alpha$. We verify these observations numerically in Figure 5.

(a)

(b)

Figure 5:  Figure (a) plots the input density, i.e., standard normal density,
and the output density when the parameter $\alpha=1000$. Figure (b) plots the
mean of the output density as a function of the parameter $\alpha$ (on a log
scale).

##### Deriving the implicit variance model.

We show that the implicit variance model can be derived from our optimization
framework. In particular, we prove the following result:

######  Theorem 10.3.

Consider an instance of the implicit variance model given by parameters
$\mu,\sigma,\sigma_{0}$. Consider instances
$I_{1}=(\Omega,f_{\mathcal{G}},\ell,\alpha,\tau_{1})$ and
$I_{2}=(\Omega,f_{\mathcal{G}},\ell,\alpha,\tau_{2})$ of
(LABEL:prog:framework), where $\Omega=\mathbb{R}$, $f_{\mathcal{G}}$ is the
normal density $N(\mu,\sigma_{0}^{2})$, $\alpha=1$, $\ell(v,x)=(x-v)^{2}$,
$\tau_{1}=\frac{1}{2}(1+\ln(2\pi\sigma_{0}^{2}))$ and
$\tau_{2}=\frac{1}{2}(1+\ln(2\pi(\sigma_{0}^{2}+\sigma^{2})))$. Then the
output density of (LABEL:prog:framework) on $I_{1}$ is $N(\mu,\sigma_{0}^{2})$
and the output density of (LABEL:prog:framework) on $I_{2}$ is
$N(\mu,\sigma_{0}^{2}+\sigma^{2})$.

###### Proof.

The result follows from Theorem 10.2. For the instance $\mathcal{I}_{1}$,
Theorem 10.2 shows that the output density of LABEL:prog:framework is the
normal density with mean $\mu$ and variance

$\frac{1}{2\pi}e^{2\tau_{1}-1}=\sigma_{0}^{2}.$

Similarly, it follows that the output density for the instance
$\mathcal{I}_{2}$ is normal with mean $\mu$ and variance

$\frac{1}{2\pi}e^{2\tau_{2}-1}=\sigma_{0}^{2}+\sigma^{2}.$

∎

##  11 Pareto density

The Pareto density is defined as follows over $\Omega=[1,\infty)$ and has a
parameter $\beta>1$:

$f_{\mathcal{P}}(x)\coloneqq\frac{\beta}{x^{\beta+1}},\quad x\in[1,\infty).$

The mean of this density is $\frac{\beta}{\beta-1}$. Thus, the mean is finite
only when $\beta>1$. Its differential entropy is
$1+\frac{1}{\beta}+\ln\frac{1}{\beta}$ [148]. We consider the loss function
$\ell(x,v)\coloneqq\ln x-\ln v$. First, we compute the expression for $I(x)$
which we use to verify the applicability of Theorem 6.1 with the above
parameters.

######  Lemma 11.1 (Expression for $I_{\alpha}(x)$).

Consider an instance $\mathcal{I}=(\Omega,f_{\mathcal{P}},\ell,\alpha,\tau)$
of PrimalOpt where $\Omega=[1,\infty)$, $\ell(x,v)\coloneqq\ln x-\ln v$, and
$f_{\mathcal{P}}$ is the Pareto density with parameter $\beta>1$. Then

$I(x)=\alpha\ln x+\frac{\alpha-1}{\beta x^{\beta}}-\frac{\alpha}{\beta}.$

###### Proof.

We use integration by parts to derive the following expression for $I(x)$:

$\displaystyle I(x)$ $\displaystyle=\alpha\int_{1}^{x}(\ln x-\ln
v)\frac{\beta}{v^{\beta+1}}d\mu(v)+\int_{x}^{\infty}(\ln x-\ln
v)\frac{\beta}{v^{\beta+1}}d\mu(v)$ $\displaystyle=\alpha\left[\frac{\ln v-\ln
x}{v^{\beta}}\right]_{1}^{x}-\alpha\int_{1}^{x}\frac{1}{v^{\beta+1}}d\mu(v)+\left[\frac{\ln
v-\ln
x}{v^{\beta}}\right]_{x}^{\infty}+\int_{x}^{\infty}\frac{1}{v^{\beta+1}}d\mu(v)$
$\displaystyle=\alpha\ln
x+\frac{\alpha}{\beta}\left(\frac{1}{x^{\beta}}-1\right)-\frac{1}{\beta
x^{\beta}}.$

∎

##### Applicability of Theorem 6.1.

Next, we verify that assumptions (A0)–(A5) hold. Since $\Omega=[1,\infty)$,
(A0) holds for any finite $\tau$. (A1) and (A2) hold due to the choice of the
loss function. (A3) can be shown to hold since $f_{\mathcal{P}}$ is a Pareto
density: To see this note that for any finite $x\geq 1$

$\displaystyle\int_{\Omega}\left|\ell(x,v)\right|f_{\mathcal{P}}(v)d\mu(v)$
$\displaystyle=\int_{1}^{x}\left(\ln(x)+\ln(v)\right)f_{\mathcal{P}}(v)d\mu(v)+\int_{x}^{\infty}\left(\ln(v)-\ln(x)\right)f_{\mathcal{P}}(v)d\mu(v)$
$\displaystyle\leq\int_{1}^{\infty}\left(\ln(x)+\ln(v)\right)f_{\mathcal{P}}(v)d\mu(v)$
$\displaystyle=\ln(x)+\int_{1}^{\infty}\ln(v)f_{\mathcal{P}}(v)d\mu(v)$
$\displaystyle=\ln(x)+\left[-\frac{\ln(v)}{v^{\beta}}\right]_{1}^{\infty}+\int_{1}^{\infty}\frac{1}{v^{\beta+1}}d\mu(v)$
$\displaystyle=\ln(x)+\left[-\frac{v^{-\beta}}{\beta}\right]_{1}^{\infty}$
$\displaystyle=\ln(x)+\frac{1}{\beta}$ $\displaystyle<\infty.$

Thus, (A3) holds. (A4) holds with, e.g., $R=2$. By Lemma 11.1,

$I(x)=\alpha\ln x+\frac{\alpha-1}{\beta x^{\beta}}-\frac{\alpha}{\beta}.$

Thus, $I(x)$ is differentiable at each $x\in\Omega$. Moreover, for all
$x\in\Omega$

$\frac{\partial I(x)}{\partial
x}=\frac{\alpha}{x}-\frac{\alpha-1}{x^{\beta+1}}=\frac{1}{x}\left(\alpha-\frac{\alpha-1}{x^{\beta}}\right)\stackrel{{\scriptstyle
x\geq 1}}{{>}}0.$

Since the derivative is positive for all $x\in\Omega=[1,\infty)$, it follows
that $I(x)$ has a unique global minimum at $x=1$. Therefore, (A5) holds. Since
assumptions (A0)–(A5) hold, we invoke Theorem 6.1 to deduce the form of
$f^{\star}$. Figure 6 plots the mean and the variance of the output density
$f^{\star}$ as a function of the parameter $\tau$ when the input density
${f_{\mathcal{D}}}$ is the standard normal density. Figure 7 plots the mean
and the variance of the output density as a function of the parameter $\alpha$
in this setting.

(a) (b)

Figure 6:  Mean and variance of the output density $f$, i.e.,
$\operatornamewithlimits{\mathbb{E}}_{x\sim f}[x]$ and ${\rm Var}_{x\sim
f}[x]$, as a function of $\tau$ when ${f_{\mathcal{D}}}$ is the Pareto
distribution with parameter 3 and $\alpha$ is fixed to 2.

(a) (b)

Figure 7:  Mean and variance of the output density $f$, i.e.,
$\operatornamewithlimits{\mathbb{E}}_{x\sim f}[x]$ and ${\rm Var}_{x\sim
f}[x]$, as a function of $\alpha$ when ${f_{\mathcal{D}}}$ is the Pareto
distribution with parameter 3 and $\tau$ is fixed to the entropy of
${f_{\mathcal{D}}}$.

######  Theorem 11.2 (Expression for $f^{\star}$ with $\alpha=1$).

Consider an instance $\mathcal{I}=(\Omega,f_{\mathcal{P}},\ell,1,\tau)$ of
PrimalOpt where $\Omega=[1,\infty)$, $\ell\coloneqq\ln x-\ln v$, and
$f_{\mathcal{P}}$ is the Pareto density with parameter $\beta>1$. Let
$f^{\star}$ be the optimal solution of instance $\mathcal{I}$. $f^{\star}$ is
a Pareto density with parameter $\beta_{\tau}$ satisfying the following
condition:

$1+\frac{1}{\beta_{\tau}}-\ln\beta_{\tau}=\tau.$

Let $m_{\tau}$ and $\sigma^{2}_{\tau}$ be the mean and variance of
$f^{\star}_{\tau}$ as a function of $\tau$. It holds that $m_{\tau}$ is
monotonically increasing in $\tau$ and $\sigma^{2}_{\tau}$ is either infinite
or monotonically increasing in $\tau$.

###### Proof.

For $\alpha=1$, Lemma 11.1 implies that

$\displaystyle I(x)$ $\displaystyle=\ln(x)-\frac{1}{\beta}.$

As shown earlier, one can invoke Theorem 6.1 for instance $\mathcal{I}$ for
any finite $\tau$, which implies that $f^{\star}$ has the following form

$f^{\star}(x)\propto e^{\frac{-\ln(x)+\frac{1}{\beta}}{\gamma^{\star}}}\propto
x^{-\frac{1}{\gamma^{\star}}}.$

where the proportionality constant and $\gamma^{\star}$ are determined by
$\int_{\Omega}f^{\star}(x)d\mu(x)=1$ and $\mathsf{Ent}(f^{\star})=\tau$.
$f^{\star}$ is a Pareto density with parameter
$\beta_{\tau}=\frac{1}{\gamma^{\star}}-1$ and, hence, has a differential
entropy of $1+\frac{1}{\beta_{\tau}}+\ln{\frac{1}{\beta_{\tau}}}$ [148]. This
combined with the condition $\mathsf{Ent}(f^{\star})=\tau$ implies that
$f^{\star}$ is the Pareto density with $\beta_{\tau}$ satisfying

$1+\frac{1}{\beta_{\tau}}+\ln{\frac{1}{\beta_{\tau}}}=\tau.$

Since $1+\frac{1}{\beta_{\tau}}+\ln{\frac{1}{\beta_{\tau}}}$ is a decreasing
function of $\beta_{\tau}$, it follows that increasing $\tau$ monotonically
decreases $\beta_{\tau}$. Note that $\beta_{\tau}>1$ for any finite $\tau$.
The mean and variance of $f^{\star}$ are
$m_{\tau}=\frac{\beta_{\tau}}{\beta_{\tau}-1}$ and
$\sigma^{2}_{\tau}=\frac{\beta_{\tau}}{(\beta_{\tau}-1)^{2}(\beta_{\tau}-2)}$
respectively. Since $m_{\tau}$ is a monotonically decreasing function of
$\beta_{\tau}$ (for $\beta_{\tau}>1$) and $\beta_{\tau}$ is a monotonically
decreasing function of $\tau$, it follows that $m_{\tau}$ is a monotonically
increasing function of $\beta_{\tau}$. The variance $\sigma^{2}_{\tau}$ is
finite when $\beta_{\tau}>2$. Moreover, if $\beta_{\tau}>2$, then
$\sigma^{2}_{\tau}$ is a monotonically decreasing function of $\beta_{\tau}$.
Since $\beta_{\tau}$ is a monotonically decreasing function of $\tau$, it
follows that $\sigma^{2}_{\tau}$ is either infinite or a monotonically
increasing function of $\beta_{\tau}$. ∎

##### Reduction in mean with increase in $\alpha$.

(a)

(b)

Figure 8:  Figure (a) plots the input Pareto density with parameter
$\beta=1.5$ and the corresponding limiting output density. Figure (b) plots
the mean of the input Pareto density and the corresponding limiting output
density as a function of the parameter $\beta$.

We show how our framework can capture a similar phenomenon as the
multiplicative-bias model of [90]. Recall that in the multiplicative-bias
model, the estimated utility of the disadvantaged group is scaled down by a
factor $\rho>1$.

Fix a parameter $\beta>1$. Consider an instance $\mathcal{I}_{1}$ given by the
parameters $I_{1}=(\Omega=[1,\infty),{f_{\mathcal{D}}},\ell,\alpha,\tau)$
where ${f_{\mathcal{D}}}$ is the Pareto density with parameter $\beta$,
$\tau=\mathsf{Ent}({f_{\mathcal{D}}})=1+\frac{1}{\beta}+\ln\frac{1}{\beta},\alpha=1$
and $\ell(x,v)=\ln x-\ln v$. As shown in Theorem 11.2, the output density
$f^{\star}$ is the same as ${f_{\mathcal{D}}}$. The proof of this result also
shows that the output density $f^{\star}(x)$ is proportional to
$e^{-I(x)/\gamma^{\star}}$, where $I(x)=\ln x-\frac{1}{\beta}$ and
$\gamma^{\star}$ is the optimal dual variable for the corresponding entropy
constraint.

The disadvantaged group is modeled by an instance $\mathcal{I}_{2}$ which has
the same parameters as that of $\mathcal{I}_{1}$ except that the parameter
$\alpha$ is larger than 1. In this case, Theorem 11.2 shows that the optimal
density $f^{\star}_{\alpha}(x)$ is proportional to
$e^{-I_{\alpha}(x)/\gamma_{\alpha}^{\star}}$, where $I_{\alpha}(x)=\alpha\ln
x+\frac{\alpha-1}{\beta x^{\beta}}-\frac{\alpha}{\beta}$ and
$\gamma_{\alpha}^{\star}$ is the corresponding optimal dual variable. For
large $\alpha$, $I_{\alpha}(x)\sim\alpha(I(x)+\frac{1}{\beta x^{\beta}}).$
Hence, the output density (for large $\alpha$) is given by

$f^{\star}_{\alpha}(x)=K\cdot
e^{-\frac{\alpha}{\gamma^{\star}_{\alpha}}\left(I(x)+\frac{1}{\beta
x^{\beta}}\right)},$

where $K$ is a normalization constant. The normalization constant $K$ and the
ratio $\gamma^{\star}/\alpha$ are given by two constraints: (i) integral of
$f^{\star}_{\alpha}$ over the domain $\Omega$ should be 1, and (ii) the
entropy of $f^{\star}_{\alpha}$ should be equal to
$\tau=\mathsf{Ent}({f_{\mathcal{D}}})$. This shows that the ratio
$\gamma^{\star}_{\alpha}/\alpha$ tends to a constant for large $\alpha$, and
hence, the output density converges to the density given by

$g^{\star}(x)=Ke^{-C\left(\ln x+\frac{1}{\beta x^{\beta}}\right)}.$

We plot the output density $g^{\star}(x)$ for $\beta=1.5$ in Figure 8(a). In
Figure 8(b), we observe that for all considered values of the parameter
$\beta$, the mean of the limiting output density $g^{\star}$ always remains
below that of the input density ${f_{\mathcal{D}}}$. It is also worth noting
that for large $\beta$, the gap between the mean of the input density and that
of the (limiting) output density diminishes. Intuitively, this happens because
as $\beta$ increases, the mean of the input (Pareto) density gets closer to 1
(recall that the mean of a Pareto density with parameter $\beta$ is equal to
$\frac{\beta}{\beta-1}$). Now the output density also places more mass closer
to 1, but gets restricted because of two conditions: (i) the entropy of the
output density must be the same as that of the corresponding input density,
and (ii) it cannot place any probability mass on values below 1. Hence, there
is not much “room” for the output density to place extra probability mass on
values close to 1 (as compared to the corresponding Pareto density). Hence its
mean cannot go much below that of the corresponding Pareto density.

##  12 Conclusion, limitations, and future work

We present a new optimization-based approach to modeling bias in evaluation
processes ((LABEL:prog:framework)). Our model has two parameters, risk
averseness $\alpha$ and resource-information trade-off $\tau$, which are well
documented to lead to evaluation biases in a number of contexts. We show that
it can generate rich classes of output densities (Theorem 3.1) and discuss how
the output densities depend on the two parameters (Section 3). Empirically, we
demonstrate that the densities arising from our model have a good fit with the
densities of biased evaluations in multiple real-world datasets and a
synthetic dataset; often, leading to a better fit than models of prior works
[90, 61] (Table 1). We use our model as a tool to evaluate different types of
bias-mitigating interventions in a downstream selection task–illustrating how
this model could be used by policymakers to explore available interventions
(Figures 2 and E.1); see also Section 4. Our work relies on the assumptions in
prior works that there are no differences (at a population level) between
$G_{1}$ and $G_{2}$; see, e.g., [89, 61, 40]. If this premise is false, then
the effectiveness of interventions can be either underestimated or
overestimated which may lead a policymaker to select a suboptimal
intervention. That said, if all the considered interventions reduce risk
aversion and/or resource constraints, then the chosen intervention should
still have a positive impact on the disadvantaged group. Our model can be
easily used to study multiple socially-salient groups by considering a group-
specific risk-aversion parameter and a group-specific information constraint.
For example, if two groups $G_{1},G_{2}$ overlap, then we can consider three
disjoint subgroups $G_{1}\cap G_{2},$ $G_{1}\backslash G_{2}$ and
$G_{2}\backslash G_{1}$. Our model of evaluation processes considers scenarios
where candidates are evaluated along a single dimension. It can also be
applied – in a dimension-by-dimension fashion – to scenarios where individuals
are evaluated along multiple dimensions, but the evaluation in any dimension
is independent of the evaluation in other dimensions. Modeling evaluation
processes involving multiple correlated dimensions is an interesting
direction. While we illustrate the use of our model in a downstream selection
task, utilities generated from biased evaluation processes are also used in
other decision-making tasks (such as regression and clustering), and studying
the downstream impact of evaluation biases on them is an important direction.
Moreover, the output of or model can be used by policymakers to assess the
impact of interventions in the supermodular set aggregation setting, where the
utility of the selected group is more than the sum of the individuals. Our
model cannot be directly used to understand the effect of interventions in the
long term. Additional work would be required to do so, perhaps as in [38], and
would be an important direction for future work. Finally, any work on
debiasing could be used adversarially to achieve the opposite goal. We need
third-party evaluators, legal protections, and available recourse for affected
parties – crucial components of any system – though beyond the scope of this
work.

##### Acknowledgments.

This project is supported in part by NSF Awards CCF-2112665 and IIS-2045951.

## References

  * [1] Is Income Implicit in Measures of Student Ability?  https://budgetmodel.wharton.upenn.edu/issues/2021/9/28/is-income-implicit-in-measures-of-student-ability#:~:text=Summary%3A%20Measures%20of%20student%20ability,in%20family%20income%20across%20students, 2021.
  * [2] Supreme Court guts affirmative action, effectively ending race-conscious admissions.  https://www.npr.org/2023/06/29/1181138066/affirmative-action-supreme-court-decision, 2023.
  * [3] Social Security Administration.  Beyond the Top 1000 Names, 2018.  https://www.ssa.gov/oact/babynames/limits.html.
  * [4] Rakesh Agrawal, Sreenivas Gollapudi, Alan Halverson, and Samuel Ieong.  Diversifying search results.  In Proceedings of the Second ACM International Conference on Web Search and Data Mining, WSDM ’09, page 5–14, New York, NY, USA, 2009. Association for Computing Machinery.
  * [5] William Aiello, Fan R. K. Chung, and Linyuan Lu.  A random graph model for massive graphs.  In STOC, pages 171–180. ACM, 2000.
  * [6] Julia Angwin, Madeleine Varner, and Ariana Tobin.  Machine bias: Facebook enabled advertisers to reach ‘jew haters’.  ProPublica, Sept, 2017.
  * [7] Another PIL seeks entrance tests in Gujarati, October 2011.  https://web.archive.org/web/20120106121134/http://articles.timesofindia.indiatimes.com/2011-10-21/ahmedabad/30306229_1_entrance-tests-regional-language-gujarati-language.
  * [8] Michael Argyle.  The psychology of interpersonal behaviour.  Penguin UK, 1994.
  * [9] S. Arora, E. Hazan, and S. Kale.  The multiplicative weights update method: a meta-algorithm and applications.  Theory of Computing, 8(1):121–164, 2012.
  * [10] Kenneth J. Arrow.  Aspects of the Theory of Risk-Bearing.  YrjöJahnsson lectures. YrjöJahnssonin Säätiö, Helsinki, 1965.
  * [11] Kenneth J. Arrow.  The Theory of Discrimination, pages 1–33.  Princeton University Press, Princeton, 1974.
  * [12] Kenneth J. Arrow.  What has economics to say about racial discrimination?  Journal of Economic Perspectives, 12(2):91–100, June 1998.
  * [13] L. Babcock and S. Laschever.  Why Women Don’t Ask: The High Cost of Avoiding Negotiations - And Positive Strategies for Change.  Little, Brown Book Group Limited, 2009.
  * [14] Stijn Baert.  Hiring a gay man, taking a risk?: A lab experiment on employment discrimination and risk aversion.  Journal of Homosexuality, 65(8):1015–1031, 2018.  PMID: 28841095.
  * [15] Katherine Baldiga.  Gender Differences in Willingness to Guess.  Management Science, 60(2):434–448, 2014.
  * [16] British Business Bank and Oliver Wyman.  Alone together: Entrepreneurship and diversity in the UK, October 2020.  https://www.british-business-bank.co.uk/wp-content/uploads/2020/10/Alone-together-Entrepreneurship-and-diversity-in-the-UK-FINAL.pdf.
  * [17] Albert-László Barabási and Réka Albert.  Emergence of scaling in random networks.  Science, 286(5439):509–512, 1999.
  * [18] Hannah Bast.  How Objective is Peer Review?, November 2020.  https://cacm.acm.org/blogs/blog-cacm/248824-how-objective-is-peer-review/fulltext.
  * [19] Surender Baswana, Partha P Chakrabarti, V Kamakoti, Yash Kanoria, Ashok Kumar, Utkarsh Patange, and Sharat Chandran.  Joint seat allocation: An algorithmic perspective, 2015.
  * [20] Surender Baswana, Partha Pratim Chakrabarti, Sharat Chandran, Yashodhan Kanoria, and Utkarsh Patange.  Centralized Admissions for Engineering Colleges in India.  In Proceedings of the 2019 ACM Conference on Economics and Computation, EC ’19, page 323–324, New York, NY, USA, 2019. Association for Computing Machinery.
  * [21] Ariane Baye and Christian Monseur.  Gender differences in variability and extreme scores in an international context.  Large-scale Assessments in Education, 4(1):1, 2016.
  * [22] Gary S. Becker.  The Economics of Discrimination.  Economic Research Studies. University of Chicago Press, 2010.
  * [23] Christian Belzil and Marco Leonardi.  Risk aversion and schooling decisions.  Annals of Economics and Statistics, pages 35–70, 2013.
  * [24] Sarah Berger.  This high school senior says he is spending $1,700 on college applications, January 2018.  https://www.cnbc.com/2018/01/02/high-price-of-college-applications.html.
  * [25] Marianne Bertrand and Esther Duflo.  Chapter 8 - field experiments on discriminationaalaura stilwell and jan zilinsky provided excellent research assistance. we thank abhijit banerjee for comments. we are particularly grateful to betsy levy paluck, our discussant, for her detailed and thoughtful review of an earlier draft.  In Abhijit Vinayak Banerjee and Esther Duflo, editors, Handbook of Field Experiments, volume 1 of Handbook of Economic Field Experiments, pages 309–393. North-Holland, 2017.
  * [26] Deepti Bhaskaran.  CBSE’s ‘Udaan’ initiative helping girl students crack IIT entrance exam.  Mint, August 2017.  https://www.livemint.com/Education/B5VfKv6ts4EOWTuXpqYxKJ/CBSEs-Udaan-initiative-helping-girl-students-crack-IIT-e.html.
  * [27] Scott Bland.  Schumer to Introduce Rules for Diverse Senate Hiring.  Politico, 2017.  https://www.politico.com/story/2017/02/schumer-diversity-nfl-rooney-rule-235477.
  * [28] Avrim Blum and Kevin Stangl.  Recovering from biased data: Can fairness constraints improve accuracy?  In FORC, volume 156 of LIPIcs, pages 3:1–3:20. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2020.
  * [29] Niclas Boehmer, L. Elisa Celis, Lingxiao Huang, Anay Mehrotra, and Nisheeth K. Vishnoi.  Subset selection based on multiple rankings in the presence of bias: Effectiveness of fairness constraints for multiwinner voting score functions.  In International Conference on Machine Learning, ICML, volume 202 of Proceedings of Machine Learning Research, pages 2641–2688. PMLR, 2023.
  * [30] Iris Bohnet.  What Works: Gender Equality by Design.  Harvard University Press, 2016.
  * [31] Robert F Boldt, John A Centra, and Rosalea G Courtney.  The validity of various methods of treating multiple sat® scores.  ETS Research Report Series, 1986(1):i–8, 1986.
  * [32] Tristan L Botelho and Mabel Abraham.  Pursuing quality: How search costs and uncertainty magnify gender-based double standards in a multistage evaluation process.  Administrative Science Quarterly, 62(4):698–730, 2017.
  * [33] Hannah Riley Bowles.  Why Women Don’t Negotiate Their Job Offers, June 2014.  https://hbr.org/2014/06/why-women-dont-negotiate-their-job-offers.
  * [34] Stephen Boyd and Lieven Vandenberghe.  Convex Optimization.  Cambridge University Press, 2004.
  * [35] Andrei Z. Broder, Ravi Kumar, Farzin Maghoul, Prabhakar Raghavan, Sridhar Rajagopalan, Raymie Stata, Andrew Tomkins, and Janet L. Wiener.  Graph structure in the web.  Comput. Networks, 33(1-6):309–320, 2000.
  * [36] Quinn Capers IV, Daniel Clinchot, Leon McDougle, and Anthony G Greenwald.  Implicit racial bias in medical school admissions.  Academic Medicine, 92(3):365–369, 2017.
  * [37] Marilyn Cavicchia.  How to fight implicit bias? With conscious thought, diversity expert tells NABE, June 2017.
  * [38] L. Elisa Celis, Chris Hays, Anay Mehrotra, and Nisheeth K. Vishnoi.  The Effect of the Rooney Rule on Implicit Bias in the Long Term.  In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 678–689, New York, NY, USA, 2021. Association for Computing Machinery.
  * [39] L. Elisa Celis, Vijay Keswani, and Nisheeth K. Vishnoi.  Data preprocessing to mitigate bias: A maximum entropy based approach.  In ICML, volume 119 of Proceedings of Machine Learning Research, pages 1349–1359. PMLR, 2020.
  * [40] L. Elisa Celis, Anay Mehrotra, and Nisheeth K. Vishnoi.  Interventions for ranking in the presence of implicit bias.  In FAT*, pages 369–380. ACM, 2020.
  * [41] Christopher P Chambers and Federico Echenique.  A Characterisation of ‘Phelpsian’ Statistical Discrimination.  The Economic Journal, 131(637):2018–2032, 08 2020.
  * [42] Tessa E. S. Charlesworth and Mahzarin R. Banaji.  Patterns of Implicit and Explicit Attitudes: I. Long-Term Change and Stability From 2007 to 2016.  Psychological Science, 30(2):174–192, 2019.  PMID: 30605364.
  * [43] Erick Chastain, Adi Livnat, Christos Papadimitriou, and Umesh Vazirani.  Algorithms, games, and evolution.  Proceedings of the National Academy of Sciences, 111(29):10620–10623, 2014.
  * [44] Mansi Choksi.  Inside India’s Cram City.  The New York Times, January 2023.  https://web.archive.org/web/20230202034926/https://www.nytimes.com/2023/01/18/magazine/india-cram-schools-kota.html.
  * [45] Stephen Coate and Glenn C. Loury.  Will affirmative-action policies eliminate negative stereotypes?  The American Economic Review, 83(5):1220–1240, 1993.
  * [46] Brian W. Collins.  Tackling unconscious bias in hiring practices: The plight of the rooney rule.  NYUL Rev., 82:870, 2007.
  * [47] Conduct IIT entrance test in Tamil also: PMK, March 2012.  https://web.archive.org/web/20140502002203/https://www.thehindu.com/todays-paper/tp-national/article2961807.ece.
  * [48] Keith Conrad.  Probability distributions and maximum entropy.  Entropy, 6(452):10, 2004.
  * [49] Bradford Cornell and Ivo Welch.  Culture, information, and screening discrimination.  Journal of political Economy, 104(3):542–571, 1996.
  * [50] Jason Dana, Robyn Dawes, and Nathanial Peterson.  Belief in the unstructured interview: The persistence of an illusion.  Judgment and Decision Making, 8(5):512–520, 2013.
  * [51] Jeffrey Dastin.  Amazon scraps secret AI recruiting tool that showed bias against women, October 2019.  https://reut.rs/2N1dzRJ.
  * [52] Debraj Deb.  Tripura govt to sponsor 30 students with NEET, JEE coaching under ‘Super 30’ scheme.  The Indian Express, May 2020.  https://indianexpress.com/article/education/tripura-to-sponsor-30-top-students-with-neet-jee-coaching-under-super-30-scheme-from-this-year-6413972/.
  * [53] Delhi govt. to provide free coaching for NEET, JEE aspirants, February 2022.  https://www.thehindu.com/news/cities/Delhi/delhi-govt-to-provide-free-coaching-for-neet-jee-aspirants/article65056297.ece.
  * [54] William Dieterich, Christina Mendoza, and Tim Brennan.  Compas risk scales: Demonstrating accuracy equity and predictive parity.  Northpoint Inc, 2016.
  * [55] Ezekiel J Dixon-Román, Howard T Everson, and John J McArdle.  Race, poverty and sat scores: Modeling the influences of family income on black and white high school students’ sat performance.  Teachers College Record, 115(4):1–33, 2013.
  * [56] Miroslav Dudik.  Maximum entropy density estimation and modeling geographic distributions of species, 2007.
  * [57] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel.  Fairness through awareness.  In ITCS, pages 214–226, New York, NY, USA, 2012. ACM.
  * [58] Tracy D Eells and C Robert Showalter.  Work-related stress in american trial judges.  Journal of the American Academy of Psychiatry and the Law Online, 22(1):71–83, 1994.
  * [59] Edith Elkind, Piotr Faliszewski, Piotr Skowron, and Arkadii Slinko.  Properties of multiwinner voting rules.  In AAMAS 2014, pages 53–60, 2014.
  * [60] Kim Elsesser.  Lawsuit Claims SAT And ACT Are Biased—Here’s What Research Says, 2019.  https://www.forbes.com/sites/kimelsesser/2019/12/11/lawsuit-claims-sat-and-act-are-biased-heres-what-research-says/?sh=177187663c42.
  * [61] Vitalii Emelianov, Nicolas Gast, Krishna P. Gummadi, and Patrick Loiseau.  On fair selection in the presence of implicit variance.  In EC, pages 649–675. ACM, 2020.
  * [62] Episode 2 of Inside the Yale Admissions Office Podcast, 2020.  https://admissions.yale.edu/podcast.
  * [63] Robert Epstein and Ronald E Robertson.  The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections.  Proceedings of the National Academy of Sciences, 112(33):E4512–E4521, 2015.
  * [64] Frederick Erickson and Jeffrey Schultz.  The Counselor as gatekeeper: Social interaction in inverviews.  Academic Press, 1982.
  * [65] Sean Fahey.  The Real Cost Of Bad Hiring Decisions (And How To Avoid Making Them), March 2022.  https://www.forbes.com/sites/forbeshumanresourcescouncil/2022/03/10/the-real-cost-of-bad-hiring-decisions-and-how-to-avoid-making-them/?sh=1e15226e5dac.
  * [66] Hanming Fang and Andrea Moro.  Chapter 5 - Theories of Statistical Discrimination and Affirmative Action: A Survey.  In Jess Benhabib, Alberto Bisin, and Matthew O. Jackson, editors, Handbook of Social Economics, volume 1, pages 133–200. North-Holland, 2011.
  * [67] Nikhil Garg, Hannah Li, and Faidra Monachou.  Standardized tests and affirmative action: The role of bias and variance.  In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 261, New York, NY, USA, 2021. Association for Computing Machinery.
  * [68] Atul Gawande.  The Checklist Manifesto: How to Get Things Right.  Henry Holt and Company, 2010.
  * [69] Tamar Szabó Gendler.  On the epistemic costs of implicit bias.  Philosophical Studies, 156(1):33, 2011.
  * [70] Daniel T. Gilbert and J. Gregory Hixon.  The trouble of thinking: Activation and application of stereotypic beliefs.  Journal of Personality and Social Psychology, 60:509–517, 1991.
  * [71] Anand M Goel and Anjan V Thakor.  Overconfidence, ceo selection, and corporate governance.  the Journal of Finance, 63(6):2737–2784, 2008.
  * [72] Claudia Goldin and Cecilia Rouse.  Orchestrating Impartiality: The Impact of “Blind” Auditions on Female Musicians.  American Economic Review, 90(4):715–741, September 2000.
  * [73] Anthony G Greenwald and Mahzarin R Banaji.  Implicit social cognition: attitudes, self-esteem, and stereotypes.  Psychological review, 102(1):4, 1995.
  * [74] Anthony G Greenwald and Linda Hamilton Krieger.  Implicit bias: Scientific foundations.  California Law Review, 94(4):945–967, 2006.
  * [75] Moritz Hardt, Eric Price, and Nati Srebro.  Equality of opportunity in supervised learning.  In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 3315–3323, 2016.
  * [76] Abigail Johnson Hess.  Rich students get better SAT scores—here’s why, October 2019.  https://www.cnbc.com/2019/10/03/rich-students-get-better-sat-scores-heres-why.html.
  * [77] The White House.  Fact Sheet: President Obama Announces New Commitments from Investors, Companies, Universities, and Cities to Advance Inclusive Entrepreneurship at First-Ever White House Demo Day, August 2015.
  * [78] J.E. Ingersoll.  Theory of Financial Decision Making.  G - Reference,Information and Interdisciplinary Subjects Series. Rowman & Littlefield, 1987.
  * [79] Abigail Z. Jacobs and Hanna Wallach.  Measurement and fairness.  In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 375–385, New York, NY, USA, 2021. Association for Computing Machinery.
  * [80] Edwin T. Jaynes.  Information theory and statistical mechanics.  Physical Review, 106:620–630, May 1957.
  * [81] Edwin T. Jaynes.  On the rationale of maximum-entropy methods.  Proceedings of the IEEE, 70(9):939–952, 1982.
  * [82] Kailash Jeenger.  Reservation Is About Adequate Representation, Not Poverty Eradication.  The Wire, May 2020.  https://thewire.in/law/supreme-court-bench-reservation.
  * [83] Yangqing Jia and Trevor Darrell.  Heavy-tailed distances for gradient based image descriptors.  In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011.
  * [84] D. Kahneman, S.P. Slovic, P. Slovic, A. Tversky, and Cambridge University Press.  Judgment Under Uncertainty: Heuristics and Biases.  Cambridge University Press, 1982.
  * [85] Irwin Katz.  Gordon Allport’s “The Nature of Prejudice”.  Political Psychology, 12(1):125–157, 1991.
  * [86] Giora Keinan.  Decision making under stress: Scanning of alternatives under controllable and uncontrollable threats.  Journal of Personality and Social Psychology, 52:639–644, 1987.
  * [87] M.E. Kite and B.E. Whitley.  Psychology of Prejudice and Discrimination: 3rd Edition.  Taylor & Francis, 2016.
  * [88] Jon Kleinberg and Sendhil Mullainathan.  Simplicity creates inequity: Implications for fairness, stereotypes, and interpretability.  In Proceedings of the 2019 ACM Conference on Economics and Computation, EC ’19, page 807–808, New York, NY, USA, 2019. Association for Computing Machinery.
  * [89] Jon M. Kleinberg, Sendhil Mullainathan, and Manish Raghavan.  Inherent trade-offs in the fair determination of risk scores.  In ITCS, volume 67 of LIPIcs, pages 43:1–43:23. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2017.
  * [90] Jon M. Kleinberg and Manish Raghavan.  Selection problems in the presence of implicit bias.  In ITCS, volume 94 of LIPIcs, pages 33:1–33:17. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2018.
  * [91] Mr. Rajeev Kumar.  Rti complaint, 2009.  Decision No. CIC/SG/C/2009/001088/5392, Complaint No. CIC/SG/C/2009/001088.
  * [92] Rahul Kumar.  SC, ST, OBC representation in Indian education is dismal, upper-caste nexus persists.  The Print, March 2021.  https://theprint.in/campus-voice/sc-st-obc-representation-in-indian-education-is-dismal-upper-caste-nexus-persists/627217/.
  * [93] Ravi Kumar, Prabhakar Raghavan, Sridhar Rajagopalan, D Sivakumar, Andrew Tomkins, and Eli Upfal.  Stochastic models for the web graph.  In Proceedings 41st Annual Symposium on Foundations of Computer Science, pages 57–65. IEEE, 2000.
  * [94] Didier Lairez.  A short derivation of boltzmann distribution and gibbs entropy formula from the fundamental postulate, 2023.
  * [95] Kevin Lang.  A Language Theory of Discrimination*.  The Quarterly Journal of Economics, 101(2):363–382, 05 1986.
  * [96] Flavien Léger.  A gradient descent perspective on sinkhorn.  Applied Mathematics & Optimization, 84(2):1843–1855, 2021.
  * [97] David Asher Levin, Yuval Peres, and Elizabeth Lee Wilmer.  Markov chains and mixing times.  American Mathematical Soc., 2009.
  * [98] Haim Levy and Myles Robinson.  Stochastic Dominance: Investment Decision Making Under Uncertainty, volume 34.  Springer, 2006.
  * [99] Tamar Lewin.  A New SAT Aims to Realign With Schoolwork, March 2014.  https://www.nytimes.com/2014/03/06/education/major-changes-in-sat-announced-by-college-board.html.
  * [100] Falk Lieder and Thomas L. Griffiths.  Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources.  Behavioral and Brain Sciences, 43:e1, 2020.
  * [101] W Lippmann.  Public opinion. new york city.  Harcourt, Brace. Lipson, J., Omidian, P.(1992, September). Health issues of Afghan refugees in California. The Western Journal of Medicine, 157(3):271–275, 1922.
  * [102] J. H. C. Lisman and M. C. A. van Zuylen.  Note on the generation of most probable frequency distributions.  Statistica Neerlandica, 26(1):19–23, March 1972.
  * [103] List of languages by number of native speakers in India.  https://en.wikipedia.org/wiki/List_of_languages_by_number_of_native_speakers_in_India.
  * [104] Karen S Lyness and Madeline E Heilman.  When fit is fundamental: performance evaluations and promotions of upper-level female and male managers.  Journal of Applied Psychology, 91(4):777, 2006.
  * [105] Michael W. Mahoney and Lorenzo Orecchia.  Implementing regularization implicitly via approximate eigenvector computation.  In ICML, pages 121–128. Omnipress, 2011.
  * [106] Anay Mehrotra, Bary S. R. Pradelski, and Nisheeth K. Vishnoi.  Selection in the presence of implicit bias: The advantage of intersectional constraints.  In FAccT ’22: 2022 ACM Conference on Fairness, Accountability, and Transparency, Seoul, Republic of Korea, June 21 - 24, 2022, pages 599–609. ACM, 2022.
  * [107] Anay Mehrotra, Bary S. R. Pradelski, and Nisheeth K. Vishnoi.  Selection in the Presence of Implicit Bias: The Advantage of Intersectional Constraints.  In FAccT, page To appear. ACM, 2022.
  * [108] Anay Mehrotra and Nisheeth K. Vishnoi.  Maximizing submodular functions for recommendation in the presence of biases.  In Ying Ding, Jie Tang, Juan F. Sequeda, Lora Aroyo, Carlos Castillo, and Geert-Jan Houben, editors, Proceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023 - 4 May 2023, pages 3625–3636. ACM, 2023.
  * [109] Robert C. Merton.  Optimum consumption and portfolio rules in a continuous-time model**i would like to thank p. a. samuelson, r. m. solow, p. a. diamond, j. a. mirrlees, j. a. flemming, and d. t. scheffman for their helpful discussions. of course, all errors are mine. aid from the national science foundation is gratefully acknowledged. an earlier version of the paper was presented at the second world congress of the econometric society, cambridge, england.  In W.T. ZIEMBA and R.G. VICKSON, editors, Stochastic Optimization Models in Finance, pages 621–661. Academic Press, 1975.
  * [110] Corinne A. Moss-Racusin, John F. Dovidio, Victoria L. Brescoll, Mark J. Graham, and Jo Handelsman.  Science faculty’s subtle gender biases favor male students.  Proceedings of the National Academy of Sciences, 109(41):16474–16479, 2012.
  * [111] Julie A Nelson.  Are women really more risk-averse than men? a re-analysis of the literature using expanded methods.  Journal of economic surveys, 29(3):566–585, 2015.
  * [112] R. E. O’Dea, M. Lagisz, M. D. Jennions, and S. Nakagawa.  Gender differences in individual variation in academic grades fail to fit expected patterns for stem.  Nature Communications, 9(1):3777, 2018.
  * [113] Tom O’Neil.  Ask the expert: How to write a CV, 2012.  http://web.archive.org/web/20170202065654/http://www.economist.com/node/21559508.
  * [114] Paula Onuchic.  Recent contributions to theories of discrimination.  arXiv preprint arXiv:2205.05994, 2022.
  * [115] Philip Oreopoulos and Diane Dechief.  Why do some employers prefer to interview matthew, but not samir? new evidence from toronto, montreal, and vancouver.  Canadian Labour Market and Skills Researcher Network Working Paper 95, 2012.
  * [116] Christina Passariello.  Tech Firms Borrow Football Play to Increase Hiring of Women, September 2016.  https://www.wsj.com/articles/tech-firms-borrow-football-play-to-increase-hiring-of-women-1474963562.
  * [117] Edmund S. Phelps.  The statistical theory of racism and sexism.  The American Economic Review, 62(4):659–661, 1972.
  * [118] J.R. Posselt.  Inside Graduate Admissions: Merit, Diversity, and Faculty Gatekeeping.  Harvard University Press, 2016.
  * [119] John W. Pratt.  4 - Risk Aversion in the Small and in the Large.  In Peter Diamond and Michael Rothschild, editors, Uncertainty in Economics, pages 59–79. Academic Press, 1978.
  * [120] Proof: Normal distribution maximizes differential entropy for fixed variance.  https://statproofbook.github.io/P/norm-maxent.html.
  * [121] Elaine D Pulakos.  Selection assessment methods.  United stated of America: Society for Human Resource Management (SHRM) Foundation, 2005.
  * [122] Manish Raghavan, Solon Barocas, Jon M. Kleinberg, and Karen Levy.  Mitigating bias in algorithmic hiring: evaluating claims and practices.  In Mireille Hildebrandt, Carlos Castillo, Elisa Celis, Salvatore Ruggieri, Linnet Taylor, and Gabriela Zanfir-Fortuna, editors, FAT* ’20: Conference on Fairness, Accountability, and Transparency, Barcelona, Spain, January 27-30, 2020, pages 469–481, Barcelona, Spain, 2020. ACM.
  * [123] Barbara F. Reskin and Debra Branch McBrier.  Why Not Ascription? Organizations’ Employment of Male and Female Managers.  American Sociological Review, 65(2):210–233, 2000.
  * [124] Retaking the SAT, Manhattan Review.  https://www.manhattanreview.com/sat-retaking.
  * [125] Dan-Olof Rooth.  Automatic associations and discrimination in hiring: Real world evidence.  Labour Economics, 17(3):523–534, 2010.
  * [126] Walter Rudin.  Real and Complex Analysis.  McGraw-Hill, 1986.
  * [127] Melody S Sadler, Joshua Correll, Bernadette Park, and Charles M Judd.  The world is not black and white: Racial bias in the decision to shoot in a multiethnic context.  Journal of Social Issues, 68(2):286–313, 2012.
  * [128] Sahitya parishad demands entrance tests in gujarati, September 2011.  https://web.archive.org/web/20120106140750/http://articles.timesofindia.indiatimes.com/2011-09-21/education/30184151_1_regional-languages-entrance-tests-raghuveer-chaudhary.
  * [129] Jad Salem and Swati Gupta.  Closing the gap: Group-aware parallelization for the secretary problem with biased evaluations.  Available at SSRN 3444283, 2019.
  * [130] Howard Schuman, Charlotte Steeh, Lawrence Bobo, and Maria Krysan.  Racial Attitudes in America: Trends and Interpretations.  Social trends in the United States. Harvard University Press, 1985.
  * [131] Deepa Seetharaman.  Facebook Is Testing the ‘Rooney Rule’ Approach to Hiring.  The Wall Street Journal, June 2015.
  * [132] Herbert A Simon.  Bounded rationality and organizational learning.  Organization science, 2(1):125–134, 1991.
  * [133] Mohit Singh and Nisheeth K. Vishnoi.  Entropy, optimization and counting.  In Proceedings of the 46th Annual ACM Symposium on Theory of Computing, pages 50–59. ACM, 2014.
  * [134] Aarzoo Snigdha.  JEE Main 2021: Over 45,000 Students Writing Exam In Regional Languages.  NDTV, February 2021.  https://www.ndtv.com/education/jee-main-2021-over-45000-students-writing-exam-in-regional-languages-2378496.
  * [135] Thomas Sowell.  Affirmative Action Around the World: An Empirical Study.  Yale University Press, 2008.
  * [136] Damian Straszak and Nisheeth K. Vishnoi.  Maximum entropy distributions: Bit complexity and stability.  COLT, 2019.
  * [137] Damian Straszak and Nisheeth K. Vishnoi.  Iteratively reweighted least squares and slime mold dynamics: connection and convergence.  Mathematical Programming, 194(1):685–717, 2022.
  * [138] R.H. Thaler.  Misbehaving: The Making of Behavioral Economics.  W. W. Norton, 2015.
  * [139] The Cost of a Bad Hire, February 2019.  https://www.northwestern.edu/hr/about/news/february-2019/the-cost-of-a-bad-hire.html.
  * [140] The Utter Uselessness of Job Interviews, April 2017.  https://www.nytimes.com/2017/04/08/opinion/sunday/the-utter-uselessness-of-job-interviews.html.
  * [141] UP govt free coaching for JEE, NEET aspirants begins today, February 2021.  https://www.livemint.com/education/news/up-govt-free-coaching-for-jee-neet-aspirants-begins-today-11613449583134.html.
  * [142] AD Van Knippenberg, AP Dijksterhuis, and Diane Vermeulen.  Judgement and memory of a criminal act: The effects of stereotypes and cognitive load.  European Journal of Social Psychology, 29(2-3):191–201, 1999.
  * [143] Nisheeth K. Vishnoi.  Algorithms for Convex Optimization.  Cambridge University Press, 2021.
  * [144] J. von Neumann, O. Morgenstern, H.W. Kuhn, and A. Rubinstein.  Theory of Games and Economic Behavior: 60th Anniversary Commemorative Edition.  Princeton Classic Editions. Princeton University Press, 2007.
  * [145] Joseph Walker.  Meet the New Boss: Big Data, September 2012.  https://www.wsj.com/articles/SB10000872396390443890304578006252019616768.
  * [146] Thomas E. Weisskopf.  Impact of Reservation on Admissions to Higher Education in India.  Economic and Political Weekly, 39(39):4339–4349, 2004.
  * [147] Christine Wennerås and Agnes Wold.  Nepotism and Sexism in Peer-Review.  Nature, 387(6631):341–343, May 1997.
  * [148] Wikipedia contributors.  Maximum entropy probability distribution — Wikipedia, the free encyclopedia.  https://en.wikipedia.org/w/index.php?title=Maximum_entropy_probability_distribution&oldid=1154992719, 2023.  [Online; accessed 16-May-2023].
  * [149] Joan C Williams.  Double jeopardy? an empirical study with implications for the debates over implicit bias and intersectionality.  Harvard Journal of Law & Gender, 37:185, 2014.
  * [150] Sarah Wood.  Colleges With The Highest Application Fees, January 2022.  https://www.usnews.com/education/best-colleges/the-short-list-college/articles/colleges-with-the-highest-application-fees.
  * [151] Ruixun Zhang, Thomas J Brennan, and Andrew W Lo.  The origin of risk aversion.  Proceedings of the National Academy of Sciences, 111(50):17777–17782, 2014.
  * [152] Jonathan C. Ziegert and Paul J. Hanges.  Employment discrimination: The role of implicit attitudes, motivation, and a climate for racial bias.  Journal of Applied Psychology, 90(3):553–562, 2005.

##  Appendix A Other related work

##### Models of bias in Economics.

There are two prominent models of discrimination in the Economics literature:
taste-based discrimination and statistical discrimination [114]. These capture
different types of biases [117, 45, 49, 12, 22, 41, 11] (also see [66, 25,
114]). Taste-based discrimination [22] models explicit biases (e.g., racism
and sexism) and, in the vanilla taste-based discrimination model, individuals
are divided into two groups and a decision-maker pays an additional cost for
interacting with individuals in the disadvantaged group. This additional
additive cost diminishes the value of disadvantaged individuals for the
decision-maker. While we do not model explicit biases, such an additive bias
also arises in our model of evaluation processes with specific parameter
choices, suggesting that additive biases may also arise due to resource
constraints and risk averseness (Section 6). Statistical discrimination models
how group-wise disparities in the noise in the inputs to a Bayesian decision-
maker propagate to systematic disparities in decisions [117, 11]. Our
mathematical model can be viewed as giving an explanation of why such
disparities may arise in the input.

##### Implicit biases in Psychology.

There is a long and rich history of the study of implicit (and explicit)
biases in Psychology, e.g., [85, 101, 74, 104, 69, 127, 149]. This body of
works proposes various theories about why implicit biases arise [69] and their
relation to real-world stimuli [130, 63, 42]. We refer the reader to [73, 87]
for an overview. [69] explains that the ease of categorizing individuals into
categories (defined by, e.g., color, race, or gender) provides an incentive to
the evaluator to use their prior (possibly biased) knowledge, and this leads
to implicit biases. In contrast, we show that even when the evaluator has the
same prior estimate for all social groups, biases can arise in evaluation
processes when the information-to-resource trade-off or the degree of risk
averseness is different for different groups. Further, since resource
constraints and risk averseness are not specific to the setting of a single
evaluator, our model of evaluation processes also models scenarios where the
evaluator denotes a group or an organization.

##### Optimization and human behavior.

The use of optimization to model human behavior dates back to (at least) von
Neumann and Morgenstern’s work that showed that, under a small number of
assumptions, the behavior of an evaluator is as if they are optimizing the
expected value of a “utility function” [144]. Since then, numerous apparent
deviations from this theory were discovered [84]. These deviations, broadly
referred to as irrational behavior or cognitive biases, laid the foundation of
Behavioral Economics [138]. Several theories have been proposed to account for
the deviations in human behavior from utility maximization. Including prospect
theory that models risk averseness of individuals – the empirical observation
that humans process losses and gains of equal amounts (monetary or otherwise)
asymmetrically [84] – bounded rationality that reconciles irrational human
behavior by proposing that humans solve underlying optimization problems
approximately (instead of optimally) [132], and resource rational analysis
that proposes that humans trade-off the utility of with the costs (e.g., such
as effort and time) required to find a solution with higher utility [100].
These works model hidden costs and constraints on humans that lead to
deviations from the traditional “rational” utility maximization. Like this
work, these works also use optimization to explain human behavior, but while
they focus on irrational behaviors and cognitive biases, our work focuses on
biases with respect to socially-salient attributes in evaluation processes.

##### Other entropy-based models.

Maximum entropy distributions have been widely deployed in machine learning
[56] and theoretical computer science [133]. Maximum entropy distributions
have been shown to be “stable” [136]. The maximum-entropy framework over
discrete distributions has been used to preprocess data to debias it [39]. In
our optimization program, we use entropy to measure the amount of
“information” in a distribution — this appears as a constraint in our program.
An entropy-constrained viewpoint is also prevalent in explanations of other
phenomena. For instance, various optimization algorithms can be captured using
an optimization- and entropy-based viewpoint [9, 96, 143], it is also known to
arise in explanations of biological phenomena [43], and leads to Page rank and
other popular random-walk-based procedures [105]. Finally, taking an
optimization viewpoint when studying a dynamical system also has additional
benefits, such as providing a “potential function” that functions that gives
an efficient and interpretable method of tracking the progress of a complex
dynamical system [137].

##  Appendix B Connection to the Gibbs equation

######  Theorem B.1 (Gibbs equation).

Consider an instance $\mathcal{I}=(\Omega,{f_{\mathcal{D}}},\ell,\alpha,\tau)$
of the optimization problem PrimalOpt that satisfies the assumptions of
Theorem 6.1. Then, the following holds:

$\displaystyle-\gamma^{\star}\ln
Z^{\star}=\mathrm{Err}_{\ell,\alpha}(f^{\star},{f_{\mathcal{D}}})+\gamma^{\star}\mathsf{Ent}(f^{\star}).$
(13)

Here, $f^{\star}(x)\propto e^{-\frac{I(x)}{\gamma^{\star}}}$ is the solution
to PrimalOpt, $Z^{\star}\coloneqq\int
e^{-\frac{I(x)}{\gamma^{\star}}}d\mu(x)$, and $\gamma^{\star}>0$ is the
solution to DualOpt. Recall that
$I(x)\coloneqq\int_{\Omega}\ell_{\alpha}(x,v){f_{\mathcal{D}}}(v)d\mu(v)$.

###### Proof.

Theorem 6.1 implies that there exists $f^{\star}(x)$ and $\gamma^{\star}>0$
such that

$f^{\star}(x)\propto e^{-\frac{I(x)}{\gamma^{\star}}}.$

Thus, if we let $Z^{\star}\coloneqq\int
e^{-\frac{I(x)}{\gamma^{\star}}}d\mu(x)$, then

$f^{\star}(x)=\frac{e^{-\frac{I(x)}{\gamma^{\star}}}}{Z^{\star}}.$

Thus, from the optimality condition (12), we obtain that there is a
$\phi^{\star}$ such that

$\displaystyle I(x)+\gamma^{\star}(1+\ln f^{\star}(x))+\phi^{\star}=0.$

Since $\gamma^{\star}>0$, we can divide by it to obtain
$Z^{\star}=e^{1+\frac{\phi^{\star}}{\gamma^{\star}}}$. We integrate the above
with respect to the density $f^{\star}(x)$ to get

$\displaystyle\mathrm{Err}_{\ell,\alpha}(f^{\star},{f_{\mathcal{D}}})+\gamma^{\star}-\gamma^{\star}\tau+\phi^{\star}=0.$

Thus, we obtain:

$\displaystyle\gamma^{\star}\tau={\gamma^{\star}}\mathsf{Ent}(f^{\star})=\mathrm{Err}_{\ell,\alpha}(f^{\star},{f_{\mathcal{D}}})+{\gamma^{\star}}\ln
Z^{\star}.$ (14)

Rearranging this equation we obtain the theorem. ∎

In analogy with the Gibbs equation in statistical physics [94], $Z^{\star}$
can be viewed as the partition function corresponding to the energy function
$I(x)$, $\gamma^{\star}$ corresponds to the temperature, $-{\gamma^{\star}}\ln
Z^{\star}$ corresponds to the free energy and
$\mathrm{Err}(f^{\star},{f_{\mathcal{D}}})$ is the internal energy.

It follows from the theorem that we can write $f^{\star}(x)$ as

$f^{\star}(x)=e^{-\tau}\exp{\left(-\frac{I(x)-\mathrm{Err}_{\ell,\alpha}(f^{\star},{f_{\mathcal{D}}})}{\gamma^{\star}}\right)}.$
(15)

##  Appendix C Exponential density

The exponential density is defined as follows over $\Omega=[0,\infty)$ and has
a parameter $\lambda>0$:

$f_{\rm Exp}(x)\coloneqq\lambda e^{-\lambda x},\quad x\in[0,\infty).$

$\lambda$ is referred to as the “rate” parameter. The mean of the exponential
density is $\frac{1}{\lambda}$. The differential entropy of $f_{\rm Exp}$ is
$1-\ln\lambda$ [148]. We consider the loss function $\ell(x,v)\coloneqq x-v$.
First, we compute the expression of $I(x)$ which we use to verify the
applicability of Theorem 6.1 with the above parameters.

######  Lemma C.1 (Expression for $I(x)$).

Consider an instance $\mathcal{I}=(\Omega,f_{\rm Exp},\ell,\alpha,\tau)$ of
PrimalOpt where $\Omega=[0,\infty)$, $f_{\rm Exp}$ is the Exponential density
with rate parameter ${\lambda}$, and $\ell(x,v)\coloneqq x-v$. Then

$I(x)=\frac{1}{\lambda}\left(\alpha(\lambda x-1)+(\alpha-1)e^{-\lambda
x}\right).$

###### Proof.

The desired integral is

$\lambda\alpha\int_{0}^{x}(x-v)e^{-\lambda
v}d\mu(v)+\lambda\int_{x}^{\infty}(x-v)e^{-\lambda v}d\mu(v).$

We do a change of variable, with $y\coloneqq x-v$ above, to get

$\displaystyle\lambda\alpha\int_{0}^{x}ye^{\lambda(y-x)}d\mu(y)+\lambda\int_{-\infty}^{0}ye^{\lambda(y-x)}d\mu(y)$
$\displaystyle=\lambda\alpha e^{-\lambda x}\left[e^{\lambda y}\frac{\lambda
y-1}{\lambda^{2}}\right]_{0}^{x}+\lambda e^{-\lambda x}\left[e^{\lambda
y}\frac{\lambda y-1}{\lambda^{2}}\right]_{-\infty}^{0}$
$\displaystyle=\frac{\alpha(\lambda x-1)}{\lambda}+\frac{\alpha e^{-\lambda
x}}{\lambda}-\frac{e^{-\lambda x}}{\lambda}$
$\displaystyle=\frac{1}{\lambda}\left(\alpha(\lambda
x-1)+(\alpha-1)e^{-\lambda x}\right).$

∎

##### Applicability of Theorem 6.1.

We show that assumptions (A0)–(A5) hold. Since $\Omega=[0,\infty)$, (A0) holds
for any finite $\tau$. (A1) and (A2) hold due to the choice of the loss
function. (A3) can be shown to hold since $f_{\mathrm{Exp}}$ is an Exponential
density: To see this note that for any finite $x\in\Omega$

$\displaystyle\int_{\Omega}\left|\ell(x,v)\right|f_{\mathrm{Exp}}(v)d\mu(v)$
$\displaystyle=\int_{1}^{x}\left(x-v\right)f_{\mathrm{Exp}}(v)d\mu(v)+\int_{x}^{\infty}\left(v-x\right)f_{\mathrm{Exp}}(v)d\mu(v)$
$\displaystyle\leq\int_{1}^{\infty}\left(\left|x\right|+\left|v\right|\right)f_{\mathrm{Exp}}(v)d\mu(v)$
$\displaystyle=\left|x\right|+\frac{1}{\lambda}$ $\displaystyle<\infty.$

Thus, (A3) holds. (A4) holds with, e.g., $R=\frac{1}{\lambda}$. By Lemma C.1,

$I(x)=\frac{1}{\lambda}\left(\alpha(\lambda x-1)+(\alpha-1)e^{-\lambda
x}\right).$

From this expression, it follows that $I(x)$ is differentiable at each
$x\in\Omega$. Moreover, for all $x\in\Omega$

$\frac{\partial^{2}I(x)}{\partial^{2}x}=\lambda(\alpha-1)e^{-\lambda x}.$

For any $\alpha>1$, this derivative is positive for all $x\in\Omega$ and,
hence, $I(x)$ strictly convex whenever $\alpha>1$ and, thus, it has a unique
global minimum. If $\alpha=1$, then $I(x)=x-\frac{1}{\lambda}$. This function
has a unique global minimum at $x=0$ over $\Omega=[0,\infty)$. Combining with
the $\alpha>1$ case, it follows that (A5) holds. Since assumptions (A0)–(A5)
hold, one can invoke Theorem 6.1 to deduce the form of $f^{\star}$.

######  Theorem C.2 (Expression for $f^{\star}$ when $\alpha=1$).

Consider an instance $\mathcal{I}=(\Omega,f_{\mathrm{Exp}},\ell,1,\tau)$ of
PrimalOpt where $\Omega=[0,\infty)$, $\ell(x,v)\coloneqq x-v$, and
$f_{\mathrm{Exp}}$ is the Exponential with rate parameter ${\lambda}$. Let
$f^{\star}$ be the optimal solution of $\mathcal{I}$. Then $f^{\star}$ is the
Exponential density with mean $e^{\tau-1}$

Thus, for $\alpha=1$, increasing $\tau$ increases the rate parameter of the
output density.

###### Proof.

Since $\alpha=1$, Lemma C.1 implies that

$\displaystyle I(x)$ $\displaystyle=x-\frac{1}{\lambda}.$

As shown earlier, one can invoke Theorem 6.1 for instance $\mathcal{I}$ for
any finite $\tau$, which implies that $f^{\star}$ has the following form

$f^{\star}(x)\propto\exp\left(\frac{-x-\frac{1}{\lambda}}{\gamma^{\star}}\right)\propto\exp\left(\frac{-x}{\gamma^{\star}}\right).$

where the proportionality constant and $\gamma^{\star}$ are determined by
$\int_{\Omega}f^{\star}(x)d\mu(x)=1$ and $\mathsf{Ent}(f^{\star})=\tau$. Since
$f^{\star}$ is an Exponential density with rate parameter
$\frac{1}{\gamma^{\star}}$, its entropy is
$\mathsf{Ent}(f^{\star})=1+\ln\left(\gamma^{\star}\right)$ [148]. Since
$\mathsf{Ent}(f^{\star})=\tau$, the previous equality implies that
$\gamma^{\star}=e^{\tau-1}$. It follows that
$f^{\star}(x)=e^{1-\tau}\cdot\exp\left(-\frac{x}{\exp\left(\tau-1\right)}\right)$
which is the Exponential density with mean $e^{\tau-1}$. ∎

##  Appendix D Laplace density

The Laplace density is defined as follows over $\Omega=\mathbb{R}$ and has
parameters $a\in\mathbb{R}$ and $b>0$:

$f_{\mathcal{L}}(x)\coloneqq\frac{1}{2b}e^{-\frac{1}{b}\left|x-a\right|},\quad
x\in\mathbb{R}.$

$a$ is referred to as the “location” parameter and $b$ as “diversity.” The
differential entropy of $f_{\mathcal{L}}$ is $1+\ln(2b)$ [148]. We consider
the loss function to be $\ell(x,v)\coloneqq\left|x-a\right|-\left|v-a\right|$
for $a\in\mathbb{R}$. First, we compute the expression for $I(x)$ that we use
to show that Theorem 6.1 is applicable with the above parameters.

######  Lemma D.1 (Expression for $I(x)$).

Consider an instance $\mathcal{I}=(\Omega,f_{\mathcal{L}},\ell,\alpha,\tau)$
of PrimalOpt where $\Omega=\mathbb{R}$, $f_{\mathcal{L}}$ is the Laplace
density with parameters $a\in\mathbb{R}$ and $b>0$, and
$\ell(x,v)\coloneqq\left|x-a\right|-\left|v-a\right|$. Then

$I(x)=\begin{cases}\alpha b(w-1)+b\frac{(\alpha-1)}{2}e^{-w}&\text{if $x\geq
a$}\\\ -b(w+1)+b\frac{(1-\alpha)}{2}e^{w}&\text{otherwise}\end{cases},$

where $w\coloneqq\frac{x-a}{b}$.

###### Proof.

The desired integral is

$\frac{\alpha}{2b}\int_{-\infty}^{x}(|x-a|-|v-a|)e^{-|v-a|/b}d\mu(v)+\frac{1}{2b}\int_{x}^{\infty}(|x-a|-|v-a|)e^{-|v-a|/b}d\mu(v).$

We perform a change of variables with $w\coloneqq\frac{x-a}{b}$ and
$y=\frac{v-a}{b}$. Then the above integral becomes

$\frac{\alpha
b}{2}\int_{-\infty}^{w}(|w|-|y|)e^{-|y|}d\mu(y)+\frac{b}{2}\int_{w}^{\infty}(|w|-|y|)e^{-|y|}d\mu(y).$

Now two cases arise (i) $w\geq 0$, or (ii) $w\leq 0$. First, consider the case
when $w\geq 0$. Then the above integral becomes:

$\displaystyle\frac{\alpha}{2}\int_{-\infty}^{0}(w+y)e^{y}d\mu(y)+\frac{\alpha}{2}\int_{0}^{w}(w-y)e^{-y}d\mu(y)+\frac{1}{2}\int_{w}^{\infty}(w-y)e^{-y}d\mu(y)$
$\displaystyle=\frac{\alpha(w-1)}{2}+\frac{\alpha(w+e^{-w}-1)}{2}-\frac{e^{-w}}{2}$
$\displaystyle=\alpha b(w-1)+\frac{(\alpha-1)b}{2}e^{-w}.$

In the second case, we get

$\displaystyle\frac{\alpha}{2}\int_{-\infty}^{w}(-w+y)e^{y}d\mu(y)+\frac{1}{2}\int_{w}^{0}(-w+y)e^{y}d\mu(y)+\frac{1}{2}\int_{0}^{\infty}(-w-y)e^{-y}d\mu(y)$
$\displaystyle=-\frac{\alpha e^{w}}{2}+\frac{e^{w}-w-1}{2}-\frac{1+w}{2}$
$\displaystyle=-(w+1)b+\frac{(1-\alpha)e^{w}b}{2}.$

∎

##### Applicability of Theorem 6.1.

We show that for any finite $a\in\mathbb{R}$ and $b>0$, assumptions (A0)–(A5)
hold for the instance $\mathcal{I}=(\Omega,f_{\mathcal{L}},\ell,\alpha,\tau)$
of PrimalOpt where $\Omega=\mathbb{R}$, $f_{\mathcal{L}}$ is the Laplace
density with parameters $a\in\mathbb{R}$ and $b>0$, and
$\ell(x,v)=\left|x-a\right|-\left|v-a\right|$. Since $\Omega=\mathbb{R}$, (A0)
holds for any finite $\tau$. (A1) and (A2) hold due to the choice of the loss
function. (A3) can be shown to hold since $f_{\mathcal{L}}$ is a Laplace
density: To see this note that for any finite $x\in\Omega$

$\displaystyle\int_{\Omega}\left|\ell(x,v)\right|f_{\mathcal{L}}(v)d\mu(v)$
$\displaystyle\leq\int_{0}^{\infty}\left(\left|x-a\right|+\left|v-a\right|\right)f_{\mathcal{L}}(v)d\mu(v)$
$\displaystyle=\left|x-a\right|+\frac{b}{2}$ $\displaystyle<\infty.$

Thus, (A3) holds. (A4) holds with, e.g., $R=b\ln(2)$. By Lemma D.1,

$I(x)=\begin{cases}\alpha{b}(w-1)+{b}\frac{(\alpha-1)}{2}e^{-w}&\text{if
$x\geq a$}\\\
-{b}(w+1)+{b}\frac{(1-\alpha)}{2}e^{w}&\text{otherwise}\end{cases},$

where $w=\frac{x-a}{b}$. One can check that $I(x)$ is continuous and
differentiable at each $x\in\Omega\setminus\left\\{a\right\\}$. Moreover, for
all $x<a$, $\frac{\partial I(x)}{\partial x}<0$ and for all $x\geq a$,
$\frac{\partial I(x)}{\partial x}\geq 0$. Hence, it follows that $I(x)$ has a
unique global minimum at $x=a$. Therefore, (A5) holds. Since assumptions
(A0)–(A5) hold, we invoke Theorem 6.1 to deduce the form of the optimal
density.

######  Theorem D.2 (Expression for $f^{\star}$ when $\alpha=1$).

Consider an instance $\mathcal{I}=(\Omega,f_{\mathcal{L}},\ell,1,\tau)$ of
PrimalOpt where $\Omega=\mathbb{R}$, ${f_{\mathcal{D}}}$ is the Laplace
density with parameters $a\in\mathbb{R}$ and $b>0$, and
$\ell(x,v)=\left|x-a\right|-\left|v-a\right|$. Let $f^{\star}$ be the optimal
solution of $\mathcal{I}$. Then $f^{\star}$ is the Laplace density with
parameters $(a,e^{\tau-1}/2)$.

Thus, for $\alpha=1$, increasing $\tau$ does not change the location
parameter, but increases the “diversity” parameter of the output density.

###### Proof.

Since $\alpha=1$, Lemma D.1 implies that

$\displaystyle I(x)$ $\displaystyle=\left|x-a\right|-\frac{b}{2}.$

As shown earlier, one can invoke Theorem 6.1 for instance $\mathcal{I}$ for
any finite $\tau$, which implies that $f^{\star}$ has the following form

$f^{\star}(x)\propto\exp\left(\frac{-\left|x-a\right|-\frac{b}{2}}{\gamma^{\star}}\right)\propto\exp\left(\frac{-\left|x-a\right|}{\gamma^{\star}}\right).$

where the proportionality constant and $\gamma^{\star}$ are determined by
$\int_{\mathbb{R}}f^{\star}(x)d\mu(x)=1$ and $\mathsf{Ent}(f^{\star})=\tau$.
Clearly, $f^{\star}$ is a Laplace density with the diversity parameter
$\gamma^{\star}$, its entropy is
$\mathsf{Ent}(f^{\star})=1+\ln\left(2\gamma^{\star}\right)$ [148]. On the
other hand, since $\mathsf{Ent}(f^{\star})=\tau$, the previous equality
implies that $\gamma^{\star}=\frac{1}{2}e^{\tau-1}$. It follows that
$f^{\star}(x)=e^{1-\tau}\cdot\exp\left(-\frac{2\left|x-a\right|}{\exp\left(\tau-1\right)}\right)$
which is the Laplace density with parameters $(a,\frac{1}{2}e^{\tau-1})$. ∎

##  Appendix E Implementation details and additional empirical results

In this section, we present additional discussions and evaluations of
intervention in the JEE setting (Section E.1), plots omitted from Section 5
(Section E.2), and implementation details of our model (Section E.3). The code
for this paper is available at https://github.com/AnayMehrotra/Bias-in-
Evaluation-Processes.

###  E.1 Case Study: Evaluating bias-mitigating interventions in IIT-JEE
admissions

In this section, we continue our study of the effectiveness of different
interventions in a downstream selection task. Like in Section 5, we consider
selection based on the JEE 2009 scores, but here consider representational
constraints actually used in admissions to IITs. We also discuss additional
interventions being implemented by the Indian state and central governments to
reduce inequity in JEE scores.

Recall that, the Indian Institutes of Technology (IITs) are a group of
engineering institutes in India. In 2009, there were 15 IITs and today this
has grown to 23. Undergraduate admissions at IITs are decided based on the
scores of candidates in the Joint Entrance Exam (JEE). JEE is conducted once
every year. In 2009, the scores, (binary) genders, and birth categories of all
candidates who appeared in JEE 2009 were released in response to a Right to
Information application filed in June 2009 [91]. The birth category of the
candidates is an official socioeconomic status label recognized by the
government of India [135].

Here, we focus on two groups of candidates: the candidates in the general
(GEN) category (the most privileged) and candidates not in the general
category. We begin by discussing some of the interventions in place to reduce
inequity in JEE scores and subsequent admissions at IITs.

Interventions used in IIT admissions. The Indian constitution allows the
central government and state governments to enforce affirmative action in the
form of quotas or lower bound constraints for official SES groups at
educational institutes, employments, and political bodies [82, 146]. In 2005
lower-bound interventions were introduced in the admissions process at the
IITs. Concretely, in 2009, out of the 7,440 seats, 3,688 (49.6%) were reserved
for students who are not in the GEN category. This means that at least 3,688
out of the $7,440$ students admitted into IITs must not be in the GEN
category. Note that this allows more than 3,688 or even all admitted students
to be outside the GEN category. We call this constraint the Reservation
constraint and, in this section, we study its effectiveness compared to other
forms of interventions.

Apart from reservations, a number of other interventions have also been
proposed and/or implemented to reduce biases in the JEE. We discuss two other
types of interventions next.

Interventions to reduce skew. Private coaching institutes that train students
for JEE have been criticized for being exorbitantly expensive and, hence,
inaccessible for students in low SES groups [44]. Lack of accessibility to
training resources can reduce the scores of candidates in low SES
groups–creating a skew in the scores. To improve accessibility to training, in
2022, the Delhi government established a new program that will provide free
training to students enrolled in government-funded schools [53]. Similar
programs have also been introduced in other states [52, 141] and by school
education boards that span multiple states [26]. In the context of our model,
these interventions can be seen as reducing this skew in the evaluation
process.

Interventions to reduce information constraint. A criticism of JEE is that it
is only offered in the English and Hindi languages. This is undesirable
because only 44% of Indians report English or Hindi as their first language
and, according to the 2011 census, less than 68% of Indians list one of these
languages among the three languages they are most comfortable with [103]. IITs
have been repeatedly criticized for not offering the exam in regional
languages [128, 7, 47]. The main concern is that the current exam reduces the
performance of students less familiar with English and Hindi. In the context
of our model, this can be thought of as placing a stronger information
constraint on candidates who do not speak English or Hindi as a first
language: these students would need to spend a higher cognitive load to
understand the questions. This constraint not only acts during the exam but
also during the preparation period because students speaking regional
languages (and not English or Hindi), have to devote additional time to
learning either English or Hindi in addition to the technical material for the
exam.

While the JEE exam itself has not been offered in regional languages yet.
Recently, in 2021, the screening test that candidates have to clear before
appearing in JEE was offered in 11 regional languages in addition to English
and Hindi [134].

In this section, we compare the effectiveness of the above three interventions
– Reservation for lower SES groups, interventions to reduce skew (change
$\alpha$ by $\Delta_{\alpha}$ percent), and interventions to reduce
information-constraint (change $\tau$ by $\Delta_{\tau}$ percent).

(a) Changing $\alpha$ by $\Delta_{\alpha}$ percent (b) Changing $\tau$ by
$\Delta_{\tau}$ percent

Figure 9:  Effectiveness of different interventions on the selection-
utility–as estimated by our model: We vary the strengths of the interventions
($\Delta_{\alpha}\in[0,1]$ and $\Delta_{\tau}\in[0,1]$) and report the
expected utilities of the subset output by all three interventions. The
$x$-axis shows the strength of the intervention changing $\alpha$ (Figure
9(a)) or $\tau$ (Figure 9(b)). The $y$-axis shows the ratio of the (true)
utility of the subset output with an intervention to the (true) utility of the
subset output without any intervention. Our main observation is that for each
of the three interventions, there is a value of the percentage change in
$\alpha$ and $\tau$ (i.e., $\Delta_{\alpha}$ and $\Delta_{\tau}$ respectively)
for which the intervention outperforms the other two interventions. Hence,
depending on the amount of change a policymaker expects a specific
intervention (e.g., providing free coaching) to have on the parameters
$\alpha$ and $\tau$, they can use our framework as a tool to inform their
decision about which intervention to enforce. Error bars represent the
standard error of the mean over 100 repetitions.  Figure 10:  The densities of
scores of students in GEN category (blue) and students not in GEN category
(orange) in JEE-2009–only for students scoring at least 80 points out of 480.

Setup (Group sizes and $k$). Admissions into IITs are highly selective. For
instance, in 2009, 384,977 applicants (232,334 from GEN; 60%) took the exam
and just 7,440 (2%) were admitted to IITs. The admission is based on the
candidates’ All India Rank (henceforth just rank)–which denotes the
candidate’s position in the list of candidates ordered in decreasing order of
their scores in JEE. Let $G_{1}$ be the group of students in GEN category and
$G_{2}$ be all other students. To study the impact of different interventions
for admissions into IITs, we fix group sizes and $k$ to match real numbers:
$\left|G_{1}\right|=232,334$, $\left|G_{2}\right|=152,643$, and $k=7,400$. We
focus on the set of candidates who scored at least 80 (out of 480) on the
exam. (The threshold 80 ensures that at least 10k candidates outside the GEN
category are considered and this is significantly lower than the $k$-th
highest score of 167). We fix $f_{\mathcal{D}}$ to be the density of utilities
of all candidates in $G_{1}$ who scored at least 80. Since $f_{\mathcal{D}}$
has a Pareto-like density (see Figure 10), we fix $\ell(x,v)=\ln(x)-\ln(v)$.
We fix $\Omega$ to be the set of all possible scores and $f_{G_{2}}$ to be the
density of all candidates in $G_{2}$ who scored at least 80. As in Section 5,
we select $\alpha$ and $\tau$ that lead to the density closest in TV distance
to $f_{G_{2}}$. The rest of the setup is the same as in Section 5.

Unlike the main body, here, we only consider high-scoring candidates (those
with a score of at least 80) because JEE is highly selective ($k/n\leq 0.02$)
and, hence, to have meaningful results the estimated density $f_{\mathcal{E}}$
should have a good fit to the density from the real-data on the top 2%
quantile, i.e., the right tail. To ensure this, we specifically consider the
right tail of the distribution (by dropping candidates with a score below 80).

Observations and discussion. We vary $\Delta_{\alpha}\in[0,1]$ and
$\Delta_{\tau}\in[0,1]$ and report the expected utilities of the subset output
by all three interventions over 100 iterations in Figure 9. Our main
observation is that for each of the three interventions, there is a value of
the percentage change in $\alpha$ and $\tau$ (i.e., $\Delta_{\alpha}$ and
$\Delta_{\tau}$ respectively) for which the intervention outperforms the other
two interventions. Hence, depending on the amount of change a policymaker
expects a specific intervention (e.g., providing free coaching) to have on the
parameters $\alpha$ and $\tau$, they can use our framework as a tool to inform
their decision about which intervention to enforce. Further, we observe that,
as expected, increasing $\Delta_{\alpha}$ and $\Delta_{\tau}$, i.e., the
percentage of change in $\alpha$ and $\tau$, improves the utility achieved by
the corresponding interventions.

Limitations and further discussion. Next, we discuss some of the limitations
of our study. First, we note that interventions such as increasing the
accessibility of education can not only reduce inequity in JEE but can also
have positive effects on other exams and hiring. Hence, such interventions can
have a larger positive (or negative) impact than suggested by our simulations.
Studying these auxiliary effects is beyond the scope of this paper. Further,
our study also does not model the response of the students, e.g., how do
interventions affect the students’ incentive to invest in skill development?
Finally, our model only predicts the effect of $\alpha$ and $\tau$ on utility
distributions. These predictions may not be accurate and a careful post-
deployment evaluation may be required to accurately assess the effectiveness
of different interventions.

###  E.2 Additional plots for simulations in Section 5

In this section, we present plots of the best-fit densities output by our
framework on different datasets.

(a) Best-fit distribution ($\alpha=3\cdot 10^{-4}$ and $\tau=1.51$) with
JEE-2009 (Birth category)

(b) Best-fit distribution ($\alpha=3\cdot 10^{-4}$ and $\tau=1.51$) with
JEE-2009 (Gender)

Figure 11:  Illustration of the best-fit distribution output by our framework
for the JEE-2009 dataset. Captions of subfigures report the best fit $\alpha$
and $\tau$.

(a) Best-fit distribution ($\alpha=22.78$ and $\tau=2.09$) with Semantic
Scholar Open Research Corpus

(b) Best-fit distribution ($\alpha=1.92$ and $\tau=3.19$) with synthetic
network data

Figure 12:  Illustration of the best-fit distribution output by our framework
for the Semantic Scholar Open Research Corpus. Captions of subfigures report
the best fit $\alpha$ and $\tau$.

###  E.3 Implementation details

####  E.3.1 Our framework and other models

In this section, we give implementation details of our model. Recall that our
model outputs the density which is the optimal solution of the following
optimization program.

$\displaystyle\textstyle\operatornamewithlimits{argmin}_{\text{$f$: density on
$\Omega$}}\ \ $
$\displaystyle\textstyle\mathrm{Err}_{\ell,\alpha}\left({f_{\mathcal{D}}},f\right)\coloneqq\int_{\Omega}\left[\int_{\Omega}\ell_{\alpha}(x,v)f(x)d\mu(x)\right]{f_{\mathcal{D}}}(v)d\mu(v)\textstyle,\hypertarget{prog:framework:appendix}{}$
(OptProg-App) such that
$\displaystyle\quad\textstyle-\int_{\Omega}f(x)\log{f(x)}d\mu(x)\geq\textstyle\tau.$

An instance of this program is specified by the following parameters.

  1. 1.

A domain $\Omega\subseteq\mathbb{R}$ (e.g., $\Omega=\mathbb{R}$ and
$\Omega=[1,\infty)$);

  2. 2.

A true density $f_{\mathcal{D}}$ over $\Omega$ with respect to the Lebesgue
measure $\mu$;

  3. 3.

A loss function $\ell\colon\Omega\times\Omega\to\mathbb{R}$ (e.g.,
$\ell(x,v)=(x-v)^{2}$ and $\ell(x,v)=\ln\left(x/v\right)$);

  4. 4.

A risk-averseness (or risk-eagerness) parameter $\alpha>0$; and

  5. 5.

A resource-information parameter $\tau>0$.

Recall that $\ell_{\alpha}$ is a risk-averse loss defined by $\ell$ and
$\alpha$ as in (4). For our simulations, we consider the shifted variant of
$\ell_{\alpha}$ mentioned in Section 2: given a shift parameter
$v_{0}\in\mathbb{R}$, a loss function
$\ell\colon\Omega\times\Omega\to\mathbb{R}$, and parameter $\alpha>0$

$\ell_{\alpha,v_{0}}(x,v)=\begin{cases}\alpha\cdot\ell(x,v+v_{0})&\text{if
}x>v+v_{0},\\\ \ell(x,v+v_{0})&\text{otherwise.}\end{cases}$

Let $f^{\star}_{\alpha,\tau,v_{0}}$ be the optimal solution to the instance
$\mathcal{I}_{\alpha,\tau,v_{0}}=\left(\Omega,f_{\mathcal{D}},\ell,\alpha,\tau,v_{0}\right)$
of (LABEL:prog:framework:appendix).

Algorithmic task. Given a “target” density $f_{\mathcal{T}}$ (denoting the
density of biased utilities in the data), risk-averse loss function
$\ell_{\alpha}$, and true density $f_{\mathcal{D}}$, the goal of our
implementation is to find $\alpha^{\circ}$, $\tau^{\circ}$, and
$v_{0}^{\circ}$ that minimize the total variation distance between
$f_{\mathcal{T}}$ and $f^{\star}_{\alpha,\tau,v_{0}}$:

$(\alpha^{\circ},\tau^{\circ},v_{0}^{\circ})\coloneqq\operatornamewithlimits{argmin}_{\alpha,\tau,v_{0}}d_{\rm
TV}(f^{\star}_{\alpha,\tau,v_{0}},f_{\mathcal{T}}).$

Algorithmic approach and implementation. We perform grid-search over all three
parameters $\alpha,\tau,$ and $v_{0}$. Given a specific $\alpha,\tau,$ and
$v_{0}$, to solve the above problem, we use the characterization in Theorem
3.1 to find $f^{\star}_{\alpha,\tau,v_{0}}$. Recall that the optimal solution
of (LABEL:prog:framework:appendix) is of the following form

$f^{\star}_{\alpha,\tau,v_{0}}(x)=C\cdot\exp\left(-I_{\alpha,v_{0}}(x)/{\gamma^{\star}}\right)$

where
$I_{\alpha,v_{0}}(x)\coloneqq\int_{\Omega}\ell_{\alpha,v_{0}}(x,v)f_{\mathcal{D}}(x)d\mu(x)$
and $C,\gamma^{\star}>0$ are constants that are uniquely specified by the
following two equations

$\int_{\Omega}f^{\star}_{\alpha,\tau,v_{0}}(x)d\mu(x)=1\quad\text{and}\quad-\int_{\Omega}f^{\star}_{\alpha,\tau,v_{0}}(x)\log\left(f^{\star}_{\alpha,\tau,v_{0}}(x)\right)d\mu(x)=\tau.$

Algorithmically, finding $C$ and $\gamma^{\star}$ requires computing a double
integral over $\Omega$. In all of the simulations in Section 5, $\Omega$ is a
discrete domain, so these integrals reduce to summations and we compute them
exactly. We also provide an implementation of our algorithm for continuous
domains. The implementation for continuous domains uses the quad function in
scipy to compute the integrals. For the grid search itself, we varied $\alpha$
over $[10^{-4},10^{2}]$, $\tau$ over $[10^{-1},10]$, and $v_{0}$ over
$\Omega$. We found this range to be sufficient for our simulation, but it
would be interesting to design a principled way of specifying the ranges given
other parameters and target density $f_{\mathcal{T}}$.

##### Implementation details of multiplicative bias model [90] and implicit
variance model [61].

Recall that the multiplicative bias and the implicit variance models are
specified by parameters $\rho$ and $\sigma$ respectively: given a fixed true
value $v\in\mathbb{R}$, the output of the multiplicative bias model is
$v/\rho$ and the output of the implicit variance model is $v+\zeta$ where
$\zeta$ is a zero-mean normal random variable with variance $\sigma^{2}$. In
addition, we allow both models to introduce a shift $v_{0}$. For the
multiplicative bias model, given a true density $f_{\mathcal{D}}$ and a target
density $f_{\mathcal{T}}$, we compute $(\rho^{\circ},v_{0}^{\circ})$ that
solves $\operatornamewithlimits{argmin}_{\rho,v_{0}}d_{\rm
TV}(f_{\rho,v_{0}},f_{\mathcal{T}})$ where $f_{\rho,v_{0}}$ is the density of
$(v/\rho)+v_{0}$ for $v\sim f_{\mathcal{D}}$. For the implicit variance model,
given a true density $f_{\mathcal{D}}$ and a target density $f_{\mathcal{T}}$,
we compute $(\sigma,v_{0})$ that solves
$\operatornamewithlimits{argmin}_{\sigma,v_{0}}d_{\rm
TV}(f_{\sigma,v_{0}},f_{\mathcal{T}})$ where $f_{\sigma,v_{0}}$ is the density
of $v+v_{0}+\zeta$ for $v\sim f_{\mathcal{D}}$ and a zero-mean normal random
variable $\zeta$ with variance $\sigma^{2}$. For both models, we compute the
optimal parameters using grid search: we vary $v_{0}$ over $\Omega$,
$(1/\rho)$ over $[0,1]$, and $\sigma$ over $[10^{-2},10]$.

####  E.3.2 Computational resources used

All simulations were run on a MacBook Pro with 16 GB RAM and an Apple M2 Pro
processor.

####  E.3.3 JEE-2009 Scores

##### Additional discussion of the dataset.

The JEE-2009 test scores were released in response to a Right to Information
application filed in June 2009 [91]. This dataset contains the scores of all
students from JEE-2009 (384,977 total) [91]; we used the version available
provided by [40]. In addition to the scores, for each student, the data
contains their self-reported (binary) gender and their birth category. The
birth category of a student is an officially designated indicator of their
socioeconomic group, where the general (GEN) category is the most privileged;
see [135, 19] for more details.

We observe that students not in the GEN category have significantly lower
average scores than students in the GEN category (18.2 vs. 35.1); this may not
imply that students not in the GEN category would perform poorly if admitted.
Indeed, among students of equal true “potential,” those from underprivileged
groups are known to perform poorer on standardized tests [60]. In the Indian
context, this could be due to many reasons, including that in India, fewer
students outside the GEN category attend primary school compared to students
from the general category, and on average a lower amount of money is spent on
the education of students in the non-general category compared to the general
category [92].

Figure 13:  Distribution of scores in the JEE dataset for different protected
groups based on birth category. See Section 5 for a discussion of the dataset.

####  E.3.4 Semantic Scholar Open Research Corpus

Cleaning and predicting author names. We follow the procedure used by [40].
Concretely, we remove papers without publication year (1.86% of total) and
predict author gender using their first name from a publicly available dataset
[3], containing first names and gender of everyone born between 1890 to 2018
and registered with the US social security administration (USSSA). We remove
authors whose first name has 2 or fewer characters, as these names are likely
to be abbreviations (retaining 75% of the total), and then categorize an
author as female (respectively male) if more than $\phi=0.9$ fraction of the
people of the same first name are female (respectively male) in the USSSA
data. We drop all uncategorized authors (32.25% of the remaining). This
results in 3,900,934 women and 5,074,426 men (43.46% females). We present the
tradeoff between the total number of authors retained and $\phi$ in Figure 14.

Counting the number of citations. We aim to ensure that the citation counts we
compute correspond to the total citations received by an author over their
lifetime (so far). Since the dataset only contains citations from 1980
onwards, we remove authors who published their first paper before 1980 as the
dataset does not have information about their earlier citations. This is the
same as the cleaning procedure used by [40]. We present the resulting
citation-distributions for male and female authors respectively in Figure 15.

Figure 14:  The tradeoff between the threshold $\phi$ used for clearing the
Semantic Scholar Open Research Corpus and the number of authors retained.
Details appear in Section E.3.4.  Figure 15:  Distributions of total citations
of men and women in the Semantic Scholar Open Research Corpus. Details appear
in Section E.3.4.