# Considering a Classical Upper Bound
on the Frobenius Number

###### Abstract

In this paper we study the (classical) Frobenius problem, namely the problem
of finding the largest integer that cannot be represented as a nonnegative
integral combination of given relatively prime (strictly) positive integers
(known as the Frobenius number). The main contribution of this paper are
observations regarding a previously known upper bound on the Frobenius number
where, in particular, we observe that a previously presented argument features
a subtle error, which alters the value of the upper bound. Despite this, we
demonstrate that the subtle error does not impact upon on the validity of the
upper bound, although it does impact on the upper bounds tightness. Notably,
we formally state the corrected result and additionally compare the relative
tightness of the corrected upper bound with the original. In particular, we
show that the updated bound is tighter in all but only a relatively “small”
number of cases using both formal techniques and via Monte Carlo simulation
techniques.

1Aled Williams Daiki Haijima

1Department of Mathematics

London School of Economics and Political Science

London, UK

<EMAIL_ADDRESS>
##  1 Introduction

Let $\boldsymbol{a}$ be a positive integral $n$-dimensional primitive vector,
i.e. $\boldsymbol{a}=(a_{1},\ldots,a_{n})^{T}\in\mathbb{Z}^{n}_{>0}$ with
$\gcd(\boldsymbol{a}):=\gcd(a_{1},\ldots,a_{n})=1$. In what follows, we
exclude the case $n=1$ and assume that the dimension $n\geq 2$. In particular,
without loss of generality, we assume the following conditions:

$\boldsymbol{a}=(a_{1},\ldots,a_{n})^{T}\in\mathbb{Z}^{n}_{>0}\,,n\geq 2\text{
and }\gcd(\boldsymbol{a}):=\gcd(a_{1},\ldots,a_{n})=1.$ (1)

The Frobenius number of $\boldsymbol{a}$, denoted by $F(\boldsymbol{a})$, is
the largest integer that cannot be represented as a nonnegative integral
combination of the $a_{i}$’s, i.e.

$F(\boldsymbol{a}):=\max\left\\{b\in\mathbb{Z}:b\neq\boldsymbol{a}^{T}\boldsymbol{z}\text{
for all }\boldsymbol{z}\in\mathbb{Z}^{n}_{\geq 0}\right\\},$

where $\boldsymbol{a}^{T}$ denotes the transpose of $\boldsymbol{a}$. It
should be noted for completeness that the Frobenius problem, namely the
problem of finding the Frobenius number, is also known by other names within
the literature including the money-changing problem (or the money-changing
problem of Frobenius, or the coin-exchange problem of Frobenius) [17, 15, 3],
the coin problem (or the Frobenius coin problem) [4, 13] and the Diophantine
problem of Frobenius [12, 10]. From a geometric viewpoint, $F(\boldsymbol{a})$
is the maximal right-hand side such that the knapsack polytope

$P(\boldsymbol{a},b)=\left\\{\boldsymbol{x}\in\mathbb{R}^{n}_{\geq
0}:\boldsymbol{a}^{T}\boldsymbol{x}=b\right\\}$

does not contain integral points. It should be noted that (1) is indeed a
necessary and sufficient condition for the existence of the Frobenius number.

Note that instead of the conditions (1), some authors instead assume the
stronger condition that all the entries of the vector are pairwise coprime,
i.e.

$\boldsymbol{a}=(a_{1},\ldots,a_{n})^{T}\in\mathbb{Z}^{n}_{>0}\,,n\geq 2\text{
and }\gcd(a_{i},a_{j})=1\text{ for any }i,j\in\\{1,2,\ldots,n\\}\text{ with
}i\neq j.$ (2)

It should be noted that not all integral vectors satisfying (1) also satisfy
the stronger conditions (2), say the vector $\boldsymbol{a}=(6,10,15)^{T}$ for
example.

There is a very rich history on Frobenius numbers and the book [1] provides a
very good survey of the problem. It is worth noting that computing the
Frobenius number in general is $\mathcal{NP}$-hard [9] (which was proved via a
reduction to the integer knapsack problem), however, if the number of integers
$n$ is fixed, then a polynomial time algorithm to calculate
$F(\boldsymbol{a})$ exists [8]. If $n=2$, it is well-known (most likely due to
Sylvester [14]) that

$\displaystyle F(a_{1},a_{2})$ $\displaystyle=a_{1}a_{2}-a_{1}-a_{2}$ (3)
$\displaystyle=(a_{1}-1)(a_{2}-1)-1\,.$

In contrast to the case when $n=2$, it was shown by Curtis [5] that no closed
formula exists for the Frobenius number if $n>2$. In light of this, there has
been a great deal of research into producing upper bounds on
$F(\boldsymbol{a})$. These bounds share the property that in the worst-case
they are of quadratic order with respect to the maximum absolute valued entry
of $\boldsymbol{a}$, which will be denoted by $\|\boldsymbol{a}\|_{\infty}$.
Further, let $\|\cdot\|_{2}$ denote the Euclidean norm. In particular, upon
assuming that $a_{1}\leq a_{2}\leq\cdots\leq a_{n}$ holds, such bounds include
the classical bound by Erdős and Graham [6, Theorem 1]

$F(\boldsymbol{a})\leq 2a_{n-1}\left\lfloor\frac{a_{n}}{n}\right\rfloor-
a_{n},$

by Selmer [12]

$F(\boldsymbol{a})\leq 2a_{n}\left\lfloor\frac{a_{1}}{n}\right\rfloor-a_{1},$

by Vitek [16, Theorem 5]

$F(\boldsymbol{a})\leq\frac{1}{2}(a_{2}-1)(a_{n}-2)-1,$

by Beck et al. [2, Theorem 9]

$F(\boldsymbol{a})\leq\frac{1}{2}\left(\sqrt{a_{1}a_{2}a_{3}\left(a_{1}+a_{2}+a_{3}\right)}-a_{1}-a_{2}-a_{3}\right),$

and by Fukshansky and Robins [7, Equation 29]

$F(\boldsymbol{a})\leq\left\lfloor\frac{(n-1)^{2}\,\Gamma(\frac{n+1}{2})}{\pi^{(n-1)/2}}\sum_{i=1}^{n}a_{i}\sqrt{\|\boldsymbol{a}\|_{2}^{2}-a_{i}^{2}}+1\right\rfloor,$

where $\Gamma(\cdot)$ and $\lfloor\,\cdot\,\rfloor$ denote Euler’s gamma and
the standard floor functions, respectively.

##  2 Preliminary and Auxiliary Results

In this section, we present some preliminary results that are essential for
establishing a requirement for upper bounds on $F(\boldsymbol{a})$ when the
more general conditions (1) on $\boldsymbol{a}$ hold. In particular, this
requirement is induced via a simple lower bound which considers the parity of
the $a_{i}$’s and is formally introduced below.

######  Proposition 1.

If an integral vector $\boldsymbol{a}$ satisfies (1), then at least one
$a_{i}$ must be odd for $i\in\\{1,2,\ldots,n\\}$.

###### Proof.

Suppose for contradiction there does not exist an odd $a_{i}$, i.e. that
$\boldsymbol{a}$ has only even elements. It follows immediately that
$\gcd(\boldsymbol{a})\geq 2$, which contradicts the assumed conditions (1) as
required. ∎

Denote by $o_{t}:=o_{t}(\boldsymbol{a})$ the $t$-th smallest odd element in
$\boldsymbol{a}$. Observe that Proposition 1 implies that $o_{1}$ necessarily
exists for any integral vector $\boldsymbol{a}$ satisfying (1).

######  Proposition 2.

If an integral vector $\boldsymbol{a}$ satisfies (1), then $o_{1}-2$ is a
lower bound for $F(\boldsymbol{a})$.

###### Proof.

Firstly, observe that since $o_{1}$ is the smallest odd element in
$\boldsymbol{a}$, it follows that any odd number strictly less than $o_{1}$
cannot be expressed as $\sum_{i=1}^{n}a_{i}x_{i}$ for
$x_{i}\in\mathbb{Z}_{\geq 0}$ for $i\in\\{1,2,\ldots,n\\}$. In particular,
$o_{1}-2$ cannot be expressed as a nonnegative integral linear combination of
the $a_{i}$’s. Thus, the Frobenius number $F(\boldsymbol{a})$ is at least
$o_{1}-2$ as required. ∎

The propositions outlined can be applied to establish a requirement for upper
bounds on the Frobenius number, particularly where the weaker conditions (1)
concerning the vector $\boldsymbol{a}$ are met.

######  Lemma 1.

If an integral vector $\boldsymbol{a}=(a_{1},a_{2},\ldots,a_{n})^{T}$
satisfies (1) with $a_{1}\leq a_{2}\leq\cdots\leq a_{n}$, then any general
upper bound on the Frobenius number $F(\boldsymbol{a})$ must inherently depend
on the largest element $a_{n}$.

###### Proof.

Let us suppose for simplicity that the vector $\boldsymbol{a}$ has the form
that $a_{i}$ is even for each $i\in\\{1,2,\ldots,n-1\\}$ while the final entry
$a_{n}$ is odd. Notice that here $o_{1}=a_{n}$ and, in light of Proposition 2,
it immediately follows that $F(\boldsymbol{a})\geq a_{n}-2$.

Suppose for contradiction that there exists an upper bound on
$F(\boldsymbol{a})$ that does not depend on $a_{n}$, i.e. that there exists
some function $f:\mathbb{R}^{n-1}\rightarrow\mathbb{R}$ satisfying
$F(\boldsymbol{a})\leq f(a_{1},a_{2},\ldots,a_{n-1})$. If we set
$a_{n}=f(a_{1},a_{2},\ldots,a_{n-1})+3$ if $f(a_{1},a_{2},\ldots,a_{n-1})$ is
even and $a_{n}=f(a_{1},a_{2},\ldots,a_{n-1})+4$ if
$f(a_{1},a_{2},\ldots,a_{n-1})$ is odd, then observe that
$a_{n}-2>f(a_{1},a_{2},\ldots,a_{n-1})$ holds. In particular, notice that the
lower bound on the Frobenius number is strictly larger than the (assumed)
upper bound, which is a contradiction as required. ∎

It should be emphasised that this result suggests that any (general) upper
bound on the Frobenius number which does not depend on the maximal entry of
$\boldsymbol{a}$ does not necessarily hold in general without stronger
assumptions than the conditions (1).

The following results provide a rather surprisingly useful property that holds
when $\boldsymbol{a}$ satisfies the stronger conditions (2) that all the
entries of the vector are pairwise coprime.

######  Lemma 2.

If an integral vector $\boldsymbol{a}=(a_{1},a_{2},\ldots,a_{n})^{T}$
satisfies (2), then for any $i,j\in\\{1,2,\ldots,n\\}$ with $i\neq j$ we have

$F(\boldsymbol{a})\leq(a_{i}-1)(a_{j}-1)-1.$

###### Proof.

Firstly, notice that given any pair $a_{i}$ and $a_{j}$ with $i\neq j$ that in
light of the conditions (2) it follows that $\gcd(a_{i},a_{j})=1$. Thus, the
Frobenius number $F(a_{i},a_{j})$ corresponding to the pair $a_{i}$ and
$a_{j}$ exists and takes finite value. Furthermore, it follows in light of (3)
that the equality

$F(a_{i},a_{j})=(a_{i}-1)(a_{j}-1)-1$

holds. By the definition of Frobenius number, we deduce that all integers
strictly greater than $(a_{i}-1)(a_{j}-1)-1$ can be expressed as
$a_{i}x_{i}+a_{j}x_{j}$ for some $x_{i},x_{j}\in\mathbb{Z}_{\geq 0}$. Thus, it
immediately follows that all integers strictly greater than
$(a_{i}-1)(a_{j}-1)-1$ can be expressed as $\sum_{k=1}^{n}a_{k}x_{k}$ for
$x_{k}\in\mathbb{Z}_{\geq 0}$ for $k\in\\{1,2,\ldots,n\\}$ (upon setting
$x_{k}=0$ when $k\neq i,j$ whenever necessary). In particular, this shows that
the Frobenius number $F(\boldsymbol{a})$ satisfies the inequality
$F(\boldsymbol{a})\leq(a_{i}-1)(a_{j}-1)-1$ for any $i\neq j$ as required. ∎

The following corollary follows immediately from Lemma 2.

######  Corollary 1.

If an integral vector $\boldsymbol{a}=(a_{1},a_{2},\ldots,a_{n})^{T}$
satisfies (2) and $a_{1}\leq a_{2}\leq\cdots\leq a_{n}$, then

$F(\boldsymbol{a})\leq(a_{1}-1)(a_{2}-1)-1.$

It should be emphasised that the above results tell us that the well-known
result (3) of Sylvester [14] extends naturally to provide an upper bound for
the Frobenius number $F(\boldsymbol{a})$ under the (stronger) assumption (2)
that the entries of the vector $\boldsymbol{a}$ are pairwise coprime.

##  3 Observations on a Previously Known Upper Bound

Recall that Beck et al. [2, Theorem 9] introduced the upper bound

$F(\boldsymbol{a})\leq\frac{1}{2}\left(\sqrt{a_{1}a_{2}a_{3}\left(a_{1}+a_{2}+a_{3}\right)}-a_{1}-a_{2}-a_{3}\right)$
(4)

on the Frobenius number upon finding bounds for Fourier-Dedekind sums. This
bound (4) is widely referenced across books and papers, however, in most of
these little attention is given to the underlying assumed conditions on
$\boldsymbol{a}$. In particular, the upper bound (4) necessitates that the
stronger conditions (2) hold, instead of the more general (weaker) conditions
(1).

######  Proposition 3.

The upper bound (4) of Beck et al. [2, Theorem 9] does not necessarily hold
unless the stronger conditions (2) hold. This requirement remains even if the
weaker conditions (1) are met.

###### Proof.

Observe that if $n=2$, then the stronger (2) and weaker conditions (1) are
equivalent. Thus, we focus here only on the case that $n\geq 3$, where we show
there are counterexamples for each $n$.

Let us consider two cases, namely $n=3$ and $n\geq 4$, respectively. If $n=3$,
then consider the integer vector $\boldsymbol{a}=(3,6,19)^{T}$ with
$F(\boldsymbol{a})=35$. In such case, notice that the bound (4) yields

$\frac{1}{2}\left(\sqrt{3\cdot 6\cdot
19\,(3+6+19)}-3-6-19\right)=\frac{1}{2}\left(6\sqrt{266}-28\right)\approx
34.928519$

where, in particular, $35\not\leq 34.928519$ and hence the upper bound (4)
fails when $n=3$. If instead $n\geq 4$, then in light of Lemma 1 clearly (4)
cannot be a general upper bound for the Frobenius number. ∎

Note that in the case $n\geq 4$, any vector
$\boldsymbol{a}=(2,4,6,a_{4},\ldots,a_{n})^{T}$ satisfying (1) and
$a_{n}\geq\cdots\geq a_{4}>7$ provides a counterexample for any $n$. Indeed,
since $a_{4}>7$ by assumption, the Frobenius number is clearly greater than or
equal to 7 (since 7 cannot be expressed by $\sum_{i=1}^{n}a_{i}x_{i}$ for
$x_{i}\in\mathbb{Z}_{\geq 0}$ for all $i$). Despite this, the bound (4) yields

$\frac{1}{2}\left(\sqrt{2\cdot 4\cdot 6\,(2+4+6)}-2-4-6\right)=6,$

which demonstrates the upper bound (4) does not necessarily hold if only the
weaker conditions (1) are met.

The remarks presented in this section are intended to clarify a common
misunderstanding about the upper bound (4) as referenced in various books and
papers. Furthermore, it is crucial to highlight a subtle error in the argument
presented by Beck et al. [2], which alters the value of the upper bound. The
following result states the corrected upper bound, where the proof is outlined
in a later section.

######  Theorem 1.

If an integral vector $\boldsymbol{a}=(a_{1},a_{2},\ldots,a_{n})^{T}$
satisfies (2) with $a_{1}\leq a_{2}\leq\cdots\leq a_{n}$, then the argument of
Beck et al. [2] yields

$F(\boldsymbol{a})\leq\frac{1}{2}\left(\sqrt{\frac{1}{3}\left(a_{1}+a_{2}+a_{3}\right)\left(a_{1}+a_{2}+a_{3}+2a_{1}a_{2}a_{3}\right)+\frac{8}{3}\left(a_{1}a_{2}+a_{2}a_{3}+a_{3}a_{1}\right)}-a_{1}-a_{2}-a_{3}\right).$
(5)

It is natural to consider if the original bound (4) given by Beck et al. [2,
Theorem 9] is indeed correct provided the integral vector $\boldsymbol{a}$
satisfies the stronger conditions (2). It turns out that the upper bound (4)
remains valid. The following result states this formally, where the proof is
outlined in a later section.

######  Theorem 2.

If an integral vector $\boldsymbol{a}=(a_{1},a_{2},\ldots,a_{n})^{T}$
satisfies (2) with $a_{1}\leq a_{2}\leq\cdots\leq a_{n}$, then

$F(\boldsymbol{a})\leq\frac{1}{2}\left(\sqrt{a_{1}a_{2}a_{3}\left(a_{1}+a_{2}+a_{3}\right)}-a_{1}-a_{2}-a_{3}\right).$

Furthermore, it is natural to consider the relative tightness of (1) with (4).
This comparison will be explored in the subsequent section of the paper.

##  4 Tightness Comparison of Upper Bounds

In this section, we consider the relative tightness of the upper bounds (1)
and (4). In particular, to slightly simplify notation, let us denote by

$\displaystyle UB_{1}(\boldsymbol{a})$
$\displaystyle=UB_{1}(a_{1},a_{2},a_{3})$
$\displaystyle:=\frac{1}{2}\left(\sqrt{\frac{1}{3}\left(a_{1}+a_{2}+a_{3}\right)\left(a_{1}+a_{2}+a_{3}+2a_{1}a_{2}a_{3}\right)+\frac{8}{3}\left(a_{1}a_{2}+a_{2}a_{3}+a_{3}a_{1}\right)}-a_{1}-a_{2}-a_{3}\right)$

and

$UB_{2}(\boldsymbol{a})=UB_{2}(a_{1},a_{2},a_{3}):=\frac{1}{2}\left(\sqrt{a_{1}a_{2}a_{3}\left(a_{1}+a_{2}+a_{3}\right)}-a_{1}-a_{2}-a_{3}\right).$

Recall that Theorem 1 and 2 imply that $UB_{1}(\boldsymbol{a})$ and
$UB_{2}(\boldsymbol{a})$ are valid upper bounds provided $\boldsymbol{a}$
satisfies (2) and $a_{1}\leq a_{2}\leq\cdots\leq a_{n}$, however, we instead
are interested in which bound is tighter. The first result of this section
shows that $UB_{2}(\boldsymbol{a})$ is tighter than $UB_{1}(\boldsymbol{a})$
is only a relatively “small” (finite) number of cases, where the proof is
excluded given this was completed via simple enumeration.

######  Theorem 3.

If an integral vector $\boldsymbol{a}=(a_{1},a_{2},\ldots,a_{n})^{T}$
satisfies (2) with $a_{1}\leq a_{2}\leq\cdots\leq a_{n}$, then
$F(\boldsymbol{a})$ satisfies $F(\boldsymbol{a})\leq UB_{1}(\boldsymbol{a})$
and $F(\boldsymbol{a})\leq UB_{2}(\boldsymbol{a})$, where
$UB_{2}(\boldsymbol{a})$ is sharper only when

$\displaystyle(a_{1},a_{2},a_{3})\in\big{\\{}$
$\displaystyle(1,2,3),(1,2,5),(1,2,7),(1,2,9),(1,2,11),(1,2,13),(1,2,15),$
$\displaystyle(1,2,17),(1,2,19),(1,2,21),(1,2,23),(1,2,25),(1,3,4),(1,3,5),$
$\displaystyle(1,3,7),(1,3,8),(1,3,10),(1,3,11),(1,3,13),(1,3,14),(1,4,5),$
$\displaystyle(1,4,7),(1,4,9),(1,4,11),(1,5,6),(1,5,7),(1,5,8),(1,5,9),(1,6,7),(2,3,5)\big{\\}}.$

It should be emphasised that this result suggests that in “almost all” cases,
$UB_{1}(\boldsymbol{a})$ provides a tighter bound than
$UB_{2}(\boldsymbol{a})$. In order to compare the relative tightness of the
bounds, for completeness, we apply Monte Carlo simulation techniques (see e.g.
[11, Chapter 2]) and present the results below.

During this simulation, we firstly randomly generate integral vectors
$\boldsymbol{a}$ satisfying the conditions (2) with ordering $a_{1}\leq
a_{2}\leq\cdots\leq a_{n}$, before computing the values of
$UB_{1}(\boldsymbol{a})$ and $UB_{2}(\boldsymbol{a})$. This process is
iteratively repeated 100,000 times. During the sampling, we set
$\|\boldsymbol{a}\|_{\infty}=\max_{i}|a_{i}|\leq 1000$ for convenience. Note
that in each graph in Figure 1, the vertical axis corresponds to the
difference $UB_{1}(\boldsymbol{a})-UB_{2}(\boldsymbol{a})$, where a large
vertical value illustrates that the corrected upper bound (5) is much tighter
than the originally stated bound (4).

(a) $UB_{1}(\boldsymbol{a})-UB_{2}(\boldsymbol{a})$ upon increasing $a_{3}$.

(b) $UB_{1}(\boldsymbol{a})-UB_{2}(\boldsymbol{a})$ upon increasing
$a_{1}a_{2}$.

(c) $UB_{1}(\boldsymbol{a})-UB_{2}(\boldsymbol{a})$ upon increasing
$a_{1}a_{2}a_{3}$.

(d) $UB_{1}(\boldsymbol{a})-UB_{2}(\boldsymbol{a})$ upon increasing
$a_{1}+a_{2}+a_{3}$.

Figure 1: This figure illustrates how the difference
$UB_{1}(\boldsymbol{a})-UB_{2}(\boldsymbol{a})$ varies dependent upon the
$a_{i}$’s.

Figure 1(a) demonstrates the difference
$UB_{1}(\boldsymbol{a})-UB_{2}(\boldsymbol{a})$ grows rapidly with increases
in $a_{3}$, while the difference remains small if $a_{3}$ is small. It should
be emphasised that one would not expect the difference to be significant when
$a_{3}$ is small given that the entries of $\boldsymbol{a}$ are ordered by
assumption. Figure 1(b) shows how this differences varies upon increases in
$a_{1}a_{2}$. Notably, the figure suggests that the minimum difference
increases linearly with $a_{1}a_{2}$, whereas the maximum difference seems to
grow sublinearly with $a_{1}a_{2}$. Furthermore, if $a_{1}a_{2}$ is large
(around 1,000,000), then the variance of the difference
$UB_{1}(\boldsymbol{a})-UB_{2}(\boldsymbol{a})$ appears small. This can be
explained by considering the difference

$\displaystyle
UB_{1}(\boldsymbol{a})-UB_{2}(\boldsymbol{a})=\frac{1}{2}\Big{(}$
$\displaystyle\sqrt{a_{1}a_{2}a_{3}\left(a_{1}+a_{2}+a_{3}\right)}$
$\displaystyle-\sqrt{\frac{1}{3}a_{1}a_{2}a_{3}\left(a_{1}+a_{2}+a_{3}\right)+\left(a_{1}+a_{2}+a_{3}\right)^{2}+\frac{8}{3}\left(a_{1}a_{2}+a_{2}a_{3}+a_{3}a_{1}\right)}\,\Big{)},$

which is maximised for fixed $a_{1}a_{2}$ when $a_{3}$ is large and is
minimised when $a_{3}$ is small. In particular, note that if $a_{3}$ is small,
then the assumed ordering implies that $a_{3}\approx a_{2}$. In such case, the
value of the difference can be well approximated by

$\frac{1}{2}a_{1}a_{2}\left(\sqrt{3}-\sqrt{2+\frac{11}{a_{1}a_{2}}}\,\right),$

which grows roughly linearly. If instead $a_{1}a_{2}$ is large (say around
1,000,000), then the assumed ordering and upper bound on the $a_{i}$’s
restricts variance in both $UB_{1}(\boldsymbol{a})$ and
$UB_{2}(\boldsymbol{a})$, respectively. Figure 1(c) demonstrates the
difference $UB_{1}(\boldsymbol{a})-UB_{2}(\boldsymbol{a})$ grows sublinearly
with the product $a_{1}a_{2}a_{3}$, while the variance once more appears
small, which can be similarly explained via careful algebraic analysis. Figure
1(d) shows how this difference varies upon increases in $a_{1}+a_{2}+a_{3}$,
where the variance decreases significantly as the value of $a_{1}+a_{2}+a_{3}$
grows beyond 2000. Notably, the final figure suggests that one should expect
the difference $UB_{1}(\boldsymbol{a})-UB_{2}(\boldsymbol{a})$ to be small
only in the scenario that $a_{1}$ is “reasonably small”, which follows in
light of the assumed ordering.

##  5 Proof of Theorem 1

In this section, we follow closely the argument presented by Beck et al. [2,
Theorem 9] to demonstrate that it actually yields the upper bound (5) instead
of (4). It should be noted that the proof presented by Beck et al. [2] instead
provides an upper bound for

$\displaystyle F^{*}(\boldsymbol{a}):$
$\displaystyle=\max\left\\{b\in\mathbb{Z}:b\neq\boldsymbol{a}^{T}\boldsymbol{z}\text{
for all }\boldsymbol{z}\in\mathbb{Z}^{n}_{>0}\right\\}$
$\displaystyle=F(\boldsymbol{a})+a_{1}+a_{2}+\cdots+a_{n},$

which is the largest integer that cannot be represented as a (strictly)
positive integral combination of the $a_{i}$’s.

Let $A=\\{a_{1},a_{2},\ldots,a_{n}\\}$ be a set of pairwise coprime positive
integers and

$p_{A}^{\prime}(b)=\\#\left\\{\left(m_{1},\ldots,m_{n}\right)\in\mathbb{Z}^{n}_{>0}:\sum_{k=1}^{n}m_{k}\,a_{k}=b\right\\},$

where notice that $F^{*}(\boldsymbol{a})$ is simply the largest value for $b$
for which $p_{A}^{\prime}(b)=0$.

Let $c_{1},c_{2},\ldots,c_{n}\in\mathbb{Z}$ be relatively prime to
$c\in\mathbb{Z}$, and $t\in\mathbb{Z}$. We define the Fourier–Dedekind sum as

$\sigma_{t}\left(c_{1},\ldots,c_{n};c\right)=\frac{1}{c}\sum_{\lambda^{c}=1\neq\lambda}\frac{\lambda^{t}}{\left(\lambda^{c_{1}}-1\right)\cdots\left(\lambda^{c_{n}}-1\right)}.$

Note that one particularly noteworthy expression (which follows by
periodicity) [2] is

$\sigma_{t}(a,b;c)=\sum_{m=0}^{c-1}\left(\left(\frac{-a^{-1}(bm+t)}{c}\right)\right)\left(\left(\frac{m}{c}\right)\right)-\frac{1}{4c}$
(6)

with $aa^{-1}\equiv 1\pmod{c}$ and where $((x))=x-\lfloor
x\rfloor-\frac{1}{2}$ is a sawtooth function.

###### Proof.

Firstly, note that it is easy to verify that

$F^{*}(\boldsymbol{a})=F^{*}(a_{1},a_{2},\ldots,a_{n})\leq
F^{*}(a_{1},a_{2},a_{3})+a_{3}+a_{4}+\cdots+a_{n}.$

We closely follow [2] by focusing on the case where $n=3$ and the $a_{i}$’s
are pairwise coprime. In order to slightly simplify notation, let $a,b,c$
denote pairwise relatively prime positive integers. Upon using the the Cauchy-
Schwartz inequality we find

$\displaystyle\sigma_{t}(a,b;c)$
$\displaystyle\geqslant-\sum_{m=0}^{c-1}\left(\left(\frac{m}{c}\right)\right)^{2}-\frac{1}{4c}=\sum_{m=0}^{c-1}\left(\frac{m}{c}-\frac{1}{2}\right)^{2}-\frac{1}{4c}$
$\displaystyle=-\frac{1(2c-1)(c-1)c}{c^{2}}+\frac{1}{c}\frac{c(c-1)}{2}-\frac{c}{4}-\frac{1}{4c}$
$\displaystyle=-\frac{c}{12}-\frac{5}{12c}\,.$

It should be noted that in [2], the right-hand side of the expression
previously discussed differs from the one presented here. We can now utilise
the above inequality to obtain

$\displaystyle p_{\\{a,b,c\\}}^{\prime}(t)\geq$
$\displaystyle\,\frac{t^{2}}{2abc}-\frac{t}{2}\left(\frac{1}{ab}+\frac{1}{ac}+\frac{1}{bc}\right)+\frac{1}{12}\left(\frac{3}{a}+\frac{3}{b}+\frac{3}{c}+\frac{a}{bc}+\frac{b}{ac}+\frac{c}{ab}\right)$
$\displaystyle\,\hskip
19.91692pt-\frac{1}{12}(a+b+c)-\frac{5}{12}\left(\frac{1}{a}+\frac{1}{b}+\frac{1}{c}\right)$
$\displaystyle=$
$\displaystyle\,\frac{t^{2}}{2abc}-\frac{t}{2}\left(\frac{1}{ab}+\frac{1}{ac}+\frac{1}{bc}\right)+\frac{1}{12}\left(\frac{a}{bc}+\frac{b}{ac}+\frac{c}{ab}\right)$
$\displaystyle\,\hskip
19.91692pt-\frac{1}{12}(a+b+c)-\frac{1}{6}\left(\frac{1}{a}+\frac{1}{b}+\frac{1}{c}\right)\,,$

which upon algebraic manipulation yields the upper bound

$F^{*}(a,b,c)\leq\frac{1}{2}(a+b+c)+\frac{1}{2}\sqrt{\frac{1}{3}(a+b+c)(a+b+c+2abc)+\frac{3}{8}(ab+bc+ca)}$

Thus, upon replacing $a,b$ and $c$ with $a_{1},a_{2}$ and $a_{3}$,
respectively, we deduce that

$\displaystyle F^{*}(\boldsymbol{a})$ $\displaystyle\leq
F^{*}(a_{1},a_{2},a_{3})+a_{3}+a_{4}+\cdots+a_{n}$
$\displaystyle\leq\left(\frac{1}{2}\sqrt{\frac{1}{3}\left(a_{1}+a_{2}+a_{3}\right)\left(a_{1}+a_{2}+a_{3}+2a_{1}a_{2}a_{3}\right)+\frac{8}{3}\left(a_{1}a_{2}+a_{2}a_{3}+a_{3}a_{1}\right)}+a_{1}+a_{2}+a_{3}\right)$
$\displaystyle\,\hskip 56.9055pt+a_{3}+a_{4}+\cdots+a_{n}\,$

which yields that

$\displaystyle F(\boldsymbol{a})$
$\displaystyle=F^{*}(\boldsymbol{a})-a_{1}-a_{2}-\cdots-a_{n}$
$\displaystyle\leq\frac{1}{2}\left(\sqrt{\frac{1}{3}\left(a_{1}+a_{2}+a_{3}\right)\left(a_{1}+a_{2}+a_{3}+2a_{1}a_{2}a_{3}\right)+\frac{8}{3}\left(a_{1}a_{2}+a_{2}a_{3}+a_{3}a_{1}\right)}-a_{1}-a_{2}-a_{3}\right)$

as required, which concludes the proof. ∎

##  6 Proof of Theorem 2

Recall the notation

$UB_{1}(\boldsymbol{a}):=\frac{1}{2}\left(\sqrt{\frac{1}{3}\left(a_{1}+a_{2}+a_{3}\right)\left(a_{1}+a_{2}+a_{3}+2a_{1}a_{2}a_{3}\right)+\frac{8}{3}\left(a_{1}a_{2}+a_{2}a_{3}+a_{3}a_{1}\right)}-a_{1}-a_{2}-a_{3}\right)$

and

$UB_{2}(\boldsymbol{a}):=\frac{1}{2}\left(\sqrt{a_{1}a_{2}a_{3}\left(a_{1}+a_{2}+a_{3}\right)}-a_{1}-a_{2}-a_{3}\right).$

In this section, we show that $UB_{2}(\boldsymbol{a})$ is a valid upper bound
on the Frobenius number $F(\boldsymbol{a})$ provided $\boldsymbol{a}$
satisfies (2) and $a_{1}\leq a_{2}\leq\cdots\leq a_{n}$. It should be noted
that the stronger condition (2) with the (assumed) ordering of the $a_{i}$’s
implies that $a_{1}<a_{2}<\cdots<a_{n}$ provided that $a_{1}>1$. Before
discussing the correctness of the upper bound $UB_{2}(\boldsymbol{a})$, we
firstly consider how this bound varies with changes in $a_{3}$.

######  Proposition 4.

If $a_{1}<a_{2}<a_{3}$, then $UB_{2}(\boldsymbol{a})$ is a strictly increasing
function in $a_{3}$. If instead $a_{1}\leq a_{2}\leq a_{3}$ holds, then
$UB_{2}(\boldsymbol{a})$ is a nondecreasing function in $a_{3}$.

###### Proof.

Upon partial differentiation with respect to $a_{3}$, observe that
$UB_{2}(\boldsymbol{a})$ becomes

$\frac{\partial UB_{2}(\boldsymbol{a})}{\partial
a_{3}}=\frac{1}{2}\left(\frac{a_{1}a_{2}\left(a_{1}+a_{2}+2a_{3}\right)}{2\sqrt{a_{1}a_{2}a_{3}\left(a_{1}+a_{2}+a_{3}\right)}}-1\right).$

If $a_{1}<a_{2}<a_{3}$, then note that
$a_{1}a_{2}\left(a_{1}+a_{2}+2a_{3}\right)>4a_{3}$ holds. Upon simple
algebraic manipulation, we deduce that
$a_{1}^{2}a_{2}^{2}\left(a_{1}+a_{2}+2a_{3}\right)^{2}>4a_{1}a_{2}a_{3}\left(a_{1}+a_{2}+a_{3}\right)$,
which implies that

$\frac{a_{1}a_{2}\left(a_{1}+a_{2}+2a_{3}\right)}{2\sqrt{a_{1}a_{2}a_{3}\left(a_{1}+a_{2}+a_{3}\right)}}>1.$

It follows that $\frac{\partial UB_{2}(\boldsymbol{a})}{\partial a_{3}}>0$ and
hence $UB_{2}(\boldsymbol{a})$ is a strictly increasing function in $a_{3}$
when $a_{1}<a_{2}<a_{3}$. If instead $a_{1}\leq a_{2}\leq a_{3}$, then a
similar argument yields that $\frac{\partial UB_{2}(\boldsymbol{a})}{\partial
a_{3}}\geq 0$ and hence $UB_{2}(\boldsymbol{a})$ is a nondecreasing function
in $a_{3}$ as required. ∎

We now proceed to present a detailed proof to establish the validity of
Theorem 2.

###### Proof.

Observe that if $UB_{2}(\boldsymbol{a})\geq UB_{1}(\boldsymbol{a})$, then
clearly the upper bound $UB_{2}(\boldsymbol{a})$ is valid in consequence to
the validity of Theorem 1. Thus, we consider only the setting where
$UB_{2}(\boldsymbol{a})<UB_{1}(\boldsymbol{a})$. Notice that upon simple
algebraic manipulation the inequality $UB_{2}(\boldsymbol{a})\leq
UB_{1}(\boldsymbol{a})$ is equivalent to

$a_{1}a_{2}a_{3}\left(a_{1}+a_{2}+a_{3}\right)\leq
a_{1}^{2}+a_{2}^{2}+a_{3}^{2}+10\left(a_{1}a_{2}+a_{2}a_{3}+a_{1}a_{3}\right).$
(7)

It is sufficient to here consider only the case that $a_{1}<a_{2}<a_{3}$. In
particular, this follows because otherwise we require $a_{1}=1$ in light of
the assumed conditions (2) and hence, in such case, we yield that
$F(\boldsymbol{a})=-1\leq UB_{2}(\boldsymbol{a})$ holds as required.

In order to satisfy (7), the (strict) inequality $a_{1}a_{2}<33$ is necessary.
Indeed, if instead $a_{1}a_{2}\geq 33$, then

$\displaystyle a_{1}a_{2}a_{3}\left(a_{1}+a_{2}+a_{3}\right)$
$\displaystyle=a_{1}a_{2}a_{3}^{2}+a_{1}a_{2}a_{3}\left(a_{1}+a_{2}\right)$
$\displaystyle>a_{1}a_{2}a_{3}^{2}$ $\displaystyle\geq 33a_{3}^{2}$
$\displaystyle=3a_{3}^{2}+10(a_{3}^{2}+a_{3}^{2}+a_{3}^{2})$
$\displaystyle>a_{1}^{2}+a_{2}^{2}+a_{3}^{2}+10\left(a_{1}a_{2}+a_{2}a_{3}+a_{1}a_{3}\right)$

holds, where the final inequality follows since $a_{1},a_{2}<a_{3}$. Thus, we
need only to consider the cases that $a_{1}a_{2}\leq 32$ and $a_{1}<a_{2}$
with $\gcd(a_{1},a_{2})=1$. The pairs $(a_{1},a_{2})$ satisfying these
conditions are:

  1. (i)

if $a_{1}=1$, then
$\left(a_{1},a_{2}\right)=(1,2),(1,3),\ldots,(1,31),(1,32)$,

  2. (ii)

if $a_{1}=2$, then
$\left(a_{1},a_{2}\right)=(2,3),(2,5),(2,7),(2,9),(2,11),(2,13),(2,15)$,

  3. (iii)

if $a_{1}=3$, then $\left(a_{1},a_{2}\right)=(3,4),(3,5),(3,7),(3,8),(3,10)$,

  4. (iv)

if $a_{1}=4$, then $\left(a_{1},a_{2}\right)=(4,5),(4,7)$, and

  5. (v)

if $a_{1}=5$, then $\left(a_{1},a_{2}\right)=(5,6)$.

It should be emphasised that if $UB_{2}(\boldsymbol{a})$ is a valid upper
bound in each of the above cases, then it follows that
$UB_{2}(\boldsymbol{a})$ is a valid upper bound as required. In order to
complete the proof we now consider each of these cases in turn. It should be
noted that cases (ii) - (v) use the properties
$F(\boldsymbol{a})\leq(a_{1}-1)(a_{2}-1)-1$ (which follows by Corollary 1) and
$UB(a_{1},a_{2},a_{3}^{\prime})>UB(a_{1},a_{2},a_{3})$ when
$a_{3}^{\prime}>a_{3}$ (which follows by Proposition 4).

  1. (i)

$a_{1}=1$: In this case, notice that since the entries of the vector
$\boldsymbol{a}$ are coprime by assumption, it follows that we have $a_{2}\geq
2$ and $a_{3}\geq 3$. Note that we have
$\left(a_{2}-1\right)\left(a_{3}-1\right)\geq 2$ and $a_{2}a_{3}\geq
1+a_{2}+a_{3}$. These inequalities imply that

$a_{2}a_{3}\left(1+a_{2}+a_{3}\right)\geq\left(1+a_{2}+a_{3}\right)^{2}$

which, upon rearranging algebraically, yields that

$\frac{1}{2}\left(\sqrt{a_{2}a_{3}\left(1+a_{2}+a_{3}\right)}-\left(1+a_{2}+a_{3}\right)\right)=UB_{2}(1,a_{2},a_{3},\ldots,a_{n})\geq
0.$

Finally, observe that the equality $F(1,a_{2},a_{3},\ldots,a_{n})=-1$ holds
for all $a_{2},a_{3},\ldots,a_{n}$ and, thus, $UB_{2}(\boldsymbol{a})$ is a
valid upper bound in this scenario.

  2. (ii)

$a_{1}=2$: In this case, notice that

$F(2,3,a_{3},a_{4},\ldots,a_{n})\leq(2-1)(3-1)-1=1<3.660254=UB_{2}(2,3,5)\leq
UB_{2}(2,3,5),$

where the strict inequality follows since if $(a_{1},a_{2})=(2,3)$, then
$a_{3}\geq 5$ by the conditions (2). In a similar fashion, notice that

$F(2,5,a_{3},a_{4},\ldots,a_{n})\leq(2-1)(5-1)-1=3<8.652476=UB_{2}(2,5,7)\leq
UB_{2}(2,5,a_{3}),$
$F(2,7,a_{3},a_{4},\ldots,a_{n})\leq(2-1)(7-1)-1=5<14.811762=UB_{2}(2,7,9)\leq
UB_{2}(2,7,a_{3}),$
$F(2,9,a_{3},a_{4},\ldots,a_{n})\leq(2-1)(9-1)-1=7<22=UB_{2}(2,9,11)\leq
UB_{2}(2,9,a_{3}),$
$F(2,11,a_{3},a_{4},\ldots,a_{n})\leq(2-1)(11-1)-1=9<30.116122=UB_{2}(2,11,13)\leq
UB_{2}(2,11,a_{3}),$
$F(2,13,a_{3},a_{4},\ldots,a_{n})\leq(2-1)(13-1)-1=11<39.083269=UB_{2}(2,13,15)\leq
UB_{2}(2,13,a_{3}),$
$F(2,15,a_{3},a_{4},\ldots,a_{n})\leq(2-1)(15-1)-1=13<48.8407169=UB_{2}(2,15,17)\leq
UB_{2}(2,15,a3).$

  3. (iii)

$a_{1}=3$: In this case, notice that

$F(3,4,a_{3},a_{4},\ldots,a_{n})\leq(3-1)(4-1)-1=5<7.416408=UB_{2}(3,4,5)\leq
UB_{2}(3,4,a_{3}),$
$F(3,5,a_{3},a_{4},\ldots,a_{n})\leq(3-1)(5-1)-1=9<12.343135=UB_{2}(3,5,7)\leq
UB_{2}(3,5,a_{3}),$
$F(3,7,a_{3},a_{4},\ldots,a_{n})\leq(3-1)(7-1)-1=11<18.495454=UB_{2}(3,7,8)\leq
UB_{2}(3,7,a_{3}),$
$F(3,8,a_{3},a_{4},\ldots,a_{n})\leq(3-1)(8-1)-1=13<27.105118=UB_{2}(3,8,11)\leq
UB_{2}(3,8,a_{3}),$
$F(3,10,a_{3},a_{4},\ldots,a_{n})\leq(3-1)(10-1)-1=17<32.497191=UB_{2}(3,10,11)\leq
UB_{2}(3,10,a_{3}).$

  4. (iv)

$a_{1}=4$: In this case, notice that

$F(4,5,a_{3},a_{4},\ldots,a_{n})\leq(4-1)(5-1)-1=11<15.6643191=UB_{2}(4,5,7)\leq
UB_{2}(4,5,a_{3}),$
$F(4,7,a_{3},a_{4},\ldots,a_{n})\leq(4-1)(7-1)-1=17<25.496479=UB_{2}(4,7,9)\leq
UB_{2}(4,7,a_{3}).$

  5. (v)

$a_{1}=5$: In this case, notice that

$F(5,6,a_{3},a_{4},\ldots,a_{n})\leq(5-1)(6-1)-1=19<21.740852=UB_{2}(5,6,7)\leq
UB_{2}(5,6,a_{3}).$

Thus, provided that $a_{1}a_{2}\leq 32$, then
$F(\boldsymbol{a})<UB_{2}(a_{1},a_{2},a_{3})=UB_{2}(\boldsymbol{a})$. Thus, it
follows that $UB_{2}(\boldsymbol{a})$ is indeed a valid upper bound in all
cases, which concludes the proof. ∎

## References

  * [1] Jorge L Ramírez Alfonsín.  The diophantine Frobenius problem.  OUP Oxford, 2005.
  * [2] Matthias. Beck, Ricardo. Diaz, and Sinai. Robins.  The frobenius problem, rational polytopes, and fourier–dedekind sums.  Journal of number theory, 96(1):1–21, 2002.
  * [3] Sebastian Böcker and Zsuzsanna Lipták.  The money changing problem revisited: computing the frobenius number in time $o(ka1)$.  In International Computing and Combinatorics Conference, pages 965–974. Springer, 2005.
  * [4] Valentin Boju and Louis Funar.  The math problems notebook.  Springer Science & Business Media, 2007.
  * [5] Frank Curtis.  On formulas for the frobenius number of a numerical semigroup.  Mathematica Scandinavica, 67(2):190–192, 1990.
  * [6] Paul Erdős and Ronald Graham.  On a linear diophantine problem of frobenius.  Acta Arithmetica, 21(1):399–408, 1972.
  * [7] Lenny Fukshansky and Sinai Robins.  Frobenius problem and the covering radius of a lattice.  Discrete & Computational Geometry, 37(3):471–483, 2007.
  * [8] Ravi Kannan.  Lattice translates of a polytope and the frobenius problem.  Combinatorica, 12(2):161–177, 1992.
  * [9] Jorge L Ramirez-Alfonsin.  Complexity of the frobenius problem.  Combinatorica, 16(1):143–147, 1996.
  * [10] Öystein J Rödseth.  On a linear diophantine problem of frobenius.  Journal für die reine und angewandte Mathematik, 1978.
  * [11] Sheldon M Ross.  Simulation.  Academic Press, 5 edition, 2013.
  * [12] Ernst S Selmer.  On the linear diophantine problem of frobenius.  Journal für die reine und angewandte Mathematik, 1977(293-294):1–17, 1977.
  * [13] Michael Z Spivey.  Quadratic residues and the frobenius coin problem.  Mathematics Magazine, 80(1):64–67, 2007.
  * [14] James J Sylvester.  Problem 7382.  Educational Times, 37:26, 1884.
  * [15] Amitabha Tripathi.  On a variation of the coin exchange problem for arithmetic progressions.  Integers: Electronic Journal of Combinatorial Number Theory, 3(5), 2003.
  * [16] Yehoshua Vitek.  Bounds for a linear diophantine problem of frobenius.  Journal of the London Mathematical Society, 2(1):79–85, 1975.
  * [17] Herbert S Wilf.  A circle-of-lights algorithm for the money-changing problem.  The American Mathematical Monthly, 85(7):562–565, 1978.


# Medical Image Debiasing by Learning Adaptive Agreement from a Biased Council

Luyang Luo  Member, IEEE  Xin Huang  Minghao Wang  Zhuoyue Wan  and Hao Chen
Senior Member, IEEE This work was supported by the Pneumoconiosis Compensation
Fund Board, HKSARS (Project No. PCFB22EG01), Hong Kong Innovation and
Technology Fund (Project No. ITS/028/21FP), Shenzhen Science and Technology
Innovation Committee Fund (Project No. SGDX20210823103201011), and the Project
of Hetao Shenzhen-Hong Kong Science and Technology Innovation Cooperation Zone
(HZQB-KCZYB-2020083). (Corresponding author: Hao Chen.)Luyang Luo and Xin
Huang are with the Department of Computer Science and Engineering, The Hong
Kong University of Science and Technology, Hong Kong, China (e-mail:
cseluyang@ust.hk).Minghao Wang is with the Department of Chemical and
Biological Engineering, The Hong Kong University of Science and Technology,
Hong Kong, China.Zhuoyue Wan is with Department of Computing, The Hong Kong
Polytechnic University, Hong Kong, China.Hao Chen is with the Department of
Computer Science and Engineering and Department of Chemical and Biological
Engineering, The Hong Kong University of Science and Technology, Hong Kong,
China. Hao Chen is also affiliated with HKUST Shenzhen-Hong Kong Collaborative
Innovation Research Institute, Futian, Shenzhen, China. (e-mail:
jhc@cse.ust.hk).

###### Abstract

Deep learning could be prone to learning shortcuts raised by dataset bias and
result in inaccurate, unreliable, and unfair models, which impedes its
adoption in real-world clinical applications. Despite its significance, there
is a dearth of research in the medical image classification domain to address
dataset bias. Furthermore, the bias labels are often agnostic, as identifying
biases can be laborious and depend on post-hoc interpretation. This paper
proposes learning Adaptive Agreement from a Biased Council (Ada-ABC), a
debiasing framework that does not rely on explicit bias labels to tackle
dataset bias in medical images. Ada-ABC develops a biased council consisting
of multiple classifiers optimized with generalized cross entropy loss to learn
the dataset bias. A debiasing model is then simultaneously trained under the
guidance of the biased council. Specifically, the debiasing model is required
to learn adaptive agreement with the biased council by agreeing on the
correctly predicted samples and disagreeing on the wrongly predicted samples
by the biased council. In this way, the debiasing model could learn the target
attribute on the samples without spurious correlations while also avoiding
ignoring the rich information in samples with spurious correlations. We
theoretically demonstrated that the debiasing model could learn the target
features when the biased model successfully captures dataset bias. Moreover,
to our best knowledge, we constructed the first medical debiasing benchmark
from four datasets containing seven different bias scenarios. Our extensive
experiments practically showed that our proposed Ada-ABC outperformed
competitive approaches, verifying its effectiveness in mitigating dataset bias
for medical image classification. The codes and organized benchmark datasets
will be released via https://github.com/LLYXC/PBBL.

Shortcut Learning, Dataset Bias, Trustworthy Artificial Intelligence, Deep
Learning

##  1 Introduction

Artificial intelligence (AI), typically represented by deep learning, has
achieved expert-level performance in many domains of medical image analysis
[1]. However, the trustworthiness of deep learning models is challenged by
their preference of learning from spurious correlations caused by shortcuts,
or dataset biases [2, 3]. Concerns have also been raised in the medical image
classification domain that deep models could learn biases other than the
targeted features [3, 4, 5, 6, 7], leading to misdiagnosis and unfairness for
the less-represented groups. Consequently, there are rising calls to include
more evaluation procedures to ensure that a deep learning model is unbiased
before deployment as a medical product [8, 9]. Hence, mitigating dataset bias
to develop trustworthy medical models plays a significant role in facilitating
the integration of deep learning into real-world clinical applications.

Figure 1: Dataset bias in medical image classification could lead to
inaccurate and untrustworthy results. Here, the source of data and whether the
patient contains pneumonia are spuriously correlated. A biased model would
make decisions based on the data source while ignoring the patterns of the
lesions. Our goal is to learn a robust model that can make bias-invariant
decisions from the biased training set.

Specifically, dataset biases, or shortcuts, are often referred to the features
that spuriously correlate to the target patterns. Previous works argue that
such features would be preferred by the deep learning models as they are much
easier to learn [4, 10, 11]. In particular, the training of stochastic
gradient descent (SGD) tend to find simple solutions, which is also called
simplicity bias by some and deemed one of the reasons why deep neural networks
generalize well [12, 13, 14]. However, such a preference can also prevent
models from learning more complex patterns, providing a precondition for
models to use shortcuts to quickly fit the entire training data [11, 2, 10].
Simplicity bias is especially harmful when non-causal factors have spurious
correlations with the target patterns. For example, a convolutional neural
network may identify pneumothorax patients based on the patterns of the chest
drain [3, 15], a common treatment to remove air or fluid from the pleural
space. Compared with the complex and ambiguous signs of pneumothorax, chest
drains show clearer patterns on the radiograph and are more easily recognized.
Similar findings have been reported on the preference of learning data source
over thoracic diseases (as shown in Fig. 1) [4], gender signs over
pneumothorax [6], or even laterality markers 111The laterality marker is a
sign of ”L” or ”R” put on a chest radiograph to indicate the side of a
patient. over COVID-19 lesions [5]. Despite being reported frequently, there
are few benchmarks and solutions on mitigating dataset biases in the medical
image domain.

Re-collecting data could remove dataset biases, which is conceptually simple
but practically infeasible [16]. Therefore, many studies attempted to learn a
robust model by up-weighting the minority group of samples (e.g., pneumothorax
cases without chest drains) [17, 18]. Another broad direction proposes
learning invariant representations across different environments [19, 20, 21,
22, 23]. These methods replace the need for data collection with algorithmic
solutions using explicit labels of the dataset bias, which are still less
practical as the dataset bias is often unknown until careful evaluation and
interpretation of the trained model [8, 5], and explicitly labeling the
dataset bias is tedious and expertise-dependent.

More recent studies explored alleviating shortcut learning without explicit
dataset bias labels [24, 10, 25, 26, 27, 4], which can be roughly categorized
into two-stage methods and one-stage methods. The two-stage methods first
capture and predict the bias information and then develop group-robust
learning models based on the predicted bias information [24, 25, 27, 4].
However, these approaches are sensitive to the convergence of biased models
and could bring in noise during the group-robust learning process. In
contrast, the one-stage methods typically develop debiasing models by
comparing its loss with that of a simultaneously trained biased model [10,
26]. Nevertheless, these approaches are mainly based on heuristic loss
weighting functions, which could lead to over-weighing of the samples without
spurious correlations and prevent the model from learning the target patterns.
Taking binary classification as an example, let two binary variables $t$ and
$b$ represent the target feature and bias feature, respectively. A biased
model could majorly learn from the samples with spurious correlation, e.g.,
$t=b$, and make decisions according to $b$. When restricted to only learning
from the data without spurious correlation, the debiasing model could probably
learn another biased decision, e.g., $t\neq b$, and still make decisions
according to $b$. In this sense, heuristically up-weighing the samples without
spurious correlation could be harmful, and a good debias model should learn
from both types of samples.

To this end, this paper proposes Adaptive Agreement from Biased Council, a
one-stage algorithm that debiases via balancing the learning of agreement and
disagreement from the guidance of a biased model. Specifically, a biased model
was trained with the general cross entropy loss which helps capture shortcuts
by encouraging the model to learn easier samples. To foster the bias learning
ability, we introduced the bias council, an ensemble of classifiers learned
from diversified training subsets. To learn a debiasing model, instead of
using heuristic loss weighting functions, we proposed an adaptive agreement
objective by requiring the model to agree with the correct decisions and
disagree with the wrong decisions made by the biased model. Essentially, the
right or wrong decisions by the biased model indicated how likely the samples
were with or without spurious correlations. Hence, learning agreement
prevented the debias model from ignoring largely the rich information
contained in samples with spurious correlations, and learning disagreement
further drove the model to learn a different minimal via the samples without
spurious correlations. Ada-ABC could then be derived by training
simultaneously the bias council and the debiasing model. Further, we provided
theoretic analysis to demonstrate that the adaptive agreement loss enforced
the debiasing model to learn different features from those captured by the
biased model. To demonstrate the effectiveness of our proposed Ada-ABC on
mitigating dataset biases in medical images, we carried out extensive
experiments under seven different scenarios on four medical image datasets
with various dataset biases. We highlight our main contributions as follows:

  * •

We proposed Ada-ABC, a novel one-stage bias label-agnostic framework that
alleviates dataset bias in medical image classification.

  * •

We demonstrated theoretically that with our proposed algorithm, the debiasing
model could learn the target feature when the biased model captures the bias
information.

  * •

To our best knowledge, we provided the first medical debiasing benchmark with
four datasets under seven different scenarios covering various medical dataset
biases.

  * •

We validated the effectiveness of our proposed Ada-ABC in alleviating medical
dataset biases under various situations based on the benchmark.

##  2 Related Works

###  2.1 Dataset Bias in Medical Images

There are many studies reported that deep learning models prefer bias
information other than targeted patterns in the domain of medical image
analysis. Taking chest X-ray (CXR), the commonest medical imaging, as an
example, Zech et al.discovered that CNN generalized poorly on the testing set
from external sources (i.e., a different hospital). Luo et al.[28] showed that
classification models could learn pattern other than disease signs with
quantitative analysis. Viviano et al.[29] further found that CNNs could learn
unwanted features outside the lungs even when restricted to learning from
thoracic disease masks. More specifically, some consistent but medically
irrelevant patterns have been identified. Chest drains, a common treatment for
pneumothorax, were found to be used to identify the disease condition [3]. In
the study by Degrave et al.[5], laterality marker was found to be an evidence
for the model to recognize COVID-19 signs. Recent studies further found that
imbalance of gender [6], race [7], and even socioeconomic [30] could also
cause unfairness in deep learning models for under-represented groups [31].
Moreover, biases could also exist when applying deep learning models to other
medical imaging domains, such as mammography [32] and magnetic resonance
imaging [33]. Despite many reports of dataset biases, works on combating
dataset bias in medical image classification are still scarce. We deem that
one of the main reasons is the lack of benchmarks with at least bias labels in
the testing set. In this paper, we provide a medical debiasing benchmark with
four datasets under seven different scenarios with various dataset biases.

###  2.2 Deep Debiased Learning

There has been an increasing interest in developing debiased models in both
the natural image domain and the medical image domain. We here broadly
categorize the related works in the following two.

Methods using bias labels. A broad branch of work uses resampling or
reweighting strategies to robustly learn representations for both the majority
and the minority groups. Li et al.[17] proposed a minimax algorithm to
automatically learning resampling weights over the training samples. Sagawa et
al.[18] proposed group distributional robust optimization (G-DRO) to
prioritize the learning on worst-performing groups. Another type of studies
emphasizes learning invariant representations. Arjovsky et al.[19] proposed
invariant risk minimization (IRM) to enforce learning invariant
representations across different environment. Zhou et al.[21] further impose
sparsity regularization into IRM to alleviate the overfitting problem caused
by overparameterization. Similarly, contrastive learning [22] and mutual
information minimization [23] have also been utilized to learn more compact
and invariant features across different environments. In this paper, we study
more practical situations where the biases are not explicitly labeled. We will
also show that our proposed method even achieved comparable results to the
approach that used the bias labels.

Methods without bias labels. Labeling biases could be tedious, and finding
biases might rely on post-hoc interpretation of the model [5]. Efforts have
also been devoted to developing debiased models without explicit bias labels.
Two-stage methods often estimate the bias distribution first and then develop
debiasing model with the estimated bias information. Sohoni et al.[24]
estimated the bias information via clustering techniques and then debiasing
with G-DRO. Liu et al.[25] proposed a simple yet effective twice-training
strategy that first learns an ERM model and then develops a debiasing model
based on the sampling weights given by the ERM model. Luo et al.[4] estimated
the Bayesian distribution of the biases and target labels, and then adopted
bias-balanced learning [34] for the second-stage debiasing training.
Nevertheless, these methods were highly sensitive to the convergence of the
biased model, and wrong bias predictions could introduce much noise into the
second stage.

One-stage approaches typically develop the biased and debiasing models
simultaneously. Nam et al.[10] proposed a heuristic loss weighting strategy,
where a debiased model was learned by comparing its loss with another
simultaneously trained biased model. Based on this scheme, Lee et al.[26]
further introduced feature augmentation by swapping the features between the
two networks, Kim et al.[35] proposed to pre-train the model first, and then
use an ensemble of biased models to stabilize the learning of bias
information. The debiasing objective is then a cross entropy loss inversely
weighted by the number of correct predictions given by the biased models.
However, the heuristic loss functions may over-weigh the samples without
spurious correlations and insufficiently utilize the rich information
contained in the majority groups. On the contrary, our proposed Ada-ABC could
assist in balancing the learning of different samples, thus outperforming
other competitive algorithms.

##  3 Methodology

###  3.1 Problem Setup

Generally, a sample data $x$ could contain different attributes, e.g., whether
a chest X-ray contains pneumothorax, whether the patient is male, whether a
chest drain is applied, etc. Let $(t,b)$ be the pair of (possibly latent)
target and bias attributes, the values of $t$ and $b$ are binary, where 0 and
1 represent whether the attribute is absent or present, respectively.
Specifically, $t$ is used for labeling the dataset, and $b$ may not be
recorded due to limited labeling budget or privacy reasons. We can obtain a
dataset
$\mathcal{D}=\\{(\mathit{x}_{1},\mathit{y}_{1}),(\mathit{x}_{2},\mathit{y}_{2}),\cdots\\}$,
where $y_{i}$ represents the label of $x_{i}$. As the data is labeled
according to the target attribute, we have $y_{i}=t_{i}$. The dataset is
biased as the bias attribute is spuriously correlated to the target attribute,
i.e., $b_{i}=y_{i}$ for most of the samples. Therefore, $b$ can be almost as
predictive as $t$. We define a sample with spurious correlation if
$b_{i}=y_{i}$ or without spurious correlation if $b_{i}\neq y_{i}$. Following
previous works [10, 26, 4], we strictly focus on the situations where the
dataset bias is known to exist while not explicitly labeled. Our main goal is
to develop debiasing models that use the target attribute instead of the bias
attribute for making decisions.

To this end, we propose a one-stage debiasing algorithm, Adaptive Agreement
from Biased Council (Ada-ABC), to learn a debiased model without explicit
labeling of the bias attributes. As depicted in Fig. 2, Ada-ABC trains two
networks simultaneously. A model $f_{\theta}$ will be trained with empirical
risk minimization (ERM) to learn the shortcuts as much as possible, where $f$
represents the mapping function of the model and $\theta$ represents the
model’s parameters. The other model $f_{\tilde{\theta}}$ will be trained at
the same time via learning adaptive agreement from $f_{\theta}$.

Figure 2: The Framework of Ada-ABC. The goal is to develop a debiasing model
which is robust to dataset biases (e.g., caused by spurious correlation
between the data source and health condition). A bias council with multiple
classification heads is trained with empirical risk minimization, e.g.,
minimization of generalized cross entropy loss. A second model is
simultaneously trained and required to agree with the correct predictions made
by the ERM model and disagree with the wrong predictions. Under such an
adaptive agreement learning scheme, a different decision-making rule can be
learned from the samples w/o spurious correlations, while rich information
from the samples w/ spurious correlation can be preserved as well.

###  3.2 Learning Adaptive Agreement

Essentially, deep neural networks optimized by empirical risk minimization
(e.g., minimization of the cross entropy loss) prioritize learning from the
simple patterns [36]. Consequently, when an easy bias feature (e.g., chest
drain) is spuriously correlated with a harder target feature (e.g.,
pneumothorax sign), a model optimized by empirical risk minimization prefers
learning the biases [10, 25, 4]. Hence, samples with spurious correlation
could be more correctly classified by $f_{\theta}$, while those without
spurious correlation tend to be misclassified more frequently. Our key
motivation is that, if $f_{\theta}$ is well trained, it would be sufficient to
learn a debiasing model by letting $f_{\tilde{\theta}}$ learn to agree with
the right decisions and disagree with the wrong decisions made by
$f_{\theta}$.

For simplicity, let $p$ and $\tilde{p}$ be the prediction by
$f_{\theta,y=1}(x)$ and $f_{\tilde{\theta},y=1}(x)$, respectively. To learn
agreement, we can set $f_{\tilde{\theta}}$ to be also optimized by the cross
entropy loss:

$\mathcal{L}_{\rm ce}=-\log\tilde{p}.$ (1)

As aforementioned, minimizing $\mathcal{L}_{\rm ce}$ here minimizes the
empirical risk over the entire training set. To achieve disagreement,
intuitively, $f_{\tilde{\theta}}$ should make opposite predictions to
$f_{\theta}$, i.e., $\tilde{p}$ should tend to 0 or 1 when $p$ approaches to 1
or 0, respectively. Motivated by [37], we implement the following loss to
drive $f_{\tilde{\theta}}$ to make an opposite prediction to $f_{\theta}$:

$\mathcal{L}_{\rm opp}=-\log(\tilde{p}(1-p)+p(1-\tilde{p})+\epsilon),$ (2)

where $\epsilon$ is a small value for numerical stabilization.

While bias labels are not available, one of our main challenge here is when to
learn agreement or disagreement, which is essentially the question of when the
biased model would make right or wrong decisions. Particularly, the prediction
by $f_{\theta}$ reveals whether a sample has spurious correlation, as
$f_{\theta}$ is learned with ERM and can be used as an indicator for the
training of the debiasing model. A large or small value of $p$ indicates that
the sample has a high potential for exhibiting spurious correlation or not. In
this way, we propose the following adaptive agreement learning loss:

$\begin{split}\mathcal{L}_{\rm ad}&=p\mathcal{L}_{ce}+(1-p)\mathcal{L}_{\rm
opp}\\\ &=\mathcal{L}_{\rm agr}+\mathcal{L}_{\rm dis}.\end{split}$ (3)

In the above, $p$ adaptively assigns different weights for the samples, where
more agreement learning will be put on the samples with spurious correlations,
and more disagreement will be put on the samples without spurious
correlations. In this way, $\mathcal{L}_{\rm ad}$ can be applied to all
samples with $p$ adjusting the learning of agreement and disagreement.

Importantly, when the ERM model successfully captures the dataset bias $b$, it
can be shown that $f_{\tilde{\theta}}$ would be driven to learn the patterns
of $t$ by the following.

###### Theorem 1

(Eq. 3 encourages learning the target pattern.) Given a joint data
distribution $\mathcal{D}$ of triplets of random variables $(T,B,Y)$ taking
values into $\\{0,\ 1\\}^{3}$, where $T$ represents the target feature and $B$
represents the bias feature. Assuming that an ERM model learned the posterior
distribution $\mathbb{P}_{1}(Y=1|T=t,\ B=b)=b$, meaning that it is invariant
to feature $t$. Then, the posterior solving $\mathcal{L}_{ad}$ objective will
be $\mathbb{P}_{2}(Y=1|T=t,\ B=b)=t$, invariant to feature $b$.

######  Proof 3.2.

Let $T$, $B$, and $Y$ represent the random variables for the target feature,
bias feature, and ground truth label. The training set is a joint distribution
$\mathcal{D}$ of triplets of $(T,\ B,\ Y)$ taking values in $\\{0,\ 1\\}^{3}$.
For simplicity, we further let $\mathop{\mathcal{D}}_{t=b}$ and
$\mathop{\mathcal{D}}_{t\neq b}$ to be uniform on $\\{T,B\\}$, but the
following still holds if the distribution is not uniform. In other words,

$\begin{split}&\mathbb{P}_{\mathcal{D}}(t=0,b=1|t\neq
b)=\mathbb{P}_{\mathcal{D}}(t=1,b=0|t\neq b)=1/2,\\\
&\mathbb{P}_{\mathcal{D}}(t=1,b=1|t=b)=\mathbb{P}_{\mathcal{D}}(t=0,b=0|t=b)=1/2.\end{split}$
(4)

Let $\mathbb{P}$ and $\mathbb{\tilde{P}}$ be the learned distribution of the
biased model and debiasing model, respectively. We further assume that the
biased model learned the posterior distribution $\mathbb{P}(Y=1|T=t,\ B=b)=b$.

Training the debiasing model would minimize the expectation of
$\mathcal{L}_{ad}$ objective over $\mathcal{D}$:

$\begin{split}\min&\mathop{\mathbb{E}}\limits_{(t,\
b)\sim\mathcal{D}}\left[\mathcal{L}_{\rm agr}+\mathcal{L}_{\rm dis}\right]\\\
&=\mathop{\mathbb{E}}_{(t,\
b)\sim\mathcal{D}}\left[p\mathcal{L}_{ce}\right]+\mathop{\mathbb{E}}_{(t,\
b)\sim\mathcal{D}}\left[(1-p)\mathcal{L}_{\rm opp}\right].\end{split}$ (5)

By the mentioned conditions, the biased model successfully captures the bias.
Eq. 5 can then be further re-written to:

$\min\mathop{\mathbb{E}}_{(t,\ b)\sim\mathcal{D}\atop
t=b}\left[\mathcal{L}_{ce}\right]+\mathop{\mathbb{E}}_{(t,\
b)\sim\mathcal{D}\atop t\neq b}\left[\mathcal{L}_{\rm opp}\right],$ (6)

where the first term in Eq. 6 is minimized for agreement in the distribution
of bias-aligned data, which is the empirical risk minimization over
$\mathop{\mathcal{D}}\limits_{t=b}$:

$\begin{cases}&\mathbb{\tilde{P}}(Y=1|t=1,\ b=1)=1,\\\
&\mathbb{\tilde{P}}(Y=1|t=0,\ b=0)=0.\end{cases}$ (7)

The second term in Eq. 6 becomes:

$\begin{split}&\mathop{\mathbb{E}}\limits_{(t,\ b)\sim\mathcal{D}\atop t\neq
b}\left[-\log(\tilde{p}(1-p)+p(1-\tilde{p}))\right]=\\\
&\frac{1}{2}\left[-\log\left(1-\mathbb{\tilde{P}}(Y=1|t=0,\
b=1)\right)\right]\\\
&+\frac{1}{2}\left[-\log\left(\mathbb{\tilde{P}}(Y=1|t=1,\
b=0)\right)\right]\end{split}$ (8)

which is minimized for

$\begin{cases}&\mathbb{\tilde{P}}(Y=1|t=0,\ b=1)=0,\\\
&\mathbb{\tilde{P}}(Y=1|t=1,\ b=0)=1.\end{cases}$ (9)

Combining Eq. 7 and Eq. 9, the posterior learned by the debiasing model
according to our proposed adaptive agreement learning loss will be:

$\mathbb{\tilde{P}}(Y=1|T=t,\ B=b)=t,$ (10)

which shows that the debiasing model learns the targeted feature invariant to
the bias.

###  3.3 Learning the Bias Council

By Eq. 3, the samples on which $f_{\tilde{\theta}}$ should learn agreement or
disagreement are decided by $p$. Then, the following challenge is how to make
$p$ successfully indicate the dataset bias information. In other words, the
next goal is to make $f_{\theta}$ as biased as possible.

To this end, we presented the combination of the generalized cross entropy
(GCE) loss [38] and a diversely trained ensemble of classifiers. The GCE loss
was first proposed for learning from noisy labels, where the samples with
clean labels are often regarded easier samples than those with noisy labels:

$\mathcal{L}_{\rm gce}=\frac{1-f_{\theta}(x)^{q}}{q},$ (11)

where $q\in(0,1]$ is a hyper-parameter balancing the behavior of the loss. For
each class (i.e., the target label is 1), $\mathcal{L}_{\rm gce}$ generates to
the mean absolute error loss when $q=1$ and behaves like conventional cross
entropy loss when $q\to 1$. This can be seen by its gradient:

$\begin{split}\frac{\partial\mathcal{L}_{\rm
gce}(f_{\theta}(x))}{\partial\theta}&=-{f_{\theta}(x)}^{q-1}\frac{\partial
f_{\theta}(x)}{\partial\theta}\\\
&={f_{\theta}(x)}^{q}(-f_{\theta}(x)^{-1}\frac{\partial
f_{\theta}(x)}{\partial\theta})\\\
&={f_{\theta}(x)}^{q}\frac{\partial\mathcal{L}_{\rm
ce}(f_{\theta}(x))}{\partial\theta},\end{split}$ (12)

where $\mathcal{L}_{\rm ce}(f_{\theta}(x))=-{\rm log}(f_{\theta}(x))$. The
above shows that the gradient of the GCE loss is a weighted version of the
gradient of the cross entropy loss, and the weight is given by its own
prediction. In other words, this loss encourages the model to be confident in
its prediction by up-weighing the samples with high predicted probabilities
and down-weighing the samples otherwise. As mentioned, the samples with higher
probabilities are highly potentially the easy samples with spurious
correlations. Hence, $f_{\theta}$ would focus on learning from these samples
to capture the dataset bias.

To further facilitate robust bias learning, we proposed to train a biased
council, which consists of an ensemble of diverse GCE-optimized classifiers.
Specifically, we introduced a group of classification heads
$\\{h^{i}_{\phi_{i}}\\}_{1}^{n}$ to $f_{\theta}$, where $\phi_{i}$ represents
the parameters of $h^{i}$. The $i$-th head would be trained with a subset
$\mathcal{D^{\prime}}_{i}$ randomly sampled from $\mathcal{D}$. The parameters
were also randomly and independently initialized for each head. Thus, the
increased diversity in the training set and parameters would promote the
classifiers to learn a more robust ensemble [39]. Finally, the prediction by
$f$ is set to be the average of the head predictions, i.e.,
$p=f_{\theta}(x)=\sum_{1}^{n}h^{i}_{\phi_{i}}(z)$, where $z$ is the feature
fed to the classifiers.

###  3.4 Holistic Training of Ada-ABC

Combining the above, the training objective for learning adaptive agreement
from bias council can be derived as:

$\mathop{\arg\min}_{\tilde{\theta},\theta}\mathcal{L}_{\rm
agr}(\tilde{\theta})+\lambda\mathcal{L}_{\rm
dis}(\tilde{\theta})+\mathcal{L}_{gce}(\theta),$ (13)

where we add a hyper-parameter $\lambda$ to help balance the learning of
agreement and disagreement in case the samples are too imbalanced. Note that
$\mathcal{L}_{\rm agr}$ and $\mathcal{L}_{\rm dis}$ will not be back-
propagated to $f_{\theta}$ to avoid influence from the gradient of
$f_{\tilde{\theta}}$. The above loss terms can be optimized simultaneously and
need not the convergence of a biased model as a first step.

Algorithm 1 details the one-stage training process. We emphasize that Ada-ABC
does not require any explicit bias labels and only requires the knowledge that
a training set is biased. Furthermore, with the convergence of the biased
model, the proposed adaptive learning scheme enables the debiasing model to
learn a different feature from the wrongly predicted samples and also keeps
the rich knowledge from sufficient samples with spurious correlations.

Algorithm 1 Ada-ABC Training

Input: Dataset $\mathcal{D}=\\{\mathcal{X},\mathcal{Y}\\}$; Parameters of
biased model $\theta$; Parameters of debiased model $\tilde{\theta}$; Hyper-
parameter $\lambda$.
Output: Debiased model $f_{\tilde{\theta}}$.

1:Initialize parameters $\theta$, and $\tilde{\theta}$.

2:while not converge do

3: $(X,Y)\sim\mathcal{D}$

4: $P\xleftarrow{}f_{\theta}(X)$

5: $\ell_{\rm agr}\xleftarrow{}\mathcal{L}_{\rm agr}(\tilde{\theta};X,Y,P)$
$\triangleright$ (1),(3)

6: $\ell_{\rm dis}\xleftarrow{}\mathcal{L}_{\rm dis}(\tilde{\theta};X,Y,P)$
$\triangleright$ (2),(3)

7: $\ell_{\rm ad}\xleftarrow{}\ell_{\rm agr}+\lambda\ell_{\rm dis}$
$\triangleright$ (3)

8:
$\tilde{\theta}\xleftarrow{}\tilde{\theta}-\eta\nabla_{\tilde{\theta}}\ell_{\rm
ad}$

9: $\ell_{\rm gce}\xleftarrow{}\mathcal{L}_{\rm gce}(\theta;X,Y)$
$\triangleright$ (11)

10: $\theta\xleftarrow{}\theta-\eta\nabla_{\theta}\ell_{\rm gce}$

11:end while

##  4 Experiments

###  4.1 Medical Debiasing Benchmark

We constructed the first medical debiasing benchmark (MBD) with four real-
world medical image datasets, including Source-biased Pneumonia classification
dataset (SbP), Gender-biased Pneumothorax classification dataset (GbP), Chest
Drain-biased Pneumothorax classification dataset (DbP), and the OL3I Dataset,
with totally seven different dataset bias scenarios. Detailed numbers of data
used for each dataset can be found in Table 1. In the following, we use $t$
and $b$ to represent the target and bias attributes, respectively.

####  4.1.1 Source-biased Pneumonia Classification (SbP)

Chest X-rays (CXRs) generated from different clinical centers could have
distribution shifts caused by factors such as imaging parameters, vendor
types, patient cohort differences, etc. [40]. SbP is a pneumonia
classification dataset containing most pneumonia cases from MIMIC-CXR [41] and
most healthy cases (no findings) from NIH-CXR [42]. Here,
$t=health\\_condition$, and $b=data\\_source$. Further, there are three
training sets with the ratios of bias-aligned samples of 99%, 95%, and 90%,
respectively. The three scenarios share the same validation and testing sets,
which have uniform distributions on the $t$ and $b$ variables, i.e.,
containing equal numbers of different groups (with pneumonia or without
pneumonia; from NIH or MIMIC-CXR).

####  4.1.2 Gender-biased Pneumothorax Classification (GbP)

Significant performance decreases of a pneumothorax classifier have been
witnessed when training with male cases and testing on female cases (and vice
versa) [6]. GbP dataset was collected from the NIH-CXR dataset, where
$t=health\\_condition$ and $b=gender$. It contains two training sets with most
male patients (case 1) and most female patients (case 2), respectively. The
validation and testing sets are with uniform distributions of $t$ and $b$.

####  4.1.3 Chest Drain-biased Pneumothorax Classification (DbP)

Chest drain is a common treatment for pneumothorax and have been reported as a
type of dataset biases [15, 3]. DbP was collected from the NIH-CXR dataset,
and the chest drain labels were provided by [3]. In the training set, most
pneumothorax cases contain chest drains, and all healthy cases do not contain
chest drains. Here, $t=health\\_condition$, and $b=chest\\_drain$.

####  4.1.4 Age-biased Ischemic Heart Disease Prognosis (OL3I)

The Opportunistic L3 Ischemic heart disease (OL3I) dataset [43] provided
abdominopelvic computed tomography images at the third lumbar vertebrae (L3)
level for opportunistic assessment of ischemic heart disease risk. In this
paper, we predicted the ischemic heart disease risk one year after the
examination. According to [44], age is spuriously correlated to the one-year
risk, where individuals with age larger than 60 is less likely to be healthy
and more likely to obtain ischemic heart disease within one year. In other
words, $t=ischemic\\_heart\\_disease\\_risk$, and $b=age$. We followed the
original split of the OL3I dataset.

Table 1: Detailed number of data in different datasets. $t$ and $b$ represent
the target and bias attribute, respectively. The meaning of $t$ and $b$ for
each dataset can be found in Sec. 4.1.

Dataset | $t$ | Training |  | Validation |  | Testing |
---|---|---|---|---|---|---|---
SbP |  | $b=0$ | $b=1$ | $b=0$ | $b=1$ | $b=0$ | $b=1$
($\rho$=99%) | $1$ | 5,000 | 50 | 200 | 200 | 400 | 400
| $0$ | 50 | 5,000 | 200 | 200 | 400 | 400
($\rho$=95%) | $1$ | 5,000 | 250 | 200 | 200 | 400 | 400
| $0$ | 250 | 5,000 | 200 | 200 | 400 | 400
($\rho$=90%) | $1$ | 5,000 | 500 | 200 | 200 | 400 | 400
| $0$ | 500 | 5,000 | 200 | 200 | 400 | 400
GbP |  | $b=0$ | $b=1$ | $b=0$ | $b=1$ | $b=0$ | $b=1$
(Case1) | $1$ | 800 | 100 | 150 | 150 | 250 | 250
| $0$ | 100 | 800 | 150 | 150 | 250 | 250
(Case2) | $1$ | 100 | 800 | 150 | 150 | 250 | 250
| $0$ | 800 | 100 | 150 | 150 | 250 | 250
DbP |  | $b=0$ | $b=1$ | $b=0$ | $b=1$ | $b=0$ | $b=1$
| $1$ | 500 | 50 | 50 | 50 | 100 | 100
| $0$ | 0 | 1,000 | 0 | 200 | 0 | 400
OL3I |  | $b=0$ | $b=1$ | $b=0$ | $b=1$ | $b=0$ | $b=1$
| $1$ | 87 | 141 | 13 | 43 | 25 | 141
| $0$ | 3,512 | 1,487 | 830 | 417 | 1,060 | 478

####  4.1.5 Evaluation Metrics

(a)

(b)

(c)

(d)

(e)

(f)

Figure 3: Effects of the hyper-parameters $\lambda$ and number of heads. The
first row shows the results on SbP dataset with $\rho=99\%$: (a) The changes
of aligned AUC and conflicting AUC w.r.t. the change of $\lambda$ (# heads =
16). (b) The changes of overall AUC and balanced AUC w.r.t. the change of
$\lambda$ (# heads = 16). (c) The changes of overall AUC and balanced AUC
w.r.t. the change of number of heads ($\lambda$ = 100). The second row shows
the results on OL3I dataset: (d) The changes of aligned AUC and conflicting
AUC w.r.t. the change of $\lambda$ (# heads = 8). (e) The changes of overall
AUC and balanced AUC w.r.t. the change of $\lambda$ (# heads = 8). (f) The
changes of overall AUC and balanced AUC w.r.t. the change of number of heads
($\lambda$ = 300).

In the testing phase, following [10, 26, 4], a sample is called bias-aligned
if its attributes are spuriously correlated in the training data, e.g.,
pneumothorax cases with chest drains. A sample is called bias-conflicting if
it has attributes contradict to the bias-aligned samples, e.g., pneumothorax
cases without chest drains. To observe the debiasing results on different
samples, we compute four types of area under the receive operating
characteristic curve (AUC): i) bias-aligned AUC, which is the AUC computed on
bias-aligned samples; ii) bias-conflicting AUC computed on the bias-
conflicting samples; iii) balanced AUC, which is the average of bias-aligned
AUC and bias-conflicting AUC; and iv) overall AUC computed on all samples.

The bias-aligned AUC and the bias-conflicting AUC are mainly used as a
reference to tell whether the model is highly biased. For example, when the
bias-aligned AUC is too higher than the bias-conflicting AUC, the model could
correctly classify samples with $t=b$ but not sample with $t\neq b$, which
means it’s highly biased. As it’s important to correctly classify all groups
of data, the balanced AUC and overall AUC are used as a fair evaluation for
different models.

###  4.2 Analysis of Ada-ABC

####  4.2.1 Analysis with a Toy Example

We first evaluate the effects of different learning schemes with a toy
example, where the model is optimized to distinguish the samples ($\triangle$
vs. $\square$) according to their coordinates. Fig. 4(a) shows the decision
boundary of a vanilla model optimized by empirical risk minimization, where
the minority groups of samples are misclassified. By learning purely
agreement, i.e., $\mathcal{L}_{\rm agr}$, a second model could learn a similar
decision boundary, as shown in Fig. 4(b). Notably, learning purely
disagreement, i.e., $\mathcal{L}_{\rm dis}$, a second model could generate the
exactly opposite decisions to the vanilla model. Finally, we trained a
debiasing model using our proposed adaptive agreement learning in Eq. 3 with
$\lambda$ set to 1, and a correct decision boundary could be achieved as shown
in Fig. 4(d).

(a)

(b)

(c)

(d)

Figure 4: The decision boundaries by (a) an ERM model that learns a simple solution; another model that learns to (b) purely agree with the ERM model, or (c) purely disagree with the ERM model, (d) or adaptively agree or disagree with the ERM model. Details are best appreciated when enlarged. Dataset | $\lambda$ | # heads
---|---|---
SbP bias90 | 5 | 2
SbP bias95 | 10 | 64
SbP bias99 | 100 | 16
GbP case1 | 1 | 128
GbP case2 | 0.1 | 64
DbP | 1 | 4
OL3I | 300 | 8
Table 2: Summary of hyper-parameters for each dataset. Table 3: Comparison on
SbP. For methods do not use bias labels, the best and second-best performance
(on the balanced AUC and overall AUC) are in red and blue, respectively. The
last row is the average of the mean overall AUC from all seven scenarios.
$\rho$: the ratio of bias-aligned samples in the training set. $\dagger$ means
that the method uses ground truth bias labels.

Dataset | Metric | G-DRO† [18] | ERM | D-BAT [37] | JTT [25] | PBBL [4] | LfF [10] | DFA [26] | Ada-ABC
---|---|---|---|---|---|---|---|---|---
SbP ($\rho=99\%$) | Aligned | 74.30±2.28 | 99.03±0.95 | 45.40±16.17 | 97.02±1.07 | 72.40±0.71 | 77.50±11.08 | 69.33±1.74 | 75.11±6.32
Conflicting | 85.18±1.26 | 4.93±3.68 | 73.18±14.68 | 19.54±4.33 | 77.61±0.45 | 64.38±8.75 | 75.48±2.61 | 81.20±5.32
Balanced | 79.74±0.55 | 51.98±1.60 | 59.29±2.68 | 58.28±1.87 | 75.00±0.18 | 70.94±1.30 | 72.40±0.48 | 78.15±0.50
Overall | 79.71±0.40 | 59.21±3.76 | 61.37±0.92 | 64.37±1.45 | 74.70±0.14 | 71.86±1.72 | 72.49±0.45 | 77.99±0.34
SbP ($\rho=95\%$) | Aligned | 68.65±1.21 | 97.91±0.75 | 64.55±15.39 | 92.09±3.86 | 71.72±6.65 | 69.56±2.01 | 69.04±4.21 | 84.70±1.66
Conflicting | 89.86±0.67 | 20.45±5.96 | 67.82±16.60 | 45.75±11.94 | 84.68±3.49 | 86.43±1.67 | 84.94±2.56 | 73.64±2.22
Balanced | 79.26±0.47 | 59.18±2.61 | 66.18±1.75 | 68.92±4.11 | 78.20±0.20 | 77.99±0.18 | 76.99±0.85 | 79.17±0.47
Overall | 79.80±0.36 | 67.11±1.85 | 66.93±1.96 | 71.79±2.35 | 78.04±3.46 | 78.28±0.22 | 77.26±0.49 | 79.12±0.40
SbP ($\rho=90\%$) | Aligned | 70.02±2.20 | 96.51±0.26 | 82.84±4.47 | 87.36±1.25 | 76.82±2.80 | 68.57±2.16 | 74.63±4.61 | 83.38±3.02
Conflicting | 89.80±0.87 | 31.21±3.04 | 67.66±5.12 | 63.58±2.86 | 85.75±0.32 | 87.46±2.17 | 83.30±3.96 | 77.31±3.81
Balanced | 79.94±0.68 | 63.86±1.39 | 75.26±0.76 | 75.47±0.95 | 80.49±0.20 | 78.02±0.18 | 78.96±0.33 | 80.34±0.39
Overall | 80.23±0.37 | 69.84±1.32 | 75.52±0.73 | 76.25±1.35 | 78.78±3.02 | 78.26±0.18 | 78.76±0.15 | 80.07±0.21
GbP (case 1) | Aligned | 85.81±0.16 | 89.42±0.25 | 86.88±1.35 | 86.99±0.56 | 90.17±0.42 | 88.73±1.34 | 86.12±0.46 | 88.08±0.45
Conflicting | 83.96±0.17 | 77.21±0.33 | 83.43±0.79 | 78.80±1.09 | 77.07±1.73 | 77.47±0.09 | 77.92±0.23 | 78.51±0.59
Balanced | 84.86±0.05 | 83.31±0.05 | 80.00±0.58 | 82.89±0.80 | 83.62±0.68 | 83.10±0.64 | 82.02±0.31 | 83.30±0.52
Overall | 84.93±0.01 | 83.75±0.05 | 83.60±0.87 | 83.16±0.77 | 84.13±0.56 | 83.46±0.71 | 82.23±0.30 | 83.59±0.53
GbP (case 2) | Aligned | 83.76±1.59 | 89.39±0.85 | 87.56±0.77 | 89.30±0.87 | 86.34±0.64 | 87.25±0.62 | 80.44±0.58 | 88.80±0.36
Conflicting | 85.14±0.31 | 76.13±0.93 | 84.39±0.52 | 80.82±0.42 | 81.69±2.67 | 79.07±0.96 | 85.51±0.57 | 81.88±1.33
Balanced | 84.45±0.65 | 82.76±0.78 | 81.23±1.01 | 85.06±0.23 | 84.02±1.01 | 83.16±0.45 | 82.98±0.19 | 85.34±0.48
Overall | 84.42±0.61 | 82.93±0.78 | 84.40±0.93 | 85.20±0.30 | 84.03±0.97 | 83.19±0.44 | 83.09±0.21 | 85.44±0.45
DbP | w/ Drain | 87.99±0.84 | 87.50±0.64 | 87.19±0.72 | 86.93±0.84 | 87.01±1.06 | 86.78±0.48 | 87.31±0.65 | 88.25±0.31
w/o Drain | 77.32±1.68 | 75.27±2.03 | 75.87±0.83 | 73.57±0.93 | 76.90±4.17 | 72.81±0.52 | 74.60±0.04 | 76.96±1.32
Overall | 82.60±1.24 | 81.39±1.31 | 81.53±0.72 | 80.25±0.79 | 80.88±0.65 | 79.79±0.49 | 80.96±0.31 | 82.61±0.82
Ol3I | Aligned | 71.53±4.33 | 87.02±1.13 | 78.13±7.23 | 88.86±1.19 | 86.92±0.32 | 71.73±4.69 | 89.46±2.25 | 89.42±2.21
Conflicting | 42.69±1.83 | 34.52±1.76 | 35.90±4.90 | 31.17±6.84 | 33.62±5.76 | 62.07±1.83 | 37.31±3.82 | 37.16±4.97
Balanced | 57.11±2.92 | 61.27±0.66 | 57.01±3.20 | 60.02±3.43 | 60.27±2.88 | 66.90±1.43 | 63.38±0.78 | 63.29±1.87
Overall | 62.05±3.36 | 72.43±0.96 | 64.52±4.18 | 71.34±2.16 | 71.21±1.03 | 61.79±1.03 | 74.27±0.73 | 74.15±1.00
Averaged Overall AUC | 79.11 | 73.81 | 73.98 | 76.05 | 78.82 | 76.66 | 78.44 | 80.42

####  4.2.2 Effects of Hyper-parameters

We then show the effects of the proposed adaptive agreement learning and the
bias council in Fig. 3. The first and second rows illustrate the results on
the SbP dataset with $\rho=99\%$ and the OL3I dataset, respectively.

We first set the number of classification heads fixed for the debiasing
models, and evaluated the effects of $\lambda$. Generally, $\lambda$ balances
the learning on the bias-aligned samples and the bias-conflicting samples, as
can be observed by the changes of aligned AUC and conflicting AUC from Figs.
3(a) and 3(d). Also, the overall and balanced AUC would be harmed if $\lambda$
was set too small or large, as can be observed from Figs. 3(b) and 3(e).
Essentially, as the agree-disagree loss is applied on all samples, $\lambda$
plays an important role in balancing the preference between learning agreement
and learning disagreement. Decreasing or increasing $\lambda$ would encourage
the learning on the bias-aligned samples or the bias-conflicting samples,
respectively. This effect is more obvious on the SbP dataset where the ratio
of biases in training and testing sets are controlled for better debiasing
demonstration. Then, we set $\lambda$ to a fixed value and varied the number
of classification heads. Generally, using a set of classifiers as a bias
council in the biased model would help debiasing, as can be observed from
Figs. 3(c) and 3(f). Overall, it can be demonstrated that Ada-ABC can robustly
learn from both the bias-aligned samples and the bias-conflicting samples and
manage to mitigate the dataset biases.

###  4.3 Comparative Study with MDB

####  4.3.1 Compared Approaches

The compared methods include i) the ERM model which is trained using cross
entropy loss; ii) Group Distribution Robust Optimization (G-DRO) [18] which
optimizes the performance of the worst-performing group with the knowledge of
bias labels; iii) D-BAT [37], an out-of-distribution generalization method
that trains a set of different classifiers different from each other using an
unlabeled OOD set. The validation sets are used as the OOD sets for this
method. iv) Just Train Twice (JTT) [25], a two-stage approach that first
trains a biased ERM model and then develops the debiased model with a sampling
ratio generated from the ERM model; v) Pseudo Bias-Balanced Learning (PBBL)
[4], a two-stage method which estimates the Bayes distribution of biases and
target labels first and then uses the prior for debiased model training; (vi)
Learning from Failure (LfF) [10], a one-stage algorithm that developed
debiased model with a loss-weighting strategy assisted by a highly biased
model; and (vii) Disentangled Feature Augmentation (DFA) [26], a one-stage
method that further introduces feature disentanglement and augmentation into
LfF.

####  4.3.2 Implementation Details

As different datasets were with different bias scenarios, different value of
$\lambda$ and number of heads were chosen, as shown in Table 2. Moreover, the
hyper-parameter $q$ in the generalized cross entropy loss is set to 0.7 as
recommended in [38].

Our implementations used the PyTorch framework on a GeForce RTX™ 3090 GPU. For
SbP, GbP, and DbP, all methods were finetuned from DenseNet-121 [45], whereas
large-scale CXR pre-trained weights [46] were adopted for the CXR datasets.
For OL3I, experiments were conducted based on ResNet-18 with ImageNet pre-
trained weights, following [44]. Adam [47] with a learning rate of 1e-4 was
used as the optimizer. The results on the testing sets were obtained by the
models with the best overall AUC on the validation set. We constructed three
runs for each method, and the averaged results as well as the standard
deviation were reported.

####  4.3.3 Quantitative Evaluation

Source-biased Pneumonia Classification. As can be observed in Table 3, the ERM
model could still be biased when finetuned on the biased dataset despite
having been pre-trained on large-scale CXR dataset. All debiasing algorithms
mitigated dataset biases to certain levels. Specifically, the one-stage
methods LfF and DFA could prefer learning bias-conflicting samples. On the
other hand, our proposed Ada-ABC increased the AUC on bias-conflicting samples
without a large sacrifice on the bias-aligned AUC. As a result, Ada-ABC showed
robust performance on all three cases with overall AUCs of 77.99%, 79.12%, and
80.07% when $\rho=$ 99%, 95%, and 90%, respectively, achieving consistent
improvement compared with other methods that do not use bias label
information. It’s worth noting that, Ada-ABC gradually achieved comparable
performance to that of G-DRO with the decrease of $\rho$, demonstrating robust
features learning capability.

Gender-biased Pneumothorax Classification. By Table 3, our proposed Ada-ABC
showed high AUC on the bias-aligned samples with increased performance on the
bias-conflicting samples. As a result, Ada-ABC achieved 83.59% and 85.44%
overall AUC under cases of GbP1 and GbP2, respectively. Notably, for GbP2,
Ada-ABC could even surpass G-DRO on the balanced AUC and Overall AUC, showing
an effective and robust solution to the gender bias in medical image
classification.

Drain-biased Pneumothorax Classification. As healthy cases do not have chest
drains, we computed the AUC with healthy samples and pneumothorax cases with
or without chest drains. We also provided the overall AUC. As reported in
Table 3, there is a clear performance gap between the AUCs computed on cases
with or without chest drains, showing that it was much easier for the models
to learn to distinguish pneumothorax with chest drains. We found that other
methods (except G-DRO and D-BAT) may perform even worse than the ERM model,
which indicates that the dataset bias in this case is harder to mitigate. In
contrast, our proposed Ada-ABC demonstrated its effectiveness by achieving the
best performance for all cases, with or without chest drains, showing that it
learned more bias-invariant features.

Age-biased Ischemic Heart Disease Prognosis. The OL3I dataset is not only
biased but also highly imbalanced, posing a hard challenge to the debiasing
algorithms. In our experiment, most of the compared algorithms were not even
comparable to the ERM model. The disadvantage of LfF was also amplified here,
where much weight was put on the high-loss samples by the GCE model, yet the
large proportion of low-loss samples was ignored, leading to high conflicting
AUC and low aligned AUC. Notably, G-DRO achieved a low overall AUC, which is
in line with the findings by Zong et al.[44], and we deemed that it also over-
weighed the minority groups. In this dataset, DFA achieved the best overall
AUC, mostly due to that its feature augmentation also helped alleviate the
class imbalance. The proposed Ada-ABC achieved the second-best overall AUC
with a marginal difference compared to DFA, demonstrating robust feature
learning under scenarios with more complex sub-group distribution shifts.

Overall Results. The last row of Table 3 reports the average of the mean of
overall AUC across the seven scenarios. Our proposed Ada-ABC achieved 80.42%
averaged result on the medical debiasing benchmark, with clear improvement
compared with either the two-stage or one-stage debiasing approaches. We found
that Ada-ABC could even surpass the performance of G-DRO, showing robust
feature learning capability and consistent improvement.

####  4.3.4 Qualitative Visualization

We visualize the saliency maps [48] of the ERM model and the debiasing model
developed with Ada-ABC on the four datasets in Fig. 5. In particular, $t=b=1$
for all the images shown here, and both models gave correct predictions.
However, the ERM model tended to use the wrong regions to identify the
patient. In contrast, the debiasing model developed using Ada-ABC could
successfully mitigate the bias on the shown samples, attending to the correct
regions corresponding to the disease signs. In other words, our proposed model
could learn to make the right decisions for the right reasons. This
observation further highlights the significance of addressing dataset bias for
robust and trustworthy medical image analysis.

Figure 5: The saliency maps by the ERM model (2nd row) and the debiasing model
by Ada-ABC (3rd row). Samples from columns 1-4 are from SbP, GbP, DbP, and
OL3I, respectively. Both models made correct predictions but were looking for
different reasons.

##  5 Conclusion

In summary, this paper proposes a simple yet effective one-stage debiasing
framework, Adaptive Agreement from Biased Council (Ada-ABC). Ada-ABC is based
on simultaneous training of a biased network and a debiasing network. The
biased model is developed to capture the bias information in the dataset,
using a bias council trained with the generalized cross entropy loss to
amplify the learning preference on the samples with spurious correlation.
Then, the debiasing model adaptively learns to agree or disagree with the
biased model on the samples with or without spurious correlation,
respectively, under the supervision of our proposed adaptive learning loss. We
provided theoretical analysis to prove that the debiasing model could learn
the targeted feature when the biased model successfully captures the bias
information. Further, we constructed the first medical debiasing benchmark
(MBD) to our best knowledge, which consists of four datasets with seven
different bias scenarios. Based on MBD, we validated the effectiveness of Ada-
ABC in mitigating dataset bias with extensive experiments and showed that it
consistently achieves state-of-the-art performance in most of the studied
cases. We demonstrated that, both theoretically and practically, Ada-ABC
provides a promising way for more accurate, fair, and trustworthy medical
image analysis.

## References

  * [1] E. J. Topol, “High-performance medicine: the convergence of human and artificial intelligence,” Nature medicine, vol. 25, no. 1, pp. 44–56, 2019.
  * [2] R. Geirhos et al., “Shortcut learning in deep neural networks,” Nature Machine Intelligence, vol. 2, no. 11, pp. 665–673, 2020.
  * [3] L. Oakden-Rayner et al., “Hidden stratification causes clinically meaningful failures in machine learning for medical imaging,” in Proceedings of the ACM conference on health, inference, and learning, pp. 151–159, 2020.
  * [4] L. Luo et al., “Pseudo bias-balanced learning for debiased chest x-ray classification,” in Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part VIII, pp. 621–631, Springer, 2022.
  * [5] A. J. DeGrave, J. D. Janizek and S.-I. Lee, “Ai for radiographic covid-19 detection selects shortcuts over signal,” Nature Machine Intelligence, vol. 3, no. 7, pp. 610–619, 2021.
  * [6] A. J. Larrazabal et al., “Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis,” Proceedings of the National Academy of Sciences, vol. 117, no. 23, pp. 12592–12594, 2020.
  * [7] J. W. Gichoya et al., “Ai recognition of patient race in medical imaging: a modelling study,” The Lancet Digital Health, vol. 4, no. 6, pp. e406–e414, 2022.
  * [8] D. A. Bluemke et al., “Assessing radiology research on artificial intelligence: a brief guide for authors, reviewers, and readers—from the radiology editorial board,” 2020.
  * [9] S. Taylor-Phillips et al., “Uk national screening committee’s approach to reviewing evidence on artificial intelligence in breast cancer screening,” The Lancet Digital Health, vol. 4, no. 7, pp. e558–e565, 2022.
  * [10] J. Nam et al., “Learning from failure: De-biasing classifier from biased classifier,” Advances in Neural Information Processing Systems, vol. 33, pp. 20673–20684, 2020.
  * [11] H. Shah et al., “The pitfalls of simplicity bias in neural networks,” Advances in Neural Information Processing Systems, vol. 33, pp. 9573–9585, 2020.
  * [12] D. Kalimeris et al., “Sgd on neural networks learns functions of increasing complexity,” Advances in neural information processing systems, vol. 32, 2019.
  * [13] K. Hermann and A. Lampinen, “What shapes feature representations? exploring datasets, architectures, and training,” Advances in Neural Information Processing Systems, vol. 33, pp. 9995–10006, 2020.
  * [14] D. Teney et al., “Evading the simplicity bias: Training a diverse set of models discovers solutions with superior ood generalization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16761–16772, 2022.
  * [15] J. Rueckel et al., “Impact of confounding thoracic tubes and pleural dehiscence extent on artificial intelligence pneumothorax detection in chest radiographs,” Investigative Radiology, vol. 55, no. 12, pp. 792–798, 2020.
  * [16] P. Rouzrokh et al., “Mitigating bias in radiology machine learning: 1. data handling,” Radiology: Artificial Intelligence, vol. 4, no. 5, p. e210290, 2022.
  * [17] Y. Li and N. Vasconcelos, “Repair: Removing representation bias by dataset resampling,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9572–9581, 2019.
  * [18] S. Sagawa et al., “Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization,” in International Conference on Learning Representations, 2020.
  * [19] M. Arjovsky et al., “Invariant risk minimization,” arXiv preprint arXiv:1907.02893, 2019.
  * [20] G. Zhang et al., “Quantifying and improving transferability in domain generalization,” Advances in Neural Information Processing Systems, vol. 34, pp. 10957–10970, 2021.
  * [21] X. Zhou et al., “Sparse invariant risk minimization,” in International Conference on Machine Learning, pp. 27222–27244, PMLR, 2022.
  * [22] E. Tartaglione, C. A. Barbano and M. Grangetto, “End: Entangling and disentangling deep representations for bias correction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13508–13517, 2021.
  * [23] W. Zhu et al., “Learning bias-invariant representation by cross-sample mutual information minimization,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15002–15012, 2021.
  * [24] N. Sohoni et al., “No subclass left behind: Fine-grained robustness in coarse-grained classification problems,” Advances in Neural Information Processing Systems, vol. 33, pp. 19339–19352, 2020.
  * [25] E. Z. Liu et al., “Just train twice: Improving group robustness without training group information,” in International Conference on Machine Learning, pp. 6781–6792, PMLR, 2021.
  * [26] J. Lee et al., “Learning debiased representation via disentangled feature augmentation,” Advances in Neural Information Processing Systems, vol. 34, pp. 25123–25133, 2021.
  * [27] E. Kim, J. Lee and J. Choo, “Biaswap: Removing dataset bias with bias-tailored swapping augmentation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14992–15001, 2021.
  * [28] L. Luo et al., “Rethinking annotation granularity for overcoming shortcuts in deep learning–based radiograph diagnosis: A multicenter study,” Radiology: Artificial Intelligence, vol. 4, no. 5, p. e210299, 2022.
  * [29] J. D. Viviano et al., “Saliency is a possible red herring when diagnosing poor generalization,” in ICLR, 2020.
  * [30] L. Seyyed-Kalantari et al., “Chexclusion: Fairness gaps in deep chest x-ray classifiers,” in BIOCOMPUTING 2021: proceedings of the Pacific symposium, pp. 232–243, World Scientific, 2020.
  * [31] L. Seyyed-Kalantari et al., “Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations,” Nature medicine, vol. 27, no. 12, pp. 2176–2182, 2021.
  * [32] B. Zufiria et al., “Analysis of potential biases on mammography datasets for deep learning model development,” in Applications of Medical Artificial Intelligence: First International Workshop, AMAI 2022, Held in Conjunction with MICCAI 2022, Singapore, September 18, 2022, Proceedings, pp. 59–67, Springer, 2022.
  * [33] Q. Zhao, E. Adeli and K. M. Pohl, “Training confounder-free deep learning models for medical applications,” Nature communications, vol. 11, no. 1, p. 6010, 2020.
  * [34] Y. Hong and E. Yang, “Unbiased classification through bias-contrastive and bias-balanced learning,” Advances in Neural Information Processing Systems, vol. 34, 2021.
  * [35] N. Kim et al., “Learning debiased classifier with biased committee,” in Advances in Neural Information Processing Systems, 2022.
  * [36] D. Arpit et al., “A closer look at memorization in deep networks,” in International conference on machine learning, pp. 233–242, PMLR, 2017.
  * [37] M. Pagliardini et al., “Agree to disagree: Diversity through disagreement for better transferability,” in ICLR, 2023.
  * [38] Z. Zhang and M. Sabuncu, “Generalized cross entropy loss for training deep neural networks with noisy labels,” Advances in neural information processing systems, vol. 31, 2018.
  * [39] G. Nam et al., “Diversity matters when learning from ensembles,” Advances in neural information processing systems, vol. 34, pp. 8367–8377, 2021.
  * [40] L. Luo et al., “Deep mining external imperfect data for chest x-ray disease screening,” IEEE transactions on medical imaging, vol. 39, no. 11, pp. 3583–3594, 2020.
  * [41] A. E. Johnson et al., “Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports,” Scientific data, vol. 6, no. 1, pp. 1–8, 2019.
  * [42] X. Wang et al., “Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2097–2106, 2017.
  * [43] J. M. Zambrano Chaves et al., “Opportunistic assessment of ischemic heart disease risk using abdominopelvic computed tomography and medical record data: a multimodal explainable artificial intelligence approach,” Scientific Reports, vol. 13, no. 1, p. 21034, 2023.
  * [44] Y. Zong, Y. Yang and T. Hospedales, “Medfair: Benchmarking fairness for medical imaging,” in The Eleventh International Conference on Learning Representations, 20223.
  * [45] G. Huang et al., “Densely connected convolutional networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 4700–4708, 2017.
  * [46] J. P. Cohen et al., “Torchxrayvision: A library of chest x-ray datasets and models,” in International Conference on Medical Imaging with Deep Learning, pp. 231–249, PMLR, 2022.
  * [47] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. Int. Conf. Learn. Representations, 2015.
  * [48] B. Zhou et al., “Learning deep features for discriminative localization,” in CVPR, pp. 2921–2929, 2016.