# Accelerate Cycle-Level Full-System Simulation of Multi-Core RISC-V Systems
with Binary Translation

Xuan Guo<EMAIL_ADDRESS>University of CambridgeCambridgeUK and  Robert
Mullins<EMAIL_ADDRESS>University of CambridgeCambridgeUK

(2020)

###### Abstract.

It has always been challenging to balance the accuracy and performance of
instruction set simulators (ISSs). Register-transfer level (RTL) simulators or
systems such as gem5 (Binkert et al., 2011) are used to execute programs in a
cycle-accurate manner but are often prohibitively slow. In contrast,
functional simulators such as QEMU (Bellard, 2005) can run large benchmarks to
completion in a reasonable time yet capture few performance metrics and fail
to model complex interactions between multiple cores. This paper presents a
novel multi-purpose simulator that exploits binary translation to offer fast
cycle-level full-system simulations. Its functional simulation mode
outperforms QEMU and, if desired, it is possible to switch between functional
and timing modes at run-time. Cycle-level simulations of RISC-V multi-core
processors are possible at more than 20 MIPS, a useful middle ground in terms
of accuracy and performance with simulation speeds nearly 100 times those of
more detailed cycle-accurate models.

††journalyear: 2020††copyright: ccby††conference: Fourth Workshop on Computer
Architecture Research with RISC-V; May 29, 2020; Virtual Workshop††doi:

##  1\. Introduction

RISC-V is a free, open, and extensible ISA. With the ongoing ecosystem
development of RISC-V and an increasing number of companies and institutions
switching to RISC-V for both production and research, RISC-V has become the
test bed instruction set of computer architecture research. A key tool when
exploring new architectural trade-offs is the instruction-set simulator (ISS).
Fast cycle-level simulation allows new ideas to be validated quickly at an
appropriate level of abstraction and without the complexities of hardware
development. In particular we focus on the challenge of simulating multi-core
RISC-V systems.

The design of a processor can broadly be divided into the design of the core
and memory subsystem. Characterising the performance of the core pipeline in
isolation is often a simpler task to that of characterising the memory system.
While smaller synthetic benchmarks are useful at the core level, larger more
complex and longer running workloads are often needed to understand the memory
system and the potential interactions between cores.

For example, the smaller synthetic MCU benchmark CoreMark (Consortium, 2020)
executes at a magnitude of $10^{8}$ instructions per iteration, while SPEC2017
(Corporation, 2017), a larger and more realistic benchmark running real-life
applications, requires a magnitude of $10^{12}$ instructions for a single run
(Limaye and Adegbija, 2018). SPEC takes from hours to days to run even on real
machines, and hardly possible for simulators to run to completion. It is
therefore helpful to have a fast simulator.

Of course, fast simulation involves a trade-off between the fidelity of the
model and the speed at which simulations can be completed. Unfortunately, we
are currently forced to choose between slow cycle-accurate simulators or fast
functional-only simulators such as QEMU. In particular, there is a lack of
fast full-system simulators that can accurately model cache-coherent multi-
core processors.

In this paper, we present the Rust RISC-V Virtual Machine (R2VM). R2VM is
written in the increasingly popular high-level system programming language
Rust (Team, 2020). R2VM is released 111Available at
https://github.com/nbdd0121/r2vm. under permissive MIT/Apache-2.0 licenses in
the hope to encourage its adoption and expansion by the broader community. To
our knowledge, this is the first binary translated simulator that supports
cycle-level multi-core simulation. It can accurately model cache coherency
protocols and shared caches. Cycle-level simulations are possible at more than
20 MIPS, while the performance of functional-only simulations can outperform
QEMU and exceed 400 MIPS.

##  2\. Background

###  2.1. Instruction Set Simulators

ISSs can be classified as either execution-driven, emulation-driven or trace-
driven (Brais et al., 2020). We omit a detailed discussion of execution-driven
simulators such as Cachegrind (Nethercote and Seward, 2007) that modify
programs with binary instrumentation and execute them natively, because they
require the host and the guest ISA to be identical and do not support full-
system simulation. Emulation-driven simulators emulate the program execution,
and gather performance metrics on-the-fly; in contrast, trace-driven
simulators run emulation before-hand and gather traces from the program, e.g.
branches or memory accesses, and later replay the trace against a specific
model. Traces allow ideas to be evaluated quickly without the need to simulate
in detail, but cannot easily capture effects that may alter the instructions
that are executed, e.g. inter-core interactions or speculative execution
(Brais et al., 2020). Moreover, storage space required for traces grows
linearly with the length of execution, making trace-driven simulators
incapable of simulating large benchmarks. R2VM is an emulation-driven
simulator and the remainder of this paper will focus exclusively on emulation-
driven simulators.

Simulators can also be categorised by their levels of abstraction. One
category of simulators is functional simulators. Functional simulators
simulate the effects of instructions without taking microarchitectural details
into account. Because less information is needed, aggressive optimisations can
be performed, and the performance is usually several magnitudes faster than
timing simulators. QEMU falls into this category. It should be noted though
that while QEMU itself is a purely functional simulator, it can be modified to
collect metadata for off-line or on-line cache simulation (Van Dung et al.,
2014).

The other category is timing simulators. Out of timing simulators, RTL
simulators can model processor microarchitectures very precisely, but the
difficulty in implementing a feature in RTL simulator is not much different
from implementing it in hardware directly. RTL simulators are also poor in
performance, usually run at a magnitude of kIPS (Ta et al., 2018).

At a higher level, there are cycle-level microarchitectural simulators. These
are able to omit RTL implementation details to improve performance while
retaining a detailed microarchitectural model. An popular example is the gem5
simulator running with In-Order or O3 mode (Binkert et al., 2011). For faster
performance, we can give up some extra microarchitectural details and predict
the number of cycles taken for each non-memory instruction instead of
computing them in real-time, and in the extreme case, assume all non-memory
operation only takes 1 cycle to execute as gem5’s “timing simple” CPU model
assumes. This approach is no longer cycle-accurate, but this cycle-approximate
model is often adequate to perform cache and memory simulations.

###  2.2. Binary Translation

Binary translation is a technique that accelerates instruction set
architecture (ISA) simulation or program instrumentation (Hazelwood, 2011). An
interpreter will fetch, decode and execute the instruction pointed by the
current program counter (PC) one-by-one, while binary translation will, either
ahead of time (static binary translation) or in the runtime, i.e. when the
block of code if first executed (dynamic binary translation (DBT)), translate
one or more basic blocks from the simulated ISA to the host’s native code,
cache the result, and use the translation result next time the same block is
executed.

QEMU uses binary translation for cross-ISA simulation or when there is no
hardware virtualisation support (Bellard, 2005). Böhm et al. proposed a method
to introduce binary translation to single-core timing simulation in 2010 (Böhm
et al., 2010).

###  2.3. Multi-core Simulation

Extending single core simulators to handle multiple cores is complicated by
the performance implications of the ways in which cores may interact. As cores
share caches and memory, simulations of individual cores cannot simply be run
independently. For example, accurate modelling of cache coherence, atomic
memory operations and inter-processor interrupts (IPIs) must be considered.

Böhm et al.’s modified ARCSim simulator (Böhm et al., 2010) can model single-
core processors with high accuracy and reasonable performance; however, Almer
et al.’s extension to Böhm et al.’s work (Almer et al., 2011) that essentially
runs multiple copies of the single-core simulator in parallel threads to
provide multi-core support is limited in its fidelity. The author comments
“detailed behaviour of the shared second-level cache, processor interconnect
and external memory of the simulated multi-core platform” cannot be modelled
accurately. QEMU is able to exploit multiple cores to emulate a multi-core
guest but provides only a functional simulation mode and supports no timing or
modelling of the memory system.

An accurate model of cache coherence and the memory hierarchy requires that
multiple cores are simulated in lockstep (or in a way that guarantees
equivalent results). Simulators that forego this are unable to properly
simulate race conditions and shared resources. Existing cycle-level simulators
such as gem5 achieve lockstep by iterating through all simulated cores each
cycle. This causes a significant performance drop. Spike (or riscv-isa-sim),
on the other hand, switches the active core to simulate less frequently. Its
default compilation option only switches the core every 1000 cycles, making it
impossible to model race conditions where all cores are trying to acquire a
lock simultaneously. No existing binary translated simulators can model multi-
core interaction in lockstep, and therefore none of these can model cache
coherency or shared second-level cache properly.

##  3\. Implementation

###  3.1. Overview

The high-level control flow of R2VM, as shown in LABEL:fig:control, is similar
to other binary translators. When an instruction at a particular PC is to be
executed, the code cache is looked up and the cached translated binary is
executed directly if found; otherwise, the binary translator is invoked and an
entire basic block is fetched, decoded, and translated. We have used a variety
of techniques to improve the binary translator performance that are often
found in other binary translators, such as block chaining (Bellard, 2005).

Figure 1. Control flow overview of the simulator

figure]fig:control

As full-system simulation is supported, we have to deal with the case that a
4-byte uncompressed instruction spans two pages. We handle this by creating a
stub that reads the 2 bytes that lie on the second page each time the stub is
executed, and patches the generated code if 2 bytes read are different from
that of initial translation.

Cota et al. (Cota and Carloni, 2019) suggests sharing a code cache between
multiple cores to promote code reuse and boost performance. In contrast, we
provide each hardware thread its own code cache. This allows different code to
be generated for each core, e.g. in the case of heterogeneous cores. This also
lessens the synchronisation requirements when modifying the code cache,
simplify the implementation.

###  3.2. Pipeline Simulation

The main difference between our simulator’s flow and existing ones such as
QEMU is that we introduce “pipeline model”s, which comprises several hooks.
Hooks can process relevant instructions and generate essential
microarchitectural simulation code if necessary. The hooks can also indicate
the number of cycles it would take for the instruction to complete. It should
be noted that this is only for the execution pipeline, while memory systems
and cache are in a separate component.

For simple models, such as gem5’s “timing simple” model where each instruction
takes 1 cycle to execute, implementation is straightforward as shown in
LABEL:listing:simple.

⬇

#[derive(Default)]

pub struct SimpleModel;

\parimpl PipelineModel for SimpleModel {

fn after_instruction(&mut self, compiler: &mut DbtCompiler, _op: &Op,
_compressed: bool) {

compiler.insert_cycle_count(1);

}

\parfn after_taken_branch(&mut self, compiler: &mut DbtCompiler, _op: &Op,
_compressed: bool) {

compiler.insert_cycle_count(1);

}

}

Figure 2. Timing simple model implementation

listing]listing:simple

We have also implemented and validated an in-order pipeline model that
accurately models a classic 5-stage pipeline with a static branch predictor.
Our implementation captures pipeline hazards, such as data hazards caused by
load-use dependency and stalls due to a branch/jump into a misaligned 4-byte
instruction. Unlike Böhm et al.’s simulator (Böhm et al., 2010) which needs to
call a “pipeline” function after each instruction, our implementation models
pipeline behaviours during DBT code generation and reflects them as number of
cycles taken, therefore requires no explicit code to be executed in runtime.

More complex processors may need either to make an estimation of pipeline
states (and sacrifice some accuracy) or generate custom assembly in the hooks
to maintain these states during execution (and sacrifice some performance).

###  3.3. Multi-core Simulation

The techniques we described in the previous section works well for single-core
systems. But as described in the background section, running them in parallel
or switch between them in a coarse-grained manner has a huge impact on
simulation accuracy of multi-threaded programs. The ideal scheduling
granularity is, therefore, a cycle, i.e. having all simulated cores run in
lockstep. This is, however, difficult to achieve for binary translators.

We experimented the idea of using thread barriers to synchronise multiple
threads each simulating a single core. It turns out we could only synchronise
1 million times per second even after careful optimisation at the assembly
level.

####  3.3.1. Lockstep Simulation

The approach we use takes inspiration from fibers, sometimes also referred to
as coroutines or green threads. Fibers are cooperatively scheduled by the
user-space application, and they voluntarily “yield” to other fibers, in
contrast to traditional threads which are preemptively scheduled by the
operating system and are generally heavy-weight constructs. Fibers are often
used in I/O heavy, highly concurrent workloads, such as network programming,
but this time we borrowed it to our simulator.

In our implementation, we create one fiber for each hardware thread simulated,
plus a fiber for the event loop. Each time the pipeline model instructs the
DBT to wait for a few cycles, we will generate a number of yields.
LABEL:listing:example shows an example of generated code under timing simple
model.

⬇

mov rax, qword [rbp+0x78] ; \ mov qword [rbp+0x70], rax ; | add a4, zero, a5

call fiber_yield_raw ; /

mov eax, dword [rbp+0x78] ; \ add eax, -0x1 ; |

cdqe ; | addiw a5, a5, -1

mov qword [rbp+0x78], rax ; |

call fiber_yield_raw ; /

mov eax, dword [rbp+0x70] ; \ imul eax, dword [rbp+0x50] ; |

cdqe ; | mulw a0, a4, a0

mov qword [rbp+0x50], rax ; |

call fiber_yield_raw ; /

Figure 3. Example of generated code with yield calls. RBP points to the array
of RISC-V registers.

listing]listing:example

Figure 4. Memory layout of fibers

figure]fig:layout

Different from normal fiber implementation, we engineered the fiber’s memory
layout to look like LABEL:fig:layout to suit the need of a simulator. Each
fiber is allocated with a 2M memory aligned to 2M boundary, and the stack for
running code under the fiber is contained within this memory range. The
alignment requirement allows the fiber’s start address to be recovered from
the stack pointer by simply masking out least significant 21 bits. The base
pointer points to the end of fixed fiber structures rather than the beginning,
so that positive offsets from the base pointer can be used freely by the DBT-
ed code, while the negative offsets are used for fiber management.

The ABI of the host platform for DBT-ed code is not respected; we rather
specify all registers other than the base pointer and stack pointer to be
volatile, or caller-saved. By doing so, fiber_yield_raw does not need to bear
the cost of saving any registers. To yield in non-DBT-ed code, we can
alternatively push ABI-specified callee-saved registers into the stack and
switch.

This careful design makes fiber switching lightning fast; the fiber_yield_raw
function is as simple as 4 instructions on AMD64, shown in
LABEL:listing:yield.

⬇

fiber_yield_raw:

mov [rbp - 32], rsp ; Save current stack pointer

mov rbp, [rbp - 16] ; Move to next fiber

mov rsp, [rbp - 32] ; Restore stack pointer

ret

Figure 5. Implementation of the fiber yielding code

listing]listing:yield

####  3.3.2. Synchronisation Points

Simply yielding a few cycles after every executed instruction will severely
limit performance and in many cases will be unnecessary. In practice, we only
need to synchronise at points where the execution pipeline can produce visible
side-effects to other cores and/or the rest of the system, or where the rest
of the system’s behaviour would affect the running pipeline.

We observe that there are three ways that a pipeline interacts with another:

  * •

An memory operation is performed.

  * •

An control register operation is performed. This includes read/write to
performance monitor registers, or control registers related to the memory
system.

  * •

An interrupt happens.

For the first two types of interaction, we insert a synchronisation point
before and after they are executed. For the third case (interrupts), because
it is generally difficult to interrupt an DBT-ed code mid-way, we choose to
check for interrupts only at the end of basic blocks. We believe that this
decision will not affect the accuracy of our simulation due to the inherent
entropy of I/O operations.

The positioning of yielding that lies in between two synchronisation points,
therefore, would have no visible side-effects and cannot be distinguished. Our
implementation postpones all yielding until the next synchronisation point. We
tweaked our yield implementation as shown in LABEL:listing:yield slightly to
allow multi-cycle yield, and it demonstrates around 10% performance gain
compared to naive yielding.

###  3.4. Memory Simulation

Figure 6. Control flow for memory accesses

figure]fig:memflow

Previous sections described how we simulate each core’s processing pipeline
and how we achieve simulation in lockstep. The techniques we described and
implemented speeds up pipeline simulation, but the speedup could be very
limited when all memory accesses are still simulated. Moreover, the
instruction cache and translation-lookaside buffers (TLBs) would also need to
be simulated for accuracy.

####  3.4.1. L0 Data Cache

For memory operations, each running core has its own “L0 data cache”. When a
core needs to read from or write to a memory address, it first checks if it is
in the L0 data cache. If it hits, then memory access is performed entirely
within DBT-ed code, bypassing the memory model entirely.

As a result, the memory model will not intercept all memory accesses. It is
therefore important to control what could be in the L0 data cache. We maintain
a property that if an access hits the L0 data cache, then it must be a cache
hit would the memory access reach the memory model. We speed up TLB simulation
with a similar approach in our previous work (Guo and Mullins, 2019).

In our previous TLB simulation work, the property mandates an invariant that
all entries in L0 TLB are in the L1 data TLB. The invariant kept in R2VM is
that all L0 data cache entries are contained both in L1 data TLB and L1 data
cache. Therefore, as shown in LABEL:fig:memflow, when entries are evicted from
either the simulated TLB or cache model, corresponding entries need to be
flushed from the L0 data cache for the inclusiveness property.

Figure 7. Memory layout of a tag entry in L0 data cache

figure]fig:l0d

We carefully engineered the memory layout of L0 data cache entries for maximum
efficiency. The L0 data cache is direct-mapped, with each entry representing a
cache line. Each entry has a memory layout like LABEL:fig:l0d. It does not
store actual memory contents; it rather stores a translation from the virtual
tag to a physical address. In a sense, it is more like a TLB with cache-line
granularity than a cache. We pack the XOR-ed value of guest physical address
and corresponding guest virtual address, plus a bit indicating if the cache
line is read-only to a single machine word.

For each memory access, the L0 data cache is indexed into using the virtual
tag. For read access, we compare if `T >> 1` is equal to vtag. For write
access, we compare if `vtag << 1` is equal to `T`. If the check passes, the
requested virtual address is XOR-ed with `A` to produce the address to access
directly within DBT-ed code. If the check fails, the cold path is executed and
the memory model is invoked. The memory model will simulate both TLB and data
cache, and either triggers a page fault or inserts an entry into the L0 data
cache.

The existence of L0 data cache promises the performance of R2VM’s fast-path,
because it requires only 3 memory operations for each memory operation
simulated. In the default configuration, because the memory model does not
intercept all memory accesses, replacement policies such as least-recently
used (LRU) cannot be used for the simulated TLB and cache. Generally, we
believe this is an acceptable accuracy loss to trade vastly better simulation
performance. If LRU-like policies are really needed, the L0 data cache could
be bypassed and the memory model be invoked for each memory access, in
sacrifice of performance.

####  3.4.2. L0 Instruction Cache

R2VM also simulates instruction TLB and caches similar to the data cache. Each
core also has its own L0 instruction cache, with a simpler entry layout
because read/write permission needs not to be distinguished. To keep the
overhead of simulating instruction cache down, instead of accessing it each
time an instruction is executed, we instead do it only when a basic block
begins, or when the instruction being translated is in a different cache line
compared to the previous instruction. For a cache line size of 64 bytes, this
means that we only need to generate a single L0 instruction cache access for
every 16-32 instructions.

We also creatively use the L0 instruction cache to optimise jumps across
pages. Traditionally, because the page mapping might change and therefore the
actual target of jump instruction might change, DBT s have to conservatively
not link these blocks together. We instead check the L0 instruction cache
(which we would need to check anyway when next block begins) and see if the
target is the same as the cached target. If so, the cached target is used and
the control does not go back to the main loop.

####  3.4.3. Cache Coherency

Our design for the memory system inherently supports the use of cache
coherency. Whenever the cache coherency protocol requires an invalidation, it
can be flushed from the L0 data cache of the target core. Because all
simulated cores execute in lockstep, and there are synchronisation points
before all memory accesses, the effect of the invalidation will be visible
before the next memory access.

###  3.5. Runtime Reconfiguration

R2VM is capable of doing user-level simulation, supervisor-level simulation
and machine-level simulation. For user-level simulation, Linux syscalls are
emulated, and for supervisor-level, supervisor binary interface (SBI) calls
are emulated.

In many cases, we want to gather cache statistics with the behaviour of
operating system (OS) taken into account, but we do not want to count the OS
booting and workload preparation steps before the region of interest, and do
not want to pay for the performance overhead of detailed models for these
portions. The design of R2VM takes this into account, and both pipeline and
memory models can be switched dynamically in the runtime. The switching is
controlled by writing a special control and status register (CSR) in the
vendor-specific CSR range.

R2VM supports pipeline model switching by simply flushing the code cache for
translated binary, and let the DBT engine to use the new model’s hooks for
code generation. Moreover, since as mentioned previously in Section 3.1, each
core has its own code cache for DBT-ed code, we allow the pipeline models to
be specified per core rather than at once.

The memory model is switched in the runtime by flushing the L0 data cache and
the instruction cache. The cache line size is also a runtime-configurable
property. For example, if both TLB and cache are simulated, the cache line
size can be set to 64 bytes. If only TLB is simulated, the cache line size can
be set to 4096 bytes, turning L0 data cache effectively into an L0 data TLB.

If the memory model permits, R2VM can also switch between lockstep execution
and parallel execution like other binary translators during the runtime.
Parallel execution is enabled on the “atomic” memory model. When paired with
the “atomic” pipeline model this behaves functionally equivalent to QEMU and
gem5’s atomic model which permits fast-forwarding of aforementioned booting
and preparation steps.

##  4\. Evaluation

As described in Section 3, R2VM offers a range of pipeline models and memory
models to select from, and allows switching between them mid-simulation. Each
model shows different trade-offs. The list of pre-implemented pipeline and
memory models can be found in Table 1 and Table 2.

Name | Description
---|---
Atomic | Cycle count not tracked
Simple | Each non-memory instruction takes one cycle
InOrder | Models a simple 5-stage in-order scalar pipeline
Table 1. List of pre-implemented pipeline models Name |  Description
---|---
Atomic |  Memory accesses not tracked
TLB |  TLB hit rate collected; cache not simulated
Cache |  Cache hit rate collected; TLB and cache coherency not modelled; parallel execution allowed
MESI |  A directory-based MESI cache coherency protocol with a shared L2. Lockstep execution required.
Table 2. List of pre-implemented memory models

###  4.1. Accuracy and Validation

For pipeline models, we validated the accuracy of the in-order model against
an actual RTL implementation of a RISC-V core using CoreMark (Consortium,
2020). CoreMark is particularly helpful for this validation, as CoreMark’s
working set is small enough to fit into caches and therefore the memory system
of the RTL implementation would not affect the benchmark result. In our run,
the RTL implementation reports 2.10 CoreMark/MHz where the in-order model,
when paired with the atomic memory model, reports 2.09 CoreMark/MHz. The
difference is less than 1%. The “simple” model is simply validated by checking
that all cores have their MCYCLE and MINSTRET CSR equal.

For memory models, we used a few micro-benchmarks to cover the use case for
each model. For TLB and cache simulation, we used a single-core micro-
benchmark that is similar to the MemLat tool from the 7-zip LZMA benchmark
(Pavlov, 2019). For the MESI cache-coherency model, we used a micro-benchmark
to simulate a scenario where two cores are heavily contending over a shared
spin-lock. The memory model under test is used together with the validated in-
order pipeline model, and we compare the number of cycles taken to execute a
benchmark in R2VM and in RTL simulation. The error is around for the 10% for
the cache coherency model and lower for non-coherent models. Though not as
accurate as the pipeline model, we believe at this accuracy the simulation can
provide representative-enough metrics for exploring design decisions.

###  4.2. Performance

Figure 8. Performance comparison between models and other simulators

figure]fig:perf

We evaluated the performance of R2VM against QEMU using the deduplication
workload from PARSEC (Bienia et al., 2008) on 4 cores to test the integer
performance of the simulator (as both R2VM and QEMU interprets floating-point
operations). The kIPS numbers of the gem5 simulator are from Saidi et al.’s
presentation (Saidi and Sandberg, [n. d.]).

As shown in LABEL:fig:perf, the techniques we use lead to superb performance.
When caches are not simulated and therefore cores can run in parallel threads,
R2VM runs at $>$300 MIPS per core, even outperforming QEMU. Lockstep execution
brings down performance by 10x to $\sim$30 MIPS (for 4 simulated cores in a
single-threaded), but this is still significantly faster than gem5.

Thanks to our pipeline model design which moves most simulation to DBT
compilation time rather than runtime, and to our memory model design which
offloads most memory accesses by using L0 caches, simulating pipelines and
cache coherency protocols did not add a significant overhead themselves,
compared to the overhead of lockstep execution.

##  5\. Conclusion

We have introduced R2VM, a multi-purpose binary translating simulator that is
able to simulate multi-core RISC-V systems at the cycle-level at high-speed.
This is done by leveraging the use of fibers to support fast lockstep
execution. Overall, optimisations made R2VM possible to achieve functional
simulation performance that exceeds that of QEMU and cycle-level simulation
nearly 100 times faster than gem5.

## References

  * (1)
  * Almer et al. (2011) Oscar Almer, Igor Böhm, Tobias Edler Von Koch, Björn Franke, Stephen Kyle, Volker Seeker, Christopher Thompson, and Nigel Topham. 2011.  Scalable multi-core simulation using parallel dynamic binary translation. In _2011 International Conference on Embedded Computer Systems: Architectures, Modeling and Simulation_. IEEE, 190–199.
  * Bellard (2005) Fabrice Bellard. 2005\.  QEMU, a fast and portable dynamic translator.. In _USENIX Annual Technical Conference, FREENIX Track_ , Vol. 41. 46.
  * Bienia et al. (2008) Christian Bienia, Sanjeev Kumar, Jaswinder Pal Singh, and Kai Li. 2008.  The PARSEC benchmark suite: Characterization and architectural implications. In _Proceedings of the 17th international conference on Parallel architectures and compilation techniques_. ACM, 72–81.
  * Binkert et al. (2011) Nathan Binkert, Bradford Beckmann, Gabriel Black, Steven K Reinhardt, Ali Saidi, Arkaprava Basu, Joel Hestness, Derek R Hower, Tushar Krishna, Somayeh Sardashti, et al. 2011\.  The gem5 simulator.  _ACM SIGARCH computer architecture news_ 39, 2 (2011), 1–7.
  * Böhm et al. (2010) Igor Böhm, Björn Franke, and Nigel Topham. 2010\.  Cycle-accurate performance modelling in an ultra-fast just-in-time dynamic binary translation instruction set simulator. In _2010 International Conference on Embedded Computer Systems: Architectures, Modeling and Simulation_. IEEE, 1–10.
  * Brais et al. (2020) Hadi Brais, Rajshekar Kalayappan, and Preeti Ranjan Panda. 2020\.  A Survey of Cache Simulators.  _ACM Computing Surveys (CSUR)_ 53, 1 (2020), 1–32.
  * Consortium (2020) The Embedded Microprocessor Benchmark Consortium. 2020.  CoreMark.  https://www.eembc.org/coremark/.  Accessed: 2020-04-14.
  * Corporation (2017) The Standard Performance Evaluation Corporation. 2017.  SPEC CPU® 2017\.  https://www.spec.org/cpu2017/.  Accessed: 2020-04-14.
  * Cota and Carloni (2019) Emilio G Cota and Luca P Carloni. 2019.  Cross-ISA machine instrumentation using fast and scalable dynamic binary translation. In _Proceedings of the 15th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments_. 74–87.
  * Guo and Mullins (2019) Xuan Guo and Robert Mullins. 2019.  Fast TLB Simulation for RISC-V Systems. In _Third Workshop on Computer Architecture Research with RISC-V_.
  * Hazelwood (2011) Kim Hazelwood. 2011\.  Dynamic binary modification: Tools, techniques, and applications.  _Synthesis Lectures on Computer Architecture_ 6, 2 (2011), 1–81.
  * Limaye and Adegbija (2018) Ankur Limaye and Tosiron Adegbija. 2018.  A workload characterization of the SPEC CPU2017 benchmark suite. In _2018 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)_. IEEE, 149–158.
  * Nethercote and Seward (2007) Nicholas Nethercote and Julian Seward. 2007.  Valgrind: a framework for heavyweight dynamic binary instrumentation.  _ACM Sigplan notices_ 42, 6 (2007), 89–100.
  * Pavlov (2019) Igor Pavlov. 2019\.  7-Zip LZMA Benchmark.  https://www.7-cpu.com/.  Accessed: 2020-04-14.
  * Saidi and Sandberg ([n. d.]) Ali Saidi and Andreas Sandberg. [n. d.].  gem5 Virtual Machine Acceleration.  http://www.m5sim.org/wiki/images/c/c3/2012_12_gem5_workshop_kvm.pdf.
  * Ta et al. (2018) Tuan Ta, Lin Cheng, and Christopher Batten. 2018.  Simulating Multi-Core RISC-V Systems in gem5. In _Workshop on Computer Architecture Research with RISC-V_.
  * Team (2020) The Rust Team. 2020\.  Rust Programming Language.  https://www.rust-lang.org/.  Accessed: 2020-04-14.
  * Van Dung et al. (2014) Tran Van Dung, Ittetsu Taniguchi, and Hiroyuki Tomiyama. 2014\.  Cache simulation for instruction set simulator QEMU. In _2014 IEEE 12th International Conference on Dependable, Autonomic and Secure Computing_. IEEE, 441–446.


11institutetext: DTU Compute, Technical University of Denmark
11email: {ninwe, afhar<EMAIL_ADDRESS>22institutetext: Fraunhofer Institute
for Digital Medicine MEVIS, Germany

# Slicing Through Bias: Explaining Performance Gaps in Medical Image Analysis
using Slice Discovery Methods

Vincent Olesen 11 Nina Weng 11 Aasa Feragen 11 Eike Petersen 1122

###### Abstract

Machine learning models have achieved high overall accuracy in medical image
analysis. However, performance disparities on specific patient groups pose
challenges to their clinical utility, safety, and fairness. This can affect
known patient groups – such as those based on sex, age, or disease subtype –
as well as previously unknown and unlabeled groups. Furthermore, the root
cause of such observed performance disparities is often challenging to
uncover, hindering mitigation efforts. In this paper, to address these issues,
we leverage Slice Discovery Methods (SDMs) to identify interpretable
underperforming subsets of data and formulate hypotheses regarding the cause
of observed performance disparities. We introduce a novel SDM and apply it in
a case study on the classification of pneumothorax and atelectasis from chest
x-rays. Our study demonstrates the effectiveness of SDMs in hypothesis
formulation and yields an explanation of previously observed but unexplained
performance disparities between male and female patients in widely used chest
X-ray datasets and models. Our findings indicate shortcut learning in both
classification tasks, through the presence of chest drains and ECG wires,
respectively. Sex-based differences in the prevalence of these shortcut
features appear to cause the observed classification performance gap,
representing a previously underappreciated interaction between shortcut
learning and model fairness analyses.

###### Keywords:

Slice Discovery Methods Algorithmic Fairness Shortcut Learning Chest X-ray
Model Debugging

##  1 Introduction

Machine learning models have shown great promise in medical image-based
diagnosis, sometimes with performance claims that rival human experts.
However, reported performance may overstate these models’ clinical utility and
safety [28]. Specifically, models may underperform or fail systematically on
critical subsets of data even while overall average accuracy remains high. In
computer vision research, such subsets are called _underperforming slices_ or
_blind spots_ [7, 22], where ‘slice’ refers to a subset of samples with
similar characteristics, such as an attribute familiar to a domain expert.

In medical image analysis, a slice of patients may be underperforming for a
wide range of reasons, including group under-representation, increased input
or label noise, fundamental differences in the difficulty of the prediction
task, and shortcut learning [5, 11, 19, 21]. Performance disparities between
patient groups have been observed in many medical image analysis domains [4,
8, 16, 24, 31], raising concerns about the potential unfairness resulting from
the application of such models. However, properly _mitigating_ such
performance disparities requires identifying their root cause, which is often
challenging [18, 21, 27]. The challenge is further compounded by the fact that
the feature that causally distinguishes high-performing from low-performing
patients is often unknown and, thus, not annotated. This renders simple
subgroup analyses based on available metadata insufficient for identifying the
causes of performance disparities.

Figure 1: A general overview of the key elements of slice discovery methods.

To address the issue of unknown distinguishing features, various methods for
the unsupervised discovery of underperforming slices have been proposed in the
computer vision literature. Such methods are variously known as _Slice
Discovery Methods_ (SDMs) [7] or _Blindspot Discovery Methods_ (BDMs) [22].
Typically, these methods perform a cluster analysis on the input space and
then select poorly performing clusters, or _slices_ , for further analysis;
refer to fig. 1 for a high-level overview. SDMs can aid machine learning
practitioners and domain experts in identifying underperforming sets of data,
as well as in forming hypotheses about the _causes_ of this underperformance.
With few exceptions [19], SDMs have not yet found widespread use in the
medical imaging domain.

In this study, we explore the use of SDMs for the analysis of performance
disparities in medical image analysis. Our contributions are twofold. Firstly,
we provide a general overview of SDMs in medical image analysis and we propose
a novel SDM, rigorously justifying all of our design choices. We demonstrate
the effectiveness of our proposed SDM for hypothesis formulation in a case
study of pneumothorax and atelectasis classification on two public chest X-ray
datasets (NIH-CXR 14 [26] and CheXpert [9]). Secondly, by further analyzing
the hypotheses generated using our SDM, we show that chest drain shortcut
learning causes a previously unexplained yet variously reproduced performance
gap between male and female subjects in pneumothorax classification. This
constitutes an important link between shortcut learning and model fairness
analyses that has, to the authors’ knowledge, not been described before. In
addition, using our SDM, we discover a new shortcut feature (the presence of
ECG cables) that may explain male–female performance disparities in
atelectasis classification. The code for our analyses will be made publicly
available after acceptance.

##  2 Related work

###  2.1 Bias and shortcuts in chest x-ray analysis

Algorithmic fairness in medical image analysis, and performance disparities
between patient groups in particular, have recently come under rapidly
increasing scrutiny [4, 14, 19, 23, 24, 27]. In this context, the fairness of
chest x-ray-based disease classification models has received particularly
broad attention [1, 8, 14, 24, 27, 29]. Larrazabal et al. [14] demonstrated
that such models had better classification performance for a particular
patient group (based on biological sex) if that group was represented more
strongly in the training dataset. While not the focus of their study, their
results also indicated significant differences between model performance on
male and female subjects, with the classification models performing better for
either group in different diseases. These (sometimes large) performance gaps
persisted even in the case of sex-balanced training sets. This observation
prompted Weng et al. [27] to investigate the hypothesis that biological sex
differences were causing these unexplained performance gaps. Based on their
results, the authors rejected this hypothesis, leaving the performance gaps
unexplained. Zhang et al. [29] employed standard algorithmic fairness
mitigation approaches to the chest x-ray case, finding that simple group
balancing was one of the most robust approaches – which did not, however,
mitigate the performance gaps observed by Larrazabal et al. [14].

In a separate development, it has been widely demonstrated that chest x-ray-
based disease classification models are prone to relying on shortcut learning
[1, 5, 11, 19]. Both Oakden-Rayner et al. [19] and Jiménez-Sánchez et al. [11]
demonstrate how pneumothorax classification tends to rely heavily on the
presence of chest drains, which represent the standard treatment for
pneumothorax. Connecting the two challenges of shortcut learning and fairness,
many authors have raised concerns about the potential for deep learning models
to exploit spurious correlations between sensitive attributes, such as age,
gender, or ethnicity, and the prediction target [1, 8]. To the authors’
knowledge, the fact that shortcut learning relying on _non-_ sensitive
features (such as the presence of chest drains) can explain performance
disparities between sensitive groups (such as gender groups) has not been
discussed explicitly before. Notably, Jiménez-Sánchez et al. [11] took
important first steps in this direction, by differentially reporting the
effect of shortcut learning on different gender groups.

###  2.2 Slice discovery methods

Slice discovery methods (SDMs) are a recently emerging tool for the
performance analysis and subsequent improvement of deep learning models. In
particular, they aim to solve the problem that the features that identify
underperforming groups of inputs might not be known a priori. To this end,
SDMs typically perform unsupervised clustering of the input data, in order to
identify semantically similar ‘slices’ of data that the model under analysis
performs poorly on. In more detail, SDMs usually consist of the following
steps: (1) the input data is embedded into a latent representation space, (2)
some SDMs perform dimensionality reduction, (3) an unsupervised learning
method, such as clustering, is used to extract slices of the data, and (4) the
extracted slices are prioritized based on a performance metric, such as
accuracy. In table 1, adapted from Plumb et al. [22], we summarize previously
proposed SDMs and their respective high-level design.

Table 1: A summary of slice discovery methods. Clf: The representation used by the classification model under analysis. Adapted from Plumb et al. [22]. Method | Rep. | Dim. reduction | Clustering
---|---|---|---
Algorithmic measurement [19] | Clf |  | KNN
MultiAccuracy Boost [13] | VAE |  | Linear model
GEORGE [25] | Clf | UMAP (d=1,2) | GMM
Spotlight [6] | Clf |  | GMM
Planespot [22] | Clf | scvis (d=2) | GMM
Domino [7] | CLIP | PCA (d=128) | GMM
Failure mode distillation [10] | CLIP |  | SVM
Bias-Aware Hierarchical Clustering [17] | Clf | UMAP(d=2) | Mod. K-Means
Proposed SDM (Ours) | Clf | FC layer (d=128) | GMM

The most common methods to extract slices are clustering algorithms, namely
the Gaussian Mixture Model (GMM) and its variants. The most common choice of
image embedding is the latent space representation computed by the
classification model under scrutiny, i.e., its penultimate layer’s outputs.
However, recent methods instead utilize multi-modal pre-trained models like
CLIP to enable the generation of text descriptions for extracted slices [7,
10]. Interestingly, the crucial dimensionality reduction step has received
relatively little attention yet, as recently pointed out by Plumb et al. [22].
Possibly due to their relatively recent emergence, SDMs have not yet been
widely applied in the medical image domain. In this regard, the work of
Oakden-Rayner et al. [19] represents a very notable early exception that
precedes more recent SDM developments.

##  3 Methodology

###  3.1 Proposed slice discovery method

This section introduces our proposed SDM and motivates our design choices. The
proposed method consists of the following four steps, following fig. 1:

Image representation.

We use the image representation computed by the penultimate layer of the
classification model under scrutiny. As opposed to SDMs that use a separate
model (such as CLIP) to obtain image embeddings, this approach relies only on
information also available to the model’s final classification layer.

Dimensionality reduction.

We insert a single fully connected layer with $n$-dimensional output between
the classification model’s penultimate layer and its final classification
head, and we train (just) this additional layer following the same training
procedure that was used for the classification model itself. Similarly to the
previous step, and contrary to standard choices such as PCA, t-SNE, or UMAP,
this approach is _supervised_ such that information relevant to the
classification decision is preferably preserved.

Clustering.

We use a Gaussian Mixture Model (GMM) for clustering and the Bayesian
Information Criterion (BIC) for choosing the number of clusters. We perform
clustering on disease-positive and -negative samples separately to extract
clusters of the same error type, similar to Oakden-Rayner et al. [19] who only
considered positive cases.

Cluster selection.

We propose using the Brier score (BS), a proper scoring rule, to prioritize
under- and overperforming slices, equivalent to mean squared error between
model confidence and binary target labels. The main motivation for our choice
is that, as opposed to classification accuracy, the BS is threshold-
independent. In addition, it captures both the model’s discriminative ability
and its calibration [2]. As opposed to AUROC [12], per-cluster BS can be
meaningfully compared between groups, and as opposed to many calibration
metrics [20, 23], it can be meaningfully compared between clusters of
different sizes. We quantify BS uncertainty by simple bootstrapping of each
cluster and select the cluster with the lowest 97.5-quantile as the best, and
the cluster with the highest 2.5-quantile as the worst.

###  3.2 Datasets

We consider a case study on two public datasets, NIH-CXR14 [26]111Provided by
NIH Clinical Center, https://nihcc.app.box.com/v/ChestXray-NIHCC. and CheXpert
[9]222https://stanfordmlgroup.github.io/competitions/chexpert/.. Both datasets
slightly over-represent male subjects. Amelia Jiménez-Sánchez kindly provided
the chest drain labels, also used in previous studies [3, 11], for 3543 random
cases with pneumothorax in the NIH dataset and 972 cases with and without
pneumothorax in CheXpert. For NIH, we observe a larger prevalence of chest
drains among pneumothorax-positive male subjects compared to female subjects
(49.5% vs. 42.8%). For CheXpert, we observe a larger prevalence of chest
drains among pneumothorax-negative male subjects (23.0% vs. 14.7% in females)
but comparable chest drain prevalence across sexes for pneumothorax-positive
subjects (50.5% in males vs. 50.2% in females).

###  3.3 Experiments

We conduct a case study on the pneumothorax classification task, following the
experimental setup of Weng et al. [27]. Specifically, to reduce the potential
for label noise to affect our analyses, we select one sample per patient, with
a preference for pneumothorax-positive samples and an equal sex ratio.
Withholding the chest drain-annotated samples, we split the datasets into
60%/10%/30% train, validation, and test sets, resampling the splits ten times.
We resize the images to 224x224 pixels and train a ResNet50 (Adam optimizer,
learning rate $10^{-6}$, batch size 64, 20 epochs). We use data augmentation
for the training dataset, including horizontal flipping, rotation up to 15
degrees, and scaling up to 10% with a 50% probability for each augmentation.
We then carry out our proposed SDM and report the distribution of
comorbidities and chest drains (for annotated chest drain samples). We repeat
the same analyses for atelectasis classification. Based on the findings of our
SDM, we conduct further post-hoc analyses in both cases.

##  4 Results

Figure 2: The comorbidity and chest drain distribution in pneumothorax-
positive chest drain annotated samples of NIH-CXR14 for the worst-performing
(left column) and best-performing (right column) slices by Brier score. The
pneumothorax-negative case is omitted as chest drain annotations were not
available for these samples.

#### Pneumothorax classification.

Fig. 2 and fig. 1 in the supplementary material show the best- and worst-
performing slices based on the Bier score in the NIH-CXR14 and CheXpert
datasets. In both datasets, the underperforming slices for pneumothorax-
negative samples have a lower-than-average chest drain proportion, while the
opposite holds for pneumothorax-positive cases.

Figure 3:  AUROC on CheXpert with male and female test subjects, following the
natural (‘unbalanced’) distribution of chest drains and balanced by chest
drain presence.

Based on these results (and based on prior work [11, 19]), we hypothesized
that the presence of chest drains is used to classify pneumothorax. Indeed, we
observe that computed model confidence (the softmax output of the model for
the disease-positive class) is consistently higher in samples with chest
drains across datasets, sexes, and pneumothorax labels (positive/negative),
lending strong support to this hypothesis (fig. 2 in the supplementary
material). Furthermore, as both datasets have varying chest drain prevalences
by sex, we hypothesized that this could contribute to the gender performance
gap. To assess this hypothesis, we determined AUROC by sex in (a) a test set
following the natural distribution of chest drains, and (b) a test set with
equalized chest drain prevalence in the male and female populations (fig. 3).
We observe a significant difference in performance in the first case
($p<0.001$) but no significant difference in the second, chest-drain-balanced
case ($p>0.1$), indicating that chest drain shortcut learning is the cause of
a large part of the male–female performance gap observed in prior work [14,
27]. Statistical significance was assessed using a Mann-Whitney U-test.

#### Atelectasis classification.

We repeated our SDM analysis for atelectasis. (Slice statistics not shown here
due to space constraints.) Prompted by a visual inspection of the recordings
in the best and worst-performing slices, we hypothesized that the two slices
differed substantially in their prevalence of ECG cables in the recordings. To
test this hypothesis, we randomly selected and (as non-experts) labeled 100
samples each for the best- and worst-performing slices on atelectasis-positive
and -negative cases according to whether they displayed ECG cables or not. Of
atelectasis-negative samples, 95% of the labeled recordings in the worst-
performing slice contained ECG cables, compared to 10% in the best-performing
slice. Of the atelectasis-positive cases, 50% of the labeled recordings in the
worst-performing slice had ECG cables, compared to 99% in the best-performing
slice.

##  5 Discussion and conclusion

We have proposed a novel slice discovery method (SDM), which differs from
previously proposed methods in several key elements. Both in the
representation extraction as well as in the dimensionality reduction steps, we
prioritize preserving features that are used by the classification model under
test. In addition, we propose using the Brier score (BS) for selecting highly
and poorly performing clusters because it is threshold-independent, accounts
for both discriminative ability and calibration, and enables meaningful
comparisons between clusters of different sizes. In a case study on chest
x-ray-based disease classification, our SDM successfully recovered a
previously known case of shortcut learning (chest drains for pneumothorax
classification) and suggested a new, previously unknown case (ECG cables for
atelectasis classification). The latter case also demonstrated another benefit
of using SDMs: reducing the required labeling efforts, because these can be
specifically targeted at the worst and best clusters. Our case study shows
that our proposed SDM, and SDMs in general, can aid researchers in generating
hypotheses regarding the causes of model underperformance on subsets of data,
which is crucial for leveling _up_ performance [21].

Consistent with the observations made by Weng et al. [27], our findings
challenge the notion that biological differences are the primary driver of the
previously observed [14] but unexplained male–female performance gaps in chest
x-ray-based disease classification. Instead, our results suggest that shortcut
learning in conjunction with a difference in chest drain prevalence between
males and females causes the observed performance disparity. This newly gained
knowledge opens up the possibility for the targeted application of shortcut
learning mitigation techniques, instead of relying on blind group performance
equalization approaches that often result in leveling _down_ performance [21,
29, 30].

While our study builds upon the chest drain annotations of [3] and [11], for
which the former show a high level of agreement between expert and non-expert
annotations, caution is warranted. Non-expert annotations, although showing
agreement, may not necessarily represent the ground truth or offer a
representative sample (images without consensus between multiple labelers were
disregarded [11]). This caution is emphasized by a significant difference in
the prevalence of chest drains among pneumothorax-positive samples in NIH-
CXR14 between studies [3, 19]. Moreover, the mitigation of shortcut learning
is a highly active research area, and mitigating many different shortcuts
simultaneously, and with limited label availability, remains challenging [15].
Finally, the interpretation of identified slices in more challenging cases
represents a crucial challenge for future research. Both chest drains and ECG
cables are visible to the human eye (though maybe not the non-expert), but
other potentially problematic features may not be.

(a) CheXpert, Pneumothorax Negative

(b) CheXpert, Pneumothorax Positive

Figure 4: The comorbidity and chest drain distribution in (a) pneumothorax-
negative (top row) and (b) pneumothorax-positive chest drain annotated samples
of CheXpert for the worst-performing (left column) and best-performing (right
column) slices by upper 95% bootstrapped Brier scores.

(a) NIH-CXR14 (b) CheXpert

Figure 5: Distribution of confidences (the softmax output of the model for the
disease-positive class) for pneumothorax classification by sex, presence of
pneumothorax, and chest drain for NIH-CXR14 (left) and CheXpert (right).
Across datasets, sex and pneumothorax presence, subjects without chest drains
are more likely to be classified as pneumothorax-negative than subjects with
chest drains.

(a) NIH-CXR14 (b) CheXpert

Figure 6: Histogram of Brier scores for pneumothorax classification by sex,
presence of pneumothorax, and chest drain for NIH-CXR14 (left) and CheXpert
(right) across ten samplings of the train-validation-test sets. Across
datasets and sex, pneumothorax-negative subjects with chest drains and
pneumothorax-positive subjects without chest drains are underperforming,
compared to pneumothorax-positive subjects with chest drains and pneumothorax-
negative subjects without chest drains.

### Funding acknowledgements

Work on this project was partially funded by the Independent Research Fund
Denmark (DFF, grant number 9131-00097B), Denmark’s Pioneer Centre for AI (DNRF
grant number P1), and the Novo Nordisk Foundation through the Center for Basic
Machine Learning Research in Life Science (MLLS, grant number NNF20OC0062606).
The listed funding sources had no involvement in the conception of the
manuscript, the writing of the report, or the decision to submit the article
for publication.

## References

  * [1] Brown, A., Tomasev, N., Freyberg, J., Liu, Y., Karthikesalingam, A., Schrouff, J.: Detecting shortcut learning for fair medical AI using shortcut testing. Nature Communications 14(1) (2023). https://doi.org/10.1038/s41467-023-39902-7
  * [2] Bröcker, J.: Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the Royal Meteorological Society 135(643), 1512–1519 (2009). https://doi.org/10.1002/qj.456
  * [3] Damgaard, C., Eriksen, T.N., Juodelyte, D., Cheplygina, V., Jiménez-Sánchez, A.: Augmenting chest x-ray datasets with non-expert annotations, http://arxiv.org/abs/2309.02244
  * [4] Daneshjou, R., Vodrahalli, K., Novoa, R.A., Jenkins, M., Liang, W., Rotemberg, V., et al.: Disparities in dermatology AI performance on a diverse, curated clinical image set. Science Advances 8(32) (2022). https://doi.org/10.1126/sciadv.abq6147
  * [5] DeGrave, A.J., Janizek, J.D., Lee, S.I.: AI for radiographic COVID-19 detection selects shortcuts over signal. Nature Machine Intelligence 3(7), 610–619 (2021). https://doi.org/10.1038/s42256-021-00338-7
  * [6] d’Eon, G., d’Eon, J., Wright, J.R., Leyton-Brown, K.: The Spotlight: A general method for discovering systematic errors in deep learning models. In: ACM FAccT. p. 1962–1981 (2022). https://doi.org/10.1145/3531146.3533240
  * [7] Eyuboglu, S., Varma, M., Saab, K.K., Delbrouck, J.B., Lee-Messer, C., Dunnmon, J., et al.: Domino: Discovering systematic errors with cross-modal embeddings. In: ICLR (2022), https://openreview.net/forum?id=FPCMqjI0jXN
  * [8] Glocker, B., Jones, C., Roschewitz, M., Winzeck, S.: Risk of bias in chest radiography deep learning foundation models. Radiology: Artificial Intelligence 5(6) (2023). https://doi.org/10.1148/ryai.230060
  * [9] Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., et al.: CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison. In: AAAI/IAAI/EAAI (2019). https://doi.org/10.1609/aaai.v33i01.3301590
  * [10] Jain, S., Lawrence, H., Moitra, A., Madry, A.: Distilling model failures as directions in latent space. In: ICLR (2023), https://openreview.net/forum?id=99RpBVpLiX
  * [11] Jiménez-Sánchez, A., Juodelyte, D., Chamberlain, B., Cheplygina, V.: Detecting Shortcuts in Medical Images - A Case Study in Chest X-Rays. In: ISBI. IEEE, Cartagena, Colombia (2023). https://doi.org/10.1109/ISBI53787.2023.10230572
  * [12] Kallus, N., Zhou, A.: The fairness of risk scores beyond classification: Bipartite ranking and the xAUC metric. In: NeurIPS. vol. 32. Curran Associates, Inc. (2019)
  * [13] Kim, M.P., Ghorbani, A., Zou, J.: Multiaccuracy: Black-box post-processing for fairness in classification. In: AIES. pp. 247–254. ACM (2019). https://doi.org/10.1145/3306618.3314287
  * [14] Larrazabal, A.J., Nieto, N., Peterson, V., Milone, D.H., Ferrante, E.: Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis. Proceedings of the National Academy of Sciences 117(23), 12592–12594 (2020). https://doi.org/10.1073/pnas.1919012117
  * [15] Li, Z., Evtimov, I., Gordo, A., Hazirbas, C., Hassner, T., Ferrer, C.C., et al.: A whac-a-mole dilemma: Shortcuts come in multiples where mitigating one amplifies others. In: CVPR. pp. 20071–20082 (2023). https://doi.org/10.1109/CVPR52729.2023.01922
  * [16] Lin, M., Li, T., Yang, Y., Holste, G., Ding, Y., Van Tassel, S.H., et al.: Improving model fairness in image-based computer-aided diagnosis. Nature Communications 14(1) (2023). https://doi.org/10.1038/s41467-023-41974-4
  * [17] Misztal-Radecka, J., Indurkhya, B.: Bias-aware hierarchical clustering for detecting the discriminated groups of users in recommendation systems. Information Processing & Management 58(3), 102519 (2021). https://doi.org/10.1016/j.ipm.2021.102519
  * [18] Mukherjee, P., Shen, T.C., Liu, J., Mathai, T., Shafaat, O., Summers, R.M.: Confounding factors need to be accounted for in assessing bias by machine learning algorithms. Nature Medicine 28(6), 1159–1160 (2022). https://doi.org/10.1038/s41591-022-01847-7
  * [19] Oakden-Rayner, L., Dunnmon, J., Carneiro, G., Re, C.: Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. In: CHIL. pp. 151–159. ACM (2020). https://doi.org/10.1145/3368555.3384468
  * [20] Petersen, E., Ganz, M., Holm, S.H., Feragen, A.: On (assessing) the fairness of risk score models. In: FAccT. ACM (2023). https://doi.org/10.1145/3593013.3594045
  * [21] Petersen, E., Holm, S., Ganz, M., Feragen, A.: The path toward equal performance in medical machine learning. Patterns 4(7) (2023). https://doi.org/10.1016/j.patter.2023.100790
  * [22] Plumb, G., Johnson, N., Cabrera, A., Talwalkar, A.: Towards a more rigorous science of blindspot discovery in image classification models. Transactions on Machine Learning Research (2023), https://openreview.net/forum?id=MaDvbLaBiF
  * [23] Ricci Lara, M.A., Mosquera, C., Ferrante, E., Echeveste, R.: Towards Unraveling Calibration Biases in Medical Image Analysis, pp. 132–141. Springer Nature Switzerland (2023). https://doi.org/10.1007/978-3-031-45249-9_13
  * [24] Seyyed-Kalantari, L., Zhang, H., McDermott, M.B.A., Chen, I.Y., Ghassemi, M.: Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature Medicine 27(12), 2176–2182 (2021). https://doi.org/10.1038/s41591-021-01595-0
  * [25] Sohoni, N., Dunnmon, J., Angus, G., Gu, A., Ré, C.: No subclass left behind: Fine-grained robustness in coarse-grained classification problems. In: NeurIPS. vol. 33, pp. 19339–19352 (2020)
  * [26] Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: ChestX-Ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: CVPR. pp. 2097–2106 (2017). https://doi.org/10.1109/CVPR.2017.369
  * [27] Weng, N., Bigdeli, S., Petersen, E., Feragen, A.: Are sex-based physiological differences the cause of gender bias for chest x-ray diagnosis? In: MICCAI Workshop on Fairness of AI in Medical Imaging. pp. 142–152. Springer (2023). https://doi.org/10.1007/978-3-031-45249-9_14
  * [28] Wynants, L., Van Calster, B., Collins, G.S., Riley, R.D., Heinze, G., Schuit, E., et al.: Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ p. m1328 (2020). https://doi.org/10.1136/bmj.m1328
  * [29] Zhang, H., Dullerud, N., Roth, K., Oakden-Rayner, L., Pfohl, S., Ghassemi, M.: Improving the fairness of chest x-ray classifiers. In: Conference on Health, Inference, and Learning. pp. 204–233. PMLR (2022)
  * [30] Zietlow, D., Lohaus, M., Balakrishnan, G., Kleindessner, M., Locatello, F., Schölkopf, B., Russell, C.: Leveling down in computer vision: Pareto inefficiencies in fair deep classifiers. In: CVPR. IEEE (2022). https://doi.org/10.1109/cvpr52688.2022.01016
  * [31] Zong, Y., Yang, Y., Hospedales, T.: MEDFAIR: Benchmarking fairness for medical imaging. In: ICLR (2023), https://openreview.net/forum?id=6ve2CkeQe5S