# Tyche: Stochastic In-Context Learning for
Medical Image Segmentation

Marianne Rakic
CSAIL MIT
Broad Institute
<EMAIL_ADDRESS>Hallee E. Wong
CSAIL MIT & MGH
Jose Javier Gonzalez Ortiz
MosaicML DataBricks
& MIT
Beth Cimini
Broad Institute
John Guttag
CSAIL MIT
Adrian V. Dalca
CSAIL MIT
& HMS, MGH

###### Abstract

Existing learning-based solutions to medical image segmentation have two
important shortcomings. First, for most new segmentation task, a new model has
to be trained or fine-tuned. This requires extensive resources and machine-
learning expertise, and is therefore often infeasible for medical researchers
and clinicians. Second, most existing segmentation methods produce a single
deterministic segmentation mask for a given image. In practice however, there
is often considerable uncertainty about what constitutes the correct
segmentation, and different expert annotators will often segment the same
image differently. We tackle both of these problems with _Tyche_ , a model
that uses a context set to generate stochastic predictions for previously
unseen tasks without the need to retrain. Tyche differs from other in-context
segmentation methods in two important ways. (1) We introduce a novel
convolution block architecture that enables interactions among predictions.
(2) We introduce in-context test-time augmentation, a new mechanism to provide
prediction stochasticity. When combined with appropriate model design and loss
functions, Tyche can predict a set of plausible diverse segmentation
candidates for new or unseen medical images and segmentation tasks without the
need to retrain. Code available at: https://github.com/mariannerakic/tyche/.

Figure 1: Tyche: the first in-context stochastic segmentation framework. Human
annotators (top) can handle a wide variety of tasks, and different annotators
often produce differing segmentations. Existing automated methods (middle) are
typically task-specific and provide only one segmentation per image. _Tyche_
(bottom) can capture the disagreement among annotators across many modalities
and anatomies without retraining or fine-tuning.

##  1 Introduction

Segmentation is a core step in medical image analysis, for both research and
clinical applications. However, current approaches to medical image
segmentation fall short in two key areas. First, segmentation typically
involves training a new model for each new modality and biomedical domain,
which quickly becomes infeasible given the resources and expertise available
in biomedical research and clinical environments. Second, models most often
provide a single solution, whereas in many cases, the target image contains
ambiguous regions, and there isn’t a single correct segmentation. This
ambiguity can arise from noisy or low contrast images, variation in the task
definition, or human raters’ interpretations and downstream goals [12, 55].
Failure to take this ambiguity into account can affect downstream analysis,
diagnosis, and treatment.

Recent work tackles these issues separately. _In-context learning_ methods
generalize to unseen medical image segmentation tasks, employing an input
context or prompt to guide inference [19, 129, 128]. These methods are
deterministic and predict a single segmentation for a given input image and
task.

Separately, stochastic or probabilistic segmentation methods output multiple
plausible segmentations at inference, reflecting the task uncertainty [11, 67,
95]. Each such model is trained for a specific task, and can only output
multiple plausible segmentations at inference for that task. Training or fine-
tuning a model for a new task requires technical expertise and computational
resources that are often unavailable in biomedical settings.

We present _Tyche_ , a framework for stochastic in-context medical image
segmentation (Figure 1). _Tyche_ includes two variants for different settings.
The first, _Tyche-TS_ (Train-time Stochasticity), is a system explicitly
designed to produce multiple candidate segmentations. The second, _Tyche-IS_
(Inference-time Stochasticity), is a test time solution that leverages a
pretrained deterministic in-context model.

_Tyche_ takes as input the image to be segmented (target), and a context set
of image-segmentation pairs that defines the task. This enables the model to
perform unseen segmentation tasks upon deployment, omitting the need to train
new models. _Tyche-TS_ learns a distribution of possible label maps, and
predicts a set of plausible stochastic segmentations. _Tyche-TS_ encourages
diverse predictions by enabling the internal representations of the different
predictions to interact with each other through a novel convolutional
mechanism, a carefully chosen loss function and noise as an additional input.
In _Tyche-IS_ , we show that applying test time augmentation to both the
target and context set in combination with a trained in-context model leads to
competitive segmentation candidates.

We make the following contributions.

  * •

We present the first solution for probabilistic segmentation for in-context
learning. We develop two variants to our framework: _Tyche-TS_ that is trained
to maximize the quality of the best prediction, and _Tyche-IS_ , that can be
used straightaway with an existing in-context model.

  * •

For _Tyche-TS_ , we introduce a new mechanism, _SetBlock_ , to encourage
diverse segmentation candidates. Simpler than existing stochastic methods,
_Tyche-TS_ predicts all the segmentation candidates in a single forward pass.

  * •

Through rigorous experiments and ablations on a set of twenty unseen medical
imaging tasks, we show that both frameworks produce solutions that outperform
existing in-context and interactive segmentation benchmarks, and can match the
performance of specialized stochastic networks trained on specific datasets.

##  2 Related Work

Biomedical segmentation is a widely-studied problem, with recent methods
dominated by UNet-like architectures [107, 6, 51]. These models tackle a wide
variety of tasks, such as different anatomical regions, different structures
to segment within a region, different image modalities, and different image
settings. With most methods, a new model has to be trained or fine-tuned for
each combination of these. Additionally, most models don’t take into account
image ambiguity, and provide a single deterministic output.

Multiple Predictions. Uncertainty estimation can help users decide how much
faith to put in a segmentation [26] and guide downstream tasks. Uncertainty is
often categorized into aleatoric, uncertainty in the data, and epistemic,
uncertainty in the model [29, 61]. In this work, we focus on aleatoric
uncertainty. Medical images are also heteroscedastic in that the degree of
uncertainty varies across the image.

Different strategies exist to capture uncertainty. One can assign a
probability to each pixel [62, 45, 52, 76], or use contour strategies and
difference loss functions to predict the largest and smallest plausible
segmentations [73, 132]. These strategies however do not capture the
correlations across pixels. To address this, some methods generate multiple
plausible label maps given an image [95, 11, 67, 68, 130]. To achieve this,
one can directly model pixel correlations, such as through a multivariate
Gaussian distribution (with low rank) covariance [95], or more complex
distributions [15]. Alternatively, various frameworks combine potentially
hierarchical representations for UNet-like architectures with variational
auto-encoders [67, 11, 68]. More recently, diffusion models have been used for
ensembling [130] or to produce stochastic segmentations [105, 133]. Some
methods explicitly model the different annotators to capture ambiguity [48,
100, 111, 124]. But these methods do not apply to our framework where the
number of annotators and their characteristics are unknown.

Most of the models above involve sophisticated modeling or lengthy runtimes,
and need to be trained on each segmentation task. In _Tyche_ , we build on
intuition across these methods, but combine a more efficient mechanism with an
in-context strategy to predict segmentation candidates.

In-context Learning. Few-Shot frameworks use a small set of examples to
generalize to new tasks [31, 80, 98, 117, 101, 113, 134], sometimes by fine-
tuning an existing pretrained model [32, 99, 125, 120]. In-context learning
segmentation methods (ICL) use a small set of examples directly as input to
infer label maps for a task [19, 9, 63, 129, 129, 63]. This enables them to
generalize to new tasks. For example, UniverSeg uses an enhanced UNet-based
architecture to generalize to medical image segmentation tasks unseen during
training [19]. We build on these ideas to enable segmentation of new tasks
without the need to re-train, but expand this paradigm to model stochastic
segmentations.

Test Time Augmentation. The test-time augmentation (TTA) strategy uses
perturbations of a test input and ensembles the resulting predictions.
Existing TTA frameworks model accuracy [116, 33, 119, 64], robustness [25],
and estimates of uncertainty [90, 5]. Test-time augmentation has been applied
to diverse anatomies and modalities including brain MRI and retinal fundus [3,
5, 49, 97, 53, 127]. Prior work has formalized the variance of a model’s
predictions over a set of input transformations as capturing aleatoric
uncertainty [126, 127, 5].

_Tyche_ ’s use of TTA is distinct from prior work. Instead of ensembling
segmentations over perturbations of a test input or pixel-wise estimates of
uncertainty, _Tyche_ extends TTA to the in-context setting and uses the
individual TTA predictions to model uncertainty.

Figure 2: Tyche Model Schematic. The target $x^{t}$, context set
${(x^{t}_{j},y^{t}_{j})}_{j=1}^{S}$, and noise images $\\{z_{k}\\}_{k=1}^{K}$
are inputs to the network. The architecture employs UNet-like levels, but uses
_SetBlocks_ that enable interactions between the context set and the target
segmentation candidates. Figure 3: CrossBlock Mechanism The CrossBlock
involves interactions between a single feature and a set of features and
outputs new feature for the target and new features for each. Figure 4:
SetBlock Mechanism. _SetBlock_ enables interactions between the set of
features from the context set and the set of features from the prediction
candidates. It outputs two sets of features, one for the context and one for
the prediction candidates.

##  3 Method

For segmentation task $t$, let $\\{(x^{t}_{j},y^{t}_{j})\\}_{j=1}^{N}$ be a
dataset with images $x^{t}$ and label maps $y^{t}$. Typical segmentation
models learn a different function $\hat{y}^{t}=g_{\theta^{t}}(x^{t})$ with
parameters $\theta^{t}$ for each task $t$, where $\hat{y}^{t}$ is a single
segmentation map prediction.

We design _Tyche_ as an in-context learning (ICL) model using a single
function for all tasks:

$\hat{y}_{k}^{t}=f_{\theta}(x^{t},z_{k},\mathcal{S}^{t}).$ (1)

This function, with global parameters $\theta$, captures a distribution of
label maps $\\{\hat{y}^{t}_{k}\\}_{k=1}^{K}$, given target $x^{t}$, context
set $\mathcal{S}^{t}=\\{x^{t}_{j},y^{t}_{j}\\}_{j=1}^{S}$ defining task t, and
noise $z_{k}\sim\mathcal{N}(\bm{0},\mathbb{I})$. We use this modelling
strategy in two ways: we either explicitly train a network to approximate the
model $f_{\theta}(\cdot)$ in _Tyche-TS_ , or design a test-time strategy to
approximate $f_{\theta}(\cdot)$ using an existing (pretrained) deterministic
in-context network in _Tyche-IS_.

###  3.1 Tyche-TS

In _Tyche-TS_ , we explicitly train a neural network for $f_{\theta}(\cdot)$
that can make different predictions given the same image input $x^{t}$ but
different noise channels $z_{k}$. We model interaction between predictions,
and employ a loss that encourages diverse solutions (Figure 2).

####  3.1.1 Neural Network

We use a convolutional architecture focused on interacting representations of
sets of flexible sizes using a modified version of the usual UNet structure
[107].

Inputs. _Tyche-TS_ takes as input the target $x^{t}$, a set of K Gaussian
noise channels $z_{k}$, and a context set, $\mathcal{S}^{t}$.

Layers. Each level of the UNet takes as input a set of K candidate
representations and S context representations. We design each level to
encourage communication between the intermediate elements of the sets, and
between the two features of the segmentation candidates. The size $K$ is
flexible and can vary with iterations.

SetBlock. We introduce a new operation called _SetBlock_ , which interacts the
candidate representations $U=\\{u_{i}\\}_{i=1}^{K}$, with the context
representations $V=\\{v_{i}\\}_{i=1}^{S}$, illustrated in Figure 4. We use the
CrossBlock [19] as a building block for this new layer. The
$\text{CrossBlock}(u,V)\rightarrow(u^{\prime},V^{\prime})$ compares an image
representation $u$ to a context set representation $V$ through convolutional
and averaging operations, and outputs a new image representation $u^{\prime}$
and a new set representation $V^{\prime}$ (Figure 3).
$\text{SetBlock}(U,V)\rightarrow(U^{\prime},V^{\prime})$ builds on CrossBlock
and performs a set to set interaction of the entries of $U$ and $V$:

$\displaystyle\bar{u}$ $\displaystyle=1/m{\textstyle\sum}_{i=1}^{m}u_{i}$ (2)
$\displaystyle\bar{u}^{\prime},V^{\prime}$
$\displaystyle=\text{CrossBlock}(\bar{u},V)$ (3) $\displaystyle
u_{i}^{\prime}$
$\displaystyle=\text{Conv}_{m}\left(u_{i}||\bar{u}^{\prime}\right),\quad
i=1,\ldots,K\,$ (4) $\displaystyle u_{i}^{\prime}$
$\displaystyle=\text{Conv}_{u}\left(u_{i}^{\prime}\right),\quad
i=1,\ldots,\,K$ (5) $\displaystyle v_{i}^{\prime}$
$\displaystyle=\text{Conv}_{v}\left(v_{i}\right),\quad i=1,\ldots,\,S,$ (6)

where $||$ is the concatenation operation along the feature dimension. The
CrossBlock interacts the context representation with the mean candidate. The
$\textit{Conv}_{m}$ step communicates this result to all candidate
representation. $\textit{Conv}_{u}$ and $\textit{Conv}_{v}$ then update all
representations. All convolution operations include a non-linear activation
function.

####  3.1.2 Best candidate Loss

Typical loss function compute the loss of a single prediction relative to a
single target, but _Tyche-TS_ produces multiple predictions and has one or
more corresponding label maps.We optimize

$\small\mathcal{L}(\theta;\mathcal{T})=\mathbb{E}_{t\in\mathcal{T}}\left[\mathbb{E}_{(x^{t},y^{t}_{r}),\mathcal{S}^{t}}\left[\mathcal{L}_{seg}\left(\\{\hat{y}_{k}\\},y^{t}_{r}\right)\right]\right],$
(7)

with

$\small\mathcal{L}_{seg}(\\{\hat{y}_{k}\\},y)=\,\min_{k}\;\mathcal{L}_{Dice}\left(y_{k},y\right),$
(8)

where $y_{r}^{t}$ is a segmentation from rater $r$, and $\mathcal{L}_{Dice}$
is a weighted sum of soft Dice loss [94] and categorical crossentropy. By only
back-propagating through the best prediction among $K$ candidates, the network
is encouraged to produce diverse solutions [22, 41, 66, 81].

####  3.1.3 Training Data

We employ a large dataset of single- and multi-rater segmentations across
diverse biomedical domains. We then use data augmentation [19], as described
in B.3.

We add synthetic multi-annotator data by modelling an image as the average of
four blobs representing four raters (Figure 10). Each blob is white disk
$b_{i}$ deformed by a random smoothed deformation field $\phi_{i}$. The
synthetic image is a noisy weighted sum of raters:
$\sum_{i=1}^{4}w_{i}(b_{i}\circ\phi_{i})$ where $\circ$ represents the spacial
warp operation.

####  3.1.4 Implementation Details

We use a UNet-like architecture of 4 _SetBlock_ layers for the encoder and
decoder, with 64 features each and Leaky ReLU as activation function. We use
the Adam optimizer and a learning rate of 0.0001. At training, we have a fixed
number of candidates per sample $K_{\text{tr}}=8$. At inference, we consider
different numbers of candidates.

###  3.2 Tyche-IS

In _Tyche-IS_ , we first train (or use an existing trained) deterministic in-
context segmentation system:

$\hat{y}^{t}=h_{\theta}(x^{t},\mathcal{S}^{t}).$

We then introduce a test-time in-context augmentation strategy to provide
stochastic predictions:

$\displaystyle\hat{y}^{t}_{k}$
$\displaystyle=f_{\theta}(x^{t},z_{k},\mathcal{S}^{t})$ (9)
$\displaystyle=h_{\theta}(aug(x^{t},z_{k},\mathcal{S}^{t})),$ (10)

where $\tilde{x}^{t},\tilde{\mathcal{S}}^{t}=aug(x^{t},z_{k},\mathcal{S}^{t})$
is an augmentation function.

####  3.2.1 Augmentation Strategy

Test time augmentation for single task networks $y=g_{t}(x)$ applies different
transforms to an input image $x$:

$\small\tilde{x}_{k}=a_{\phi}(x,z_{k}),$ (11)

where $\phi$ are augmentation parameters and $z_{k}$ is a random vector. A
final prediction is then obtained by combining several predictions of
augmented images. Most commonly, the combining function averages the
predictions:

$y=\frac{1}{k}\sum_{k}g_{t}(\tilde{x}_{k}),$ (12)

where the sum operates pixel-wise.

We introduce in-context test-time augmentation (ICTTA) prediction as another
mechanism to generate diverse stochastic predictions.

We apply augmentation to both the test target $x^{t}$ and the context set
$\mathcal{S}^{t}$:

$\displaystyle\small(\tilde{x_{i}}^{t},y^{t}_{i})=(a_{\phi}(x^{t}_{i}),y^{t})$
(13)
$\displaystyle\tilde{\mathcal{S}}^{t}=\\{a_{\phi}\left(x_{j}^{t}\right),y_{j}^{t}\\}_{j=1}^{S}.$
(14)

We repeat this process $K_{i}$ times to obtain $K_{i}$ stochastic predictions:

$\displaystyle\hat{y}_{k}=f_{\theta}(\tilde{x_{i}}^{t},z_{k},\tilde{\mathcal{S}^{t}})$
(15)

We only apply intensity based transforms, to avoid the need to invert the
segmentations back. We apply Gaussian noise, blurring and pixel intensity
inversion. We detail the specific augmentations in B.2.

Figure 5: Visualization of predictions for three different samples, 1 per row.
Left: LIDC-IDRI. Right: Hippocampus dataset. The leftmost columns are raters’
annotations. The 4 last columns are model predictions. _Tyche_ provides a set
of prediction that is diverse and matches the raters, for tasks unseen at
training time. Figure 6: Single annotator visualization for different models.
We show three example images that show very different corresponding
segmentation. _Tyche_ can output plausible segmentation for single annotator
data with varying degrees of variability in the segmentation. Methods with an
asterisk are upper baselines.

##  4 Experimental Setup

###  4.1 Data

We evaluate our method using a large collection of biomedical and synthetic
datasets. Most datasets include a single manual segmentation for each example,
while a few have several raters per image.

Data Splits. We partition each dataset into development, validation, and test
splits. We assign each dataset to an in-distribution set (I.D.) or an out-of-
distribution set (O.D.). We train exclusively on the development splits of the
I.D. datasets, and use the validation splits of the I.D. datasets to tune
parameters. We use the validation splits of the O.D. datasets for final model
selection. We report results on the test splits of the O.D. datasets.

For each use case, we sample the context from each dataset’s corresponding
development set. As a result, the network doesn’t see any of the O.D. datasets
at training time.

We distinguish between single annotator data and multi-annotator data.

Single-Annotator Data. For single annotator data, we build on MegaMedical used
in recent publications [19, 131] and employ a collection of 73 datasets, of
different public biomedical domains and different modalities [14, 1, 56, 77,
39, 59, 79, 86, 103, 122, 110, 115, 104, 138, 84, 139, 83, 19, 35, 137, 21,
34, 24, 16, 17, 42, 54, 57, 74, 75, 82, 78, 85, 118, 87, 121, 106, 69, 96, 7,
36, 43, 71, 72, 88, 89, 91, 112, 40, 50, 2, 135]. MegaMedical spans a variety
of anatomies and modalities, including brain MRI, cardiac ultrasound, thoracic
CT and dental X-ray. We also use synthetic data involving simulated shapes,
intensities, and image artifacts [19, 44]. The single-annotator datasets used
for out-of-domain (O.D.) testing are: PanDental [1], WBC [139], SCD [104],
ACDC [14], and SpineWeb [138].

Multi-Annotator Data. For multi-annotator I.D. data, we use four datasets from
Qubiq [92]: Brain Growth, Brain Lesions, Pancreas Lesions, and Kidney. We also
simulate a multi-rater dataset consisting of random shapes (blobs). For the
O.D. multi-annotator data we use four datasets. One contains hippocampus
segmentation maps on brain MRIs from a large hospital. We crop the volumes
around the hippocampus [67] to focus on the areas where the raters disagree.
The second is a publicly available lung nodule dataset, LIDC-IDRI [4]. This
dataset is notable for the substantial inter-rater variability. It contains
1018 thoracic CT scans, each annotated by 4 annotators from a pool of 12
annotators. Finally, we also use retinal fundus images, STARE [47], annotated
by 2 raters, and prostate data from the MICCAI 2021 QUBIQ challenge [92],
annotated by 6 raters on two tasks. Single and multi-annotator combined, our
O.D. group contains 20 tasks unseen at training time (some datasets have
several tasks).

###  4.2 Evaluation

We evaluate our method by analysing individual prediction quality and
distribution of predictions, both qualitatively and quantitatively. We also
examine model choices through an ablation study.

A main use case of stochastic segmentation is to propose a small set of
segmentations to a human rater, who can select the most appropriate one for
their purpose. For this scenario, a model can be viewed as good if at least
one prediction matches what the rater is looking for. We thus employ the best
candidate Dice metric.

In the multi-annotator setting, we evaluate using both best candidate Dice
score, also called maximum Dice score, as well as Generalized Energy Distance
(GED)[13, 109, 123]. GED is commonly used in the stochastic segmentation
literature to asses the difference between the distribution of predictions and
the distribution of annotations [67, 11, 95, 105, 133]. GED has limitations,
such as rewarding excessive prediction diversity [105]. Let $\mathcal{Y}$ and
$\hat{\mathcal{Y}}$ be the set of annotations, GED is defined as:

$\small
D^{2}_{GED}(\mathcal{Y},\hat{\mathcal{Y}})=2\mathbb{E}\left[d(p,\hat{p})\right]-\mathbb{E}\left[d(p,p^{\prime})\right]-\mathbb{E}\left[d(\hat{p},\hat{p}^{\prime})\right],$
(16)

where $p,p^{\prime}\sim\mathcal{Y}$,
$\hat{p},\hat{p}^{\prime}\sim\hat{\mathcal{Y}}$ and $d(\cdot,\cdot)$ is a
distance metric. We use the Dice score [30].

###  4.3 Benchmarks

_Tyche_ is the first method to produce stochastic segmentation predictions in-
context. Consequently, we compare _Tyche_ to existing benchmarks, each of
which achieves only a subset of our goals.

| In-Context | Stochastic | Automatic
---|---|---|---
SENet | ✓ |  | ✓
UniverSeg | ✓ |  | ✓
SegGPT | ✓ |  | ✓
Prob. UNet |  | ✓ | ✓
PhiSeg |  | ✓ | ✓
CIMD |  | ✓ | ✓
SAM-based | ✓ | ✓ |
Tyche | ✓ | ✓ | ✓
Table 1: Summary of evaluated methods used and their properties. Only _Tyche_
is both stochastic and in-context, and does not require user interaction.

In-Context Methods. We compare to deterministic frameworks that can leverage a
context set: a few-shot method, SENet [108], and two in-context learning (ICL)
methods, UniverSeg [19] and SegGPT [129]. We train UniverSeg and SENet with
the same data split strategies and the same sets of augmentation transforms as
for _Tyche_. For SegGPT, we use the public model, trained on a mix of natural
and medical images. Figure 13 in the Supplemental Material shows that
UniverSeg trained with additional data outperforms its public version.

Stochastic Upper Bounds. We compare to task-specialized probabilistic
segmentation methods that are trained-on and perform well on specific
datasets. We independently train Probabilistic UNet [67], PhiSeg[11] and CIDM,
a recent diffusion network [105], on each of the 20 held-out tasks. For each
task, we train three model variants: no augmentation, weak augmentation, and
as much augmentation as for the _Tyche_ targets. For each benchmark variant,
we train on a O.D. development split and select the model that performs best
on the corresponding O.D. validation split. We then compare these benchmarks
to _Tyche_ on the held-out O.D. test splits.

These models are explicitly optimized for the datasets on which they are
evaluated, unlike _Tyche_ , which does not use those datasets for training.
Since these are trained, tuned and evaluated on the O.D. datasets splits,
something we explicitly aim to avoid in the problem set up as it is not easily
done in many medical settings, they serve as upper bounds on performance.

Interactive Segmentation Methods. We compare to two interactive methods: SAM
[66] and SAM-Med2D [23]. These methods can provide multiple segmentations,
but, unlike _Tyche_ , require human interaction, which is outside our scope.
SAM also has a functionality to segment all elements in an image, however less
optimized for medical imaging. We assume that the SAM-based models have access
to the same information as the ICL methods: several image-segmentation pairs
as context to guide the segmentation task. We fine-tune SAM using our I.D.
development datasets. To replace the human interaction, we provide a bounding
box, the average context label map, and 10 clicks, 5 positive and 5 negative
as input. With SAM-Med2D, we use a bounding box, and several positive and
negative clicks as input. For both SAM and SAM-Med2D, we generate clicks and
bounding box from the average context label map.

We use one iteration of interaction, and sample different plausible
segmentation candidates by sampling different sets of clicks and different
averaged context sets.

Table 1 summarizes the features of all the methods. Additional information on
the benchmarks is provided in the Supplemental Material.

$GED^{2}$ ($\downarrow$) | Hippocampus | LIDC-IDRI | Prostate Task 1 | Prostate Task 2 | STARE
---|---|---|---|---|---
| SAM | $0.57\pm 0.02$ | $0.90\pm 0.01$ | $0.20\pm 0.03$ | $0.31\pm 0.06$ | $0.89\pm 0.06$
Interactive | SAM-Med2d | $0.93\pm 0.02$ | $1.01\pm 0.01$ | $0.80\pm 0.09$ | $0.78\pm 0.11$ | $1.52\pm 0.05$
| Tyche-IS | $\bm{0.21\pm 0.01}$ | $0.41\pm 0.01$ | $0.12\pm 0.02$ | $0.20\pm 0.05$ | $0.73\pm 0.03$
I-C & Stochastic (Ours) | Tyche-TS | $0.22\pm 0.01$ | $\bm{0.40\pm 0.01}$ | $\bm{0.09\pm 0.02}$ | $\bm{0.15\pm 0.03}$ | $\bm{0.62\pm 0.03}$
| PhiSeg | $0.14\pm 0.01$ | $0.33\pm 0.01$ | $0.12\pm 0.01$ | $0.17\pm 0.05$ | $1.22\pm 0.02$
| ProbaUNet | $0.13\pm 0.01$ | $0.51\pm 0.01$ | $0.08\pm 0.01$ | $0.18\pm 0.05$ | $0.76\pm 0.06$
Stochastic Upper Bound | CIDM | $0.17\pm 0.01$ | $0.42\pm 0.01$ | $0.14\pm 0.02$ | $0.26\pm 0.04$ | $0.87\pm 0.05$
Table 2: Generalized Energy Distance for different models with a context size of 16 for in-context methods and a number of predictions set to 8. Lower is better. _Tyche_ outperforms interactive and in-context baselines, and matches stochastic upper bounds. Max Dice ($\uparrow$) | Hippocampus | LIDC-IDRI | Prostate Task 1 | Prostate Task 2 | STARE
---|---|---|---|---|---
| UniverSeg | $0.84\pm 0.01$ | $0.67\pm 0.01$ | $0.91\pm 0.01$ | $0.88\pm 0.03$ | $0.51\pm 0.02$
| SegGPT | $0.10\pm 0.01$ | $0.68\pm 0.01$ | $0.94\pm 0.01$ | $0.89\pm 0.03$ | $0.02\pm 0.01$
In-Context | SENet | $0.68\pm 0.01$ | $0.00\pm 0.00$ | $0.83\pm 0.02$ | $0.83\pm 0.02$ | $0.30\pm 0.03$
| SAM | $0.71\pm 0.01$ | $0.55\pm 0.01$ | $0.90\pm 0.01$ | $0.85\pm 0.03$ | $0.50\pm 0.03$
Interactive | SAM-Med2d | $0.52\pm 0.01$ | $0.42\pm 0.01$ | $0.62\pm 0.04$ | $0.64\pm 0.06$ | $0.21\pm 0.03$
| Tyche-IS | $0.87\pm 0.01$ | $0.90\pm 0.00$ | $0.94\pm 0.01$ | $0.91\pm 0.01$ | $0.52\pm 0.03$
I-C & Stochastic (Ours) | Tyche-TS | $\bm{0.88\pm 0.01}$ | $\bm{0.91\pm 0.00}$ | $\bm{0.95\pm 0.01}$ | $\bm{0.93\pm 0.01}$ | $\bm{0.60\pm 0.02}$
| PhiSeg | $0.88\pm 0.00$ | $0.91\pm 0.00$ | $0.93\pm 0.01$ | $0.91\pm 0.02$ | $0.15\pm 0.01$
| ProbaUNet | $0.91\pm 0.00$ | $0.86\pm 0.01$ | $0.95\pm 0.00$ | $0.91\pm 0.03$ | $0.59\pm 0.02$
Stochastic Upper Bound | CIMD | $0.84\pm 0.01$ | $0.92\pm 0.00$ | $0.93\pm 0.01$ | $0.87\pm 0.02$ | $0.41\pm 0.04$
Table 3: Best candidate Dice score for different models with a context size of
16 for in-context methods and a number of predictions set to 8. Higher is
better. _Tyche_ outperforms interactive and in-context baselines, and matches
stochastic upper bounds.

###  4.4 Experiments

We evaluate all models on the multi-annotator and single-annotator O.D. data.
We then analyze the _Tyche_ variants individually and perform an ablation
study on each to validate parameter choices. Finally, we compare the GPU
inference runtimes and model parameters.

In the Supplemental Material, we analyze further the noise given as input, the
context set, the number of predictions, the _SetBlock_ and the candidate loss.
We also provide additional performance metrics and per dataset results. We
also compare the performance of _Tyche_ and PhiSeg in a few-shot setting.
Finally, we provide additional visualizations.

Inference Setting. We use a fixed context of 16 image-segmentation pairs,
because existing in-context learning systems show minimal improvements beyond
this size [19]. Because there is variability in performance depending on the
context sampled, we sample 5 different context sets for each datapoint and
average performance. Similarly, for the stochastic upper bounds and
interactive methods, we do 5 rounds of sampling $K_{i}$ samples.

##  5 Results

###  5.1 Comparison to Benchmarks

Multi-Annotator O.D. Data. We evaluate on the datasets where multiple
annotations exist for each sample. Figure 5 shows that, for both the lung
nodules and the hippocampus datasets, _Tyche_ predictions are diverse and
capture rater diversity, even though these datasets are out-of-domain. Tables
2 and 3 show that both versions of _Tyche_ outperform the interactive and
deterministic benchmarks on all datasets except for Prostate Task 1, on which
SegGPT has similar performance. Using a paired Student t-test, we find that
_Tyche-TS_ outperforms _Tyche-IS_ in terms of maximum Dice score, with
$p<10^{-10}$. We find no statistical difference between the two methods in
terms of GED.

Figure 7: Best candidate Dice Score for single annotator data aggregated per
task. _Tyche_ outperforms the in-context and interactive segmentation
benchmarks, and approaches the stochastic upper bounds. Error bars represent
the 95% confidence interval.

Single-Annotator Data. Figure 6 shows examples of predictions for _Tyche_ and
the corresponding benchmarks for the single-annotator datasets. _Tyche_
produces a more diverse set of candidates than its competitors. Figure 7
compares all plausible models in terms of aggregate best candidate Dice score,
except for CIDM, which underperformed for single-annotator data with a mean
best candidate Dice score of: $0.673\pm 0.032$. For clarity, we only present
the full Figure in the Supplemental Material. _Tyche_ performs better than the
deterministic and interactive frameworks, and similarly to Probabilistic UNet,
one of the upper bound benchmarks that is trained on the O.D. data. A paired
Student t-test shows that _Tyche-IS_ produces statistically higher GED
($p=0.044$) than _Tyche-TS_ , but we find no statistical difference in terms
of best candidate Dice. We hypothesize that _Tyche-IS_ is competitive because
of the implicit annotator characterization provided by the context.

###  5.2 Tyche Analysis

We analyze _Tyche_ variants and study the influence of two important
parameters: the number of inference-time candidate predictions $K_{i}$ and the
size of the context set $\|\mathcal{S}\|$.

Influence of the number of prediction $K_{i}$. We study how the number of
predictions impacts the best candidate Dice score, keeping the context size
constant. Figure 8 shows that for _Tyche-TS_ , the best candidate Dice score
rises with the number of predictions, but with diminishing returns.

Figure 8: Best candidate Dice Score as the number of candidate prediction
increases. The largest improvements are usually obtained for a small number of
predictions. The error bars represent the 95% confidence interval.

Influence of context size $\|\mathcal{S}\|$. Figure 9 shows that _Tyche-TS_ is
capable of leveraging the increased context size to improve its best candidate
Dice score, and that a context size of 16 is sufficient to achieve most of the
gain.

Figure 9: Best candidate Dice Score per dataset as context size increases. A
context size of 16 is already large enough to obtain a reasonable best
candidate Dice. The error bars represent the 95% confidence interval.

Blob | SetBlock | Std | Max. DSC($\uparrow$) |  $GED^{2}$($\downarrow$)
---|---|---|---|---
✗ |  |  | $0.810\pm 0.01$ | $0.349\pm 0.04$
| ✗ |  | $0.771\pm 0.02$ | $0.425\pm 0.05$
|  | ✓ | $0.802\pm 0.01$ | $0.425\pm 0.05$
✓ | ✓ |  | $\bm{0.811\pm 0.01}$ | $\bm{0.298\pm 0.03}$

Target | CS | CS+ | Max. DSC($\uparrow$) |  $GED^{2}$ ($\downarrow$)
---|---|---|---|---
✓ |  |  | $0.776\pm 0.02$ | $0.477\pm 0.04$
| ✓ |  | $0.700\pm 0.02$ | $0.410\pm 0.05$
|  | ✓ | $0.561\pm 0.02$ | $0.867\pm 0.05$
✓ | ✓ |  | $\bm{0.813\pm 0.01}$ | $\bm{0.333\pm 0.04}$
✓ | ✓ | ✓ | $0.808\pm 0.01$ | $0.358\pm 0.04$

Table 4: Ablation Study for Tyche variants. Top: _Tyche-TS_ , without
simulated multi-annotator data, with _SetBlock_ , with Standard Deviation in
_SetBlock_. Bottom: _Tyche TeS_ , with Target, Context and Large Context
augmentations.

Ablation. Table 4 illustrates several ablations on _Tyche_ design choices. For
_Tyche-TS_ , we evaluate the following variants: no simulated multi-annotator
images, no _SetBlock_ and finally, using the standard deviation of candidate
feature representations in addition to the mean in the _SetBlock_ (“Std”). We
compare the models using best candidate Dice averaged across tasks.

Table 4 shows that the simulated multi-annotator data provides negligible
improvement, as does adding the standard deviation. However, _SetBlock_ is a
crucial part to improve the best candidate Dice score.

We study performances for three types of TTA in _Tyche-IS_ : on the target, on
the context (CS), and on the context while also including the non-augmented
context (CS+): $(S,\mathcal{G}(S))$. Table 4 shows that adding noise to only
one of the target and context yields sub-optimal performance, while augmenting
both the target and context improves performances.

###  5.3 Inference Runtime

We compare the inference runtime by predicting 8 segmentation candidates with
each method, and repeat the process 300 times. We use an NVIDIA V100 GPU.
Table 5 shows that _Tyche_ is significantly faster and smaller than SegGPT and
CIDM, yet, not as fast as some task-specific stochastic models. _Tyche-IS_ has
fewer parameters than _Tyche-TS_ , but needs additional inference time.

| Inference Time (ms) | Parameters
---|---|---
UniverSeg | $96.62\pm 0.61$ |  $1.2$M
SegGPT | $2857.19\pm 4.38$ |  $370$M
SENet | $14.91\pm 0.21$ |  $0.89$M
FT-SAM | $1036.75\pm 4.61$ |  $94$M
SAM-Med2D | $188.8\pm 7.58$ |  $91$M
PhiSeg | $11.35\pm 0.672$ |  $21.1$M
ProbaUNet | $8.44\pm 0.46$ |  $5$M
CIDM | $1.7\times 10^{5}\pm 2748$ |  $85.6$M
Tyche-IS | $128.57\pm 2.626$ |  $1.2$M
Tyche-TS | $18.09\pm 0.61$ |  $1.7$M
Table 5: Inference Runtime and Model Parameters for 8 predictions and a
context size of 16.

##  6 Conclusion

We introduced _Tyche_ , the first framework for stochastic in-context
segmentation. For any (new) segmentation task, _Tyche_ can directly produce
diverse segmentation candidates, from which practitioners can select the most
suitable one, and draw a better understanding of the underlying uncertainty.
_Tyche_ can generalize to images from data unseen at training and outperforms
in-context and interactive benchmarks. In addition, _Tyche_ often matches
stochastic models on tasks for which those models have been specifically
trained. _Tyche_ has two variants, one designed to optimize the best
segmentation candidate, with fast inference time, and a test-time augmentation
variant that can be used in combination with existing in-context learning
methods. We are excited to further study the different types of uncertainty
captured by _Tyche-TS_ and _Tyche-IS_. We will also extend the capabilities of
_Tyche_ with more complex support sets, including variable annotators and
multiple image modalities.

##  7 Acknowledgement

Research reported in this paper was supported by the National Institute of
Biomedical Imaging and Bioengineering of the National Institutes of Health
under award number R01EB033773. This work was also supported in part by
funding from the Eric and Wendy Schmidt Center at the Broad Institute of MIT
and Harvard as well as Quanta Computer Inc. Finally, some of the computation
resources required for this research was performed on computational hardware
generously provided by the Massachusetts Life Sciences Center.

## References

  * Abdi et al. [2015] Amir Hossein Abdi, Shohreh Kasaei, and Mojdeh Mehdizadeh.  Automatic segmentation of mandible in panoramic x-ray.  _Journal of Medical Imaging_ , 2(4):044003, 2015.
  * Al-Dhabyani et al. [2020] Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy.  Dataset of breast ultrasound images.  _Data in Brief_ , 28:104863, 2020.
  * Amiri et al. [2020] Mina Amiri, Rupert Brooks, Bahareh Behboodi, and Hassan Rivaz.  Two-stage ultrasound image segmentation using u-net and test time augmentation.  _International journal of computer assisted radiology and surgery_ , 15:981–988, 2020.
  * Armato III and et al. [2011] Samuel G Armato III and et al.  The lung image database consortium (lidc) and image database resource initiative (idri): a completed reference database of lung nodules on ct scans.  _Medical physics_ , 38(2):915–931, 2011.
  * Ayhan and Berens [2022] Murat Seckin Ayhan and Philipp Berens.  Test-time data augmentation for estimation of heteroscedastic aleatoric uncertainty in deep neural networks.  In _Medical Imaging with Deep Learning_ , 2022.
  * Badrinarayanan and et al. [2017] Vijay Badrinarayanan and et al.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  _IEEE transactions on pattern analysis and machine intelligence_ , 39(12):2481–2495, 2017.
  * Baid et al. [2021] Ujjwal Baid, Satyam Ghodasara, Suyash Mohan, Michel Bilello, Evan Calabrese, Errol Colak, Keyvan Farahani, Jayashree Kalpathy-Cramer, Felipe C Kitamura, Sarthak Pati, et al.  The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification.  _arXiv preprint arXiv:2107.02314_ , 2021.
  * Bakas et al. [2017] Spyridon Bakas, Hamed Akbari, Aristeidis Sotiras, Michel Bilello, Martin Rozycki, Justin S Kirby, John B Freymann, Keyvan Farahani, and Christos Davatzikos.  Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features.  _Scientific data_ , 4(1):1–13, 2017.
  * Balažević and et al. [2023] Ivana Balažević and et al.  Towards in-context scene understanding.  _arXiv preprint arXiv:2306.01667_ , 2023.
  * Bano et al. [2020] Sophia Bano, Francisco Vasconcelos, Luke M Shepherd, Emmanuel Vander Poorten, Tom Vercauteren, Sebastien Ourselin, Anna L David, Jan Deprest, and Danail Stoyanov.  Deep placental vessel segmentation for fetoscopic mosaicking.  In _Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part III 23_ , pages 763–773. Springer, 2020.
  * Baumgartner et al. [2019] Christian F Baumgartner, Kerem C Tezcan, Krishna Chaitanya, Andreas M Hötker, Urs J Muehlematter, Khoschy Schawkat, Anton S Becker, Olivio Donati, and Ender Konukoglu.  Phiseg: Capturing uncertainty in medical image segmentation.  In _Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part II 22_ , pages 119–127. Springer, 2019.
  * Becker et al. [2019] Anton S Becker, Krishna Chaitanya, Khoschy Schawkat, Urs J Muehlematter, Andreas M Hötker, Ender Konukoglu, and Olivio F Donati.  Variability of manual segmentation of the prostate in axial t2-weighted mri: a multi-reader study.  _European journal of radiology_ , 121:108716, 2019.
  * Bellemare et al. [2017] Marc G Bellemare, Ivo Danihelka, Will Dabney, Shakir Mohamed, Balaji Lakshminarayanan, Stephan Hoyer, and Rémi Munos.  The cramer distance as a solution to biased wasserstein gradients.  _arXiv preprint arXiv:1705.10743_ , 2017.
  * Bernard et al. [2018] Olivier Bernard, Alain Lalande, Clement Zotti, Frederick Cervenansky, Xin Yang, Pheng-Ann Heng, Irem Cetin, Karim Lekadir, Oscar Camara, Miguel Angel Gonzalez Ballester, et al.  Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved?  _IEEE transactions on medical imaging_ , 37(11):2514–2525, 2018.
  * Bhat et al. [2022] Ishaan Bhat, Josien PW Pluim, and Hugo J Kuijf.  Generalized probabilistic u-net for medical image segementation.  In _International Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging_ , pages 113–124. Springer, 2022.
  * Bilic et al. [2019] Patrick Bilic, Patrick Ferdinand Christ, Eugene Vorontsov, Grzegorz Chlebus, Hao Chen, Qi Dou, Chi-Wing Fu, Xiao Han, Pheng-Ann Heng, Jürgen Hesser, et al.  The liver tumor segmentation benchmark (lits).  _arXiv preprint arXiv:1901.04056_ , 2019.
  * Bloch et al. [2015] Nicholas Bloch, Anant Madabhushi, Henkjan Huisman, John Freymann, Justin Kirby, Michael Grauer, Andinet Enquobahrie, Carl Jaffe, Larry Clarke, and Keyvan Farahani.  Nci-isbi 2013 challenge: automated segmentation of prostate structures.  _The Cancer Imaging Archive_ , 370(6):5, 2015.
  * Buda et al. [2019] Mateusz Buda, Ashirbani Saha, and Maciej A Mazurowski.  Association of genomic subtypes of lower-grade gliomas with shape features automatically extracted by a deep learning algorithm.  _Computers in biology and medicine_ , 109:218–225, 2019.
  * Butoi et al. [2023] Victor Ion Butoi, Jose Javier Gonzalez Ortiz, Tianyu Ma, Mert R Sabuncu, John Guttag, and Adrian V Dalca.  Universeg: Universal medical image segmentation.  _arXiv preprint arXiv:2304.06131_ , 2023.
  * Caicedo et al. [2019] Juan C. Caicedo, Allen Goodman, Kyle W. Karhohs, Beth A. Cimini, Jeanelle Ackerman, Marzieh Haghighi, CherKeng Heng, Tim Becker, Minh Doan, Claire McQuin, Mohammad Rohban, Shantanu Singh, and Anne E. Carpenter.  Nucleus segmentation across imaging experiments: the 2018 Data Science Bowl.  _Nature Methods_ , 16(12):1247–1253, 2019.
  * [21] Albert Cardon, Stephan Saalfeld, Stephan Preibisch, Benjamin Schmid, Anchi Cheng, Jim Pulokas, Pavel Tomancak, and Volker Hartenstein.  Isbi challenge: Segmentation of neuronal structures in em stacks.
  * Charpiat et al. [2008] Guillaume Charpiat, Matthias Hofmann, and Bernhard Schölkopf.  Automatic image colorization via multimodal predictions.  In _Computer Vision–ECCV 2008: 10th European Conference on Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part III 10_ , pages 126–139. Springer, 2008.
  * Cheng et al. [2023] Junlong Cheng, Jin Ye, Zhongying Deng, Jianpin Chen, Tianbin Li, Haoyu Wang, Yanzhou Su, Ziyan Huang, Jilong Chen, Lei Jiang, Hui Sun, Junjun He, Shaoting Zhang, Min Zhu, and Yu Qiao.  SAM-Med2D, 2023.  arXiv:2308.16184 [cs].
  * Codella et al. [2017] Noel C. F. Codella, David A. Gutman, M. Emre Celebi, Brian Helba, Michael A. Marchetti, Stephen W. Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin K. Mishra, Harald Kittler, and Allan Halpern.  Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (ISIC).  _CoRR_ , abs/1710.05006, 2017.
  * Cohen et al. [2023] Seffi Cohen, Niv Goldshlager, Lior Rokach, and Bracha Shapira.  Boosting anomaly detection using unsupervised diverse test-time augmentation.  _Information Sciences_ , 626:821–836, 2023.
  * Czolbe et al. [2021] Steffen Czolbe, Kasra Arnavaz, Oswin Krause, and Aasa Feragen.  Is segmentation uncertainty useful?  In _Information Processing in Medical Imaging: 27th International Conference, IPMI 2021, Virtual Event, June 28–June 30, 2021, Proceedings 27_ , pages 715–726. Springer, 2021.
  * Decenciere et al. [2013] Etienne Decenciere, Guy Cazuguel, Xiwei Zhang, Guillaume Thibault, J-C Klein, Fernand Meyer, Beatriz Marcotegui, Gwénolé Quellec, Mathieu Lamard, Ronan Danno, et al.  Teleophta: Machine learning and image processing methods for teleophthalmology.  _Irbm_ , 34(2):196–203, 2013.
  * Degerli et al. [2021] Aysen Degerli, Morteza Zabihi, Serkan Kiranyaz, Tahir Hamid, Rashid Mazhar, Ridha Hamila, and Moncef Gabbouj.  Early detection of myocardial infarction in low-quality echocardiography.  _IEEE Access_ , 9:34442–34453, 2021.
  * Der Kiureghian and Ditlevsen [2009] Armen Der Kiureghian and Ove Ditlevsen.  Aleatory or epistemic? does it matter?  _Structural safety_ , 31(2):105–112, 2009.
  * Dice [1945] Lee R Dice.  Measures of the amount of ecologic association between species.  _Ecology_ , 26(3):297–302, 1945.
  * Ding et al. [2023] Hao Ding, Changchang Sun, Hao Tang, Dawen Cai, and Yan Yan.  Few-shot medical image segmentation with cycle-resemblance attention.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 2488–2497, 2023.
  * Finn et al. [2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _International conference on machine learning_ , pages 1126–1135. PMLR, 2017.
  * Gaillochet et al. [2022] Mélanie Gaillochet, Christian Desrosiers, and Hervé Lombaert.  Taal: Test-time augmentation for active learning in medical image segmentation.  In _Data Augmentation, Labelling, and Imperfections_. Springer Nature Switzerland, 2022.
  * Gamper et al. [2003] J Gamper, NA Koohbanani, K Benes, S Graham, M Jahanifar, SA Khurram, A Azam, K Hewitt, and N Rajpoot.  Pannuke dataset extension, insights and baselines. arxiv. 2020 doi: 10.48550.  _ARXIV_ , 2003.
  * Gerhard et al. [2013] Stephan Gerhard, Jan Funke, Julien Martel, Albert Cardona, and Richard Fetter.  Segmented anisotropic ssTEM dataset of neural tissue.  _figshare_ , pages 0–0, 2013.
  * Gollub et al. [2013] Randy L Gollub, Jody M Shoemaker, Margaret D King, Tonya White, Stefan Ehrlich, Scott R Sponheim, Vincent P Clark, Jessica A Turner, Bryon A Mueller, Vince Magnotta, et al.  The mcic collection: a shared repository of multi-modal, multi-site brain image data from a clinical investigation of schizophrenia.  _Neuroinformatics_ , 11:367–388, 2013.
  * Gousias et al. [2008] Ioannis S Gousias, Daniel Rueckert, Rolf A Heckemann, Leigh E Dyet, James P Boardman, A David Edwards, and Alexander Hammers.  Automatic segmentation of brain mris of 2-year-olds into 83 regions of interest.  _Neuroimage_ , 40(2):672–684, 2008.
  * Gousias et al. [2012] Ioannis S Gousias, A David Edwards, Mary A Rutherford, Serena J Counsell, Jo V Hajnal, Daniel Rueckert, and Alexander Hammers.  Magnetic resonance imaging of the newborn brain: manual segmentation of labelled atlases in term-born and preterm infants.  _Neuroimage_ , 62(3):1499–1509, 2012.
  * Graham et al. [2019] Simon Graham, Quoc Dang Vu, Shan E Ahmed Raza, Ayesha Azam, Yee Wah Tsang, Jin Tae Kwak, and Nasir Rajpoot.  Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images.  _Medical Image Analysis_ , 58:101563, 2019.
  * Gut [2021] Daniel Gut.  X-ray images of the hip joints.  1, 2021.  Publisher: Mendeley Data.
  * Guzman-Rivera et al. [2012] Abner Guzman-Rivera, Dhruv Batra, and Pushmeet Kohli.  Multiple choice learning: Learning to produce multiple structured outputs.  _Advances in neural information processing systems_ , 25, 2012.
  * Heller et al. [2020] Nicholas Heller, Fabian Isensee, Klaus H Maier-Hein, Xiaoshuai Hou, Chunmei Xie, Fengyi Li, Yang Nan, Guangrui Mu, Zhiyong Lin, Miofei Han, et al.  The state of the art in kidney and kidney tumor segmentation in contrast-enhanced ct imaging: Results of the kits19 challenge.  _Medical Image Analysis_ , page 101821, 2020.
  * Hernandez Petzsche et al. [2022] Moritz R Hernandez Petzsche, Ezequiel de la Rosa, Uta Hanning, Roland Wiest, Waldo Valenzuela, Mauricio Reyes, Maria Meyer, Sook-Lei Liew, Florian Kofler, Ivan Ezhov, et al.  Isles 2022: A multi-center magnetic resonance imaging stroke lesion segmentation dataset.  _Scientific data_ , 9(1):762, 2022.
  * Hoffmann et al. [2021] Malte Hoffmann, Benjamin Billot, Douglas N Greve, Juan Eugenio Iglesias, Bruce Fischl, and Adrian V Dalca.  Synthmorph: learning contrast-invariant registration without acquired images.  _IEEE transactions on medical imaging_ , 41(3):543–558, 2021.
  * Hong et al. [2021] Sungmin Hong, Anna K Bonkhoff, Andrew Hoopes, Martin Bretzner, Markus D Schirmer, Anne-Katrin Giese, Adrian V Dalca, Polina Golland, and Natalia S Rost.  Hypernet-ensemble learning of segmentation probability for medical image segmentation with ambiguous labels.  _arXiv preprint arXiv:2112.06693_ , 2021.
  * Hoopes et al. [2022] Andrew Hoopes, Malte Hoffmann, Douglas N. Greve, Bruce Fischl, John Guttag, and Adrian V. Dalca.  Learning the effect of registration hyperparameters with hypermorph.  pages 1–30, 2022.
  * Hoover et al. [2000] AD Hoover, Valentina Kouznetsova, and Michael Goldbaum.  Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response.  _IEEE Transactions on Medical imaging_ , 19(3):203–210, 2000.
  * Hu et al. [2023] Qingqiao Hu, Hao Wang, Jing Luo, Yunhao Luo, Zhiheng Zhangg, Jan S Kirschke, Benedikt Wiestler, Bjoern Menze, Jianguo Zhang, and Hongwei Bran Li.  Inter-rater uncertainty quantification in medical image segmentation via rater-specific bayesian neural networks.  _arXiv preprint arXiv:2306.16556_ , 2023.
  * Huang et al. [2021] Xiaoqiong Huang, Zejian Chen, Xin Yang, Zhendong Liu, Yuxin Zou, Mingyuan Luo, Wufeng Xue, and Dong Ni.  Style-invariant cardiac image segmentation with test-time augmentation.  In _Statistical Atlases and Computational Models of the Heart. M &Ms and EMIDEC Challenges: 11th International Workshop, STACOM 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 4, 2020, Revised Selected Papers 11_, pages 305–315. Springer, 2021.
  * [50] Humans in the Loop.  Teeth segmentation dataset.
  * Isensee et al. [2021] Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein.  nnu-net: a self-configuring method for deep learning-based biomedical image segmentation.  _Nature methods_ , 18(2):203–211, 2021.
  * Islam and Glocker [2021] Mobarakol Islam and Ben Glocker.  Spatially varying label smoothing: Capturing uncertainty from expert annotations.  In _Information Processing in Medical Imaging: 27th International Conference, IPMI 2021, Virtual Event, June 28–June 30, 2021, Proceedings 27_ , pages 677–688. Springer, 2021.
  * Jha et al. [2021] Debesh Jha, Pia H Smedsrud, Dag Johansen, Thomas de Lange, Håvard D Johansen, Pål Halvorsen, and Michael A Riegler.  A comprehensive study on colorectal polyp segmentation with resunet++, conditional random field and test-time augmentation.  _IEEE journal of biomedical and health informatics_ , 25(6):2029–2040, 2021.
  * Ji et al. [2022] Yuanfeng Ji, Haotian Bai, Jie Yang, Chongjian Ge, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhang, Wanling Ma, Xiang Wan, et al.  Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation.  _arXiv preprint arXiv:2206.08023_ , 2022.
  * Joskowicz et al. [2019] Leo Joskowicz, D Cohen, N Caplan, and Jacob Sosna.  Inter-observer variability of manual contour delineation of structures in ct.  _European radiology_ , 29:1391–1399, 2019.
  * Karim et al. [2013] Rashed Karim, R James Housden, Mayuragoban Balasubramaniam, Zhong Chen, Daniel Perry, Ayesha Uddin, Yosra Al-Beyatti, Ebrahim Palkhi, Prince Acheampong, Samantha Obom, et al.  Evaluation of current algorithms for segmentation of scar tissue from late gadolinium enhancement cardiovascular magnetic resonance of the left atrium: an open-access grand challenge.  _Journal of Cardiovascular Magnetic Resonance_ , 15(1):1–17, 2013.
  * Kavur et al. [2019a] Ali Emre Kavur, M. Alper Selver, Oğuz Dicle, Mustafa Barış, and N. Sinem Gezer.  CHAOS - Combined (CT-MR) Healthy Abdominal Organ Segmentation Challenge Data.  2019a.
  * Kavur et al. [2019b] Ali Emre Kavur, M. Alper Selver, Oğuz Dicle, Mustafa Barış, and N. Sinem Gezer.  CHAOS - Combined (CT-MR) Healthy Abdominal Organ Segmentation Challenge Data, 2019b.
  * Kavur et al. [2021a] A. Emre Kavur, N. Sinem Gezer, Mustafa Barış, Sinem Aslan, Pierre-Henri Conze, Vladimir Groza, Duc Duy Pham, Soumick Chatterjee, Philipp Ernst, Savaş Özkan, Bora Baydar, Dmitry Lachinov, Shuo Han, Josef Pauli, Fabian Isensee, Matthias Perkonigg, Rachana Sathish, Ronnie Rajan, Debdoot Sheet, Gurbandurdy Dovletov, Oliver Speck, Andreas Nürnberger, Klaus H. Maier-Hein, Gözde Bozdağı Akar, Gözde Ünal, Oğuz Dicle, and M. Alper Selver.  CHAOS Challenge - combined (CT-MR) healthy abdominal organ segmentation.  _Medical Image Analysis_ , 69:101950, 2021a.
  * Kavur et al. [2021b] A. Emre Kavur, N. Sinem Gezer, Mustafa Barış, Sinem Aslan, Pierre-Henri Conze, Vladimir Groza, Duc Duy Pham, Soumick Chatterjee, Philipp Ernst, Savaş Özkan, Bora Baydar, Dmitry Lachinov, Shuo Han, Josef Pauli, Fabian Isensee, Matthias Perkonigg, Rachana Sathish, Ronnie Rajan, Debdoot Sheet, Gurbandurdy Dovletov, Oliver Speck, Andreas Nürnberger, Klaus H. Maier-Hein, Gözde Bozdağı Akar, Gözde Ünal, Oğuz Dicle, and M. Alper Selver.  CHAOS Challenge - combined (CT-MR) healthy abdominal organ segmentation.  _Medical Image Analysis_ , 69:101950, 2021b.
  * Kendall and Gal [2017] Alex Kendall and Yarin Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  _Advances in neural information processing systems_ , 30, 2017.
  * Kendall et al. [2015] Alex Kendall, Vijay Badrinarayanan, and Roberto Cipolla.  Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding.  _arXiv preprint arXiv:1511.02680_ , 2015.
  * Kim et al. [2023] Donggyun Kim, Jinwoo Kim, Seongwoong Cho, Chong Luo, and Seunghoon Hong.  Universal few-shot learning of dense prediction tasks with visual token matching.  _arXiv preprint arXiv:2303.14969_ , 2023.
  * Kim et al. [2021] Kwanyoung Kim, Dongwon Park, Kwang In Kim, and Se Young Chun.  Task-aware variational adversarial active learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8166–8175, 2021.
  * Kiranyaz et al. [2020] Serkan Kiranyaz, Aysen Degerli, Tahir Hamid, Rashid Mazhar, Rayyan El Fadil Ahmed, Rayaan Abouhasera, Morteza Zabihi, Junaid Malik, Ridha Hamila, and Moncef Gabbouj.  Left ventricular wall motion estimation by active polynomials for acute myocardial infarction detection.  _IEEE Access_ , 8:210301–210317, 2020.
  * Kirillov and et al. [2023] Alexander Kirillov and et al.  Segment anything.  _arXiv preprint arXiv:2304.02643_ , 2023.
  * Kohl et al. [2018] Simon Kohl, Bernardino Romera-Paredes, Clemens Meyer, Jeffrey De Fauw, Joseph R Ledsam, Klaus Maier-Hein, SM Eslami, Danilo Jimenez Rezende, and Olaf Ronneberger.  A probabilistic u-net for segmentation of ambiguous images.  _Advances in neural information processing systems_ , 31, 2018.
  * Kohl et al. [2019] Simon AA Kohl, Bernardino Romera-Paredes, Klaus H Maier-Hein, Danilo Jimenez Rezende, SM Eslami, Pushmeet Kohli, Andrew Zisserman, and Olaf Ronneberger.  A hierarchical probabilistic u-net for modeling multi-scale ambiguities.  _arXiv preprint arXiv:1905.13077_ , 2019.
  * Krönke et al. [2022] Markus Krönke, Christine Eilers, Desislava Dimova, Melanie Köhler, Gabriel Buschner, Lilit Schweiger, Lemonia Konstantinidou, Marcus Makowski, James Nagarajah, Nassir Navab, et al.  Tracked 3d ultrasound and deep neural network-based thyroid segmentation reduce interobserver variability in thyroid volumetry.  _Plos one_ , 17(7):e0268550, 2022.
  * Kuhn [1955] Harold W Kuhn.  The hungarian method for the assignment problem.  _Naval research logistics quarterly_ , 2(1-2):83–97, 1955.
  * Kuijf et al. [2019] Hugo J Kuijf, J Matthijs Biesbroek, Jeroen De Bresser, Rutger Heinen, Simon Andermatt, Mariana Bento, Matt Berseth, Mikhail Belyaev, M Jorge Cardoso, Adria Casamitjana, et al.  Standardized assessment of automatic segmentation of white matter hyperintensities and results of the wmh segmentation challenge.  _IEEE transactions on medical imaging_ , 38(11):2556–2568, 2019.
  * Kuklisova-Murgasova et al. [2011] Maria Kuklisova-Murgasova, Paul Aljabar, Latha Srinivasan, Serena J Counsell, Valentina Doria, Ahmed Serag, Ioannis S Gousias, James P Boardman, Mary A Rutherford, A David Edwards, et al.  A dynamic 4d probabilistic atlas of the developing brain.  _NeuroImage_ , 54(4):2750–2763, 2011.
  * Lambert et al. [2023] Benjamin Lambert, Florence Forbes, Senan Doyle, and Michel Dojat.  Triadnet: Sampling-free predictive intervals for lesional volume in 3d brain mr images.  In _Uncertainty for Safe Utilization of Machine Learning in Medical Imaging – MICCAI 2023_. Springer Nature Switzerland, 2023.
  * Lambert et al. [2020] Zoé Lambert, Caroline Petitjean, Bernard Dubray, and Su Kuan.  Segthor: segmentation of thoracic organs at risk in ct images.  In _2020 Tenth International Conference on Image Processing Theory, Tools and Applications (IPTA)_ , pages 1–6. IEEE, 2020.
  * Landman et al. [2015] Bennett Landman, Zhoubing Xu, J Igelsias, Martin Styner, T Langerak, and Arno Klein.  Miccai multi-atlas labeling beyond the cranial vault–workshop and challenge.  In _Proc. MICCAI Multi-Atlas Labeling Beyond Cranial Vault—Workshop Challenge_ , page 12, 2015.
  * Larrazabal et al. [2021] Agostina Larrazabal, Cesar Martinez, Jose Dolz, and Enzo Ferrante.  Maximum entropy on erroneous predictions (meep): Improving model calibration for medical image segmentation.  _arXiv preprint arXiv:2112.12218_ , 2021.
  * Leclerc et al. [2019] Sarah Leclerc, Erik Smistad, Joao Pedrosa, Andreas Østvik, Frederic Cervenansky, Florian Espinosa, Torvald Espeland, Erik Andreas Rye Berg, Pierre-Marc Jodoin, Thomas Grenier, et al.  Deep learning for segmentation using an open large-scale dataset in 2d echocardiography.  _IEEE transactions on medical imaging_ , 38(9):2198–2210, 2019.
  * Lemaître et al. [2015] Guillaume Lemaître, Robert Martí, Jordi Freixenet, Joan C Vilanova, Paul M Walker, and Fabrice Meriaudeau.  Computer-aided detection and diagnosis for prostate cancer based on mono and multi-parametric mri: a review.  _Computers in biology and medicine_ , 60:8–31, 2015.
  * Li et al. [2020] Mingchao Li, Yuhan Zhang, Zexuan Ji, Keren Xie, Songtao Yuan, Qinghuai Liu, and Qiang Chen.  Ipn-v2 and octa-500: Methodology and dataset for retinal image segmentation.  _arXiv preprint arXiv:2012.07261_ , 2020.
  * Li et al. [2022] Yiwen Li, Yunguan Fu, Iani Gayo, Qianye Yang, Zhe Min, Shaheer Saeed, Wen Yan, Yipei Wang, J Alison Noble, Mark Emberton, et al.  Prototypical few-shot segmentation for cross-institution male pelvic structures with spatial registration.  _arXiv preprint arXiv:2209.05160_ , 2022.
  * Li et al. [2018] Zhuwen Li, Qifeng Chen, and Vladlen Koltun.  Interactive image segmentation with latent diversity.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 577–585, 2018.
  * Litjens et al. [2014] Geert Litjens, Robert Toth, Wendy van de Ven, Caroline Hoeks, Sjoerd Kerkstra, Bram van Ginneken, Graham Vincent, Gwenael Guillard, Neil Birbeck, Jindang Zhang, et al.  Evaluation of prostate segmentation algorithms for mri: the promise12 challenge.  _Medical image analysis_ , 18(2):359–373, 2014.
  * Ljosa et al. [2012] Vebjorn Ljosa, Katherine L Sokolnicki, and Anne E Carpenter.  Annotated high-throughput microscopy image sets for validation.  _Nature methods_ , 9(7):637–637, 2012.
  * Löffler et al. [2020] Maximilian T Löffler, Anjany Sekuboyina, Alina Jacob, Anna-Lena Grau, Andreas Scharr, Malek El Husseini, Mareike Kallweit, Claus Zimmer, Thomas Baum, and Jan S Kirschke.  A vertebral segmentation dataset with fracture grading.  _Radiology: Artificial Intelligence_ , 2(4):e190138, 2020.
  * Luo et al. [2021] Xiangde Luo, Wenjun Liao, Jianghong Xiao, Tao Song, Xiaofan Zhang, Kang Li, Guotai Wang, and Shaoting Zhang.  Word: Revisiting organs segmentation in the whole abdominal region.  _arXiv preprint arXiv:2111.02403_ , 2021.
  * Ma et al. [2021] Yuhui Ma, Huaying Hao, Jianyang Xie, Huazhu Fu, Jiong Zhang, Jianlong Yang, Zhen Wang, Jiang Liu, Yalin Zheng, and Yitian Zhao.  Rose: a retinal oct-angiography vessel segmentation dataset and new model.  _IEEE Transactions on Medical Imaging_ , 40(3):928–939, 2021.
  * Macdonald et al. [2023] Jacob A. Macdonald, Zhe Zhu, Brandon Konkel, Maciej Mazurowski, Walter Wiggins, and Mustafa Bashir.  Duke liver dataset (MRI) v2, 2023.
  * Marcus et al. [2007] Daniel S Marcus, Tracy H Wang, Jamie Parker, John G Csernansky, John C Morris, and Randy L Buckner.  Open access series of imaging studies (oasis): cross-sectional mri data in young, middle aged, nondemented, and demented older adults.  _Journal of cognitive neuroscience_ , 19(9):1498–1507, 2007.
  * Marek et al. [2011] Kenneth Marek, Danna Jennings, Shirley Lasch, Andrew Siderowf, Caroline Tanner, Tanya Simuni, Chris Coffey, Karl Kieburtz, Emily Flagg, Sohini Chowdhury, et al.  The parkinson progression marker initiative (ppmi).  _Progress in neurobiology_ , 95(4):629–635, 2011.
  * Matsunaga et al. [2017] Kazuhisa Matsunaga, Akira Hamada, Akane Minagawa, and Hiroshi Koga.  Image classification of melanoma, nevus and seborrheic keratosis by deep neural network ensemble.  _arXiv preprint arXiv:1703.03108_ , 2017.
  * Mazurowski et al. [2017] Maciej A Mazurowski, Kal Clark, Nicholas M Czarnek, Parisa Shamsesfandabadi, Katherine B Peters, and Ashirbani Saha.  Radiogenomics of lower-grade glioma: algorithmically-assessed tumor shape is associated with tumor genomic subtypes and patient outcomes in a multi-institutional study with the cancer genome atlas data.  _Journal of neuro-oncology_ , 133:27–35, 2017.
  * Menze et al. [2021] Bjoern Menze, Leo Joskowicz, Spyridon Bakas, Andras Jakab, Ender Konukoglu, Anton Becker, Amber Simpson, and Richard D.  Quantification of uncertainties in biomedical image quantification 2021.  _4th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2021)_ , 2021.
  * Menze et al. [2014] Bjoern H Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, et al.  The multimodal brain tumor image segmentation benchmark (brats).  _IEEE transactions on medical imaging_ , 34(10):1993–2024, 2014.
  * Milletari et al. [2016] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi.  V-net: Fully convolutional neural networks for volumetric medical image segmentation.  In _2016 fourth international conference on 3D vision (3DV)_ , pages 565–571. Ieee, 2016.
  * Monteiro et al. [2020] Miguel Monteiro, Loïc Le Folgoc, Daniel Coelho de Castro, Nick Pawlowski, Bernardo Marques, Konstantinos Kamnitsas, Mark van der Wilk, and Ben Glocker.  Stochastic segmentation networks: Modelling spatially correlated aleatoric uncertainty.  _Advances in Neural Information Processing Systems_ , 33:12756–12767, 2020.
  * Montoya et al. [2016] Anna Montoya, Hasnin, kaggle446, shirzad, Will Cukierski, and yffud.  Ultrasound nerve segmentation, 2016.
  * Moshkov et al. [2020] Nikita Moshkov, Botond Mathe, Attila Kertesz-Farkas, Reka Hollandi, and Peter Horvath.  Test-time augmentation for deep learning-based cell segmentation on microscopy images.  _Scientific reports_ , 10(1):5068, 2020.
  * Nguyen and Todorovic [2019] Khoi Nguyen and Sinisa Todorovic.  Feature weighting and boosting for few-shot segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 622–631, 2019.
  * Nichol et al. [2018] Alex Nichol, Joshua Achiam, and John Schulman.  On first-order meta-learning algorithms.  _arXiv preprint arXiv:1803.02999_ , 2018.
  * Nichyporuk et al. [2022] Brennan Nichyporuk, Jillian Cardinell, Justin Szeto, Raghav Mehta, Jean-Pierre R Falet, Douglas L Arnold, Sotirios A Tsaftaris, and Tal Arbel.  Rethinking generalization: The impact of annotation style on medical image segmentation.  _arXiv preprint arXiv:2210.17398_ , 2022.
  * Pandey et al. [2023] Prashant Pandey, Mustafa Chasmai, Tanuj Sur, and Brejesh Lall.  Robust prototypical few-shot organ segmentation with regularized neural-odes.  _IEEE Transactions on Medical Imaging_ , 2023.
  * Payette et al. [2021] Kelly Payette, Priscille de Dumast, Hamza Kebiri, Ivan Ezhov, Johannes C Paetzold, Suprosanna Shit, Asim Iqbal, Romesa Khan, Raimund Kottke, Patrice Grehten, et al.  An automatic multi-tissue human fetal brain segmentation benchmark using the fetal tissue annotation dataset.  _Scientific Data_ , 8(1):1–14, 2021.
  * Porwal et al. [2018] Prasanna Porwal, Samiksha Pachade, Ravi Kamble, Manesh Kokare, Girish Deshmukh, Vivek Sahasrabuddhe, and Fabrice Meriaudeau.  Indian diabetic retinopathy image dataset (idrid), 2018.
  * Radau et al. [2009] Perry Radau, Yingli Lu, Kim Connelly, Gideon Paul, AJWG Dick, and Graham Wright.  Evaluation framework for algorithms segmenting short axis cardiac mri.  _The MIDAS Journal-Cardiac MR Left Ventricle Segmentation Challenge_ , 49, 2009.
  * Rahman et al. [2023] Aimon Rahman, Jeya Maria Jose Valanarasu, Ilker Hacihaliloglu, and Vishal M Patel.  Ambiguous medical image segmentation using diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11536–11546, 2023.
  * Rister et al. [2020] Blaine Rister, Darvin Yi, Kaushik Shivakumar, Tomomi Nobashi, and Daniel L. Rubin.  CT-ORG, a new dataset for multiple organ segmentation in computed tomography.  _Scientific Data_ , 7(1):381, 2020.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  In _Medical Image Computing and Computer Assisted Intervention – MICCAI 2015_. Springer International Publishing, 2015.
  * Roy et al. [2020] Abhijit Guha Roy, Shayan Siddiqui, Sebastian Pölsterl, Nassir Navab, and Christian Wachinger.  Squeeze & excite guided few-shot segmentation of volumetric images.  _Medical image analysis_ , 59:101587, 2020.
  * Salimans et al. [2018] Tim Salimans, Han Zhang, Alec Radford, and Dimitris Metaxas.  Improving gans using optimal transport.  _arXiv preprint arXiv:1803.05573_ , 2018.
  * Saporta et al. [2021] Adriel Saporta, Xiaotong Gui, Ashwin Agrawal, Anuj Pareek, SQ Truong, CD Nguyen, Van-Doan Ngo, Jayne Seekins, Francis G Blankenberg, AY Ng, et al.  Deep learning saliency maps do not accurately highlight diagnostically relevant regions for medical image interpretation.  _MedRxiv_ , 2021.
  * Schmidt et al. [2023] Arne Schmidt, Pablo Morales-Álvarez, and Rafael Molina.  Probabilistic modeling of inter-and intra-observer variability in medical image segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 21097–21106, 2023.
  * Seibold et al. [2022] Constantin Seibold, Simon Reiß, Saquib Sarfraz, Matthias A. Fink, Victoria Mayer, Jan Sellner, Moon Sung Kim, Klaus H. Maier-Hein, Jens Kleesiek, and Rainer Stiefelhagen.  Detailed annotations of chest x-rays via ct projection for report understanding.  In _Proceedings of the 33th British Machine Vision Conference (BMVC)_ , 2022.
  * Seo et al. [2022] Jun Seo, Young-Hyun Park, Sung Whan Yoon, and Jaekyun Moon.  Task-adaptive feature transformer with semantic enrichment for few-shot segmentation.  _arXiv preprint arXiv:2202.06498_ , 2022.
  * Serag et al. [2012] Ahmed Serag, Paul Aljabar, Gareth Ball, Serena J Counsell, James P Boardman, Mary A Rutherford, A David Edwards, Joseph V Hajnal, and Daniel Rueckert.  Construction of a consistent high-definition spatio-temporal atlas of the developing brain using adaptive kernel regression.  _Neuroimage_ , 59(3):2255–2265, 2012.
  * Setio et al. [2017] Arnaud Arindra Adiyoso Setio, Alberto Traverso, Thomas De Bel, Moira SN Berens, Cas Van Den Bogaard, Piergiorgio Cerello, Hao Chen, Qi Dou, Maria Evelina Fantacci, Bram Geurts, et al.  Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the luna16 challenge.  _Medical image analysis_ , 42:1–13, 2017.
  * Shanmugam et al. [2021] Divya Shanmugam, Davis Blalock, Guha Balakrishnan, and John Guttag.  Better aggregation in test-time augmentation.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 1214–1223, 2021.
  * Shen et al. [2022] Qianqian Shen, Yanan Li, Jiyong Jin, and Bin Liu.  Q-net: Query-informed few-shot medical image segmentation.  _arXiv preprint arXiv:2208.11451_ , 2022.
  * Simpson et al. [2019] Amber L Simpson, Michela Antonelli, Spyridon Bakas, Michel Bilello, Keyvan Farahani, Bram Van Ginneken, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, et al.  A large annotated medical image dataset for the development and evaluation of segmentation algorithms.  _arXiv preprint arXiv:1902.09063_ , 2019.
  * Sinha et al. [2019] Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell.  Variational adversarial active learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5972–5981, 2019.
  * Snell et al. [2017] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Song et al. [2022] Yuxin Song, Jing Zheng, Long Lei, Zhipeng Ni, Baoliang Zhao, and Ying Hu.  CT2US: Cross-modal transfer learning for kidney segmentation in ultrasound images with synthesized data.  _Ultrasonics_ , 122:106706, 2022.
  * Staal et al. [2004] Joes Staal, Michael D Abràmoff, Meindert Niemeijer, Max A Viergever, and Bram Van Ginneken.  Ridge-based vessel segmentation in color images of the retina.  _IEEE transactions on medical imaging_ , 23(4):501–509, 2004.
  * Székely and Rizzo [2013] Gábor J Székely and Maria L Rizzo.  Energy statistics: A class of statistics based on distances.  _Journal of statistical planning and inference_ , 143(8):1249–1272, 2013.
  * Tanno et al. [2019] Ryutaro Tanno, Ardavan Saeedi, Swami Sankaranarayanan, Daniel C Alexander, and Nathan Silberman.  Learning from noisy labels by regularized estimation of annotator confusion.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 11244–11253, 2019.
  * Vinyals et al. [2016] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al.  Matching networks for one shot learning.  _Advances in neural information processing systems_ , 29, 2016.
  * Wang et al. [2019a] Guotai Wang, Wenqi Li, Michael Aertsen, Jan Deprest, Sébastien Ourselin, and Tom Vercauteren.  Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks.  _Neurocomputing_ , 338:34–45, 2019a.
  * Wang et al. [2019b] Guotai Wang, Wenqi Li, Sébastien Ourselin, and Tom Vercauteren.  Automatic brain tumor segmentation using convolutional neural networks with test-time augmentation.  In _Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 4th International Workshop, BrainLes 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Revised Selected Papers, Part II 4_ , pages 61–72. Springer, 2019b.
  * Wang et al. [2023a] Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang.  Images speak in images: A generalist painter for in-context visual learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6830–6839, 2023a.
  * Wang et al. [2023b] Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang.  Seggpt: Towards segmenting everything in context.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1130–1140, 2023b.
  * Wolleb et al. [2021] Julia Wolleb, Robin Sandkühler, Florentin Bieder, Philippe Valmaggia, and Philippe C. Cattin.  Diffusion Models for Implicit Image Segmentation Ensembles.  In _Medical Imaging with Deep Learning_ , 2021.
  * Wong et al. [2023] Hallee E Wong, Marianne Rakic, John Guttag, and Adrian V Dalca.  Scribbleprompt: Fast and flexible interactive segmentation for any medical image.  _arXiv preprint arXiv:2312.07381_ , 2023.
  * Ye et al. [2023] Andre Ye, Quan Ze Chen, and Amy Zhang.  Confidence contours: Uncertainty-aware annotation for medical semantic segmentation.  _arXiv preprint arXiv:2308.07528_ , 2023.
  * Zbinden et al. [2023] Lukas Zbinden, Lars Doorenbos, Theodoros Pissas, Adrian Thomas Huber, Raphael Sznitman, and Pablo Márquez-Neila.  Stochastic segmentation with conditional categorical diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1119–1129, 2023.
  * Zhang et al. [2019] Chi Zhang, Guosheng Lin, Fayao Liu, Rui Yao, and Chunhua Shen.  Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 5217–5226, 2019.
  * Zhang et al. [2022a] Yingtao Zhang, Min Xian, Heng-Da Cheng, Bryar Shareef, Jianrui Ding, Fei Xu, Kuan Huang, Boyu Zhang, Chunping Ning, and Ying Wang.  Busis: A benchmark for breast ultrasound image segmentation.  In _Healthcare_ , page 729. MDPI, 2022a.
  * Zhang et al. [2022b] Yingtao Zhang, Min Xian, Heng-Da Cheng, Bryar Shareef, Jianrui Ding, Fei Xu, Kuan Huang, Boyu Zhang, Chunping Ning, and Ying Wang.  Busis: A benchmark for breast ultrasound image segmentation.  In _Healthcare_ , page 729. MDPI, 2022b.
  * Zhao et al. [2022] Qi Zhao, Shuchang Lyu, Wenpei Bai, Linghan Cai, Binghao Liu, Meijing Wu, Xiubo Sang, Min Yang, and Lijiang Chen.  A multi-modality ovarian tumor ultrasound image dataset for unsupervised cross-domain semantic segmentation.  _CoRR_ , abs/2207.06799, 2022.
  * Zheng et al. [2017] Guoyan Zheng, Chengwen Chu, Daniel L Belavỳ, Bulat Ibragimov, Robert Korez, Tomaž Vrtovec, Hugo Hutt, Richard Everson, Judith Meakin, Isabel Lŏpez Andrade, et al.  Evaluation and comparison of 3d intervertebral disc localization and segmentation methods for 3d t2 mr data: A grand challenge.  _Medical image analysis_ , 35:327–344, 2017.
  * Zheng et al. [2018] Xin Zheng, Yong Wang, Guoyou Wang, and Jianguo Liu.  Fast and robust segmentation of white blood cell images by self-supervised learning.  _Micron_ , 107:55–71, 2018.

Supplementary Material

##  Appendix A Overview

We first present a brief overview of the analysis and information in this
Supplemental Material.

Tyche Model Choices.

We present additional data details, including for MegaMedical, multi-annotator
data, and simulated data. We give additional details on the _Tyche-TS_
training strategy, as well as the augmentations used at inference for _Tyche-
IS_. We also detail how we trained each benchmark, and their respective
limitations.

We provide an intuition behind the best candidate Dice loss, and show that
with _Tyche_ , it is possible to optimize for objectives that apply to the
candidate predictions as a group, and show results when optimizing GED.

Tyche Analysis.

For _Tyche-TS_ , we investigate four aspects: noise, context set, number of
predictions, and SetBlock. We show how different noise levels impact the
predictions. Moreover, we give examples of how prediction changes when the
context or the noise changes. We also show that the number of predictions at
inference impacts the diversity of the segmentation candidates predicted.

For _Tyche-IS_ , we investigate how different augmentations affect
performance.

We also analyze a scenario where only few of annotated examples are available,
using the LIDC-IDRI dataset. We compare _Tyche_ with these samples in the
context to PhiSeg, trained on these few annotated samples.

Further Evaluation.

Given the nature of stochastic segmentation tasks, no single metric fits all
purposes, and the optimal metric depends on the downstream goals. We provide
sample diversity and Hungarian Matching, showing that _Tyche_ performs well on
these as well.

We also present performance per dataset and show that the relative performance
of each model stays generally unchanged, even though the difficulty of each
dataset is widely different.

We provide additional visualizations on how _Tyche_ and the baselines perform
for each datasets, both on single and multi-annotator data.

Data split. All analysis results in this supplemental material use the
validation set of the out-of-distribution datasets, to avoid making modeling
decisions on the test sets.

##  Appendix B Tyche Model Choices

We present the implementation details for the data, our _Tyche_ models, and
the benchmarks.

###  B.1 Medical Image Data

Megamedical. We build on the dataset collection proposed in [19, 131], using
the similar preprocessing methods. The complete list of datasets is presented
Table 10.

Processing. Each image is normalized between 0 and 1 and is resized to
$128\times 128$. When the full 3D volume is available, we take two slices for
each task: the slice in the middle of the volume and the slice with the
largest count of pixels labelled for that task.

Task Definition. We consider a task as labelling a certain structure from a
certain modality for a certain dataset. If a given medical image has different
structures labeled, we consider each structure a different task. For 3D
volumes, we consider each axis a different task.

Synthetic Data. We use a set of synthetic tasks to enhance the performance of
our networks, similar to [19]. Some examples are shown in Figure 11.

Synthetic Multi-Annotator Data. We use synthetic data to encourage diversity
in our predictions. Figure 10 shows examples of blob targets with the
corresponding simulated annotations.

Figure 10: Synthetic Multi-Annotated Examples. Each blob is generated by
deforming a white disk using a random smooth deformation field, each
representing a different rater. The blobs are then averaged to form the target
image to segment. Figure 11: Synthetic Data Examples. Example of training
images on the single annotator synthetic data. Each row is an example of a
target-context pair corresponding to a different task.

###  B.2 Training of Tyche-TS

We train _Tyche-TS_ with a context size of 16 and a batch size of 4. We use a
learning rate of 0.0001, Adam as an optimizer, and a kernel size of 3. We
apply the augmentations shown Table 6. To obtain $K$ segmentation candidates,
we generate $K$ noise samples $z_{k}\sim\mathcal{N}(0,\mathbb{I})$. We
duplicate the input $K$ times and concatenate each noise sample with a
duplicated target to form $K$ stochastic inputs to the network.

Augmentations | $p$ | Parameters
---|---|---
|  | degrees $\in[-25,25]$
|  | translate$\in[0,0.1]$
Random Affine | 0.25 | scale$\in[0.9,1.1]$
|  | brightness$\in[-0.1,0.1]$,
Brightness Contrast | 0.5 | contrast$\in[0.5,1.5]$
|  | $\alpha\in[1,2.5]$
Elastic Transform | 0.8 | $\sigma\in[7,9]$
Sharpness | 0.25 | sharpness$=5$
Flip Intensities | 0.5 | None
|  | $\sigma\in[0.1,1.0]$
Gaussian Blur | 0.25 | k=5
|  | $\mu\in[0,0.05]$
Gaussian Noise | 0.25 | $\sigma\in[0,0.05]$
(a) In-Task Augmentation

Augmentations | $p$ | Parameters
---|---|---
|  | degrees $\in[0,360]$
|  | translate$\in[0,0.2]$
Random Affine | 0.5 | scale$\in[0.8,1.1]$
|  | brightness$\in[-0.1,0.1]$,
Brightness Contrast | 0.5 | contrast$\in[0.8,1.2]$
|  | $\sigma\in[0.1,1.1]$
Gaussian Blur | 0.5 | $k=5$
|  | $\mu\in[0,0.05]$
Gaussian Noise | 0.5 | $\sigma\in[0,0.05]$
|  | $\alpha\in[1,2]$
Elastic Transform | 0.5 | $\sigma\in[6,8]$
Sharpness | 0.5 | sharpness$=5$
Horizontal Flip | 0.5 | None
Vertical Flip | 0.5 | None
Sobel Edges Label | 0.5 | None
(b) Task Augmentation

Table 6: Set of augmentations used to train Tyche-TS. We distinguish between
augmentations aimed at increasing the diversity inside a task (Top) and the
augmentations aimed at increasing the diversity of tasks (Bottom). An
augmentation is applied with probability $p$ (second column). Figure 12:
Example Augmentations for Tyche-IS. We show for samples from different
datasets how three new segmentation candidates are generated. Starting from
the top row, augmentations are applied to both the target and context set to
obtain new prediction. Each row represents a new target-context pair. We do
not show the corresponding labels as for _Tyche-IS_ , we only augment with
intensity-based transforms.

###  B.3 Network for Tyche-IS

For the _Tyche-IS_ network, we use the baseline UniverSeg [19], trained with
the same data as _Tyche-TS_ , same batch size, and same context size. At test
time, we use the augmentations in Table 7. Figure 12 shows example
augmentation on different samples.

_Tyche-IS_ Augment. | $p$ | Parameters
---|---|---
|  | $\sigma\in[0.1,1.0]$
Gaussian Blur | 0.25 | $k=5$
|  | $\mu\in[0,0.05]$
Gaussian Noise | 0.25 | $\sigma\in[0,0.05]$
Flip Intensities | 0.5 | None
Sharpness | 0.25 | sharpness=5
|  | brightness$\in[-0.1,0.1]$,
Brightness Contrast | 0.25 | contrast$\in[0.5,1.5]$
Table 7: Augmentations used for _Tyche-IS_. We focus on intensity transforms,
to avoid inverting the prediction. For each image, an augmentation is sampled
with probability $p$. Figure 13: Improvement of UniverSeg by training with
more data. By training UniverSeg on more data, we obtain a better average Dice
score than the official UniverSeg release.

###  B.4 Benchmarks

We distinguish three types of benchmarks: in-context methods, that can take as
input a context set, the interactive frameworks, and the task-trained
specialized upper bounds.

In-context baselines. Because they were trained on medical data, we use SegGPT
and SAM-Med2D as provided in their official release. We train from scratch
UniverSeg and SENet. We use a batch size of 4 and the same set of data
augmentation transforms as the ones used for _Tyche-TS_ to encourage within
task and across task diversity. Training UniverSeg on the same datasets as
_Tyche_ improves performances compared to the official UniverSeg release, as
shown in Figure 13.

Specialized benchmarks. For the specialized models (CIDM, PhiSeg, and
Probabilistic UNet), we train a model for each task, for a total of 20 tasks.
For each task, we train three model variants, one for each different data
augmentation scheme, shown Table 8. We select the model with the data
augmentation strategy that does best on the out-of-distribution validation
set. We use a batch size of 4. For CIDM and Probabilistic UNet, we use the
official PyTorch release. We found Probabilistic UNet particularly unstable to
train on some datasets when applying augmentations. To avoid diverging loss
values, we had to initialize the Probabilistic UNet models from versions
trained on datasets without augmentation. For PhiSeg, the official TensorFLow
release was unstable, and we worked with the authors to run a more recent
PyTorch version of their code. Per the authors’ request, we smooth the
predicted segmentations of CIDM to remove irregularities in the final
segmentations.

Augmentation Set 1 | $p$ | Parameters
---|---|---
None | 1 | None
(a) No Augmentation. The images are given to the specialized model as is.

Light Augmentation | $p$ | Parameters
---|---|---
|  | $\sigma\in[0.1,1.5]$
Gaussian Blur | 0.5 | $k=7$
|  | $\mu\in[0,0.1]$
Gaussian Noise | 0.5 | $\sigma\in[0,0.1]$
|  | $\alpha\in[1,2]$
Variable Elastic Transform | 0.25 | $\sigma\in[6,8]$
(b) Light Augmentation. Gaussian bur, Gaussian noise, elastic transforms are
each applied to the target with probability $p$.

Heavy Augmentation | $p$ | Parameters
---|---|---
|  | $\sigma\in[0.1,1.5]$
Gaussian Blur | 0.5 | $k=7$
|  | $\mu\in[0,0.1]$
Gaussian Noise | 0.25 | $\sigma\in[0,0.1]$
|  | $\alpha\in[1,2]$
Elastic Transform | 0.25 | $\sigma\in[6,8]$
|  | degrees $\in[0,360]$
|  | translate$\in[0,0.2]$
Random Affine | 0.5 | scale$\in[0.8,1.1]$
|  | brightness$\in[-0.1,0.1]$,
Brightness Contrast | 0.5 | contrast$\in[0.5,1.5]$
Horizontal Flip | 0.5 | None
Vertical Flip | 0.5 | None
Sharpness | 0.5 | sharpness$=5$
(c) Full Set of Augmentations. To form the last augmentation set, we add the
following transforms: Affine transform, Sharpness, Brightness and Flips.

Table 8: Set of Augmentations used to train the specialized benchmarks. For
evaluation, we select the model with the training augmentations that does best
on the out-of-distribution validation set.

Interactive Methods. We use the SAM-Med2D and SAM as interactive segmentation
baselines. In our setting, user interaction is not available. We simulate it
by averaging the ground truth labels from the context set and sample clicks
and bounding boxes from averages.

We use SAM-Med2D’s official release as it is an adaptation of SAM exclusively
meant for 2D medical image segmentation tasks. We found empirically that SAM-
Med2D performs best when 3 positive and 2 negative clicks are sampled. For the
negative clicks, we sample areas inside the bounding box that are not covered
by the context set.

We also fine-tune SAM on our data. We sample 5 positive clicks and 5 negative
clicks uniformly. We also provide a bounding box and the average of the
context label maps as input.

##  Appendix C Advantages of candidates loss

###  C.1 Intuition behind best candidate loss

The best candidate loss only evaluates the candidate that yields the lowest
loss. We provide intuition for this loss function (Figure 14).

Given as input a target $x^{t}$ and a context $\mathcal{S}^{t}$, the model
outputs $K$ segmentation candidates $\hat{y}_{j}$. Despite potentially high
target ambiguity, for each training iteration we use only one annotation
$y^{t}_{r}$, .

The loss element $l(\hat{y}_{k},y^{t}_{r})$ captures volume overlap between a
candidate prediction $\hat{y}_{k}$ and the rater annotation $y^{t}_{r}$. If a
regular loss is used across all predictions, then each prediction tends
towards the lowest expect cost over the space of possible segmentations for
that image, leading to a mean segmentation among raters. This is particularly
harmful when target ambiguity is high (high rater disagreement). Then, the
mean segmentation may be very different than any one rater and may not be
representative of the ambiguity. For the best candidate loss, the model is
encouraged to make very different guesses for different segmentation
candidates. This increases the chance that one prediction matches the rater
being used as ground truth at that iteration.

Figure 14: Best candidate Loss Explanation. Because of ambiguity in the
target, there are multiple plausible ground truths. With a standard Dice loss,
all the candidates segmentation converge to the mean segmentation. The best
candidate loss enables the model to explore different plausible segmentation,
since even one proposal matching the rater segmentation used at that iteration
leads to a good loss value. Figure 15: _Tyche-TS_ with GED loss (yellow)
improved test-time GED performance (lower is better).

###  C.2 Distribution loss function

Since _Tyche-TS_ enables multiple candidates that can coordinate with one
another, we can employ loss functions that apply to a collection of raters and
predictions. We demonstrate this concept by fine-tuning _Tyche-TS_ , using a
$GED^{2}$ loss function.

Figure 15 shows that the corresponding model produces samples with lower
$GED^{2}$, as expected.

Figure 16: Prediction variability as a function of injected noise. Examples of
predictions for three subjects with three different types of input noise. Top
to Bottom: zero noise, Gaussian noise constant across segmentation candidates,
and randomly sampled Gaussian noise. The context set is fixed. With random
noise as an input _Tyche_ can output diverse segmentation candidates. Figure
17: Prediction variability as a function of the context set. The context set
significantly contributes to the variability of the output. For three
different subjects, we show how the corresponding segmentation candidates are
affected by different context sets. The random Gaussian noise given as input
is fixed for each set of candidates.

Figure 18: _Tyche-TS_ with no noise gives similar results to UniverSeg.
Setting the noise to 0 for _Tyche-TS_ gives similar best candidate Dice score
and similar $GED^{2}$ to UniverSeg. Top: Generalized Energy Distance. Bottom:
Best candidate Dice Score. Figure 19: Setting a number of predictions larger
than 1 allows for more diversity. The outputs of _Tyche-TS_ are a lot more
uniform when the number of prediction is set to 1 than when it is set to 5.
Figure 20: Removing the SetBlock leads to less diversity in the output. The
segmentation candidates output by _Tyche-TS_ are more diverse than the
candidates of a _Tyche-TS_ model trained without the SetBlock interaction.

##  Appendix D Tyche Analysis

We first analyse the different components that lead to variability in _Tyche-
TS_ : the noise, the context set, the number of prediction in on forward pass,
and the SetBlock mechanism. We then investigate how the size of the context
set and the number of predictions impact performances. For _Tyche-IS_ , we
investigate different plausible sets of augmentations to apply. Finally, we
show that _Tyche_ can produce good results even in data scarcity settings when
a stochastic methods trained solely on a few samples would have limited
performance.

###  D.1 Diversity from noise

Figure 16 shows how three predictions vary depending on three noise variants:
zero-noise, constant noise across candidates, and variable noise. Both zero
noise and constant noise lead systematically to identical segmentation
candidates. For each example of segmentation candidates, the context set is
fixed.

Figure 18 shows that setting the noise to 0 for _Tyche-TS_ produces similar
performances to UniverSeg, both for GED and best candidate Dice score.

###  D.2 Diversity from context

Figure 17 shows example predictions for three subjects and three context sets.
Different contexts lead to different predictions, even if the noise is the
same, sometimes drastically. For instance, the predictions for subject 1 and
context 3 contain a candidate with no annotations.

###  D.3 Diversity in the number of predictions

While _Tyche-IS_ predicts each segmentation sample sequentially, _Tyche-TS_
has a one-shot mechanism that predicts all the candidates at once. One may
wonder how the number of predictions requested at inference impacts the output
of _Tyche-TS_. Figure 19 shows that with the number of prediction set to 1,
_Tyche-TS_ loses a lot of its diversity compared to 5 predictions.

This is also shown quantitatively in Figure 21, where Generalized Energy
Distance decreases as the number of predictions increases.

###  D.4 Diversity from the SetBlock

_Tyche-TS_ has an intrinsic mechanism to encourage the segmentation candidates
to be diverse: SetBlock. Figure 20 visually compares the segmentations
predicted with the mechanisms and the segmentations predicted by a _Tyche_
model trained without the SetBlock. Without the SetBlock, the predictions
output by _Tyche_ are a lot less diverse.

###  D.5 Size of the Context Set

Generally, a larger context set improves performances. Figure 22 shows that
Generalized Energy Distance improves as the size of the context set increases.
However, the improvement decreases beyond 16 samples.

Figure 21: $GED^{2}$ for Tyche as a function of the number of predictions,
$K_{i}$. Performances improve with the number of prediction but with
diminishing returns. Figure 22: $GED^{2}$ for Tyche as a function of the
inference context size. Performances improve as the context size increases but
with diminishing returns.

###  D.6 Augmentations

We study different augmentation for _Tyche-IS_ and how the influence the
quality of predictions. We consider four settings: no augmentation, only light
augmentation such as Gaussian Noise and Gaussian Blur, described Table 7, the
_Tyche-IS_ augmentations shown Table 7 and the _Tyche-IS_ with slightly
stronger parameters. shown Table 9. The results are shown in Figure 23 both
aggregated across datasets and for each dataset individually. Overall, adding
augmentations improves the best candidate Dice score. However, it can degrades
the quality of the predictions, for instance for STARE. The augmentation we
selected for _Tyche-IS_ is the most promising so far, without requiring
inversion of the transform applied.

Figure 23: Performance of different augmentation schemes for _Tyche-IS_. We consider four augmentations: none, light, the one used in _Tyche-IS_ and a stronger version of the later. Top: Per Dataset. Bottom: Overall. The one that we selected for _Tyche-IS_ is the most promising. _Tyche-IS_ High Aug. | $p$ | Parameters
---|---|---
|  | $\sigma\in[0.5,1.0]$
Gaussian Blur | 0.25 | $k=5$
|  | $\mu\in[0.4,0.5]$
Gaussian Noise | 0.5 | $\sigma\in[0.1,0.2]$
Flip Intensities | 0.5 | None
Sharpness | 0.25 | sharpness=5
|  | brightness$\in[-0.1,0.1]$,
Brightness Contrast | 0.25 | contrast$\in[0.5,1.5]$
Table 9: High augmentations used to validate _Tyche-IS_. We focus on intensity
transforms, to avoid inverting the prediction. For each image, an augmentation
is sampled with probability $p$.

###  D.7 Few Shot Regime

We train PhiSeg on a subset of LIDC-IDRI, $\tilde{\mathcal{S}}$, and examine
how this network generalizes compared to Tyche, where the context set is
$\tilde{\mathcal{S}}$. We investigate four few-shot settings: 3, 5, 8 and 16.
For each, we train 30 PhiSegs: 10 seeds to account for variability in our
samples and three data augmentation regimes (none, light, normal). For each
seed and each few-shot setting, we select the PhiSeg that does best on the
validation set. Figure 24 shows that _Tyche_ can leverage the data available
much more effectively than PhiSeg, which fails to learn with so little
samples.

Figure 24: Few-Shot Regime. Comparison between _Tyche-TS_ and PhiSeg trained
on the few-shot examples. PhiSeg fails to learn from very few samples both on
max. Dice score and GED, compared to _Tyche_.

##  Appendix E Further Evaluation

Figure 25: Sample Diversity on Multi-Annotator Data. Since the sample
diversity of the deterministic methods is 0, we do not show them here. _Tyche-
IS_ produces the most diverse samples, while Fine-Tuned SAM has very low
diversity, despite varying clicks and bounding box locations. Higher is
better. Figure 26: Hungarian Matching on Multi-Annotator Data. Both _Tyche-IS_
and UniverSeg perform well. _Tyche-TS_ performs best. Higher is better.

###  E.1 Performances on other Metrics

Evaluating the quality of different predictions can be challenging especially
when different annotators are available. We proposed best candidate Dice score
and Generalized Energy Distance. Some also analysed sample diversity [95], and
Hungarian Matching [68, 133]. Sample diversity consists in measuring the
agreement between candidate predictions $\hat{\mathcal{Y}}$, rewarding most
diverse sets of candidates:

$D_{SD}(\hat{\mathcal{Y}})=\mathbb{E}\left[d(\hat{p},\hat{p}^{\prime})\right],$
(17)

where $\hat{p},\hat{p}^{\prime}\sim\hat{\mathcal{Y}}$ and $d(\cdot,\cdot)$ is
Dice score. One limitation of this metric is that it blindly rewards diversity
without taking into account the natural ambiguity in the target. Ideally, when
there is high ambiguity in the target, the samples are very diverse, and
inversely, when there is low ambiguity, the segmentation candidates are not
diverse.

In the context of stochastic predictions, Hungarian Matching [70] consists in
matching the set of predictions with the set of annotations, so that an
overall metric is minimized. We use negative Dice score. One limitation of
this method is that it has to be adapted when the number of annotators does
not match the number of prediction. The most widely used fix is to
artificially inflate the number of predictions and the number of annotations
to reach the least common multiple [68]. We use this strategy here as well.

Figures 25 and 26 show the performances for sample diversity and Hungarian
Matching respectively.

Figure 27: Best candidate Dice score for the Single-Annotator Datasets. Top to
Bottom: ACDC, PanDental, SCD, SpineWeb and WBC. _Tyche_ performs well in
general except for SpineWeb.

###  E.2 Per Dataset Results

We show for each dataset and each method the best candidate Dice score. We
also show for _Tyche_ and the different benchmarks Generalized Energy
Distance, Hungarian Matching and Sample Diversity for the datasets with
multiple annotations.

Best candidate Dice Score. Figure 27 shows the best candidate Dice score for
the single-annotator datasets. _Tyche_ performs well across datasets. Both
versions of _Tyche_ seem to dominate both types of benchmarks and be
comparable to the upper bounds, except for one dataset: SpineWeb. We
hypothesize that part of the performance drop is due to the nature of the
structure to segment in SpineWeb: individual vertebra. Most of our training
data contains single structures to segment.

Figure 28 shows the best candidate Dice score for the multi-annotator
datasets. Similar conclusions can be drawn as for the single-annotator data.
Some benchmarks are particularly sensitive to the data they are evaluated on,
for instance SegGPT. This methods performs really well on the Prostate data
but quite poorly on the STARE and the Hippocampus data. We assume that because
SegGPT was designed for images of 448x448, our images of dimension 128x128
might affect performances.

Figure 28: Best candidate Dice Score for Multi-Annotator Datasets Top to
bottom: Hippocampus, LIDC-IDRI, Prostate Task 1, Prostate Task 2 and STARE.
_Tyche_ performs well across datasets. (Higher is better.)

Generalized Energy Distance. Figure 29 shows Generalized Energy distance for
the multi-rater datasets.

Figure 29: Generalized Energy Distance for Multi-Annotator Datasets. Top to
bottom: Hippocampus, LIDC-IDRI, Prostate Task 1, Prostate Task 2 and STARE.
_Tyche_ performs well across datasets. (Lower is better.)

Hungarian Matching. Figure 30 shows the Hungarian Matching metric for the
multi-rater datasets. We find that UniverSeg performs particularly well. We
suspect that because we have to artificially duplicate our samples to compute
this metric, the resulting scenario favors methods that are closer to the
mean.

Figure 30: Hungarian Matching Dice for Multi-Annotator Datasets. Top to
bottom: Hippocampus, LIDC-IDRI, Prostate Task 1, Prostate Task 2 and STARE.
_Tyche_ performs well across datasets. (Higher is better.)

Sample Diversity. Figure 31 shows sample diversity for the multi-annotator
datasets. We only show the sample diversity for the methods outputing more
than one segmentation candidate. For UniverSeg, SegGPT and SENet, the sample
diversity is trivially 0.

Figure 31: Sample Diversity for Multi-Annotator Datasets. Top to bottom:
Hippocampus, LIDC-IDRI, Prostate Task 1, Prostate Task 2 and STARE. _Tyche_
performs well across datasets. We only show methods with diversity greater
than 0. (Higher is better.)

###  E.3 Additional Visualizations

We show additional visualization for _Tyche_ frameworks as well as all the
benchmarks.

##### Single-Annotator Data

Visualizations for ACDC are shown Figure 32. We show two PanDental examples
Figure 34 and Figure 35, one for each task. We show an example for SpineWeb
Figure 33 and one for WBC Figure 37.

Figure 32: Example Prediction for ACDC. Figure 33: Example Prediction for
SpineWeb. Figure 34: Example Prediction for PanDental, task 1. Figure 35:
Example Prediction for PanDental, task 2. Figure 36: Example Prediction for
SCD. Figure 37: Example Prediction for WBC.

##### Multi-Annotator Data

We show an example prediction for the Hippocampus data Figure 38 and one
example prediction for STARE in Figure 39. We provide visualizations for each
Prostate task Figures 40 and 41 respectively. Finally, we give two example
visualizations for LIDC-IDRI Figures 42 and 43.

Figure 38: Example Prediction for Hippocampus. Figure 39: Example Prediction for STARE. Figure 40: Example Prediction for Prostate Task 1. Figure 41: Example Prediction for Prostate Task 2. Figure 42: Example Prediction for LIDC-IDRI. Figure 43: Example Prediction for LIDC-IDRI. Dataset Name |  Description |  # of Scans |  Image Modalities
---|---|---|---
ACDC [14] |  Left and right ventricular endocardium |  99 |  cine-MRI
AMOS [54] |  Abdominal organ segmentation |  240 |  CT, MRI
BBBC003 [83] |  Mouse embryos |  15 |  Microscopy
BBBC038 [20] |  Nuclei images |  670 |  Microscopy
BUID [2] |  Breast tumors |  647 |  Ultrasound
BrainDev. [38, 37, 72, 114] |  Adult and Neonatal Brain Atlases |  53 |  multi-modal MRI
BRATS [7, 8, 93] |  Brain tumors |  6,096 |  multi-modal MRI
BTCV [75] |  Abdominal Organs |  30 |  CT
BUS [136] |  Breast tumor |  163 |  Ultrasound
CAMUS [77] |  Four-chamber and Apical two-chamber heart |  500 |  Ultrasound
CDemris [56] |  Human Left Atrial Wall |  60 |  CMR
CHAOS [60, 58] |  Abdominal organs (liver, kidneys, spleen) |  40 |  CT, T2-weighted MRI
CheXplanation [110] |  Chest X-Ray observations |  170 |  X-Ray
CT-ORG[106] |  Abdominal organ segmentation (overlap with LiTS) |  140 |  CT
DRIVE [122] |  Blood vessels in retinal images |  20 |  Optical camera
EOphtha [27] |  Eye Microaneurysms and Diabetic Retinopathy |  102 |  Optical camera
FeTA [102] |  Fetal brain structures |  80 |  Fetal MRI
FetoPlac [10] |  Placenta vessel |  6 |  Fetoscopic optical camera
HMC-QU [28, 65] |  4-chamber (A4C) and apical 2-chamber (A2C) left wall |  292 |  Ultrasound
HipXRay [40] |  Ilium and femur |  140 |  X-Ray
I2CVB [78] |  Prostate (peripheral zone, central gland) |  19 |  T2-weighted MRI
IDRID [103] |  Diabetic Retinopathy |  54 |  Optical camera
ISLES [43] |  Ischemic stroke lesion |  180 |  multi-modal MRI
KiTS [42] |  Kidney and kidney tumor |  210 |  CT
LGGFlair [18, 91] |  TCIA lower-grade glioma brain tumor |  110 |  MRI
LiTS [16] |  Liver Tumor |  131 |  CT
LUNA [115] |  Lungs |  888 |  CT
MCIC [36] |  Multi-site Brain regions of Schizophrenic patients |  390 |  T1-weighted MRI
MSD [118] |  Collection of 10 Medical Segmentation Datasets |  3,225 |  CT, multi-modal MRI
NCI-ISBI [17] |  Prostate |  30 |  T2-weighted MRI
OASIS [46, 88] |  Brain anatomy |  414 |  T1-weighted MRI
OCTA500 [79] |  Retinal vascular |  500 |  OCT/OCTA
PanDental [1] |  Mandible and Teeth |  215 |  X-Ray
PAXRay [112] |  Thoracic organs |  880 |  X-Ray
PROMISE12 [82] |  Prostate |  37 |  T2-weighted MRI
PPMI [89] |  Brain regions of Parkinson patients |  1,130 |  T1-weighted MRI
ROSE [86] |  Retinal vessel |  117 |  OCT/OCTA
SCD [104] |  Sunnybrook Cardiac Multi-Dataset Collection |  100 |  cine-MRI
SegTHOR [74] |  Thoracic organs (heart, trachea, esophagus) |  40 |  CT
SpineWeb [138] |  Vertebrae |  15 |  T2-weighted MRI
ToothSeg [50] |  Individual teeth |  598 |  X-Ray
WBC [139] |  White blood cell and nucleus |  400 |  Microscopy
WMH [71] |  White matter hyper-intensities |  60 |  multi-modal MRI
WORD [85] |  Organ segmentation |  120 |  CT
Table 10: Collection of datasets in MegaMedical 2.0. The entry number of scans is the number of unique (subject, modality) pairs for each dataset. Dataset Name |  Description |  # of Scans |  Image Modalities
---|---|---|---
LIDC-IDRI [4] |  Lung Nodules |  1018 |  CT
QUBIQ [92] |  Brain, kidney, pancreas and prostate |  209 |  MRI T1, Multimodal MRI, CT
STARE [47] |  Blood vessels in retinal images |  20 |  Optical camera
Table 11: Multi-Annotator Data. The entry number of scans is the number of
unique (subject, modality) pairs for each dataset.