2^{q-1}\frac{\mathbb{E}\left[Y^{q}\right]+\frac{\mathbb{E}[Y^{q}]^{2}}{\lambda^{q}}}{\lambda^{q}(1+\frac{\mathbb{E}[Y^{q}]}{\lambda^{q}})^{2}}=2^{q-1}\frac{\mathbb{E}\left[Y^{q}\right]}{\lambda^{q}(1+\frac{\mathbb{E}[Y^{q}]}{\lambda^{q}})}=2^{q-1}\frac{\mathbb{E}\left[Y^{q}\right]}{\lambda^{q}+\mathbb{E}[Y^{q}]}.$

##  Appendix D Technical tools

We remind the reader of Bernstein inequality, a classical concentration
inequality, this form of Bernstein inequality is borrowed from [4, Theorem
2.10].

######  Theorem 5.

Let $X_{1},\dots,X_{n}$ be independent real-valued random variables. Assume
that there exist positive numbers $v$ and $c$ such that
$\sum_{i=1}^{n}\mathbb{E}[X_{i}^{2}]\leq v$ and

$\sum_{i=1}^{n}\mathbb{E}[(X_{i})_{+}^{q}]\leq vc^{q-2}\quad\text{for all
integers }q\geq 3,$

where $x_{+}=\max(0,x)$. Then for all $t>0$

$\mathbb{P}\left(\sum_{i=1}^{n}(X_{i}-\mathbb{E}[X_{i}])\geq\sqrt{2vt}+ct\right)\leq
e^{-t}.$

The following theorem is borrowed from [1, Theorem 4], it is a concentration
inequality for suprema of sums of independent random variables.

######  Theorem 6.

Let $X_{1},\dots,X_{n}$ be independent random variables with values in a
measurable space $(\mathcal{S},\mathcal{B})$ and let $\mathcal{F}$ be a
countable class of measurable functions $f:\mathcal{S}\to\mathbb{R}$. Assume
that for every $f\in\mathcal{F}$ and every $i$, $\mathbb{E}[f(X_{i})]=0$ and
for any $\alpha\in(0,1]$ and all $i$,
$\|\sup_{f}|f(X_{i})|\|_{\psi_{\alpha}}<\infty$. Let

$Z=\sup_{f\in\mathcal{F}}\left|\sum_{i=1}^{n}f(X_{i})\right|.$

Define moreover

$\sigma^{2}=\sup_{f\in\mathcal{F}}\sum_{i=1}^{n}\mathbb{E}[f(X_{i})^{2}].$

Then, for all $0<\eta<1$ and $\delta>0$, there exists a constant
$C=C(\alpha,\eta,\delta)>0$ such that for all $t\geq 0$,

$\mathbb{P}(Z\geq(1+\eta)\mathbb{E}[Z]+t)\leq\\\
\exp\left(-\frac{t^{2}}{2(1+\delta)\sigma^{2}}\right)+3\exp\left(-\left(\frac{t}{C\|\max_{i}\sup_{f\in\mathcal{F}}|f(X_{i})|\|_{\psi_{\alpha}}}\right)^{\alpha}\right),$
(62)

and

$\mathbb{P}(Z\leq(1-\eta)\mathbb{E}[Z]-t)\leq\\\
\exp\left(-\frac{t^{2}}{2(1+\delta)\sigma^{2}}\right)+3\exp\left(-\left(\frac{t}{C\|\max_{i}\sup_{f\in\mathcal{F}}|f(X_{i})|\|_{\psi_{\alpha}}}\right)^{\alpha}\right).$
(63)

## Acknowledgements

The author would like to thank Elvezio Ronchetti, for the discussions that
gave him the ideas behind this article, and his two phd-advisors Matthieu
Lerasle and Guillaume Lecué for their advice and reviews.

## References

  * [1] [author] Adamczak, RadoslawR. et al. (2008). A tail inequality for suprema of unbounded empirical processes with applications to Markov chains. Electronic Journal of Probability 13 1000–1034.
  * [2] [author] Bahadur, R. R.R. R. (1966). A Note on Quantiles in Large Samples. Ann. Math. Statist. 37 577–580. 10.1214/aoms/1177699450
  * [3] [author] Beck, AmirA. and Sabach, ShohamS. (2015). Weiszfeld’s method: Old and new results. Journal of Optimization Theory and Applications 164 1–40.
  * [4] [author] Boucheron, S.S., Lugosi, G.G. and Massart, P.P. (2013). Concentration Inequalities: A Nonasymptotic Theory of Independence. OUP Oxford.
  * [5] [author] Catoni, OlivierO. (2012). Challenging the empirical mean and empirical variance: A deviation study. Ann. Inst. H. Poincaré Probab. Statist. 48 1148–1185. 10.1214/11-AIHP454
  * [6] [author] Catoni, OlivierO. and Giulini, IlariaI. (2017). Dimension-free PAC-Bayesian bounds for matrices, vectors, and linear least squares regression. (https://arxiv.org/abs/1712.02747).
  * [7] [author] Chen, MengjieM., Gao, ChaoC., Ren, ZhaoZ. et al. (2018). Robust covariance and scatter matrix estimation under Huber’s contamination model. The Annals of Statistics 46 1932–1960.
  * [8] [author] Cherapanamjeri, YeshwanthY., Flammarion, NicolasN. and Bartlett, Peter LP. L. (2019). Fast mean estimation with sub-gaussian rates. In Conference on Learning Theory 786–806. PMLR.
  * [9] [author] Chinot, GeoffreyG., Lecué, GuillaumeG. and Lerasle, MatthieuM. (2019). Robust statistical learning with Lipschitz and convex loss functions. Probability Theory and related fields 1–44.
  * [10] [author] De la Pena, VictorV. and Giné, EvaristE. (2012). Decoupling: from dependence to independence. Springer Science & Business Media, New York.
  * [11] [author] Debruyne, MichielM., Hubert, MiaM. and Suykens, Johan AKJ. A. (2008). Model selection in kernel based regression using the influence function. Journal of Machine Learning Research 9 2377–2400.
  * [12] [author] Depersin, JulesJ. and Lecué, GuillaumeG. (2019). Robust subgaussian estimation of a mean vector in nearly linear time.
  * [13] [author] Devroye, LucL., Lerasle, MatthieuM., Lugosi, GaborG. and Oliveira, Roberto I.R. I. (2016). Sub-Gaussian mean estimators. The Annals of Statistics 44 2695–2725. 3576558
  * [14] [author] Diakonikolas, IliasI. and Kane, Daniel M.D. M. (2019). Recent Advances in Algorithmic High-Dimensional Robust Statistics.
  * [15] [author] Diakonikolas, IliasI., Kane, Daniel M.D. M. and Pensia, AnkitA. (2020). Outlier Robust Mean Estimation with Subgaussian Rates via Stability.
  * [16] [author] Elisseeff, AndréA. and Pontil, MassimilianoM. (2002). Leave-one-out error and stability of learning algorithms with applications Stability of Randomized Learning Algorithms Source. International Journal of Systems Science - IJSySc 6\.
  * [17] [author] Fernholz, Luisa TurrinL. T. (1983). von Mises calculus for statistical functionals. Lecture Notes in Statistics 19\. Springer-Verlag, New York. 10.1007/978-1-4612-5604-5 713611
  * [18] [author] Hampel, Frank R.F. R. (1971). A General Qualitative Definition of Robustness. Ann. Math. Statist. 42 1887–1896. 10.1214/aoms/1177693054
  * [19] [author] Hampel, Frank R.F. R. (1974). The Influence Curve and Its Role in Robust Estimation. Journal of the American Statistical Association 69 383–393.
  * [20] [author] Hampel, Frank R.F. R., Ronchetti, Elvezio M.E. M., Rousseeuw, Peter J.P. J. and Stahel, Werner A.W. A. (1986). Robust Statistics: The Approach Based on Influence Functions, 1st edition ed. Wiley Series in Probability and Statistics. Wiley missing.
  * [21] [author] Hanson, D. L.D. L. and Wright, F. T.F. T. (1971). A Bound on Tail Probabilities for Quadratic Forms in Independent Random Variables. Ann. Math. Statist. 42 1079–1083. 10.1214/aoms/1177693335
  * [22] [author] He, XumingX. and Shao, Qi-ManQ.-M. (1996). A general Bahadur representation of M -estimators and its application to linear regression with nonstochastic designs. Ann. Statist. 24 2608–2630. 10.1214/aos/1032181172
  * [23] [author] Hopkins, SamS., Li, JerryJ. and Zhang, FredF. (2020). Robust and heavy-tailed mean estimation made simple, via regret minimization. Advances in Neural Information Processing Systems 33 11902–11912.
  * [24] [author] Hopkins, Samuel B.S. B. (2020). Mean Estimation with Sub-Gaussian Rates in Polynomial Time. Annals of Statistics 48 1193–1213. (https://arxiv.org/abs/1809.07425).
  * [25] [author] Hopkins, Samuel B.S. B. and Li, JerryJ. (2019). How Hard Is Robust Mean Estimation? CoRR abs/1903.07870.
  * [26] [author] Huber, Peter J.P. J. (1964). Robust Estimation of a Location Parameter. Ann. Math. Statist. 35 73–101. 10.1214/aoms/1177703732 0161415
  * [27] [author] Huber, Peter JP. J. and Ronchetti, Elvezio ME. M. (2009). Robust statistics; 2nd ed. Wiley Series in Probability and Statistics. Wiley, Hoboken, NJ.
  * [28] [author] Koh, Pang WeiP. W. and Liang, PercyP. (2017). Understanding Black-box Predictions via Influence Functions.
  * [29] [author] Lai, Kevin AK. A., Rao, Anup BA. B. and Vempala, SantoshS. (2016). Agnostic estimation of mean and covariance. In 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS) 665–674. IEEE.
  * [30] [author] Lerasle, MatthieuM. (2019). Lecture Notes: Selected topics on robust statistical learning theory.
  * [31] [author] Lugosi, GáborG. and Mendelson, ShaharS. (2019). Mean Estimation and Regression Under Heavy-Tailed Distributions: A Survey. Foundations of Computational Mathematics 19 1145–1190. 10.1007/s10208-019-09427-x
  * [32] [author] Lugosi, GáborG., Mendelson, ShaharS. et al. (2019). Sub-Gaussian estimators of the mean of a random vector. The annals of statistics 47 783–794.
  * [33] [author] Minsker, StanislavS. (2015). Geometric median and robust estimation in Banach spaces. Bernoulli 21 2308–2335. 10.3150/14-BEJ645 3378468
  * [34] [author] Minsker, StanislavS. (2018). Uniform bounds for robust mean estimators. arXiv preprint arXiv:1812.03523.
  * [35] [author] Minsker, StanislavS. and Mathieu, TimothéeT. (2019). Excess risk bounds in robust empirical risk minimization.
  * [36] [author] Ronchetti, ElvezioE. (1997). Robust inference by influence functions. Journal of Statistical Planning and Inference 57 59–72. Robust Statistics and Data Analysis, Part I. 10.1016/S0378-3758(96)00036-5
  * [37] [author] Sun, QiangQ., Zhou, Wen-XinW.-X. and Fan, JianqingJ. (2020). Adaptive huber regression. Journal of the American Statistical Association 115 254–265.
  * [38] [author] van der Vaart, Aad W.A. W. and Wellner, Jon A.J. A. (1996). Weak convergence and empirical processes. Springer Series in Statistics. Springer-Verlag, New York With applications to statistics. 1385671
  * [39] [author] Zhou, Wen-XinW.-X., Bose, KoushikiK., Fan, JianqingJ. and Liu, HanH. (2018). A new perspective on robust $M$-estimation: Finite sample theory and applications to dependence-adjusted multiple testing. Ann. Statist. 46 1904–1931. 10.1214/17-AOS1606
  * [40] [author] Zhu, BanghuaB., Jiao, JiantaoJ. and Steinhardt, JacobJ. (2020). Robust estimation via generalized quasi-gradients. arXiv preprint arXiv:2005.14073.


# Application-driven Validation of Posteriors in Inverse Problems

Tim J. Adler, Jan-Hinrich Nölke, Annika Reinke, Minu Dietlinde Tizabi,
Sebastian Gruber, Dasha Trofimova, Lynton Ardizzone, Paul F. Jaeger, Florian
Buettner, Ullrich Köthe, and Lena Maier-Hein  This project has received
funding from the European Research Council (ERC) under the European Union’s
Horizon 2020 research and innovation programme (NEURAL SPICING: grant
agreement No. [101002198]). Part of this work was funded by Helmholtz Imaging
(HI), a platform of the Helmholtz Incubator on Information and Data Science.
The work was co-funded by the European Union (ERC, TAIPO, 101088594). Views
and opinions expressed are however those of the authors only and do not
necessarily reflect those of the European Union or the European Research
Council. Neither the European Union nor the granting authority can be held
responsible for them. (T.J. Adler and J-H. Nölke are co-first authors.)
(Corresponding authors: J-H. Nölke and L. Maier-Hein. E-mail: {j.noelke,
<EMAIL_ADDRESS>T.J. Adler, J-H. Nölke, A. Reinke, M.D.
Tizabi, S. Gruber, D. Trofimova and L. Maier-Hein are/were with the German
Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical
Systems, Germany. J-H. Nölke and L. Maier-Hein are with the Faculty of
Mathematics and Computer Science, Heidelberg University, Germany. A. Reinke,
P.F. Jaeger and L. Maier-Hein are with the German Cancer Research Center
(DKFZ) Heidelberg, HI Helmholtz Imaging, Germany. M.D. Tizabi and L. Maier-
Hein are with the National Center for Tumor Diseases (NCT), NCT Heidelberg, a
partnership between DKFZ and University Medical Center Heidelberg. L.
Ardizzone and U. Köthe are/were with the Visual Learning Lab,
Interdisciplinary Center for Scientific Computing (IWR), Heidelberg
University, Heidelberg, Germany. P.F. Jaeger is with the German Cancer
Research Center (DKFZ) Heidelberg, Interactive Machine Learning Group,
Germany. F. Buettner is with the Department of Informatics and the Department
of Medicine, Goethe University Frankfurt, Germany and the German Cancer
Consortium (DKTK), partner site Frankfurt, a partnership between DKFZ and UCT
Frankfurt-Marburg, Germany and the German Cancer Research Center (DKFZ),
Heidelberg, Germany and the Frankfurt Cancer Institute, Germany. L. Maier-Hein
is with the Medical Faculty, Heidelberg University, Germany.

###### Abstract

Current deep learning-based solutions for image analysis tasks are commonly
incapable of handling problems to which multiple different plausible solutions
exist. In response, posterior-based methods such as conditional Diffusion
Models and Invertible Neural Networks have emerged; however, their translation
is hampered by a lack of research on adequate validation. In other words, the
way progress is measured often does not reflect the needs of the driving
practical application. Closing this gap in the literature, we present the
first systematic framework for the application-driven validation of posterior-
based methods in inverse problems. As a methodological novelty, it adopts key
principles from the field of object detection validation, which has a long
history of addressing the question of how to locate and match multiple object
instances in an image. Treating modes as instances enables us to perform mode-
centric validation, using well-interpretable metrics from the application
perspective. We demonstrate the value of our framework through instantiations
for a synthetic toy example and two medical vision use cases: pose estimation
in surgery and imaging-based quantification of functional tissue parameters
for diagnostics. Our framework offers key advantages over common approaches to
posterior validation in all three examples and could thus revolutionize
performance assessment in inverse problems.

###### Index Terms:

Validation, Metrics, Posterior, Deep Learning, Inverse Problems.

##  I Introduction

Figure 1:  Example of an inverse problem in medical vision. The task is to
recover the pose of an intraoperative X-ray system relative to the 3D patient
coordinate system to enable augmented reality visualization during medical
intervention. The ambiguity of the problem can be captured with an invertible
architecture, which represents multiple solutions (here: two) via modes in a
posterior distribution. Used abbreviation: Convolutional Neural Network (CNN).
Figure 2: Object detection validation methodology lends itself well to
posterior validation. This validation is subdivided into the steps of instance
localization, assignment, and computing of classification metrics. These steps
have natural analogs in the posterior validation case. Used abbreviations:
Average Precision (AP), True Positive (TP), False Positive (FP), False
Negative (FN), Standard Deviation (STD).

Deep learning has led to breakthrough successes in various areas of image
analysis. State-of-the-art approaches, however, commonly lack the capacity of
representing the fact that multiple (substantially different) plausible
solutions may exist. One medical vision example is the registration of two-
dimensional (2D) X-ray images with preoperative computed tomography (CT)
images in intraoperative surgical guidance systems (Fig. 1). To this end, the
pose of the X-ray modality relative to the patient coordinate system (given by
the preoperative three-dimensional (3D) image) has to be inferred from the 2D
X-ray images. Standard methods compute a single point estimate based on the
input images, thereby ignoring the fact that multiple different solutions may
exist. Posterior-based methods, such as conditional Diffusion Models
[batzolis2021conditional, chung2022improving] and Invertible Neural Networks
[ardizzone2018analyzing, ardizzone2019guided, nolke2021invertible], overcome
this bottleneck by converting the input to a ‘posterior’ – a full probability
distribution conditioned on the input, capable of capturing several plausible
solutions via multiple modes. While the field is currently experiencing much
progress on a methodological level, little attention is given to the adequate
validation of posterior-based methods, impeding their translation into
practice.

Most commonly, methods are either validated by extracting the maximum a
posteriori (MAP) probability location and using it as a point estimate
[ardizzone2018analyzing], by feeding the posteriors into the forward model (if
available) and choosing suitable metrics in the ‘observable space’
[ardizzone2018analyzing, zheng2022entropy, batzolis2021conditional], or
through more qualitative validation schemes as, for example, in the task of
image generation [ardizzone2019guided, bao2022all]. However, these validation
approaches are often inadequate as they neglect the requirements imposed by
the underlying application. For instance, providing the actual posteriors to
the users of a system that can handle ambiguous solutions is typically not
useful (lack of interpretability) and may not even be feasible (due to high
dimensionality). Here, validation should rather focus on the assessment of the
modes themselves, as this is what users base their decisions on in practice.
In the aforementioned clinical example (Fig. 1), images may commonly be
acquired with the patient in supine position, i.e., lying on the back. As the
prior influences the posterior, the biggest mode would correspond to the
standard position. The standard validation procedure based on MAP estimates
would ignore the small mode corresponding to a 180° rotated pose. Ignoring the
smaller mode(s) does not reflect clinical needs as a clinician could easily
choose between a small set of modes and even benefit from the model
information if a surprising mode appears.

In this paper, we therefore propose choosing a validation approach that
reflects the requirements of the driving application. Specifically, we argue
that most applications require a mode-centric validation, reflecting the fact
that domain experts (e.g., clinicians) work with concrete decisions/solutions
rather than probability distributions. In this vein, we propose metrics that
go beyond common regression errors and directly compare (multiple) predicted
to (multiple) reference modes. While this may sound trivial at first glance,
the specific implementation is not straightforward (How exactly are modes
localized? What to do in the case of mode assignment ambiguities?, etc.),
which may be one reason why the topic has – to our knowledge – not yet been
addressed in the literature. Closing this gap, our novel approach takes
inspiration from the object detection community (Fig. 2). By adopting the
principles of a localization criterion and assignment strategies, we are able
to perform a mode-centric validation with much more meaningful and better
interpretable metrics from an application perspective. In the above example,
this approach would require defining the localization criterion (and its
hyperparameters) such that the augmented reality visualization of surrounding
structures in the intraoperative X-ray can be achieved with acceptable
accuracy through the pose estimation. Classification-based performance metrics
would then be well-interpretable by the domain expert: The False Positives Per
Image (FPPI) at the computed Recall, for example, would inform the clinician
that they would need to select the most plausible pose from an average of
about FPPI+1 options during a surgery. While this would not be a problem for
FPPI = 1, it would be infeasible to choose from, for example, ten different
options.

Although it would be desirable to validate posteriors as comprehensively as
possible with both distribution-based and mode-based metrics, this may not
always be possible in real-world scenarios. In many cases, for example, a
ground truth posterior (required for distribution-based comparison) may not be
available. Moreover, the set of reference solutions may be non-exhaustive and,
for example, only contain one out of possibly multiple plausible solutions. We
address this challenge with a problem fingerprint that abstracts from the
specific problem by capturing key problem characteristics and available data
in a structured format. Guided by this fingerprint, metrics are then
recommended via a decision tree. The specific contributions of this paper are:

  1. 1.

Object detection analogy: To our knowledge, we are the first to uncover an
analogy between validation practice in an object detection setting and
validation of posteriors.

  2. 2.

Application-driven framework: Based on this analogy, we propose a posterior
validation framework that takes into account both the requirements of the
underlying application as well as the mathematical restrictions enforced by
the available validation data.

  3. 3.

Use case instantiation: An instantiation of the framework for three
complementary use cases reveals flaws in common validation practices and
showcases the benefit of a mode-centric approach.

Figure 3: Overview of metric selection framework for posterior validation.
Depending on the reference granularity (reference posterior with/without
labeled modes, exhaustive or non-exhaustive list of reference modes), the user
follows the correspondingly colored path in the decision tree. When a tree
branches, the fingerprint items determine which exact path to take.
Recommendations for distribution-based metrics (Subprocess S1) are provided in
Fig. 4. The main novelty of the proposal relates to the selection of object
detection-inspired metrics, which is presented in a separate Subprocess S2
(Fig. 5). The notation Metric1@Metric2 refers to providing the value for
Metric1 for a specific target value (e.g. Recall = 0.95) of Metric 2.

##  II Related Work

Prior work on recommendations for posterior validation is extremely sparse.
While recent efforts have focused on recommendations in the context of
classification, segmentation, and object detection [maier2022metrics], we have
not found any framework dedicated to the validation of posteriors in inverse
problems. Our analysis of the literature revealed the following common
validation principles: (1) Use of the MAP as a point estimate and application
of classic regression metrics. This validation scheme is extended by
regression metrics computed on resimulations (i.e., computing the forward
model on the posteriors), if available. Furthermore, statistical distances to
reference posteriors are commonly computed if the reference is actually given
as a posterior. Lastly, visual inspection and qualitative analyses of the
posterior (or interesting marginals) are also common practice (e.g., in
[ardizzone2018analyzing, haldemann2022exoplanet, butter2022two,
kohl2018probabilistic, monteiro2020stochastic]).

(2) In the context of conditional image generation, particular focus is put on
the quality of the generated images and their diversity. This is reflected by
commonly applied metrics such as peak signal-to-noise ratio (PSNR) or measures
of variability (e.g., variance or standard deviation (STD)) of the generated
images. At the same time, distribution-based metrics such as the Fréchet
Inception Distance (FID) are also common but rarely applied to posteriors as a
reference posterior is often lacking. Instead, validation or test images are
interpreted as samples from an unconditional distribution and compared to
samples drawn from the image generator. Depending on the exact image
generation task, direct resimulation (e.g., in super-resolution tasks) or
’resimulation via a downstream task’ (e.g., using an image classifier for
class-conditioned image generators) might be an option, in which case such
metrics are often reported (e.g., under the name of Consistency)
[ardizzone2019guided, ardizzone2021conditional, bao2022all, zheng2022entropy].

To the best of our knowledge, a mode-centric validation has not been proposed
before. Consequently, there is no prior work on using object detection
validation methodology on posterior-based inverse problem solvers.

##  III Methods

This section presents our posterior validation framework (Sec. III-A \- III-C)
as well as the conditional Invertible Neural Network (cINN)-based
architectures [ardizzone2018analyzing, ardizzone2019guided] that we developed
to instantiate the framework for medical vision problems (Sec. III-D).

Our validation framework features three main components to guide a user
through the process of application-relevant metric selection. First, to enable
an application-driven, modality-agnostic metric recommendation approach that
generalizes over domains, we encapsulate validation-relevant characteristics
of a given problem in a problem fingerprint. To this end, the parameters
listed in Tab. I are instantiated according to the domain interest. In a
second step, suitable metrics are selected based on this problem fingerprint
(Fig. 3). A key novelty in this step is the mode-centric validation
perspective inspired by the field of object detection (Fig. 5). Finally, as
this process can result in a pool of suitable metric candidates, the third
step involves the traversal of decision guides to help users understand the
tradeoffs and choose between different candidates, wherever necessary. The
following sections provide details on the three main components.

###  III-A Problem fingerprint

The fingerprint is summarized in Tab. I. While we assume the method to be
validated to provide a posterior distribution, the framework can handle
different types of references. Therefore, the most central fingerprint item is
P1: Reference granularity as it is the prerequisite for deciding whether
distribution-based metrics and/or object-inspired metrics should be used for
validation. Specifically, we distinguish four main formats in which the
reference may be available (corresponding to the colored paths in Fig. 3 and
4): posteriors with or without explicitly labeled modes, or a discrete set of
modes that may either be exhaustive or non-exhaustive. Note that a non-
exhaustive set of modes is very common in inverse problems because validation
data is often generated with a forward model for which the underlying input
serves as the (only) reference even if other inputs could have generated the
same output (see Fig. 1). Further properties will be detailed in the
following.

TABLE I: Full list of properties that comprise the inverse problem fingerprint. The first column contains the name of the property, the second column the values the property can take, and the third column a short description under which circumstances a value for a property should be chosen. Used abbreviations: conditional Invertible Neural Network (cINN), Generative Adversarial Network (GAN). [1pt] Property | Possible Options | Description
---|---|---
P1 Reference granularity | posterior with labeled modes | $Choosethisifthereferenceisaposteriorincludinglabels.$
| posterior without labeled modes | $Choosethisifthereferenceisaposteriorbutthereisnomodeinformationavailable.$
| exhaustive list of modes | $Choosethisifthereferenceisadiscretelistofmodeswhichisguaranteedtobecomplete.$
| non-exhaustive list of modes | $Choosethisifthereferenceisadiscretelistofmodesbutmodesmightbemissingonthelist.$
P2 Resimulation | available | $Choosethisiftheforwardmodel(oftenintheformofasimulation)isavailableandcanbecomputedwithacceptableresources.$
| unavailable | $Choosethisiftheforwardmodelisunavailableorprohibitivelyexpensive.$
P3 Confidence score | available | $Choosethisifyourmodedetectionmodelcanprovideconfidencescoresforeachmodeindividually.Ideally,thescoresformodesofthesameposteriorarenotinnatelycorrelated.$
| unavailable | $Choosethisifnosuitablescoreisavailable.$
P4 Prediction density | available | $Choosethisifyourmodelprovidesanexplicitlycomputabledensityfunction(e.g.,cINNs).$
| unavailable | $Choosethisifyourmodelrepresentsthedensityimplicitly(e.g.,GANs).$
P5 Natural discretization scale | available | $Choosethisifyourapplicationpermits,e.g.,amaximalnecessaryresolution.$
| unavailable | $Choosethisifyourproblemis,e.g.,toohigh-dimensional(toomanyempty/singularbins)fordiscretization.$
P6 Univariate posterior | yes | $Choosethisifthesolutionspaceoftheinverseproblemis1D.$
| no | $Choosethisifthesolutionspaceoftheinverseproblemhastwoormoredimensions.$
P7 Accurate uncertainty required | yes | $Choosethisifyourapplicationrequiresawell-calibrateduncertaintyintheformofacorrectposteriorshape.$
| no | $Choosethisifaccuratemodelocalizationissufficient.$
[1pt] |  |
Figure 4: Subprocess S1 for selecting distribution-based metrics. Based on the
exact representation of the predicted posterior and the dimensionality of the
problem, different metrics become available. Figure 5: Subprocess S2 for
selecting object detection-inspired metrics, comprising the steps of selecting
the localization criterion, the assignment strategy, and the actual
classification metric(s). The notation Metric1@Metric2 refers to providing the
value for Metric1 for a specific target value (e.g. Recall = 0.95) of Metric
2. Decision guides for selecting a suitable option from a list of candidates
are provided in section III-C. Used abbreviations: Average Precision (AP),
Free-response Receiver Operating Characteristic (FROC), False Positives Per
Image (FPPI).

###  III-B Metric selection

The workflow for metric selection, guided by the fingerprint, is provided in
Fig. 3. The two main steps are:

Selection of distribution-based metrics
If reference posteriors are provided (Property P1), distribution-based metrics
can be selected. The decision tree for selecting such a metric is depicted in
Fig. 4. The following properties are relevant in this context:

  * •

P4: Prediction density: Generative models can be categorized by whether they
give access to the underlying density of the distribution they model (e.g.,
cINNs) or not (e.g., classic Generative Adversarial Networks (GANs)
[goodfellow2020generative]). There is also a grey area where the models
provide bounds on the density (e.g., Variational Autoencoders (VAEs)
[kingma2013auto]). If the density is available, we can exploit it to gauge the
mismatch between the predicted and reference distribution using the Cross
Entropy [good1956some]. The Cross Entropy needs access to the prediction
density, but the reference density can be given as a sample. We propose the
usage of Cross Entropy as it optimally exploits the availability of the
density where it is accessible, whereas the other metrics make no explicit use
of its existence.

  * •

P5: Natural discretization scale: Many problems allow for natural
discretization, for instance, where there is a maximum necessary resolution
for an application (e.g., 1 percentage point (pp) oxygenation resolution might
be sufficient), and the range of the values is known. In such cases, the
predicted and reference posteriors can be binned with acceptable
discretization errors. Hence, the densities become mass functions, and the
(discrete) Kullback-Leibler (KL) Divergence [kullback1951information] is
accessible. We propose this metric due to its lack of hyperparameters (except
for the discretization parameters). However, if the solution space to the
inverse problem is high -dimensional, meaningful discretization is difficult
due to the curse of dimensionality. In this case, we would encounter many
empty bins and/or bins containing only a single sample. Such a binning is
inadequate to estimate the probability mass function, and we discourage the
use of the KL Divergence.

  * •

P6: Univariate posterior: In some rare cases, we are interested in a single
variable of interest as the solution to the inverse problem. If this is the
case, the posterior will be univariate and there are statistical distances
that tailor specifically to this setting. One example is the Kolmogorov-
Smirnov (KS) statistic [kolmogorov1933sulla, smirnov1948table], which gauges
the difference between two univariate distributions based on their cumulative
distribution function. The statistic itself can be used as a distance measure.
Additionally, the KS statistic is the basis of a classic hypothesis test,
which allows testing whether the posteriors significantly differ given some
$\alpha$-level. An alternative to the KS statistic is the Wasserstein Distance
[kantorovich1960mathematical], which is defined for arbitrary dimensions but
is computationally expensive for higher dimensions. Both distances have in
common that they are almost free of hyperparameters (the KS test has the
$\alpha$-level, and for the Wasserstein Distance, we have to choose the
underlying Lp-norm with a tendency to choose $p=1$ because the formula is
particularly simple), which alleviates us of the necessity to ’optimize’ the
metrics on a validation data set. The Wasserstein Distance defines a metric
(in the mathematical sense) on the space of distributions but does not
directly lead to a hypothesis test in the same way the KS statistic does.

  * •

P7: Accurate uncertainty required: In contrast to the previous properties,
which relate to ”hard facts” about the inverse problem (such as the dimension
of the solution space), this property is application-driven. In other words,
whether we are interested in accurate uncertainty quantification does not
depend on the underlying inverse problem but on the target application that
requires solving the inverse problem. While not directly visible in the
decision trees, the need for uncertainty quantification will influence the
metric selection, for example, at the localization criterion (where we can
decide to take a measure of variability of the modes into account). The
influence is elaborated in the decision guides below. Additionally, the need
for accurate uncertainty will inform the importance of the Calibration curve
suggested as a metric for discrete reference modes in Fig. 3.

If none of the properties P4 – P6 lead to a suitable distribution-based
metric, the user is left with two options (see Fig. 4). The first is the
Wasserstein Distance already introduced in the previous section. Its
disadvantage is the computational cost in higher dimensions. A pragmatic
solution is to apply the Wasserstein Distance to all 1D marginals individually
and aggregate the results. This reduces the expressiveness of the Wasserstein
Distance because there are distinct distributions with identical marginals,
which could not be distinguished by this heuristic Wasserstein Distance. The
other option is Maximum Mean Discrepancy (MMD) [gretton2012mmd], which is a
kernel method that introduces a metric on the space of distributions (at least
for suitable kernels) and whose computational costs are acceptable. Its main
downside is the sensitivity of the metric scores to the choice of the kernel
(both the family and the hyperparameters parametrizing the family). This
sensitivity often results in a separate validation set being required to
optimize the hyperparameters of the metric and also reduces the
interpretability of MMD.

Note that distribution-based metrics can also be used as a localization
criterion when using object detection-inspired metrics which will be described
in the following paragraph as depicted in Fig. 5.

Selection of object detection-inspired metrics
If the reference comes with explicit modes, the quality of the modes should be
explicitly assessed, possibly irrespective of the shape of the posterior
(which is heavily influenced by the prior). We take inspiration from object
detection validation by regarding predicted and reference modes as instances
and transferring object detection principles to our setting. Our proposal is
summarized in Fig. 2.

  * •

Localization criterion: To decide whether a mode matches the reference, a
criterion incorporating the location and (optionally) the shape of both
reference and prediction is needed. Based on the application and goal,
hyperparameters can be used to control the strictness of the criterion.

  * •

Assignment strategy: To match the correct prediction/reference pairs, an
adequate assignment strategy must be chosen. In this way, the matching of
multiple predictions to one reference mode or vice versa is avoided.

  * •

Classification metrics: Once predictions are located and assigned to reference
predictions, a confusion matrix can be computed, and meaningful classification
metrics can be calculated. Notably, these metrics should not rely on True
Negatives (TNs), which are typically not well-defined in detection tasks. We
argue that classification-based metrics offer a complementary view by treating
modes - and thus the potential solutions to a problem - as the central objects
of interest.

Note that treating modes as instances introduces a hierarchy, where each
posterior consists of one or more modes, and the data set consists of
posteriors. This hierarchy should be respected during metric aggregation
[maier2022metrics].

To choose metrics for object-centric validation (if any), the following
properties are of key importance:

  * •

P2: Resimulation (available/unavailable): While the set of reference modes may
be incomplete, it may be possible to verify whether a given mode (of the
prediction) is another plausible solution to the problem. This can be achieved
by applying the forward process (resimulation available) to the given mode and
choosing suitable metrics in the ’observable space’. The resimulation allows
to decide whether a detected mode is a True Positive (TP) or False Positive
(FP). With this information, the Precision, a highly relevant classification
metric, can be computed.

  * •

P3: Confidence score (available/unavailable): Object detection metrics
operating on the confusion matrix (e.g. the F1 Score) are highly sensitive to
the method chosen to convert (fuzzy) algorithm output to actual decisions
[godau2023deployment]. Multi-threshold metrics, such as Average Precision
(AP), overcome the need to decide on specific hyperparameters with ranking-
based approaches. Transferring these principles to posterior validation
requires the ability to rank the modes according to their likelihood of
actually being a mode. This property should be set to true if the predicted
modes come with a score that gauges the certainty of the model that the mode
actually exists. While our framework is agnostic to the source of the score,
we provide possible instantiations in our use cases in section III-D.

###  III-C Decision guides

Our framework may result in users obtaining a pool of applicable metric
candidates instead of only a single candidate. The decision guides presented
in this section aim to help the user understand the tradeoffs between
different metrics and selecting the most suitable candidate for their
underlying problem. As many of the metrics are based on the observed object
detection analogy, there are many parallels to the recommendations in
[maier2022metrics]. The following paragraphs contain the decision guides for
the ambiguous parts of the framework.

  * •

Localization criterion: The localization criterion is used to gauge the
agreement between pairs of predicted and reference modes. The choice of the
localization criterion mainly depends on two properties: first, the
granularity of the reference (P1, which is already covered in Subprocess S2 in
Fig. 5) and second, whether an accurate uncertainty is required (P7). If
uncertainty quantification is important, the shape of the posterior modes
should be taken into account when computing the mode localization. For a
reference given as a mode location (without a spread or similar), this could
take the form of computing the Mahalanobis Distance [chandra1936generalised],
which takes the covariance of the predicted mode into account. This is an
instance of the ”Centroid Distance” category. The advantage of this metric is
that it provides a continuous distance. On the other hand, the predicted mode
could be used to construct a confidence ellipsoid (or a more general
confidence region) to a given confidence level, and a match could be performed
based on whether the reference location falls within the confidence ellipsoid
(”Point inside Confidence Ellipsoid” category). This approach also takes
uncertainty into account but leads to a binary score. If the reference is
given as a distribution and accurate uncertainty is important, distribution-
based metrics should be considered as these do not only match the mode
location but incorporate the shape of the predicted and reference mode.

If accurate uncertainty estimation is less important, the localization
criterion should focus on the correct location of the mode centers. In this
case, the predicted and reference mode should be collapsed to their centers,
and a distance on these centers should be computed (”Centroid Distance”
category). The exact distance should be chosen according to the application.
Examples could be an Lp-norm for translation parameters, the cosine similarity
[schutze2008introduction] for rotational variables, or structural similarity
index [wang2004image] for images.

  * •

Assignment strategy: Whenever an uncertainty score is available, greedy
matching via the (confidence) score [maier2022metrics, everingham2015pascal]
should be applied. The rationale behind this recommendation is that models
that confidently predict wrong or far-off modes should be penalized. If no
confidence score is available, there are multiple complementary options.
Greedy matching via the localization criterion [maier2022metrics] has the
advantage of being methodologically simple and computationally fast.
Furthermore, depending on the application, it can be sensible to match the
closest modes first. An alternative would be to apply Hungarian matching
[maier2022metrics, kuhn1955hungarian], which finds an optimal matching that
minimizes the total mode distances. Such a matching can lead to a predicted
mode not being matched with its closest reference mode. Hungarian matching can
be suitable for a more theory-focused validation or method comparison
(independent of a downstream application). However, as elaborated in
[maier2022metrics], Hungarian matching can lead to overly optimistic
assignments, artificially reducing the number of FNs and FPs. Lastly,
assigning modes via a fixed localization threshold (”Matching via Localization
$>$ Fixed Threshold”) can be useful if the application requires an exact
number of predicted modes but less focus on the precise localization of the
modes. An example downstream task would be to count the occurrence of certain
structures.

  * •

Distance aggregation: An important aspect of distance aggregation is to
respect the hierarchical structure of the data, as elaborated in
[maier2022metrics]. In this posterior-based inverse problem setting, a data
set consists of data points, where each data point corresponds to a set of
reference modes and a set of predicted modes. This two-stage hierarchy implies
that first, the distances between modes per posterior should be aggregated
before these per-data-point distances should be aggregated over the whole data
set. In Fig. 3, we explicitly mention mean, median, STD, and Interquartile
Range (IQR) as aggregation methods for distance aggregation. However, these
solely represent examples of common choices. Depending on the application, it
might be advantageous to report other quantiles of the distribution (instead
of IQR) or weight the data points in the mean. Overall, it should be noted
that quantile-based aggregates (such as median or IQR) are more robust to
noise and outliers, which might make them superior to mean and STD, as many
models produce rather noisy posteriors.

  * •

Classification metrics: If a confidence score is available, we recommend
multi-threshold metrics such as AP or FPPI in almost all cases. They address
the problem of noisy modes due to imperfections in the posterior generation
and/or clustering methods. Metric@(TargetMetric=TargetValue), as introduced in
[maier2022metrics], is a notation to report the value of a metric while a
target metric is optimized on a dedicated validation split to conform to the
target value. An example would be Precision@(Recall=0.95). This type of metric
should be chosen if the application requires certain bounds, e.g., on the
frequency of FPs, as might for instance be derived from regulatory
requirements. Reporting of this form is also common practice in clinically-
focused communities. Fβ [maier2022metrics, van1979information,
chinchor1993muc] aggregates both Precision and Recall and can be useful if
there is no target value for either one, but instead, the model
(hyper-)parameters are optimized (on an additional validation set) to maximize
Fβ.

###  III-D Conditional Invertible Neural Networks for ambiguous problems

To showcase the benefit of our framework, we investigate three complementary
inverse problems that feature inherent ambiguity (see Figs. 6 – 8). In the
following, we present these use cases along with the methods whose performance
is to be assessed with our framework. Further implementation details can be
found in Appendix -A.

####  III-D1 Toy example

As a toy example, we chose a well-understood, but ambiguous, inverse problem,
namely finding the $n$-th roots of a complex number $w$ for varying $n$ (cf.
Fig. 6 (a), left). The input to the inverse problem is the complex number $w$
for which to find the root(s) and the integer $n$ describing the order of the
root. We considered two models: (1) A multi-layer perceptron (MLP) (based on
[kendall2017uncertainties]) as a naive baseline, which, given $n$ and $w$,
produces a Gaussian posterior represented by a mean and a diagonal covariance
matrix. (2) A cINN [ardizzone2019guided], which, given $n$ and $w$, produces a
posterior distribution over $z$ by sampling a latent space. As a mode
detection algorithm, we used the clustering algorithm Density-Based Spatial
Clustering of Applications with Noise (DBSCAN) [ester1996density]. To estimate
the MAP probability location, we used the mean of the largest cluster.

####  III-D2 cINNs for pose estimation in intraoperative 2D/3D registration

Image registration is the basis for many applications in the fields of medical
image computing and computer-assisted interventions. One example is the
registration of 2D X-ray images with preoperative 3D CT images in
intraoperative surgical guidance systems, as illustrated in Fig. 1. Previously
proposed methods [registration_dl, salehi2018real, miao2016, yang2016] lack
the capacity to represent the inherent ambiguity a registration problem may
contain, i.e., they cannot handle a situation where multiple substantially
different solutions exist. We address this lack with cINNs, by representing
the possible solutions to a registration problem through a non-parametric
probability distribution that encodes different plausible solutions via
multiple modes. The challenge of detecting modes in high-dimensional parameter
space is tackled by interpreting the task as a clustering problem performed on
the samples defining the posterior. The neural network architecture is
illustrated in Fig. 1. The input images are passed through a conditioning
network such that a relatively low-dimensional vector (here: 256) can be used
for conditioning the actual invertible net.

Figure 6: Results for the synthetic toy example. (a) The task consists of
computing the $n$th root(s) ($n=1,2,3$) of a non-zero complex number. While
the conditional Invertible Neural Network (cINN) captures the ambiguity of the
problem via multiple modes in the posterior, a classical multi-layer
perceptron (MLP) typically outputs the mean of plausible solutions. (b) Left:
The superiority of the cINN is not captured by classical validation methods
that treat the problem as a regression task (with a unique solution) using the
maximum a posteriori probability as the cINN estimate. Right: The explicit
mode localization and assignment offered by our framework enables the
computation of classification metrics and regression metrics applied on
matched modes. These reveal the poor performance of the MLP compared to the
cINN.

####  III-D3 cINNs for quantification of functional tissue parameters

Photoacoustic imaging is an emerging modality that enables the recovery of
functional tissue parameters. However, the underlying inverse problems are
ill-posed (Fig. 8). Specifically, the problem might have ambiguous solutions,
meaning that different tissue compositions could lead to the same
photoacoustic measurement. We address this ambiguity with a cINN-based
architecture as proposed in [nolke2021invertible]. As a naive baseline, we
chose the state-of-the-art method ”Learned Spectral Decoloring” (LSD)
[grohl2021learned] based on a fully connected neural network architecture,
which provides a single point estimate as a prediction. We optimized the
architecture and training procedure for better performance. For clustering, we
used the UniDip Clustering algorithm [maurus2016skinny], which is based on the
Hartigan-Dip test for unimodality [hartigan_dip_1985]. It provides robust
estimations with respect to resampling of the posterior and is basically
parameter-free (apart from a statistical significance level).

##  IV Experiments & Results

The purpose of the experiments was to instantiate our framework for several
use cases and showcase the added value by means of examples. Note that we did
not aim to optimize the models for the use cases or solve the underlying
tasks. Instead, the focus was on the insights that can be derived from the
proposed validation scheme.

Figure 7: Use case: pose estimation in surgery. Mode detection algorithms can
be configured to provide either higher Recall (at the cost of more False
Positives (FPs)) or higher Precision (at the cost of more False Negatives
(FNs)). Our framework captures this tradeoff by performing explicit mode
localization/matching and recommending the plotting of the Recall as a
function of the FPs per image (FPPI).

###  IV-A Synthetic toy example

The purpose of the toy experiment was to validate the framework branch for a
reference with an exhaustive list of modes. As described in Sec. III, the task
was to, given an integer $n$ and a complex number $w=R\cdot e^{i\Phi}$,
compute the $n$th root of $w$. The distinct solutions (assuming $w\neq 0$) to
this inverse problem can be explicitly enumerated as $z_{k}=\sqrt[n]{R}\cdot
e^{i\frac{\Phi+2\pi k}{n}}$ for $k=0,\dots,n-1$. The training data consisted
of tuples $(z,n,w)$, such that $z^{n}=w$. To highlight the pitfalls of
treating such a problem as a simple regression task, we trained an MLP and a
cINN to estimate $z$ from $w$ and $n$ and evaluated their performance using
the absolute error. Additionally, we instantiated our framework, which
provided us with additional metrics taking the number of modes and the
matching process into account. These additional metrics were Precision,
Recall, $F_{\beta}$ (we report $\beta=1$), AP, and the absolute error computed
for matched modes and aggregated per posterior. The ranking of modes, required
for the computation of AP was achieved by bootstrapping the posteriors. More
specifically, we resampled each posterior two times and computed the
Intersection over Union (IoU) of the new clusters with the original
clustering. The average IoU per cluster was used as confidence score.

To instantiate our training and testing sets, we drew $n$ uniformly from the
set $\\{1,2,3\\}$, $z$ uniformly from an annulus centered at $0$ with inner
radius $0.8$ and outer radius $1.2$, and ‘simulated’ the forward process via
$w=z^{n}$. The training set consisted of $10^{6}$ samples, and the testing set
of $10^{5}$ samples. $n$ was one-hot encoded, and $z$ and $w$ were represented
using their real and imaginary part, respectively. As a localization
criterion, we chose the mode center distance. The predicted and reference
modes were matched greedily by the assignment strategy.

Fig. 6 (b) left shows the classic absolute error distribution of the two
models, indicating that while both models perform poorly, the MLP might be
superior to the cINN. This is in contrast to qualitative observations (example
shown in Fig. 6 (a)), which suggest good performance of the cINN, while the
MLP seems to predict the mean of the ambiguous solutions (which is $0$).

The framework metrics unmask this performance difference between the models.
While both models perform similarly regarding Precision, the cINN outperforms
the MLP in terms of Recall, which is due to the fact that the cINN is capable
of predicting multiple modes, while the MLP is restricted to a single mode.
The absolute error of the matched modes underlines that for higher-order
roots, the cINN correctly identifies the locations of the root, while the MLP
predictions are only close to the ground truth in the unambiguous case of
$n=1$. The cINN achieved an AP of approximately 1.

###  IV-B Medical vision use case 1: Pose estimation

To showcase the potential of the framework for model optimization, we picked a
surgical use case (Fig. 1). In this setting, ambiguity in pose estimation
results from the general symmetry of the spine. To generate a validation data
set with reliable references, we simulated X-ray images taken by a C-Arm with
multiple orientations using the principle of digitally reconstructed
radiographs (DRRs) [unberath2018deepdrr]. As our experimental data set, we
used the UWSpine data set [dataset1, dataset2], which comprises spine-focused
CT volumes of 125 patients. We transformed the volumes to a homogeneous voxel
spacing and discarded images smaller than 128x256x128 as well as patients with
an asymmetric spine. For every CT volume, we sampled a set of different poses
of the C-Arm device and computed corresponding DRRs.

From an application perspective, the conversion of posteriors to modes (i.e.,
the actual solutions of interest) is a crucial step in the system. Often, mode
detection algorithms can be configured to provide either higher Recall (at the
cost of more FPs or higher Precision (at the cost of more False Negatives
(FNs)). To address this tradeoff, we applied our framework for hyperparameter
tuning. Based on the suggested mode matching, we plotted the Recall (using
only the reference modes provided by the simulation) as a function of the
(upper bound of the) FPPI for different hyperparameters of the mode clustering
algorithm. Note that we speak of an upper bound because of the non-exhaustive
list of modes. We varied the minimum samples parameter of the DBSCAN
algorithm. Given that we only worked with symmetric spines, we regarded a mode
corresponding to a left anterior oblique (LAO) angle of
LAO${}_{\text{ref}}+180$° as a TP. Based on the recommendation framework
(Figs. 3 / 4), we chose the Centroid Distance as the localization criterion
(threshold 20°) and Greedy by Localization as the assignment strategy. Fig. 7
reveals that the cluster algorithm hyperparameters corresponding to an FPPI of
approximately 0.35 provide the best tradeoff. This analysis was enabled by the
detection-driven validation approach.

###  IV-C Medical vision use case 2: Functional tissue parameter estimation

Figure 8: Use case: functional tissue parameter estimation. (a) The task is to
estimate blood oxygenation (sO2) from multispectral photoacoustic imaging
data. The potential ambiguity of the problem for a given location (e.g., a
vessel) can be resolved by changing the pose of the image modality (pose 1:
unique solution; pose 2: multiple plausible solutions). (b) Left: The
superiority of the conditional Invertible Neural Network (cINN) over a state-
of-the-art point estimation network ”Learned Spectral Decoloring” (LSD) is not
captured by classical validation methods based on maximum a posteriori
estimates. Right: The explicit mode localization and assignment offered by our
framework enable the computation of classification metrics. These reveal the
application-relevant properties of the methods, namely the Recall and the
False Positives Per Image (FPPI).

The second medical vision use case is illustrated in Fig. 8 and concerns the
quantification of tissue oxygenation from photoacoustic measurements. The
purpose of this experiment was to demonstrate that common validation methods
are not well-suited for application-driven performance assessment. To this
end, we trained two models (see Sec. III) for the given use case – a naive
baseline that treats the problem as a regression problem with a unique
solution as well as a solution based on cINNs. Since ground truth tissue
properties are unavailable for in vivo photoacoustic measurements, we
simulated a synthetic data set of human forearm images using the Monte Carlo
method for light propagation (Monte Carlo eXtreme (MCX)). Our digital tissue
model was inspired by the work of [schellenberg2022photoacoustic] and consists
of different tissue types (skin, muscle background, arteries, veins,
ultrasound gel, membrane, and water). The anatomic and optical properties are
based on knowledge from literature. The whole simulation was implemented using
the Simulation and Image Processing for Photonics and Acoustics (SIMPA)
toolkit [grohl2022simpa]. For our validation, we focused on samples that were
detected to be multimodal by our cINN. As a first naive validation approach,
we compared the MAP estimate, i.e., for the cINN, we used the median of the
largest cluster as a point estimate. As can be seen in Fig. 8 (b) left, both
methods seem to perform equally well. Note that for bimodal posterior
distributions, the MAP is not necessarily the best solution, as sometimes the
smaller mode might correspond to the reference. Point prediction methods such
as LSD usually predict a value that is either close to the largest mode of the
cINN or lies between the two modes (as in the toy example). Our framework
addresses this issue. Following the recommendation framework, we first
performed mode matching (Greedy by Localization) with a threshold of 5
percentage points (pp) sO2 difference to enable object detection-inspired
metrics. In analogy to the previous example, we then computed Recall and FPPI
upper bound. The cINN method outperforms LSD in terms of Recall (90% vs. 71%).
However, this comes at the cost of more FPPI.

##  V Discussion and conclusion

Validation of deep learning-based methods attempting to solve inverse problems
is key for both measuring progress as well as their eventual translation to
real-world applications. Currently, however, common validation practice
frequently neglects the requirements of the underlying application, leading to
the resulting metric scores often not reflecting the actual needs. This
especially holds true for posterior-based methods tackling inverse problems
for which multiple different but plausible solutions exist.

Currently, inverse problem solvers, whether using a posterior or classical
representation, are often validated in an ad hoc manner specifically tailored
to the problem at hand. Our posterior validation framework takes one step back
and proposes key properties that allow us to abstract from the specific
inverse problem and advance toward a unified, generic inverse problem
validation methodology. As we argue that flaws in common validation practice
can largely be attributed to a lack of best practices, in our opinion,
dedicating efforts towards improving common practice becomes imperative to
advance the field. Our framework provides a first step towards structured and
standardized validation practice. We hope that an according shift in research
focus exemplified by our work sparks further research on how to best validate
inverse problem methods and allows for better and more meaningful comparisons
of algorithms.

With the proposed framework, we are – to the best of our knowledge – the first
to systematically address this problem. A particular novelty is the leveraging
of object detection-inspired metrics for posterior validation, which enables a
mode-centric validation. The mode-centric view aligns naturally with
applications, for example in the medical domain, where interpretation of a
posterior distribution might be infeasible, but the scanning of a (short) list
of plausible solutions might provide a benefit over a point prediction both in
terms of predictive performance as well as uncertainty quantification.

While a direct evaluation of our proposed framework is not possible, we
instead demonstrated its value in various medical vision use cases. As this
was the primary goal of the paper, it should be noted that we did not focus on
actually solving a specific clinical problem. This is why neither the models
used nor the experimental data have been optimized for the particular use
case.

Multi-threshold metrics such as AP are widely used metrics in object detection
and, as such, are also included in our framework. However, it must be noted
that a critical requirement for their computation is the availability of a
confidence score. Natural choices for confidence scores such as the relative
mass of the mode have disadvantages such as the confidence score depending on
the number of detected modes. Future work should thus be directed toward
developing alternative confidence scores overcoming this limitation and
enabling the use of these robust metrics. Also, a future implementation of the
metrics in a library will be useful in providing the community with a
standardized and reliable resource for validation, given that previous work
highlighted the problems of non-standardized metric implementation
[maier2022metrics, reinke2023understanding]. On a further note, the pool of
available metrics in the case of non-exhaustive reference modes is currently
rather limited. We hope that the clear structure using the inverse problem
fingerprints will spark a fruitful discussion on new metric candidates
suitable for this setting.

In conclusion, our experiments clearly demonstrate the added value of mode-
centric validation compared to the standard validation approach. Our framework
could thus evolve as an important tool for posterior validation in inverse
problems.

## Acknowledgements

The authors would like to thank Melanie Schellenberg for her contribution to
figure design.

## References

|  Tim J. Adler received his Ph. D. degree in computer science from Heidelberg
University in 2023. He did his thesis at the division of Intelligent Medical
Systems (IMSY), German Cancer Research Center (DKFZ). His work focused on
uncertainty quantification in multispectral and photoacoustic imaging. He
currently holds a position as Senior Data & Applied Scientist at hema.to,
where he works on data-driven methods for leukemia diagnosis using flow
cytometry.
---|---
|  Jan-Hinrich Nölke received his M.Sc. degree in Physics from Heidelberg
University in 2021. He is currently pursuing an interdisciplinary Ph.D. at the
division of Intelligent Medical Systems (IMSY), German Cancer Research Center
(DKFZ). His research focuses on deep learning-based uncertainty quantification
in medical imaging.
---|---
|  Annika Reinke received her Ph.D. degree in computer science from Heidelberg
University in 2023. She currently holds a position as a postdoctoral
researcher at the division of Intelligent Medical Systems (IMSY), German
Cancer Research Center (DKFZ), leading the group ”Validation of Intelligent
Systems”. She further serves as an active member of several international
groups such as the MICCAI Special Interest Group on biomedical challenges.
---|---
|  Minu Dietlinde Tizabi received her Doctorate of Medicine from Heidelberg
University in 2017. She is a physician, scientist and writer in the division
of Intelligent Medical Systems (IMSY) at the German Cancer Research Center
(DKFZ).
---|---
|  Sebastian Gruber earned his M.Sc. in Physics from Heidelberg University in
2022. He completed both his Bachelor’s and Master’s theses at the division of
Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ). He
is presently employed as a Machine Learning Engineer at Onlinedoctor, a Swiss-
based health technology startup, specializing in skin disease classification.
---|---
|  Dasha Trofimova received her PhD in Physics from Heidelberg university in
2015. After work as a data scientist in industry, she joined the division of
Intelligent Medical Systems at the German Cancer Research Center (DKFZ),
focusing on medical image analysis. Her goal is to use machine learning to
make a real impact in healthcare, especially diagnostics.
---|---
|  Lynton Ardizzone completed his M.Sc. degree in Physics in 2018 at
Heidelberg University. Until 2022, he conducted research at the Visual
Learning Lab in Heidelberg for his pending PhD degree computer science. Since
2022, he has served as Head of Machine Learning at COPRESENCE AG.
---|---
|  Paul F. Jaeger is a principal investigator at the Interactive Machine
Learning Group at the German Cancer Research Center and Helmholtz Imaging.
After Studying in Karlsruhe, Stockholm, Melbourne, and Montreal, he received
his Ph.D. in Computer Science from Karlsruhe Institute of Technology. His
research focuses on image analysis algorithms, with a particular focus on
human interaction.
---|---
|  Florian Buettner is professor at Goethe University Frankfurt (Germany) and
head of department at the German Cancer Research Center (DKFZ)/German Cancer
Consortium (DKTK). His research focusses on the application-driven development
of novel machine learning algorithms in oncology.
---|---
|  Ullrich Köthe received the diploma degree in physics from the University of
Rostock, Rostock, Germany, and the PhD degree in computer science from the
University of Hamburg, Hamburg, Germany. He is currently a professor for
computer science in the Interdisciplinary Center for Scientiﬁc Computing, at
Heidelberg University. His research focuses on the connection between machine
learning and the sciences. He is particularly interested in interpretable
generative model architectures, learning methods, and applications.
---|---
|  Lena Maier-Hein is a full professor at Heidelberg University (Germany) and
division head at the German Cancer Research Center (DKFZ). She is managing
director of the National Center for Tumor Diseases (NCT) Heidelberg and of the
DKFZ Data Science and Digital Oncology cross-topic program. Her research
concentrates on machine learning-based biomedical image analysis with a
specific focus on surgical data science, computational biophotonics and
validation of machine learning algorithms.
---|---

.

.

###  -A Use case implementation details

The following sections describe the implementation details of the models used
to solve the inverse problems in the use cases. All deep learning models were
implemented using PyTorch. The cINNs further made use of the Framework for
Easily Invertible Architectures (FrEIA) [freia].

Synthetic toy example

  * •

Architecture: The MLP was implemented following the architecture proposed in
[kendall2017uncertainties]. More precisely, we implemented a six-layer fully-
connected neural network with rectified linear unit (ReLU) activations, 128
dimensions for each hidden layer, and a dropout rate of 0.2. The output was
four-dimensional, and we interpreted the first two dimensions as the mean and
the second two dimensions as the logarithmic standard deviation of a Gaussian
distribution with diagonal covariance over the solution space (i.e., the space
of possible roots). The network was trained using maximum likelihood training
under the Gaussian assumption, which corresponds to the loss

$L=\mathbb{E}_{(z,n,w)}\left[\frac{1}{2}\sum_{i=1}^{2}\left(e^{-2\delta_{i}}\cdot(z_{i}-\hat{z}_{i})^{2}+2\delta_{i}+\log(2\pi)\right)\right],$

where $\hat{z},\delta=f_{\Theta}(w,n)$ are the model predictions. We applied
Monte Carlo dropout at inference time, which led to multiple predictions
$(\hat{z}(k),\delta(k))_{k=1,\dots,N}$, which we aggregated via

$\displaystyle\hat{z}$ $\displaystyle=\frac{1}{N}\sum_{k=1}^{N}\hat{z}(k),$
$\displaystyle\hat{\sigma}_{i}(k)$ $\displaystyle=e^{\delta_{i}(k)},\text{
and}$ $\displaystyle\hat{\sigma}_{i}$
$\displaystyle=\sqrt{\frac{1}{N}\sum_{k=1}^{N}\hat{z}_{i}(k)^{2}-\hat{z}_{i}^{2}+\frac{1}{N}\sum_{k=1}^{N}\hat{\sigma}_{i}(k)^{2}},$

for $i\in\\{1,2\\}$ denoting the axes of the standard identification of
$\mathbb{R}^{2}$ with the complex numbers via real and imaginary part.

The model was trained for 1000 epochs using the AdamW
[loshchilov2017decoupled] optimizer, with a learning rate of $10^{-3}$, a
weight decay parameter of $10^{-5}$, and a batch size of 2048. For inference,
we chose $N=50$ following [kendall2017uncertainties].

The cINN was implemented using affine coupling blocks [dinh2014nice,
dinh2016density, ardizzone2018analyzing, kingma2018glow] followed by (fixed)
random permutations and a global affine transformation (i.e., an affine
transformation with learnable parameters but independent of the input to it)
[kingma2018glow, mackowiak2021generative]. We used 20 affine coupling blocks
with shallow fully-connected subnetworks with a single hidden layer with 256
dimensions and ReLU activations. The scaling of the affine coupling was soft-
clamped with a clamping constant of 2.0, and we initialized the global affine
transformation with the scaling parameter 0.7 (in FrEIA this parameter is
called global_affine_init). The cINN works by transforming the solution of the
inverse problem (z in our case) into a latent space, conditioned on the
observables (w and n), i.e., the cINN is a map g(z; w, n) that is invertible
with regard to z, given w and n. During training, a Gaussian distribution on
the latent space is enforced via maximum likelihood training:

$L=\mathbb{E}_{(z,n,w)}\left[\frac{1}{2}\sum_{i=1}^{2}g_{i}(z;n,w)^{2}-\log|\det
Jg(z;n,w)|\right],$

where $Jg$ denotes the Jacobi matrix of $g$. The architecture of the cINN is
chosen in such a way that the Jacobi matrix is triangular, such that the log-
determinant is efficiently computable. At inference time, we draw samples in
the latent space and transform them to the solution space via $g^{-1}$ (given
w and n).

The cINN was trained for 1000 epochs using the AdamW optimizer with a learning
rate of $10^{-2}$, which was reduced by a factor of 10 after epochs 200, 500,
and 900. We used a weight decay parameter of $10^{-5}$ and a batch size of
2048. In this experiment, we used 1024 latent samples to build the posterior
during inference.

Before training, z and w were normalized to zero mean and unit variance. The
one-hot encoded n was left unchanged. Furthermore, we applied noise
augmentation with a standard deviation of 0.02 to the normalized z and w
dimensions.

  * •

Mode Processing: For mode detection of the cINN posteriors, we applied the
DBSCAN [ester1996density] clustering algorithm using the scikit-learn library.
DBSCAN was applied to the denormalized data with a minimum sample size of 20
and $\varepsilon=0.2$.

Medical vision use case 1: Pose estimation

  * •

Data Set: For every CT volume, we sampled 100 different poses of the C-Arm
device and computed corresponding DRRs. The virtual C-Arm poses (relative to
the 3D volume coordinate system) were determined as follows: The translation
along the sagittal, longitudinal, and transverse axis was randomly sampled
from a continuous uniform distribution with range [-20 mm, 20 mm]. The two
angles representing the rotation around the longitudinal (LAO) and transverse
(CRAN) axis of the patient were sampled from a discrete uniform distribution
with range [-$None$, $None$] and a step size of $None$. With a probability of
0.5, the LAO angle was shifted by $None$ to capture a possible ambiguity in
the projections. We split the data into a disjoint training and test data set
(no overlap between patients) with 131,900 and 2,700 samples, respectively.
For our validation, we only considered samples with a highly symmetric spine,
which resulted in 196 samples.

  * •

Architecture: To eliminate the need for the affine coupling blocks to learn
the complete representation of the input images, a conditioning network was
applied that transformed the two input images into an intermediate
representation. The choice of the architecture of the conditioning network was
inspired by [chee2018airnet], where core elements of the registration network
are blocks with convolutional layers followed by batch normalization, dropout
layers $(p=0.2)$, and ReLU activations. In the first stage of the training, we
pre-trained the conditioning network with a mean squared error loss to predict
the pose parameters.

The cINN consisted of three affine coupling blocks, each followed by a (fixed)
random permutation. The subnetworks were implemented as fully-connected
networks with a single hidden layer with 128 dimensions, dropout layers
$p=0.02$, and tanh activations. Soft clamping was applied with a constant of
1.9. The cINN was trained with a maximum likelihood loss, batch size of 32,
and noise and contrast augmentation for both CT volume and 2D projections. The
model was trained for 3000 epochs with the Adam optimizer with a weight decay
of $10^{-4}$ and an initial learning rate of $10^{-2}$. Every 200 epochs, the
learning rate was reduced by a factor of two. During the training of the cINN,
the conditioning network was further optimized.

  * •

Mode Processing: Upon test time, CT volume and 2D projection serve as
conditioning input, and repeated sampling from the latent space (here: 1028
samples) results in a full posterior over the five-dimensional parameter
space. For mode detection, the DBSCAN clustering algorithm, as implemented in
the scikit-learn library, was used. We fixed the parameter $\varepsilon=0.19$
and varied the minimum sample size between 3 and 500 for hyperparameter
optimization. For the localization criterion and the assignment strategy, we
solely considered the LAO angle as this is the dimension with expected
ambiguous solutions.

Medical vision use case 2: Functional tissue parameter estimation

  * •

Data Set: For the functional tissue parameter quantification use case, a total
of 1100 synthetic photoacoustic images of the human forearm were simulated
(Train:Val:Test; 900:100:100 images) [schellenberg2022photoacoustic]. The
simulations were performed on 16 equidistant wavelengths between 700 and 850
nm. The optical Monte Carlo simulation was performed with $5\cdot 10^{8}$
photons with a spatial resolution of 0.15625mm. The volumes were of dimension:
75mm (transducer dim) x 20mm (planar dim) x 20mm (height). The simulated 3D
images were cropped, and additive and multiplicative Gaussian noise components
were added to match the contrast of real photoacoustic images. Finally, the
spectra of the tissue classes artery and vein were extracted, L1-normalized,
and used as input for our models.

  * •

Architecture: The original architecture of our baseline method (LSD) was
adapted, resulting in a fully connected network with two hidden layers of size
256, dropout ($p=0.5$), and ReLU activations. For the cINN, 20 coupling blocks
and (fixed) random permutations were used. The subnetworks were implemented as
fully connected networks with one hidden layer of size 1024, dropout
($p=0.5$), and ReLU activations. Soft clamping was applied with $\alpha=1.0$.
As the coupling blocks require a minimum channel dimension of two due to the
internal dimension splitting, a second dummy dimension with standard Gaussian
noise was concatenated to the one-dimensional quantity of interest
(oxygenation).

Both models were trained with a batch size of 1024 for 100 epochs. The AdamW
optimizer was used with a learning rate of $10^{-3}$ and weight decay of 0.01.
After epochs 80 and 90, the learning rate was reduced by a factor of ten.

For the cINN, 5000 posterior samples were drawn during inference time. The
UniDip clustering algorithm [maurus2016skinny] was used with a statistical
significance level of $\alpha=0.5$.

## References